Failed to connect to github.com port 443: Operation timed out

Drone Support
1

Done 0.8.4
Deployed with helm incubator/drone

Registration in github abd adding project works as expected, but pipeline itself does not work.
There is output in UI:

+ git init
Initialized empty Git repository in /drone/src/github.com***/***/.git/
+ git remote add origin https://github.com/***/***.git
+ git fetch --no-tags origin +refs/heads/feature-ci:
fatal: unable to access 'https://github.com/***/***.git/': Failed to connect to github.com port 443: Operation timed out
exit status 128

There is deployment variables:

server:
  env:
    DRONE_PROVIDER: "github"
    DRONE_OPEN: "true"
    DRONE_GITHUB: "true"
    DRONE_ORGS: "***"
    DRONE_ADMIN: "r-moiseev"
    DRONE_GITHUB: "true"
    DRONE_GITHUB_CLIENT: "***"
    DRONE_GITHUB_SECRET: "***"
    DRONE_GITHUB_PRIVATE_MODE: true

persistence:
  enabled: true

dind:
  enabled: false

And there is agent logs

{"time":"2018-04-20T07:48:22Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"received execution"}
{"time":"2018-04-20T07:48:22Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"listen for cancel signal"}
{"time":"2018-04-20T07:48:22Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","exit_code":0,"exited":false,"message":"update step status"}
{"time":"2018-04-20T07:48:22Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","exit_code":0,"exited":false,"message":"update step status complete"}
{"time":"2018-04-20T07:48:24Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","message":"log stream opened"}
{"time":"2018-04-20T07:49:22Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"pipeline lease renewed"}
{"time":"2018-04-20T07:50:22Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"pipeline lease renewed"}
{"time":"2018-04-20T07:51:22Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"pipeline lease renewed"}
{"time":"2018-04-20T07:51:59Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","message":"log stream copied"}
{"time":"2018-04-20T07:51:59Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","message":"log stream uploading"}
{"time":"2018-04-20T07:51:59Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","message":"log stream upload complete"}
{"time":"2018-04-20T07:51:59Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","message":"log stream closed"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","exit_code":1,"exited":true,"message":"update step status"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","image":"plugins/git:latest","stage":"clone","exit_code":1,"exited":true,"message":"update step status complete"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","error":"","exit_code":1,"message":"pipeline complete"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"uploading logs"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"uploading logs complete"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","error":"","exit_code":1,"message":"updating pipeline status"}
{"time":"2018-04-20T07:52:00Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"stop listening for cancel signal"}
{"time":"2018-04-20T07:52:01Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"updating pipeline status complete"}
{"time":"2018-04-20T07:52:01Z","level":"debug","message":"request next execution"}
{"time":"2018-04-20T07:52:01Z","level":"debug","repo":"***/***","build":"1","id":"1","message":"pipeline done"}

In server log there is nothing interesting

2

This looks like a network error returned by the git clone command. It would appear your kubernetes cluster is having issues with user-define Docker networks.

3

Problem is I can exec into drone container and do wget gihub.com. it is reachable

And if it was network issue then drone would not be able to authrize me show projects and control webhooks. But it does

UPD If i set DRONE_GITHUB_VERIFY then i get LibreSSL error

4

And if it was network issue then drone would not be able to authrize me show projects and control webhooks. But it does

This is not necessarily true because your drone server is probably using the default bridge network, while pipeline containers are attached to user-defined networks. Please see my comments here: http://discuss.harness.io/t/builds-fail-to-resolve-bitbucket-org-during-git-clone-while-running-on-kops-kubernetes/617/3

5

I can confitm that both server and agent are in same network. (generaly there is no concept of networks in k8s at all) and github is reachable from there

6

can confitm that both server and agent are in same network.

The git clone does not happen inside the agent or the server, so the networks used by the agent and server are not relevant. When drone executes your clone, it does so in a separate container (GitHub - drone-plugins/drone-git: Drone plugin for cloning Git repositories) that is attached to a user-defined docker network.

generaly there is no concept of networks in k8s at all

drone is not launching your containers using the kubernetes runtime. It is launching pipeline containers using the docker runtime, which is running inside a dind container (assuming you are using the default helm chart). So my prior comments about user-defined networks still apply.

I recommend this forum for kubernetes network issues. This is not the first time someone has reported issues with kubernetes networking, which can typically be resolved with host machine or dind configuration changes.

7

Ah, I see. Is there way to make agent create containers with host network? I dont use dind so agent use nodes docker.sock (if i understand right)

8

There is no way to disable or change this behavior. Iā€™m sure this can be resolved, but you may need to engage either docker or kubernetes support to figure out exactly what system configuration changes or docker daemon configuration adjustment you will need to make.

9

Thank you, you where right, there is iptables=false in kubespray`s docker service.

10

Was this resolved by setting iptables=true? If so, can you provide some more details on how you resolved it (what file you updated, etc) so that it might benefit others with the same issue? Thanks!

11

Yes, it was resolved by edit roles/docker/templates/docker-options.conf.j2 template inside kubespray. Do not know why it set to false, but can confirm that weave network still working with true.

2 Likes