I/o timeout - builds pending

Hey,

i want to setup drone on my digitalocean droplet.

I just create the server and runner by:

docker run
–volume=/var/lib/drone:/data
–env=DRONE_AGENTS_ENABLED=true
–env=DRONE_GITHUB_SERVER=https://github.com
–env=DRONE_GITHUB_CLIENT_ID=xxx
–env=DRONE_GITHUB_CLIENT_SECRET=xxx
–env=DRONE_RPC_SECRET=xxx
–env=DRONE_RPC_PROTO=http
–env=DRONE_RPC_HOST=167.172.168.81
–env=DRONE_SERVER_HOST=167.172.168.81
–env=DRONE_SERVER_PROTO=http
–env=DRONE_LOGS_DEBUG=true
–publish=80:80
–publish=443:443
–restart=always
–detach=true
–name=drone
drone/drone:latest

AND

docker run -d
-v /var/run/docker.sock:/var/run/docker.sock
-e DRONE_RPC_PROTO=http
-e DRONE_RPC_HOST=167.172.168.81
–env=DRONE_RPC_SECRET=xxx
-e DRONE_SERVER_HOST=167.172.168.81
-e DRONE_SERVER_PROTO=http
-e DRONE_RUNNER_CAPACITY=2
-e DRONE_RUNNER_NAME=${HOSTNAME}
-p 3000:3000
–restart always
–name runner
drone/drone-runner-docker:latest

Like described in the documentation.

Runner prints

time=“2020-02-06T12:37:48Z” level=error msg=“cannot ping the remote server” error=“Post http://167.172.168.81/rpc/v2/ping: dial tcp 167.172.168.81:80: i/o timeout”

Getting sick of this, maybe someone can help me :slight_smile:

Thank you!

This tells me that the runner cannot reach the server. The http request from the runner to the server times out. This indicates a networking issue with your host machine, preventing the two containers from communicating.

Experiencing the exact same issue after following step-by-step the official getting started guide.

I have tried the following things:

  • Created a network to bridge the two containers.
  • Added HTTPS (using DRONE_TLS_AUTOCERT=true environment variable)
  • Tried both domain and IP address format
  • Tried http and https

and confirm the following things:

  • DRONE_RPC_SECRET matches for both containers
  • DRONE_SERVER_HOST and DRONE_RPC_HOST matches

and have read the following threads:

None of the above get to work, with the exact same error.

Also if this is very hard to debug and very commonly encountered by people, it should be at least mentioned in the docs.

Can you advise?

We have a thread dedicated to this topic that provides suggested fixes for common issue. If you are unable to resolve the issue, the Still having issues? details a list of things that you should provide in order to receive further assistance.

http://discuss.harness.io/t/builds-are-stuck-in-pending-status/4437

  1. Provide your server configuration:
docker run \
  --volume=/var/lib/drone:/data \
  --env=DRONE_GITLAB_SERVER=https://gitlab.com \
  --env=DRONE_GITLAB_CLIENT_ID=[redacted] \
  --env=DRONE_GITLAB_CLIENT_SECRET=[redacted] \
  --env=DRONE_RPC_SECRET=b27c080a0dd00b32a5532b7217614d2d \
  --env=DRONE_SERVER_HOST=drone.ewburgers.com \
  --env=DRONE_SERVER_PROTO=http \
  --publish=80:80 \
  --publish=443:443 \
  --restart=always \
  --detach=true \
  --name=drone \
  drone/drone:1
  1. Provide your agent configuration
docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e DRONE_RPC_PROTO=http \
  -e DRONE_RPC_HOST=drone.ewburgers.com \
  -e DRONE_RPC_SECRET=b27c080a0dd00b32a5532b7217614d2d \
  -e DRONE_RUNNER_CAPACITY=2 \
  -e DRONE_RUNNER_NAME=${HOSTNAME} \
  -p 3000:3000 \
  --restart always \
  --name runner \
  drone/drone-runner-docker:1
  1. Enable DRONE_LOGS_TRACE=true on the server
  2. Enable DRONE_LOGS_TRACE=true on the agent
  3. Provide the agent logs with trace enabled
time="2021-03-28T17:43:02Z" level=info msg="starting the server" addr=":3000"
time="2021-03-28T17:43:32Z" level=error msg="cannot ping the remote server" error="Post http://drone.ewburgers.com/rpc/v2/ping: dial tcp 157.245.252.92:80: i/o timeout"
... and so on
  1. Provide the server logs with trace enabled
{"level":"info","msg":"main: internal scheduler enabled","time":"2021-03-28T18:25:37Z"}
{"acme":false,"host":"drone.ewburgers.com","level":"info","msg":"starting the http server","port":":80","proto":"http","time":"2021-03-28T18:25:37Z","url":"http://drone.ewburgers.com"}
{"interval":"30m0s","level":"info","msg":"starting the cron scheduler","time":"2021-03-28T18:25:37Z"}
{"interval":"24h0m0s","level":"info","msg":"starting the zombie build reaper","time":"2021-03-28T18:25:37Z"}
  1. Provide the Yaml configuration file (none)
  2. Provide the build details for your pending builds via this API endpoint. (no builds yet)

Thanks for the details. In this case it looks like Drone is configured correctly which means the I/O timeout is likely a networking issue, possibly DNS or firewall. Have you tried to curl the Drone server from inside your runner container to verify it can connect?

$ docker exec -it runner /bin/sh
> apk add curl
> curl http://drone.ewburgers.com/healthz

“port”:“:443”,“proto”:“http”

one minor thing … this looks off because it states Drone is listening on port 443, however, you did not configure https protocol, nor have you configured an ssl certificate or lets encrypt. These logs do not seem to match up with the docker run command that you provided. Can you double check that all the information you provided above was accurate?

when I run the docker command above I see port 80

{"acme":false,"host":"drone.ewburgers.com","level":"info","msg":"starting the http server","port":":80","proto":"http","time":"2021-03-28T18:35:44Z","url":"http://drone.ewburgers.com"}

you’re right, I’ve updated the logs (a direct copy paste). Sorry about that.

I think you’re pointing in the right direction. It seems that it is unable to resolve.
Though the command works fine on my local machine, it cannot resolve when running in the runner container.

Any ideas?

OK another detail.

Running curl -v flag.

/ # curl -v http://drone.ewburgers.com/healthz
*   Trying 157.245.252.92:80...
* connect to 157.245.252.92 port 80 failed: Operation timed out
* Failed to connect to drone.ewburgers.com port 80: Operation timed out
* Closing connection 0

It goes forever.

I tried accessing the IP 157.245.252.92 on my browser.
I was redirected to the callback url (the /login), but eventually redirected to
/login/error?message=http%3A%20named%20cookie%20not%20present.

did you install the server and runner on the same machine? If yes, the problem is likely that docker resolves the dns to the containers own network, meaning the request never leaves the container. Instead you may want to try and create a user-defined bridge network, and provide the runner with the server’s internal docker address, as opposed to the domain name. See Networking with standalone containers | Docker Documentation

since you configured DRONE_SERVER_HOST to use the domain name, you need to access Drone in your browser using the domain name, not the IP address.

Got it, so it’s not related.

Will try the very complicated networking docker method soon. Thanks!

If you plan on running the server and runner on the same machine, you may find it easier to work with docker-compose. This is a slightly modified yaml that I use to run the whole stack on my machine:

version: "3.8"
services:
    drone:
        image: drone/drone:latest
        ports:
        - "80:80"
        - "443:443"
        environment:
        - DRONE_GITLAB_CLIENT_ID=[redacted]
        - DRONE_GITLAB_CLIENT_SECRET=[redacted]
        - DRONE_RPC_SECRET=b27c080a0dd00b32a5532b7217614d2d
        - DRONE_SERVER_HOST=drone.ewburgers.com
        - DRONE_SERVER_PROTO=http
        - DRONE_LOGS_DEBUG=true
        - DRONE_CRON_DISABLED=true
        volumes:
        - /var/lib/drone:/data
    runner:
        image: drone/drone-runner-docker:latest
        environment:
        - DRONE_RPC_HOST=drone
        - DRONE_RPC_PROTO=http
        - DRONE_RPC_SECRET=b27c080a0dd00b32a5532b7217614d2d
        - DRONE_TMATE_ENABLED=true
        volumes:
        - /var/run/docker.sock:/var/run/docker.sock
1 Like