Builds stuck in Pending (using a Docker setup)

Hi,
I am using a Docker setup, running on AWS EC2, that is orchestrated via Ansible. In short, I run an Ansible playbook that connects to EC2m and then starts Drone Server, a Drone Agent, and Postgres; all of them as Docker containers.
After that I can access Drone UI and I am able to activate a Github repository that I use for testing. I can see the webhook is properly setup on Github.
Then I add and push a Drone file to test it out, Drone gets the trigger from Github webhook but the build gets stuck in Pending state.

This is the Drone file that I use for testing:

$ cat .drone.yml
kind: pipeline
type: docker
name: default

steps:
- name: tests
  image: golang:1.14.4-alpine3.12
  commands:
  - pwd

Here’s the output of docker ps after all the containers are up and running:

# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                             NAMES
99aa5ff39949        drone/agent:1.4.0   "/bin/drone-agent"       16 seconds ago      Up 15 seconds                                                                                                         agent-Nancy
841f0d39be48        drone/drone:1.4.0   "/bin/drone-server"      27 seconds ago      Up 26 seconds       0.0.0.0:8000->8000/tcp, 0.0.0.0:9000->9000/tcp, 127.0.0.1:8080->80/tcp, 127.0.0.1:8443->443/tcp   drone-server
03f4d5e39321        postgres:9.6.1      "/docker-entrypoint.…"   24 hours ago        Up 24 hours         0.0.0.0:5432->5432/tcp                                                                            postgres

And below are the environment variables that are being passed to Drone Server:

# docker inspect drone-server | grep -A22 Env
            "Env": [
                "DRONE_USER_CREATE=username:my-github-user,admin:true",
                "DRONE_DATABASE_DRIVER=postgres",
                "DRONE_SERVER_ADDR=:8000",
                "DRONE_SERVER_PROXY_PROTO=http",
                "DRONE_RPC_SECRET=MyDroneSecret",
                "DRONE_GITHUB_CLIENT_ID=MyGithubOAuthClientId",
                "DRONE_DATADOG_ENABLED=false",
                "DRONE_LOGS_DEBUG=true",
                "DRONE_DATABASE_DATASOURCE=postgres://drone:drone@p@postgres:5432/drone?sslmode=disable",
                "DRONE_GITHUB_CLIENT_SECRET=MyGithubOAuthClientSecret",
                "DRONE_GITHUB_SKIP_VERIFY=true",
                "DRONE_SERVER_PROXY_HOST=droneci.example.com",
                "DRONE_SERVER_HOST=127.0.0.1:8000",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GODEBUG=netdns=go",
                "XDG_CACHE_HOME=/data",
                "DRONE_RUNNER_OS=linux",
                "DRONE_RUNNER_ARCH=amd64",
                "DRONE_SERVER_PORT=:80",
                "DRONE_DATADOG_ENDPOINT=https://stats.drone.ci/api/v1/series"
            ],
            "Cmd": null,

And these are the environment variables passed to the Drone Agent:

# docker inspect agent-Nancy | grep -A16 Env
            "Env": [
                "DRONE_RPC_DUMP_HTTP=true",
                "DRONE_LOGS_DEBUG=true",
                "DRONE_RPC_HOST=drone-server:9000",
                "DRONE_RPC_DUMP_HTTP_BODY=true",
                "DRONE_LOGS_TRACE=true",
                "DRONE_RPC_SECRET=MyDroneSecret",
                "DRONE_RPC_SERVER=drone-server:9000",
                "DRONE_RPC_PROTO=http",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GODEBUG=netdns=go",
                "DRONE_RUNNER_OS=linux",
                "DRONE_RUNNER_ARCH=amd64",
                "DRONE_RUNNER_PLATFORM=linux/amd64",
                "DRONE_RUNNER_CAPACITY=1"
            ],
            "Cmd": null,

These are Drone Server logs:

# docker logs drone-server
{"level":"info","msg":"main: internal scheduler enabled","time":"2020-07-15T14:21:27Z"}
{"build.limit":5000,"expires":"0001-01-01T00:00:00Z","kind":"trial","level":"debug","msg":"main: license loaded","repo.limit":0,"time":"2020-07-15T14:21:27Z","user.limit":0}
{"admin":true,"level":"debug","login":"my-github-user","machine":false,"msg":"bootstrap: create account","time":"2020-07-15T14:21:27Z","token":""}
{"admin":true,"level":"debug","login":"my-github-user","machine":false,"msg":"bootstrap: updating account","time":"2020-07-15T14:21:27Z","token":""}
{"admin":true,"level":"debug","login":"my-github-user","machine":false,"msg":"bootstrap: account already up-to-date","time":"2020-07-15T14:21:27Z","token":""}
{"level":"info","msg":"main: starting the local build runner","threads":2,"time":"2020-07-15T14:21:27Z"}
{"arch":"amd64","level":"debug","machine":"841f0d39be48","msg":"runner: polling queue","os":"linux","time":"2020-07-15T14:21:27Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2020-07-15T14:21:27Z","type":"docker","variant":""}
{"acme":false,"host":"127.0.0.1:8000","level":"info","msg":"starting the http server","port":":80","proto":"http","time":"2020-07-15T14:21:27Z","url":"http://127.0.0.1:8000"}
{"interval":"30m0s","level":"info","msg":"starting the cron scheduler","time":"2020-07-15T14:21:27Z"}
{"arch":"amd64","level":"debug","machine":"841f0d39be48","msg":"runner: polling queue","os":"linux","time":"2020-07-15T14:21:27Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2020-07-15T14:21:27Z","type":"docker","variant":""}

And these are Drone Agent logs:

# docker logs agent-Nancy
{"level":"debug","msg":"successfully pinged the docker daemon","time":"2020-07-15T14:21:37Z"}
{"arch":"amd64","level":"debug","machine":"99aa5ff39949","msg":"runner: polling queue","os":"linux","time":"2020-07-15T14:21:37Z"}
2020/07/15 14:21:37 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request
2020/07/15 14:21:37 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"
2020/07/15 14:21:37 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request: retrying in 1s (30 left)
2020/07/15 14:21:38 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"
2020/07/15 14:21:38 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request: retrying in 2s (29 left)
2020/07/15 14:21:40 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"
2020/07/15 14:21:40 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request: retrying in 4s (28 left)
2020/07/15 14:21:44 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"
2020/07/15 14:21:44 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request: retrying in 8s (27 left)
2020/07/15 14:21:52 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"
2020/07/15 14:21:52 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request: retrying in 10s (26 left)
2020/07/15 14:22:02 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"
2020/07/15 14:22:02 [DEBUG] POST drone-server://localhost:8080/rpc/v1/request: retrying in 10s (25 left)

I can’t understand what’s wrong with my setup but here’s some specific questions I need help with:
1- It is not clear if the agent is able to reach the server at any time, or just after the first time and not after. That honestly looks ambiguous to me.
2- I set DRONE_RPC_PROTO env var to “http” because I thought that would fix the unsupported protocol scheme error but that did not help.
3- Also I don’t know why the agent assumes the Drone Server is running at localhost:8080. I’m setting DRONE_RPC_HOST and DRONE_RPC_SERVER env vars to drone-server:9000 but that doesn’t seem to change much.

Thanks for any help on this in advance!

I noticed you are running an old version of Drone. Since this is a new installation you should probably use the latest stable release. Also note that the drone/agent container is deprecated. The official installation docs will provide the correct images.

In terms of the issue, the problem is that your agent is trying to contact the server using localhost. This will not work because localhost refers to the network inside the agent, and the server is not running inside the agent, it is running in a sibling container which is on a different docker network.

Hi @ashwilliams1, thanks for the quick reply!
I also tried using version 1.9.x but it was the same. I was not aware that Drone Agent was deprecated. I should mention that this Drone setup has been sitting unused for a while and I’m just reactivating it these days.
About the Agent not being able to reach the server because it is using localhost, I noticed that but I did not find a way to configure this differently (at least not with the environment variables that I found in the docs).

About the Agent not being able to reach the server because it is using localhost, I noticed that but I did not find a way to configure this differently

please see the rpc host and proto variables in the runner install docs:
https://docs.drone.io/runner/docker/installation/linux/

Thanks for your help again. I did not change anything but then checked again the next day and the builds that were stuck somehow managed to run successfully. Not sure what happened though because, like I said, I did not change a thing.
What’s more, builds now run without any issues even though the Drone Agent logs still shows the same error:
2020/07/16 19:44:50 [ERR] POST drone-server://localhost:8080/rpc/v1/request request failed: Post drone-server://localhost:8080/rpc/v1/request: unsupported protocol scheme "drone-server"

the endpoint /rpc/v1/request was only used by older versions of Drone that are now deprecated. Please ensure you are using the stable release, which is drone/drone-runner-docker:1. See https://docs.drone.io/runner/docker/installation/linux/

Got it. Thanks again!