Drone builds staying grey in the UI

Drone Support
,
1

Hi

I recently shifted our Gitea repository on a Kubernetes cluster exposed through a LoadBalancer type IP address.
I also upgraded Drone to the latest version but somehow things started to fail. Whenever I start drone and drone-runner-docker, the initial build triggered either through the Drone user-interface or by committing to a monitored branch will start and run. But any subsequent build does not start anymore. If I restart the drone-runner-docker in the meantime, the next build will start, except if the time between the scheduling of the build and the restart of the drone-runner-docker is too high (like 2 minutes or similar).

Sometimes it is even required to restart he Drone sever to get things running again. I am currently running version 2.12 of the Drone server and 1.8.1 of the drone-runner-docker.

I already enabled debug logging on both server and runner, but there’s no information that would help. I also checked the log files from Gitea, that all seems perfectly fine. Even the webhooks executed successfully.

Any idea what could be wrong here?

2

Well, I cannot answer my question really but I noticed that it only ever blocks / happens when using the plugins/ansible image. All other builds run perfectly normal and do not block agents.

3

Please use the following thread to troubleshoot:
https://drone.discourse.group/t/builds-are-stuck-in-pending-status/11275

If you are unable to resolve the problem after walking through the above troubleshooting guide, please follow the instructions in the Still having issues section, which provides a list of information that we would require in order to further assist.

4

Ok. So here we go. A small explanation to what was done here:

  1. I commited some changes to a test repository in Gitea. With a PR, the build was triggered, and this time actually ran through (I restarted Drone server and agent beforehand to make sure this works). This was build number “2”.
  2. After running through, the build showed up with a green icon in Drone, but the Gitea reference still shows “pending build”.
  3. I commited another change to the same PRed branch which triggered another build, but that one did not ever start again. This was or is build number 3. And now a little update: in the meantime the build process started,…so maybe this is Docker related, I will include docker logs from the runner for that reason. But as above, the even after completing that build 3, it will show as “this build is running” in Gitea.

The server config:

#!/bin/ash
docker run \
  -v /etc/certs/drone.crt:/etc/certs/drone.crt \
  -v /etc/certs/drone.key:/etc/certs/drone.key \
  -v /etc/ssl/certs:/etc/ssl/certs \
  -v /usr/local/share/ca-certificates:/usr/local/share/ca-certificates \
  --volume=/var/lib/drone:/data \
  --env=DRONE_AGENTS_ENABLED=true \
  --env=DRONE_GITEA_SERVER=https://git.comp.ags \
  --env=DRONE_GITEA_CLIENT_ID=xxx \
  --env=DRONE_GITEA_CLIENT_SECRET=yyy \
  --env=DRONE_RPC_SECRET=yyy \
  --env=DRONE_SERVER_HOST=drone.comp.ags \
  --env=DRONE_SERVER_PROTO=https \
  --env=DRONE_TLS_CERT=/etc/certs/drone.crt \
  --env=DRONE_TLS_KEY=/etc/certs/drone.key \
  --env=DRONE_LOGS_TRACE=true \
  --publish=80:80 \
  --publish=443:443 \
  --restart=always \
  --detach=true \
  --name=drone \
  drone/drone:2.12

The runner config:

#!/bin/ash
docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \
  -e DRONE_RUNNER_VOLUMES=/usr/local/share/ca-certificates:/usr/local/share/ca-certificates,/etc/ssl/certs:/etc/ssl/certs \
  -e DRONE_RPC_PROTO=https \
  -e DRONE_RPC_HOST=drone.comp.ags \
  -e DRONE_RPC_SECRET=yyy \
  -e DRONE_RUNNER_CAPACITY=2 \
  -e DRONE_RUNNER_NAME=drone-runner-01 \
  -e DRONE_LOGS_TRACE=true \
  -p 3000:3000 \
  --restart always \
  --name runner \
  drone/drone-runner-docker:1.8.1

Drone Server Logs:

{"admin":true,"level":"debug","msg":"api: access granted","name":"build-test-repo","namespace":"inzanez","read":true,"request-id":"29ePAxt0hbTi23akTmI2xYH1iwW","time":"2022-05-25T09:20:37Z","user.login":"inzanez","visibility":"private","write":true}
{"admin":true,"level":"debug","msg":"api: access granted","name":"build-test-repo","namespace":"inzanez","read":true,"request-id":"29ePAxt0hbTi23akTmI2xYH1iwW","time":"2022-05-25T09:20:37Z","user.login":"inzanez","visibility":"private","write":true}
{"commit":"9dba7b02119eaa63b5c3cffada083b2c3ad47c34","event":"pull_request","level":"debug","msg":"trigger: received","ref":"refs/pull/1/head","repo":"inzanez/build-test-repo","time":"2022-05-25T09:20:37Z"}
{"commit":"9dba7b02119eaa63b5c3cffada083b2c3ad47c34","event":"pull_request","level":"info","msg":"trigger: skipping pipeline, does not match event","pipeline":"production-build","ref":"refs/pull/1/head","repo":"inzanez/build-test-repo","time":"2022-05-25T09:20:37Z"}
{"commit":"9dba7b02119eaa63b5c3cffada083b2c3ad47c34","event":"pull_request","level":"info","msg":"trigger: skipping pipeline, does not match event","pipeline":"staging-build","ref":"refs/pull/1/head","repo":"inzanez/build-test-repo","time":"2022-05-25T09:20:37Z"}
{"build.id":98,"build.number":5,"error":"Post \"https://git.comp.ags/api/v1/repos/Comp/other-test-repo/statuses/245c12bd42b3f9b549b341be0c7197156b22d392\": read tcp 172.17.0.2:45492-\u003e10.10.0.4:443: read: connection timed out","level":"warning","msg":"manager: cannot publish status","repo.id":7,"stage.id":98,"time":"2022-05-25T09:20:39Z"}
{"admin":true,"level":"debug","msg":"api: access granted","name":"build-test-repo","namespace":"inzanez","read":true,"request-id":"29ePBScuGiUlwvzgMI4G3NnDbAU","time":"2022-05-25T09:20:41Z","user.login":"inzanez","visibility":"private","write":true}
{"fields.time":"2022-05-25T09:20:41Z","latency":805133,"level":"debug","method":"GET","msg":"","remote":"10.20.21.20:56928","request":"/api/repos/inzanez/build-test-repo/builds?page=1\u0026per_page=50","request-id":"29ePBScuGiUlwvzgMI4G3NnDbAU","time":"2022-05-25T09:20:41Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2022-05-25T09:20:45Z","type":"docker","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2022-05-25T09:20:55Z","type":"docker","variant":""}
{"level":"debug","machine":"drone-runner-01","msg":"manager: accept stage","stage-id":101,"time":"2022-05-25T09:20:55Z"}
{"level":"debug","machine":"drone-runner-01","msg":"manager: stage accepted","stage-id":101,"time":"2022-05-25T09:20:55Z"}
{"level":"debug","msg":"manager: fetching stage details","step-id":101,"time":"2022-05-25T09:20:55Z"}
{"level":"debug","msg":"manager: updating step status","step.id":269,"step.name":"clone","step.status":"running","time":"2022-05-25T09:20:55Z"}
{"level":"debug","msg":"manager: updating step status","step.id":269,"step.name":"clone","step.status":"success","time":"2022-05-25T09:20:56Z"}
{"level":"debug","msg":"manager: updating step status","step.id":271,"step.name":"build-frontend","step.status":"running","time":"2022-05-25T09:20:56Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:21:00Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:21:00Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"windows","time":"2022-05-25T09:21:10Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"windows","time":"2022-05-25T09:21:10Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:21:40Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:21:40Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"windows","time":"2022-05-25T09:21:50Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"windows","time":"2022-05-25T09:21:50Z","type":"exec","variant":""}
{"level":"debug","msg":"manager: updating step status","step.id":270,"step.name":"build-setup-frontend","step.status":"running","time":"2022-05-25T09:20:56Z"}
{"level":"debug","msg":"manager: updating step status","step.id":271,"step.name":"build-frontend","step.status":"success","time":"2022-05-25T09:23:18Z"}
{"level":"debug","msg":"manager: updating step status","step.id":272,"step.name":"build-docker-container","step.status":"running","time":"2022-05-25T09:23:18Z"}
{"level":"debug","msg":"manager: updating step status","step.id":272,"step.name":"build-docker-container","step.status":"running","time":"2022-05-25T09:23:34Z"}
{"level":"debug","msg":"manager: updating step status","step.id":272,"step.name":"build-docker-container","step.status":"success","time":"2022-05-25T09:23:34Z"}
{"level":"debug","msg":"manager: stage is complete. teardown","stage.id":101,"time":"2022-05-25T09:23:34Z"}
{"build.id":101,"build.number":2,"level":"debug","msg":"manager: build is finished, teardown","repo.id":82,"stage.id":101,"time":"2022-05-25T09:23:34Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:23:40Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:23:40Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"windows","time":"2022-05-25T09:23:50Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"windows","time":"2022-05-25T09:23:50Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:24:20Z","type":"exec","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"windows","time":"2022-05-25T09:24:20Z","type":"exec","variant":""}

Drone Runner Logs:

time="2022-05-25T09:20:55Z" level=debug msg="stage received" stage.id=101 stage.name=development-build stage.number=1 thread=2
time="2022-05-25T09:20:55Z" level=debug msg="stage accepted" stage.id=101 stage.name=development-build stage.number=1 thread=2
time="2022-05-25T09:20:55Z" level=debug msg="stage details fetched" build.id=101 build.number=2 repo.id=82 repo.name=build-test-repo repo.namespace=inzanez stage.id=101 stage.name=development-build stage.number=1 thread=2
time="2022-05-25T09:20:55Z" level=trace msg="secret: database: no matching secret" kind=secret name=registryUrl thread=2
time="2022-05-25T09:20:55Z" level=trace msg="secret: encrypted: no matching secret" kind=secret name=registryUrl thread=2
time="2022-05-25T09:20:55Z" level=trace msg="secret: database: no matching secret" kind=secret name=registryUrl thread=2
time="2022-05-25T09:20:55Z" level=debug msg="updated stage to running" build.id=101 build.number=2 repo.id=82 repo.name=build-test-repo repo.namespace=inzanez stage.id=101 stage.name=development-build stage.number=1 thread=2
time="2022-05-25T09:20:56Z" level=debug msg="received exit code 0" build.id=101 build.number=2 repo.id=82 repo.name=build-test-repo repo.namespace=inzanez stage.id=101 stage.name=development-build stage.number=1 step.name=clone thread=2
time="2022-05-25T09:21:25Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2022-05-25T09:22:02Z" level=debug msg="received exit code 0" build.id=101 build.number=2 repo.id=82 repo.name=build-test-repo repo.namespace=inzanez stage.id=101 stage.name=development-build stage.number=1 step.name=build-setup-frontend thread=2
time="2022-05-25T09:22:05Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2022-05-25T09:22:45Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2022-05-25T09:23:18Z" level=debug msg="received exit code 0" build.id=101 build.number=2 repo.id=82 repo.name=build-test-repo repo.namespace=inzanez stage.id=101 stage.name=development-build stage.number=1 step.name=build-frontend thread=2
time="2022-05-25T09:23:25Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2022-05-25T09:23:34Z" level=debug msg="received exit code 0" build.id=101 build.number=2 repo.id=82 repo.name=build-test-repo repo.namespace=inzanez stage.id=101 stage.name=development-build stage.number=1 step.name=build-docker-container thread=2
time="2022-05-25T09:24:05Z" level=trace msg="http: no content returned: re-connect and re-try"

Build status:
{
  "id": 101,
  "repo_id": 82,
  "trigger": "inzanez",
  "number": 2,
  "parent": 1,
  "status": "success",
  "event": "pull_request",
  "action": "opened",
  "link": "https://git.comp.ags/inzanez/build-test-repo/pulls/1",
  "timestamp": 0,
  "title": "Add something",
  "message": "Add something",
  "before": "",
  "after": "9dba7b02119eaa63b5c3cffada083b2c3ad47c34",
  "ref": "refs/pull/1/head",
  "source_repo": "inzanez/build-test-repo",
  "source": "drone-build-test",
  "target": "develop",
  "author_login": "inzanez",
  "author_name": "",
  "author_email": "inzanez@comp.ags",
  "author_avatar": "https://secure.gravatar.com/avatar/ae174916a4bc5aee3bea4937bfa63422?d=identicon",
  "sender": "inzanez",
  "started": 1653470455,
  "finished": 1653470614,
  "created": 1653470437,
  "updated": 1653470455,
  "version": 3,
  "stages": [
    {
      "id": 101,
      "repo_id": 82,
      "build_id": 101,
      "number": 1,
      "name": "development-build",
      "kind": "pipeline",
      "type": "docker",
      "status": "success",
      "errignore": false,
      "exit_code": 0,
      "machine": "drone-runner-01",
      "os": "linux",
      "arch": "amd64",
      "started": 1653470455,
      "stopped": 1653470614,
      "created": 1653470437,
      "updated": 1653470614,
      "version": 4,
      "on_success": true,
      "on_failure": false,
      "steps": [
        {
          "id": 269,
          "step_id": 101,
          "number": 1,
          "name": "clone",
          "status": "success",
          "exit_code": 0,
          "started": 1653470455,
          "stopped": 1653470456,
          "version": 4,
          "image": "drone/git:latest"
        },
        {
          "id": 270,
          "step_id": 101,
          "number": 2,
          "name": "build-setup-frontend",
          "status": "success",
          "exit_code": 0,
          "started": 1653470456,
          "stopped": 1653470522,
          "version": 4,
          "depends_on": [
            "clone"
          ],
          "image": "docker.io/library/node:14-alpine"
        },
        {
          "id": 271,
          "step_id": 101,
          "number": 3,
          "name": "build-frontend",
          "status": "success",
          "exit_code": 0,
          "started": 1653470456,
          "stopped": 1653470598,
          "version": 4,
          "depends_on": [
            "clone"
          ],
          "image": "docker.io/library/node:14-alpine"
        },
        {
          "id": 272,
          "step_id": 101,
          "number": 4,
          "name": "build-docker-container",
          "status": "success",
          "exit_code": 0,
          "started": 1653470598,
          "stopped": 1653470614,
          "version": 5,
          "depends_on": [
            "build-setup-frontend",
            "build-frontend"
          ],
          "image": "docker.io/plugins/docker:latest",
          "schema": "https://drone-plugins.github.io/drone-docker/card.json"
        }
      ]
    }
  ]
}

Docker logs:

time="2022-05-25T11:20:55.276696668+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/374c270221675b5df0eaf9e7e164a401a70c202d2e7b4de174a4f26653065f71 pid=9203
time="2022-05-25T11:20:55.948742701+02:00" level=info msg="ignoring event" container=374c270221675b5df0eaf9e7e164a401a70c202d2e7b4de174a4f26653065f71 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-05-25T11:20:55.948756688+02:00" level=info msg="shim disconnected" id=374c270221675b5df0eaf9e7e164a401a70c202d2e7b4de174a4f26653065f71
time="2022-05-25T11:20:55.948840510+02:00" level=warning msg="cleaning up after shim disconnected" id=374c270221675b5df0eaf9e7e164a401a70c202d2e7b4de174a4f26653065f71 namespace=moby
time="2022-05-25T11:20:55.948858108+02:00" level=info msg="cleaning up dead shim"
time="2022-05-25T11:20:55.956291969+02:00" level=warning msg="cleanup warnings time=\"2022-05-25T11:20:55+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=9332\n"
time="2022-05-25T11:20:56.142836286+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/d1dce0a4bec02eff95a062600c2aa40fece9d1027f858703344ce3f3a6c19972 pid=9647
time="2022-05-25T11:20:56.195145800+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/23a411a34a5f116ad4184addb7648273c037e04b5623db4afd301d1e687f4bec pid=9684
time="2022-05-25T11:22:02.680422512+02:00" level=info msg="shim disconnected" id=d1dce0a4bec02eff95a062600c2aa40fece9d1027f858703344ce3f3a6c19972
time="2022-05-25T11:22:02.680500002+02:00" level=warning msg="cleaning up after shim disconnected" id=d1dce0a4bec02eff95a062600c2aa40fece9d1027f858703344ce3f3a6c19972 namespace=moby
time="2022-05-25T11:22:02.680515392+02:00" level=info msg="cleaning up dead shim"
time="2022-05-25T11:22:02.680525672+02:00" level=info msg="ignoring event" container=d1dce0a4bec02eff95a062600c2aa40fece9d1027f858703344ce3f3a6c19972 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-05-25T11:22:02.687323388+02:00" level=warning msg="cleanup warnings time=\"2022-05-25T11:22:02+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=10250\n"
time="2022-05-25T11:23:18.099672930+02:00" level=info msg="ignoring event" container=23a411a34a5f116ad4184addb7648273c037e04b5623db4afd301d1e687f4bec module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-05-25T11:23:18.099766900+02:00" level=info msg="shim disconnected" id=23a411a34a5f116ad4184addb7648273c037e04b5623db4afd301d1e687f4bec
time="2022-05-25T11:23:18.099813921+02:00" level=warning msg="cleaning up after shim disconnected" id=23a411a34a5f116ad4184addb7648273c037e04b5623db4afd301d1e687f4bec namespace=moby
time="2022-05-25T11:23:18.099824691+02:00" level=info msg="cleaning up dead shim"
time="2022-05-25T11:23:18.105883965+02:00" level=warning msg="cleanup warnings time=\"2022-05-25T11:23:18+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=10347\n"
time="2022-05-25T11:23:19.710940952+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b209eaab9b94e8419abdb0571fcec9acd2c4565bbcc647344fcbe62b2123586f pid=10522
time="2022-05-25T11:23:33.875255755+02:00" level=info msg="shim disconnected" id=b209eaab9b94e8419abdb0571fcec9acd2c4565bbcc647344fcbe62b2123586f
time="2022-05-25T11:23:33.875289635+02:00" level=info msg="ignoring event" container=b209eaab9b94e8419abdb0571fcec9acd2c4565bbcc647344fcbe62b2123586f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-05-25T11:23:33.875318555+02:00" level=warning msg="cleaning up after shim disconnected" id=b209eaab9b94e8419abdb0571fcec9acd2c4565bbcc647344fcbe62b2123586f namespace=moby
time="2022-05-25T11:23:33.875335025+02:00" level=info msg="cleaning up dead shim"
time="2022-05-25T11:23:33.881272409+02:00" level=warning msg="cleanup warnings time=\"2022-05-25T11:23:33+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=11623\n"
time="2022-05-25T11:33:24.658121508+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/927f518aeaffcf645257855ff83d6360502006e6c1f09ef63471bf11e63421e9 pid=11888
time="2022-05-25T11:33:25.327576623+02:00" level=info msg="shim disconnected" id=927f518aeaffcf645257855ff83d6360502006e6c1f09ef63471bf11e63421e9
time="2022-05-25T11:33:25.327635963+02:00" level=warning msg="cleaning up after shim disconnected" id=927f518aeaffcf645257855ff83d6360502006e6c1f09ef63471bf11e63421e9 namespace=moby
time="2022-05-25T11:33:25.327650750+02:00" level=info msg="cleaning up dead shim"
time="2022-05-25T11:33:25.327646258+02:00" level=info msg="ignoring event" container=927f518aeaffcf645257855ff83d6360502006e6c1f09ef63471bf11e63421e9 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-05-25T11:33:25.334403490+02:00" level=warning msg="cleanup warnings time=\"2022-05-25T11:33:25+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=12019\n"
time="2022-05-25T11:33:25.505736823+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e63d98791c6f19e5275dcfb2d7855d06cc8aaf27c02f714cbdd91bac02f8f22c pid=12345
time="2022-05-25T11:33:25.558847319+02:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/84bb3d6a92ab55462a60ee978a8459d00d8fb903b98056c49fd1e83b8b933bee pid=12386
time="2022-05-25T11:34:36.535974081+02:00" level=info msg="ignoring event" container=e63d98791c6f19e5275dcfb2d7855d06cc8aaf27c02f714cbdd91bac02f8f22c module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-05-25T11:34:36.535967948+02:00" level=info msg="shim disconnected" id=e63d98791c6f19e5275dcfb2d7855d06cc8aaf27c02f714cbdd91bac02f8f22c
time="2022-05-25T11:34:36.536035311+02:00" level=warning msg="cleaning up after shim disconnected" id=e63d98791c6f19e5275dcfb2d7855d06cc8aaf27c02f714cbdd91bac02f8f22c namespace=moby
time="2022-05-25T11:34:36.536047567+02:00" level=info msg="cleaning up dead shim"
time="2022-05-25T11:34:36.542113163+02:00" level=warning msg="cleanup warnings time=\"2022-05-25T11:34:36+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=12951\n"

My Gitea version is: 1.16.8.

5

And apart from that I notice that my drone runner freezes. So right now it is ‘running’ but not showing any logs anymore. And it doesn’t pick up work anymore. So it seems to be completely frozen although the last job it was executing is done and marked ‘complete’ in the Drone ui.

6

Well, it seems I do have another update: I just deployed Gitea using a hostPort exposure instead of exposing it with Metallb BGP based loadbalancer service type. And now things tend to work.

I will keep posting here until I found the issue, but it seems to be network related after all. My current guess would be that it’s got something to do with Metallb, but I need to perform more tests to be sure.

7

Ok,…so it doesn’t seem to be Metallb. I can currently only say that hostPort exposure does work without an issue. Using Metallb or PureLB in layer2 mode works without an issue as well. As soon as I enable BGP, OSPF or RIP however things start to fail. Gitea is reachable for a service exposed by the cluster, Drone runs on standalone nodes right now. I have no clue why things start to fail when enabling one of the routing protocols,…no issues with any other clients (like ‘git’ etc.)…

8

As a last update: the issues occur because of asymmetrical routing when using Metallb or any other loadbalancer type and registering routes to the Kubernetes network with the main gateway while sharing the network range of the Kubernetes nodes with the Drone machines and having the LB using a different address space.

This causes Drone to connect to the Git repo using the gateway while the Git repo will take the direct route through the Kubernetes node network back to Drone. And this leads to issues on the gateway that denies further packages from Drone to Git. Which ultimately breaks things.