Pipeline stuck in pending status

Drone Support
1

I have checked Builds are Stuck in Pending Status many times but can’t find any clue.

Brief Info

  • I’m using official charts at drone/charts
  • Trace enabled
  • DRONE_AGENTS_DISABLED and DRONE_KUBERNETES_ENABLED not set
  • Protected mode disabled
  • Recently upgraded gitea from 1.18.x to 1.19.0-rc1

Server Config

Version: 1.16.0

DRONE_GITEA_SERVER: https://git.domain.ltd
DRONE_GITEA_CLIENT_ID: a2d3b6e0-****-****-****-bd65c56fc90b
DRONE_GITEA_CLIENT_SECRET: rU9I56YjJuxNWMp38HXgkTzUr6egJi2QatSE****nDFr

DRONE_DATABASE_DRIVER: mysql
DRONE_DATABASE_DATASOURCE: user:pass@tcp(db.domain.ltd:3306)/drone?parseTime=true
DRONE_DATABASE_SECRET: fa2cda09cd1483b7f6ce7d40****10f2

DRONE_SERVER_HOST: drone.domain.ltd
DRONE_SERVER_PROTO: https
DRONE_GIT_ALWAYS_AUTH: "false"
DRONE_USER_CREATE: username:giteadmin,admin:true
DRONE_LOGS_TRACE: "true"

DRONE_RPC_SECRET: 4975fe9e8f2157b44616d765****319c

Agent Config

Version: 1.8.3

DRONE_RPC_HOST: drone:8080
DRONE_RPC_PROTO: http
DRONE_RPC_SECRET: 4975fe9e8f2157b44616d765****319c
DRONE_SECRET_PLUGIN_ENDPOINT: http://drone-kubernetes-secrets:3000
DRONE_SECRET_PLUGIN_TOKEN: 4b83c3c772e7852f4ae15cc1****fff7
DRONE_LOGS_TRACE: "true"
DRONE_RPC_DUMP_HTTP: "true"
DRONE_RPC_DUMP_HTTP_BODY: "true"
DRONE_RUNNER_NETWORK_OPTS: com.docker.network.driver.mtu:1450

Server Log

{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-03-13T10:44:39Z","type":"docker","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-03-13T10:44:39Z","type":"docker","variant":""}
{"commit":"5866d339437fefcdd8fb1db72abcb41f11a7183c","event":"push","level":"debug","msg":"webhook parsed","name":"xshop","namespace":"maoxuner","time":"2023-03-13T10:44:46Z"}
{"commit":"5866d339437fefcdd8fb1db72abcb41f11a7183c","event":"push","level":"debug","msg":"trigger: received","ref":"refs/heads/main","repo":"maoxuner/xshop","time":"2023-03-13T10:44:46Z"}
{"fields.time":"2023-03-13T10:44:47Z","latency":201323985,"level":"debug","method":"POST","msg":"","remote":"10.42.0.19:34704","request":"/hook?secret=GUSsaLAZqVxCo1hORmycpckzmKhBp6rP","request-id":"2MxLSSZn64j4GDjbbSbZOmGHPgp","time":"2023-03-13T10:44:47Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-03-13T10:45:09Z","type":"docker","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-03-13T10:45:09Z","type":"docker","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-03-13T10:45:19Z","type":"docker","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-03-13T10:45:19Z","type":"docker","variant":""}
{"fields.time":"2023-03-13T10:45:30Z","latency":4788014,"level":"debug","method":"GET","msg":"","remote":"10.42.0.19:34704","request":"/maoxuner/xshop/70","request-id":"2MxLY2gXncoGuEm0ewbJHPvZkdM","time":"2023-03-13T10:45:30Z"}
{"level":"debug","msg":"events: stream error","request-id":"2MxLLinXTgB6JuMq59SHl1pgXt0","time":"2023-03-13T10:45:30Z","user.admin":true,"user.login":"maoxuner"}
{"level":"debug","msg":"events: stream closed","request-id":"2MxLLinXTgB6JuMq59SHl1pgXt0","time":"2023-03-13T10:45:30Z","user.admin":true,"user.login":"maoxuner"}
{"fields.time":"2023-03-13T10:45:30Z","latency":98549468744,"level":"debug","method":"GET","msg":"","remote":"10.42.0.19:45838","request":"/api/stream","request-id":"2MxLLinXTgB6JuMq59SHl1pgXt0","time":"2023-03-13T10:45:30Z"}
{"fields.time":"2023-03-13T10:45:30Z","latency":4473569,"level":"debug","method":"GET","msg":"","remote":"10.42.0.19:34704","request":"/api/user","request-id":"2MxLY2vDQay5wHXuKS4w5TPPvke","time":"2023-03-13T10:45:30Z"}
{"level":"debug","msg":"events: stream opened","request-id":"2MxLY4pDcCKUlK8SsDiFYoCQwa1","time":"2023-03-13T10:45:30Z","user.admin":true,"user.login":"maoxuner"}
{"level":"debug","msg":"api: root access granted","name":"xshop","namespace":"maoxuner","request-id":"2MxLY0tmKyQghZqkHHMvm6hSldv","time":"2023-03-13T10:45:31Z","user.admin":true,"user.login":"maoxuner"}
{"fields.time":"2023-03-13T10:45:31Z","latency":21165552,"level":"debug","method":"GET","msg":"","remote":"10.42.0.19:34704","request":"/api/repos/maoxuner/xshop","request-id":"2MxLY0tmKyQghZqkHHMvm6hSldv","time":"2023-03-13T10:45:31Z"}
{"fields.time":"2023-03-13T10:45:31Z","latency":7982988,"level":"debug","method":"GET","msg":"","remote":"10.42.0.19:47586","request":"/api/user/repos?latest=true","request-id":"2MxLY9PxYT8Mbg8RaiJsXQV9RJK","time":"2023-03-13T10:45:31Z"}
{"level":"debug","msg":"api: root access granted","name":"xshop","namespace":"maoxuner","request-id":"2MxLYBbg5Ne9eJBdSr77BLshfl5","time":"2023-03-13T10:45:31Z","user.admin":true,"user.login":"maoxuner"}
{"fields.time":"2023-03-13T10:45:31Z","latency":170274181,"level":"debug","method":"GET","msg":"","remote":"10.42.0.19:47586","request":"/api/repos/maoxuner/xshop/builds/70","request-id":"2MxLYBbg5Ne9eJBdSr77BLshfl5","time":"2023-03-13T10:45:31Z"}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-03-13T10:45:49Z","type":"docker","variant":""}
{"arch":"amd64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-03-13T10:45:49Z","type":"docker","variant":""}

Agent Log

POST /rpc/v2/stage HTTP/1.1
Host: drone:8080
User-Agent: Go-http-client/1.1
Content-Length: 89
X-Drone-Token: 4975fe9e8f2157b44616d765cae2319c
Accept-Encoding: gzip

{"kind":"pipeline","type":"docker","os":"linux","arch":"amd64","variant":"","kernel":""}
POST /rpc/v2/stage HTTP/1.1
Host: drone:8080
User-Agent: Go-http-client/1.1
Content-Length: 89
X-Drone-Token: 4975fe9e8f2157b44616d765cae2319c
Accept-Encoding: gzip

{"kind":"pipeline","type":"docker","os":"linux","arch":"amd64","variant":"","kernel":""}
HTTP/1.1 204 No Content
Content-Length: 0
Cache-Control: no-cache, no-store, must-revalidate, private, max-age=0
Date: Mon, 13 Mar 2023 10:45:09 GMT
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Pragma: no-cache
X-Accel-Expires: 0

time="2023-03-13T10:45:09Z" level=trace msg="http: no content returned: re-connect and re-try"
HTTP/1.1 204 No Content
Content-Length: 0
Cache-Control: no-cache, no-store, must-revalidate, private, max-age=0
Date: Mon, 13 Mar 2023 10:45:09 GMT
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Pragma: no-cache
X-Accel-Expires: 0

time="2023-03-13T10:45:09Z" level=trace msg="http: no content returned: re-connect and re-try"
POST /rpc/v2/stage HTTP/1.1
Host: drone:8080
User-Agent: Go-http-client/1.1
Content-Length: 89
X-Drone-Token: 4975fe9e8f2157b44616d765cae2319c
Accept-Encoding: gzip

{"kind":"pipeline","type":"docker","os":"linux","arch":"amd64","variant":"","kernel":""}
POST /rpc/v2/stage HTTP/1.1
Host: drone:8080
User-Agent: Go-http-client/1.1
Content-Length: 89
X-Drone-Token: 4975fe9e8f2157b44616d765cae2319c
Accept-Encoding: gzip

{"kind":"pipeline","type":"docker","os":"linux","arch":"amd64","variant":"","kernel":""}
time="2023-03-13T10:45:49Z" level=trace msg="http: no content returned: re-connect and re-try"
HTTP/1.1 204 No Content
Content-Length: 0
Cache-Control: no-cache, no-store, must-revalidate, private, max-age=0
Date: Mon, 13 Mar 2023 10:45:49 GMT
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Pragma: no-cache
X-Accel-Expires: 0

HTTP/1.1 204 No Content
Content-Length: 0
Cache-Control: no-cache, no-store, must-revalidate, private, max-age=0
Date: Mon, 13 Mar 2023 10:45:49 GMT
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Pragma: no-cache
X-Accel-Expires: 0

time="2023-03-13T10:45:49Z" level=trace msg="http: no content returned: re-connect and re-try"

Yaml File(.drone.yaml, same as project settings)

kind: pipeline
type: docker
name: default

platform:
  os: linux
  arch: amd64

steps:
  - name: debug
    image: bash
    commands:
      - echo hello

Build Info

{
  "id": 352,
  "repo_id": 14,
  "trigger": "@hook",
  "number": 70,
  "status": "pending",
  "event": "push",
  "action": "",
  "link": "https://git.domain.ltd/maoxuner/xshop/compare/c142d0416e7ed67007aa29e0d6ccc384f499ab6a...5866d339437fefcdd8fb1db72abcb41f11a7183c",
  "timestamp": 0,
  "message": "fix: test platform\n",
  "before": "c142d0416e7ed67007aa29e0d6ccc384f499ab6a",
  "after": "5866d339437fefcdd8fb1db72abcb41f11a7183c",
  "ref": "refs/heads/main",
  "source_repo": "",
  "source": "main",
  "target": "main",
  "author_login": "maoxuner",
  "author_name": "maoxuner",
  "author_email": "maoxuner@domain.ltd",
  "author_avatar": "https://git.domain.ltd/avatars/9ba051061f02e52657cd58427002c40d",
  "sender": "maoxuner",
  "started": 0,
  "finished": 0,
  "created": 1678704286,
  "updated": 1678704286,
  "version": 1,
  "stages": [
    {
      "id": 348,
      "repo_id": 14,
      "build_id": 352,
      "number": 1,
      "name": "default",
      "kind": "pipeline",
      "type": "docker",
      "status": "pending",
      "errignore": false,
      "exit_code": 0,
      "os": "linux",
      "arch": "amd64",
      "started": 0,
      "stopped": 0,
      "created": 1678704286,
      "updated": 1678704286,
      "version": 1,
      "on_success": true,
      "on_failure": false
    }
  ]
}
2

I have upgraded gitea from 1.19.0-r1 to 1.19.0. Builds still stuck.

3

There maybe something to do with database server migration. I’m not sure.

I recently migrated database from mariadb(10) to mysql(8) with mysqldump. If I start drone with a new database, everthing goes well.