Drone Build in Pending State in EKS

Drone Support
1

Hi,
I have deployed drone-ci and drone-runner using helm charts provided in the official git repo. But when a build is run, it is staying in the pending state.

  1. I have deployed using helm chart for both server and kube-runner
  2. Enabled LOGS_TRACE in both server and runner.
  3. Runner Logs
time="2021-01-18T13:17:34Z" level=info msg="successfully pinged the remote server"
time="2021-01-18T13:17:34Z" level=info msg="polling the remote server" capacity=100 endpoint="http://drone" kind=pipeline type=kubernetes
  1. Server Logs
license: ""
authn:
  endpoint: ""
  secret: ""
  skipverify: false
agent:
  disabled: false
azureblob:
  containername: ""
  storageaccountname: ""
  storageaccesskey: ""
convert:
  extension: ""
  endpoint: ""
  secret: ""
  skipverify: false
cleanup:
  disabled: false
  interval: 24h0m0s
  running: 24h0m0s
  pending: 24h0m0s
cron:
  disabled: false
  interval: 30m0s
cloning:
  alwaysauth: false
  username: ""
  password: ""
  image: ""
  pull: IfNotExists
database:
  driver: sqlite3
  datasource: /data/database.sqlite
  secret: ""
  legacybatch: false
datadog:
  enabled: true
  endpoint: https://stats.drone.ci/api/v1/series
  token: ""
docker:
  config: ""
http:
  allowedhosts: []
  hostsproxyheaders: []
  sslredirect: false
  ssltemporaryredirect: false
  sslhost: ""
  sslproxyheaders: {}
  stsseconds: 0
  stsincludesubdomains: false
  stspreload: false
  forcestsheader: false
  browserxssfilter: true
  framedeny: true
  contenttypenosniff: false
  contentsecuritypolicy: ""
  referrerpolicy: ""
jsonnet:
  enabled: false
logging:
  debug: false
  trace: true
  color: false
  pretty: false
  text: false
prometheus:
  enableanonymousaccess: false
proxy:
  addr: https://drone-ci.hellomary.tech
  host: drone-ci.hellomary.tech
  proto: https
registration:
  closed: false
registries:
  endpoint: ""
  password: ""
  skipverify: false
repository:
  filter: []
  visibility: ""
  trusted: false
runner:
  local: false
  image: drone/controller:1
  platform: linux/amd64
  os: linux
  arch: amd64
  kernel: ""
  variant: ""
  machine: drone-5d45899ddd-cs8qb
  capacity: 2
  labels: {}
  volumes: []
  networks: []
  devices: []
  privileged: []
  environ: {}
  limits:
    memswaplimit: 0
    memlimit: 0
    shmsize: 0
    cpuquota: 0
    cpushares: 0
    cpuset: ""
nomad:
  enabled: false
  datacenters:
  - dc1
  namespace: ""
  region: ""
  prefix: drone-job-
  image: ""
  imagepull: false
  memory: 1024
  labels: {}
  cpu: 500
kube:
  enabled: false
  namespace: ""
  path: ""
  url: ""
  ttl: 300
  serviceaccountname: ""
  pullpolicy: Always
  image: ""
rpc:
  server: ""
  secret: d860eb68a128ca04fe6733c053efcb97
  debug: false
  host: drone-ci.hellomary.tech
  proto: https
s3:
  bucket: ""
  prefix: ""
  endpoint: ""
  pathstyle: false
secrets:
  endpoint: ""
  password: ""
  skipverify: false
server:
  addr: https://drone-ci.hellomary.tech
  host: drone-ci.hellomary.tech
  port: :80
  proto: https
  pprof: false
  acme: false
  email: ""
  cert: ""
  key: ""
session:
  timeout: 720h0m0s
  secret: 060HeHywpqHi4LY3vcAHAMSiTrOWZvRJ
  secure: false
  mappingfile: ""
status:
  disabled: false
  name: ""
users:
  create:
    username: ""
    machine: false
    admin: false
    token: ""
  filter: []
  minage: 0s
validate:
  endpoint: ""
  secret: ""
  skipverify: false
webhook:
  events: []
  endpoint: []
  secret: ""
  skipverify: false
yaml:
  endpoint: ""
  secret: ""
  skipverify: false
bitbucket:
  clientid: ""
  clientsecret: ""
  skipverify: false
  debug: false
gitea:
  server: https://git.acaziasoft.com
  clientid: c22f7812-0967-413c-8bba-bf087874a5fe
  clientsecret: BXh1-HDlYaAZyghUothEnW2yjVNF3coWivx1XtoUJdk=
  skipverify: false
  scope:
  - repo
  - repo:status
  - user:email
  - read:org
  debug: false
github:
  server: https://github.com
  apiserver: https://api.github.com
  clientid: ""
  clientsecret: ""
  skipverify: false
  scope:
  - repo
  - repo:status
  - user:email
  - read:org
  ratelimit: 0
  debug: false
gitlab:
  server: https://gitlab.com
  clientid: ""
  clientsecret: ""
  skipverify: false
  debug: false
gogs:
  server: ""
  skipverify: false
  debug: false
stash:
  server: ""
  consumerkey: ""
  consumersecret: ""
  privatekey: ""
  skipverify: false
  debug: false
{"fields.time":"2021-01-19T05:54:33Z","latency":17084,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:43762","request":"/","request-id":"1nH8UkiL49wa8sKLm3hgxGGe171","time":"2021-01-19T05:54:33Z"}
{"arch":"","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"","time":"2021-01-19T05:54:42Z","type":"kubernetes","variant":""}
{"fields.time":"2021-01-19T05:54:43Z","latency":17770,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:43830","request":"/","request-id":"1nH8W2avGSbb97XCN3864UUseIT","time":"2021-01-19T05:54:43Z"}
{"arch":"","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"","time":"2021-01-19T05:54:52Z","type":"kubernetes","variant":""}
{"fields.time":"2021-01-19T05:54:53Z","latency":15847,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:43874","request":"/","request-id":"1nH8XJ17Bms7dfn08cQ1bhOPzLh","time":"2021-01-19T05:54:53Z"}
{"fields.time":"2021-01-19T05:55:03Z","latency":32754,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:43912","request":"/","request-id":"1nH8YWeUDpzbL7rBwyMKiG8Lx1N","time":"2021-01-19T05:55:03Z"}
{"fields.time":"2021-01-19T05:55:13Z","latency":21088,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:43954","request":"/","request-id":"1nH8ZsUGFOgDyYLZSupK4U7RyJR","time":"2021-01-19T05:55:13Z"}
{"arch":"","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"","time":"2021-01-19T05:55:22Z","type":"kubernetes","variant":""}
{"fields.time":"2021-01-19T05:55:23Z","latency":16836,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:43996","request":"/","request-id":"1nH8b4J6KvCXKxZu4uFVjkMe5sX","time":"2021-01-19T05:55:23Z"}
{"arch":"","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"","time":"2021-01-19T05:55:32Z","type":"kubernetes","variant":""}
{"fields.time":"2021-01-19T05:55:33Z","latency":16773,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:44034","request":"/","request-id":"1nH8cKejr0C42ee4gpctdmOceBD","time":"2021-01-19T05:55:33Z"}
{"fields.time":"2021-01-19T05:55:43Z","latency":16597,"level":"debug","method":"GET","msg":"","remote":"10.0.1.164:44102","request":"/","request-id":"1nH8dYQZezF57W4lJyzo5hiCmZY","time":"2021-01-19T05:55:43Z"}
  1. Yaml configuration file
kind: pipeline
name: default
steps:
- name: build
  image: docker
  commands:
  - docker build -t 309937699281.dkr.ecr.us-west-1.amazonaws.com/chatbot_server:${DRONE_COMMIT_SHA:0:7} .

- name: publish  
  image: plugins/ecr
  settings:
    access_key:
      from_secret: access_key
    secret_key:
      from_secret: secret_key
    repo: chatbot_server
    tags: ["${DRONE_COMMIT_SHA:0:7}","latest"]
    registry: 309937699281.dkr.ecr.us-west-1.amazonaws.com


---
kind: secret
name: access_key
get:
  path: ecr
  name: access_key

---
kind: secret
name: secret_key
get:
  path: ecr
  name: secret_token
  1. Build details for your pending builds via API
HTTP/2 200
date: Tue, 19 Jan 2021 06:03:40 GMT
content-type: application/json
content-length: 1120
cache-control: no-cache, no-store, must-revalidate, private, max-age=0
expires: Thu, 01 Jan 1970 00:00:00 UTC
pragma: no-cache
vary: Origin
strict-transport-security: max-age=15724800; includeSubDomains
access-control-allow-origin: *
access-control-allow-credentials: true

{"id":7,"repo_id":8,"trigger":"@hook","number":7,"status":"killed","event":"push","action":"","link":"https://git.acaziasoft.com/HeyMary/heymary-chatbot-server/compare/42ee60925f8b93465c27b5e383291e5aa795435c...2e40a346dcb05fe0f42c0fc458294473c027e2ab","timestamp":0,"message":"Update '.drone.yml'\n","before":"42ee60925f8b93465c27b5e383291e5aa795435c","after":"2e40a346dcb05fe0f42c0fc458294473c027e2ab","ref":"refs/heads/devops","source_repo":"","source":"devops","target":"devops","author_login":"himanshu.sadadiya","author_name":"himanshu.sadadiya","author_email":"sadadiyahimanshu@gmail.com","author_avatar":"https://git.acaziasoft.com/user/avatar/himanshu.sadadiya/-1","sender":"himanshu.sadadiya","started":1611036020,"finished":1611036020,"created":1611035254,"updated":1611035254,"version":2,"stages":[{"id":5,"repo_id":8,"build_id":7,"number":1,"name":"default","kind":"pipeline","type":"docker","status":"skipped","errignore":false,"exit_code":0,"os":"linux","arch":"amd64","started":1611036021,"stopped":1611036021,"created":1611035254,"updated":1611035254,"version":2,"on_success":true,"on_failure":false}]}
2

Hello there,

Kindly go through the below document once and let us know if this helps:
http://discuss.harness.io/t/builds-are-stuck-in-pending-status/4437

Also if you still face the issue share with us :

  1. Provide your server configuration
  2. Provide your runner configuration
  3. Provide your full server logs with trace logging enabled
  4. Provide your full runner logs with trace logging enabled
    a. Enable runner variable DRONE_RPC_DUMP_HTTP=true
    b. Enable runner variable DRONE_RPC_DUMP_HTTP_BODY=true
  5. Provide your yaml configuration file
  6. Confirm you have checked all common issues described in this thread and quickly summarize how you ruled each item out.
  7. Provide the build details for your pending builds via this API endpoint.
3

I have followed the documentation. And It boiled to a problem in my configuration file. Now the job is running but I am getting the following error while using ECR plugin to build and push the image to ECR.

+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock
time="2021-01-19T12:50:50Z" level=fatal msg="Error authenticating: exit status 1"

I have checked many articles but none of them have a solution. What is the workaround for this. I have deployed everything is EKS including drone-ci

@whwalter Can you please tell me how did you configure ECR in Kubernetes with drone ci??