Runner failed to delete pod

rad1k4l · September 2, 2021, 5:10pm

I am use kubernetes runner. Kubernetes runner is deployment in kubernetes. I try simple hello world pipeline. But all tries are crashed.

Runner logs is:

time=“2021-09-02T16:31:44Z” level=debug msg=“stage received” stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:44Z” level=debug msg=“stage received” stage.id=30 stage.name=drone-test stage.number=1 thread=79
time=“2021-09-02T16:31:44Z” level=debug msg=“stage accepted” stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:44Z” level=debug msg=“stage accepted by another runner” stage.id=30 stage.name=drone-test stage.number=1 thread=79
time=“2021-09-02T16:31:44Z” level=debug msg=“poller: request stage from remote server” thread=79
time=“2021-09-02T16:31:44Z” level=debug msg=“stage details fetched” build.id=30 build.number=30 repo.id=72 repo.name=admin-api-service repo.namespace=HIDDEN_ stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:44Z” level=debug msg=“updated stage to running” build.id=30 build.number=30 repo.id=72 repo.name=admin-api-service repo.namespace=HIDDEN stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:44Z” level=debug msg=“destroying the pipeline environment” build.id=30 build.number=30 repo.id=72 repo.name=admin-api-service repo.namespace=HIDDEN stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:49Z” level=error msg=“failed to delete pod” error=“pods “drone-ehsur5m8t4uv0jzuxybc” not found” namespace=drone pod=drone-ehsur5m8t4uv0jzuxybc
time=“2021-09-02T16:31:49Z” level=debug msg=“successfully destroyed the pipeline environment” build.id=30 build.number=30 repo.id=72 repo.name=admin-api-service repo.namespace=HIDDEN stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:49Z” level=debug msg=“updated stage to complete” build.id=30 build.number=30 duration=0 repo.id=72 repo.name=admin-api-service repo.namespace=HIDDEN stage.id=30 stage.name=drone-test stage.number=1 thread=83
time=“2021-09-02T16:31:49Z” level=debug msg=“poller: request stage from remote server” thread=83
time=“2021-09-02T16:31:49Z” level=debug msg=“done listening for cancellations” build.id=30 build.number=30 repo.id=72 repo.name=admin-api-service repo.namespace=HIDDEN stage.id=30 stage.name=drone-test stage.number=1 thread=83

bradrydzewski · September 2, 2021, 5:43pm

But all tries are crashed.

what do you mean by all tries are crash? It is unclear what problem you are experiencing. Without an error message, screenshot, etc I am unable to discern the problem, and am therefore unable to offer any help.

rad1k4l · September 2, 2021, 5:58pm

Excuse me for unclear explanation

General info: I installed drone-server and drone-runner-kube on kubernetes.
I connected github and tried run simple pipeline.
Pipeline crashes after 1 - 2 seconds. I provided drone-runner-kube logs at post.

Error message from ui:

my .drone.yaml file

kind: pipeline
type: kubernetes
name: drone-test

steps:
  - name: hello
    image: alpine:3.10.3
    commands:
      - time 20
      - echo hello

bradrydzewski · September 2, 2021, 6:04pm

what version of drone server and drone runner are you using?

rad1k4l · September 2, 2021, 6:08pm

As drone runner i use this docker image :

drone/drone-runner-kube:latest

And for drone server this docker image:

drone/drone:latest

bradrydzewski · September 2, 2021, 6:09pm

also please provide the results of /api/repos/{owner}/{repo}/builds/{number} for your repository and build (be sure to replace the values in curly braces with the actual values).

rad1k4l · September 2, 2021, 6:26pm

Oh . I saw error with kubernetes service account.
But i created service account for kube runner. Even i may provide kubernetes roles yaml.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-robot
  namespace: drone

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: drone
  name: kube-runner
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - create
  - delete
  - list
  - watch
  - update

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-runner
  namespace: drone
subjects:
- kind: ServiceAccount
  name: kube-robot
  namespace: drone
roleRef:
  kind: Role
  name: kube-runner
  apiGroup: rbac.authorization.k8s.io

On url : /api/repos/Digitalks-Metrix/admin-api-service/builds/31
json output provided below.

I may provide only stages json object since it is public post;


"stages":[
      {
         "id":31,
         "repo_id":72,
         "build_id":31,
         "number":1,
         "name":"drone-test",
         "kind":"pipeline",
         "type":"kubernetes",
         "status":"error",
         "error":"pods \"drone-v281v4swrr6i6y5ev2y8\" is forbidden: error looking up service account drone/drone:kube-robot: serviceaccount \"drone:kube-robot\" not found",
         "errignore":false,
         "exit_code":255,
         "machine":"kube-runner-56c4bd567f-6mzs2",
         "os":"linux",
         "arch":"amd64",
         "started":1630602464,
         "stopped":1630602464,
         "created":1630602455,
         "updated":1630602464,
         "version":4,
         "on_success":true,
         "on_failure":false,
         "steps":[
            {
               "id":39,
               "step_id":31,
               "number":1,
               "name":"clone",
               "status":"skipped",
               "exit_code":0,
               "started":1630602464,
               "stopped":1630602464,
               "version":2,
               "image":"drone/git:latest"
            },
            {
               "id":40,
               "step_id":31,
               "number":2,
               "name":"hello",
               "status":"skipped",
               "exit_code":0,
               "started":1630602464,
               "stopped":1630602464,
               "version":2,
               "depends_on":[
                  "clone"
               ],
               "image":"docker.io/library/alpine:3.10.3"
            }
         ]
      }

bradrydzewski · September 2, 2021, 6:31pm

I see the following error message for your pipeline which indicates a configuration problem, perhaps with the service account you configured? This would explain why Drone cannot delete the Pod, since it cannot be created in the first place.

pods “drone-v281v4swrr6i6y5ev2y8” is forbidden: error looking up service account drone/drone:kube-robot: serviceaccount “drone:kube-robot” not found

@marko-gacesa is this something we can log in the runner logs?
@d1wilko would it be possible to service this error int he user interface? you can use the above payload as reference.

rad1k4l · September 2, 2021, 6:45pm

I provided above kubernetes roles.yaml. As you see yaml file has right serviceaccount configuration.
Additionally it is output of command “kubectl get serviceaccount -n drone”

NAME          SECRETS   AGE
default       1         29h
kube-robot    1         4h50m
kube-runner   1         5h58m

bradrydzewski · September 2, 2021, 6:49pm

By default, Drone uses the default namespace. The files you provided above are for the drone namespace. Did you configure the runner to use the drone namespace instead of the default namespace?

bradrydzewski · September 2, 2021, 6:51pm

and since this seems to be a configuration problem, I recommend providing the full runner configuration. We need to be able to see the full picture to advise you further.

rad1k4l · September 2, 2021, 6:52pm

Yes. I configured runner for use drone namespace. And its currently places in drone namespace.
It is drone-runner deployment yaml file:
deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-runner
  namespace: drone
  labels:
    app: kube-runner
  annotations:
spec:
  selector:
    matchLabels:
      app: kube-runner
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: kube-runner
    spec:
      containers:
        - name: kube-runner
          image: drone/drone-runner-kube:latest
          ports:
            - containerPort: 3000
          env:
            - name: DRONE_RPC_SECRET
              valueFrom:
                secretKeyRef:
                  name: drone-server-secret
                  key: DRONE_RPC_SECRET
            - name: DRONE_RPC_HOST
              value: drone-server
            - name: DRONE_RPC_PROTO
              value: http
            - name: DRONE_NAMESPACE_DEFAULT
              value: drone
            - name: DRONE_SERVICE_ACCOUNT_DEFAULT
              value: drone:kube-robot
            - name: DRONE_DEBUG
              value: true

bradrydzewski · September 2, 2021, 6:54pm

Thanks for providing the additional details. I am going to let someone else jump in, since I do not use kubernetes, and this is really beyond my area of expertise. One thing we can say for sure is that this error comes from kubernetes, but I’m not sure what you need to change in your configuration to get this resolved …

pods “drone-v281v4swrr6i6y5ev2y8” is forbidden: error looking up service account drone/drone:kube-robot: serviceaccount “drone:kube-robot” not found

rad1k4l · September 2, 2021, 7:03pm

Thank you very much for your effort. I am research this problem if I find any solve for it, I will write here.

Topic		Replies	Views
Pipeline fails with messages related to secrets in runners Drone Support	4	1527	December 20, 2022
Drone-runner-kube 1.0.0-beta.12 - Failed step blocks pipeline Drone Bugs	33	872	October 13, 2021
Runner-kube no logs Drone Support	16	1298	January 20, 2020
Kubernetes runner: operation cannot be fulfilled; the object has been modified Drone Support	3	1860	April 21, 2020
Unable to run kubernetes pipeline Drone Support	0	280	December 4, 2019

Runner failed to delete pod

Related topics