# Drone docker runner on kubernetes **URL:** \ **Category:** Drone Support\ **Created:** [June 28, 2020, 9:32pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016 "2020-06-28T21:32:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 28, 2020, 9:32pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/1 "2020-06-28T21:32:32Z")
Hi, I am tried to run drone-kube on Kuberenetes but I am getting error after it successfulll pull image next run casue the container to crash: time=“2020-06-28T15:20:45Z” level=error msg=“cannot ping the docker daemon” error=“Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?” I am trying to run drone-dokcer and 1 container is working when scaling to 2 I am getting the same error as above, how can I solve? ---
**Author:** ![ashwilliams1](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ashwilliams1/32/5549_2.png) [@ashwilliams1](https://drone.discourse.group/u/ashwilliams1)\ **Post date:** [June 28, 2020, 10:18pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/2 "2020-06-28T22:18:03Z")
Just to clarify, the docker runner (drone-runner-docker) is not meant to run on kubernetes. If you want to run pipelines on kubernetes you would use the kubenretes runner (drone-runner-kube) [1] and create kubernetes pipelines [2]. The kubernetes runner does not make any connection to the docker daemon, so the fact that you are getting a docker connection error would lead me to believe you are maybe using the wrong runner image? [1] [https://docs.drone.io/runner/kubernetes/overview/](https://docs.drone.io/runner/kubernetes/overview/) [2] [https://docs.drone.io/pipeline/overview/](https://docs.drone.io/pipeline/overview/) ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 7:34am UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/3 "2020-06-29T07:34:26Z")
@ashwilliams1 I want to use kube-runner but it written on documentation that the Kubernetes runtime is not recommended for production use [1]. and I am not about to mount /var/run/ or /var/run/doceker.sock and getting error - docker ps 2 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 3 ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 8:02am UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/4 "2020-06-29T08:02:42Z")
@ashwilliams1 kube-runner is not stable once I able to use docker ps + docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2d6b265e651f 81f5749c9058 "/bin/sh -c 'echo “$…” Less than a second ago Up Less than a second k8s\_drone-e3fhhdlgnwvncopmr0jf\_drone-34g2whzeelvf50lto3tj\_default\_d1708a3e-2ea2-4362-bfbc-4614415674fb\_1 and running againg I am getting Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 54 and no documentaion provide how to setup on kube-runner I am attach my yaml + my values.yaml ---
**Author:** ![ashwilliams1](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ashwilliams1/32/5549_2.png) [@ashwilliams1](https://drone.discourse.group/u/ashwilliams1)\ **Post date:** [June 29, 2020, 12:49pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/5 "2020-06-29T12:49:59Z")
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running This error comes from drone-runner-docker, not drone-runner-kube, so I am confused how you would get this error if you are trying to use the kubernetes runner. Please provide low-level details about how you installed and configured Drone (example commands you used to install Drone, example configuration or spec files, yaml files, etc). I am having trouble understanding the issue, and I think seeing examples of what you are doing will be very helpful ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 2:08pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/6 "2020-06-29T14:08:34Z")
Yes, I tried to run both runner-linux + runner-kube to understand this is not good Idea, runner-linux on kuberenets with /var/run/docker.sock is making the node to be unavialable, when trying to run kube-runner I am not able to connect to deamon /var/run/docker.sock as well how can I attach my yaml+value.yaml ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 2:18pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/7 "2020-06-29T14:18:12Z")
## @ashwilliams1 when running runner-kube on EKS + worker node, when I use on drone.yml vol kind: pipeline name: default type: kubernetes platform: os: linux arch: amd64 steps: - name: pull pull: default image: docker commands: - name: push-to-ecr-barnch pull: default image: plugins/ecr settings: create\_repository: true dockerfile: ./Dockerfile region: us-east-1 registry: [991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com) repo: [991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer](http://991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer) tags: - name: push-to-ecr-tag pull: default image: plugins/ecr settings: create\_repository: true dockerfile: ./Dockerfile region: us-east-1 registry: [991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com) repo: [991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer](http://991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer) repository\_policy: ecr\_repository\_policy.json tags: volumes: - name: dockersock host: path: /var/run/ if docker fails for some reason it carshes the Docker deamon on Kubernetes worker node!! ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 2:28pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/8 "2020-06-29T14:28:17Z")
and also change the permissions of the docker.sock and casuse the docker deamon on the host to be unavaialble ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 2:32pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/9 "2020-06-29T14:32:14Z")
this is from drone log Login Succeeded 46 + docker pull [registry.strln.net/thor/alpine\_hardened:latest](http://registry.strln.net/thor/alpine_hardened:latest) 47 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 2:42pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/10 "2020-06-29T14:42:06Z")
## this is one way I able to skip mount /var/run/docker.sock of the host machine kind: pipeline name: default type: kubernetes platform: os: linux arch: amd64 steps: - name: pull pull: default image: docker:dind commands: - name: push-to-ecr-barnch pull: default image: plugins/ecr settings: create\_repository: true dockerfile: ./Dockerfile region: us-east-1 registry: [991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com) repo: [991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer](http://991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer) tags: - name: push-to-ecr-tag pull: default image: plugins/ecr settings: create\_repository: true dockerfile: ./Dockerfile region: us-east-1 registry: [991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com) repo: [991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer](http://991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer) repository\_policy: ecr\_repository\_policy.json tags: # volumes: # - name: dockersock # path: /var/run/ when: event: - tag services: - name: docker image: docker:dind privileged: true volumes: - name: dockersock path: /var/run volumes: - name: dockersock temp: {} ---
**Author:** ![ashwilliams1](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ashwilliams1/32/5549_2.png) [@ashwilliams1](https://drone.discourse.group/u/ashwilliams1)\ **Post date:** [June 29, 2020, 3:56pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/11 "2020-06-29T15:56:10Z")
One thing that jumps out is I see you are mounting the docker socket from the host into the plugins/ecr step, however, this plugin uses docker-in-docker and starts its own docker daemon. Mounting a docker socket into this plugin, when the plugin tries to start its own docker-in-docker daemon and create its own docker socket, could be causing problems. Have you tried using plugins/ecr without mounting the host machine docker socket? > if docker fails for some reason it crashes the Docker deamon on Kubernetes worker node!! If running standard docker causes the host machine Docker daemon to crash on your Kubernetes node, you may consider opening an issue with the Docker project. This sounds like it would be a severe Docker bug. ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 29, 2020, 4:53pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/12 "2020-06-29T16:53:21Z")
While we generally recommend against mounting the host machine docker socket when using the docker plugin (since the plugin is meant to use docker-in-docker) you can disable the docker-in-docker daemon with the following settings: ```nohighlight image: plugins/ecr settings: daemon_off: true purge: true ``` This is important, because if you do not disable the docker-in-docker daemon it may overwrite the host machine docker socket, or may cause the plugin to fail (this could even be the reason your Docker daemon on the node is crashing, because its socket is being overwritten). Also the purge setting prevents the plugin from running `docker prune` on the host and deleting your host machine docker cache. However, I would recommend using this plugin without mounting the host machine docker socket if possible. Mounting the host machine docker socket effectively grants your pipelines root access to the host machine, which may be undesirable from a security perspective. ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 7:58pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/13 "2020-06-29T19:58:41Z")
@bradrydzewski, @ashwilliams1 this is SUPER important information which clarify the symptom I had on my system and worth to publish it for anyone have this issue. I mount the /var/run/docker.sock from the first place becasue when I building the Image on the Dockerfile the from is from private registry and I am getting the following error: - /usr/local/bin/docker build --rm=true -f ./Dockerfile -t 83a273e4ce9fbc0aaa673428f86063f45c01ecc9 . --pull=true --label org.label-schema.schema-version=1.0 --label org.label-schema.build-date=2020-06-29T19:55:20Z --label org.label-schema.vcs-ref=83a273e4ce9fbc0aaa673428f86063f45c01ecc9 --label org.label-schema.vcs-url=https://github.office.opendns.com/tlv-opsys/ops-deployer.git 83 Sending build context to Docker daemon 315.9kB 84 Step 1/25 : FROM [registry.strln.net/thor/alpine\_hardened:latest](http://registry.strln.net/thor/alpine_hardened:latest) 85 Get [https://registry.strln.net/v2/thor/alpine\_hardened/manifests/latest:](https://registry.strln.net/v2/thor/alpine_hardened/manifests/latest:) unauthorized: authentication required 86 time=“2020-06-29T19:55:20Z” level=fatal msg=“exit status 1” 87 how can I tell drone/kubernetes to use secrets to pull the image from private registry both on Pipeline step and when building an image, this is critical and if I solved it I can skip mounting /var/run/docker.sock ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 29, 2020, 8:27pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/14 "2020-06-29T20:27:59Z")
@ihakimi the docker daemon uses auth credentials to push and pull images. However, one thing I noticed is that you have two different registries (_[991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com)_ and _[registry.strln.net](http://registry.strln.net)_). Do they have different authentication credentials? Or can they use the same credentials? Are both of these ECR registries? Do they both require generated (short lived) credentials? ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 8:31pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/15 "2020-06-29T20:31:57Z")
Yes, for [991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com)]([http://991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com) Kubernetes can pull it automatically with IAM roles to this ecr, the problem is on [http://registry.strln.net/](http://registry.strln.net/) which uses temporary credentials and the login is using with CEC user+2MFA, and when do cut to dockerjson its hidden on mac chain so it quite complicated and not understand how to get this credential because of that I used this step before: - name: pull pull: default image: alpine commands: - apk update - apk add curl bash - curl [https://downloads.strln.net/install](https://downloads.strln.net/install) | bash - sl login - $(sl container registry auth generate) - docker pull [registry.strln.net/thor/alpine\_hardened:latest](http://registry.strln.net/thor/alpine_hardened:latest) environment: SL\_PKI\_DEPLOY\_KEY: from\_secret: SL\_PKI\_DEPLOY\_KEY SL\_PKI\_REQUEST\_ID: from\_secret: SL\_PKI\_REQUEST\_ID volumes: - name: dockersock path: /var/run/ when: event: - push - tag ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 29, 2020, 8:35pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/16 "2020-06-29T20:35:46Z")
> when do cut to dockerjson its hidden on mac chain so it quite complicated and not understand how to get this credential because of that I used this step before: I have a mac and have run into this issue. To get the credentials I usually run `docker login` inside a container on my mac and login and then copy the credentials: ```plaintext $ docker run -t -i docker /bin/sh # docker login # cat ~/.docker/config.json ``` You can actually store the entire contents of the `config.json` file as a secret, and then you can use this config file in your plugin, like this: ```auto image: plugins/ecr settings: config: from_secret: ... ``` Perhaps the combination of using the config.json for [strln.net](http://strln.net) and IAM for [991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com) would solve the issue? ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 29, 2020, 8:47pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/17 "2020-06-29T20:47:58Z")
@bradrydzewski BINGO image: plugins/ecr settings: config: from\_secret: auth Step 1/25 : FROM [registry.strln.net/thor/alpine\_hardened:latest](http://registry.strln.net/thor/alpine_hardened:latest) 85 latest: Pulling from thor/alpine\_hardened 86 the only probelm is this is short live credentails, and this not understand how to solve, I can do step before login but how can I pass this credentail to next step for example config: from\_file: XZZ ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 29, 2020, 9:23pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/18 "2020-06-29T21:23:54Z")
@ihakimi I feel like there are a few different options that could be employed, each with different pros and cons. One option would be to mount the _.docker_ home path as a temporary volume, run docker login to generate the _config.json_ file which would be saved to this volume, making it available to subsequent steps: ```auto steps: - name: login image: docker commands: - curl https://downloads.strln.net/install | bash - sl login - $(sl container registry auth generate) volumes: - name: dockerconfig path: /root/.docker - name: build image: plugins/ecr settings: ... volumes: - name: dockerconfig path: /root/.docker volumes: - name: dockerconfig temp: {} ``` Another option could be to provide these credentials automatically using a custom extension. You would have to write the custom extension (we provide starter templates to help simplify this process). Another option would be to fork the ecr plugin and add some extra parameters and logic to generate the credentials for [registry.strln.net](http://registry.strln.net) (I would probably consider this option). Happy to discuss these other options in greater depth if you are intersted. ---
**Author:** ![ihakimi](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/ihakimi/32/4476_2.png) [@ihakimi](https://drone.discourse.group/u/ihakimi)\ **Post date:** [June 30, 2020, 8:40am UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/19 "2020-06-30T08:40:23Z")
@bradrydzewski thank you very much! the following pipeline work!!! my question can I take the volume I create and leverage it for pull private images from steps for example - name: login image: docker commands: - apk update - apk add curl bash - curl [https://downloads.strln.net/install](https://downloads.strln.net/install) | bash - sl login - $(sl container registry auth generate) volumes: - name: dockerconfig path: /root/.docker ``` - name: private_image image: registry.strln.net/XXX/alpine_hardened:latest image_pull_secrets: from_secret_file: /root/.docker volumes: - name: dockerconfig path: /root/.docker volumes: - name: dockerconfig temp: {} ``` does it possible to get the image\_pull\_secret from file rather then secrets because of short live session credentials. ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 30, 2020, 3:44pm UTC](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016/20 "2020-06-30T15:44:58Z")
@ihakimi Drone can pull pipeline images (defined by the `image:` attribute) using a static username and password, but it cannot generate temporary usernames and passwords (for example, like what you are doing with _sl container registry auth generate_). Unfortunately there is no standard for generating temporary credentials, and every registry implements its own custom interface, which means there is no way for Drone to support this feature in a generic or universal manner. So to solve this problem we introduced registry extensions: [https://docs.drone.io/extensions/registry/](https://docs.drone.io/extensions/registry/) [https://docs.drone.io/extensions/registry/#starter-project](https://docs.drone.io/extensions/registry/#starter-project) You can create a registry extension that generates the credentials (by interface with your private registry) and return the credentials back to Drone. This is going to be your best option because it will simplify everyone’s pipeline since it will automatically handle credential configuration. Extensions are simple REST microservices and we even provide starter projects that you can use to accelerate development. [Next page](https://drone.discourse.group/t/drone-docker-runner-on-kubernetes/11016.md?page=2)