Hi,
I am tried to run drone-kube on Kuberenetes but I am getting error after it successfulll pull image next run casue the container to crash:
time=“2020-06-28T15:20:45Z” level=error msg=“cannot ping the docker daemon” error=“Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”
I am trying to run drone-dokcer and 1 container is working when scaling to 2 I am getting the same error as above, how can I solve?
Just to clarify, the docker runner (drone-runner-docker) is not meant to run on kubernetes. If you want to run pipelines on kubernetes you would use the kubenretes runner (drone-runner-kube) [1] and create kubernetes pipelines [2]. The kubernetes runner does not make any connection to the docker daemon, so the fact that you are getting a docker connection error would lead me to believe you are maybe using the wrong runner image?
[1] https://docs.drone.io/runner/kubernetes/overview/
[2] https://docs.drone.io/pipeline/overview/
@ashwilliams1 I want to use kube-runner but it written on documentation that the Kubernetes runtime is not recommended for production use [1].
and I am not about to mount /var/run/ or /var/run/doceker.sock and getting error
@ashwilliams1 kube-runner is not stable once I able to use docker ps + docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2d6b265e651f 81f5749c9058 "/bin/sh -c 'echo “$…” Less than a second ago Up Less than a second k8s_drone-e3fhhdlgnwvncopmr0jf_drone-34g2whzeelvf50lto3tj_default_d1708a3e-2ea2-4362-bfbc-4614415674fb_1
and running againg I am getting
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
54
and no documentaion provide how to setup on kube-runner
I am attach my yaml + my values.yaml
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running
This error comes from drone-runner-docker, not drone-runner-kube, so I am confused how you would get this error if you are trying to use the kubernetes runner. Please provide low-level details about how you installed and configured Drone (example commands you used to install Drone, example configuration or spec files, yaml files, etc). I am having trouble understanding the issue, and I think seeing examples of what you are doing will be very helpful
Yes, I tried to run both runner-linux + runner-kube to understand this is not good Idea, runner-linux on kuberenets with /var/run/docker.sock is making the node to be unavialable, when trying to run kube-runner I am not able to connect to deamon /var/run/docker.sock as well
how can I attach my yaml+value.yaml
kind: pipeline
name: default
type: kubernetes
platform:
os: linux
arch: amd64
steps:
name: pull
pull: default
image: docker
commands:
name: push-to-ecr-barnch
pull: default
image: plugins/ecr
settings:
create_repository: true
dockerfile: ./Dockerfile
region: us-east-1
registry: 991726456769.dkr.ecr.us-east-1.amazonaws.com
repo: 991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer
tags:
name: push-to-ecr-tag
pull: default
image: plugins/ecr
settings:
create_repository: true
dockerfile: ./Dockerfile
region: us-east-1
registry: 991726456769.dkr.ecr.us-east-1.amazonaws.com
repo: 991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer
repository_policy: ecr_repository_policy.json
tags:
volumes:
if docker fails for some reason it carshes the Docker deamon on Kubernetes worker node!!
and also change the permissions of the docker.sock and casuse the docker deamon on the host to be unavaialble
this is from drone log
Login Succeeded
46 + docker pull registry.strln.net/thor/alpine_hardened:latest
47 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
kind: pipeline
name: default
type: kubernetes
platform:
os: linux
arch: amd64
steps:
name: pull
pull: default
image: docker:dind
commands:
name: push-to-ecr-barnch
pull: default
image: plugins/ecr
settings:
create_repository: true
dockerfile: ./Dockerfile
region: us-east-1
registry: 991726456769.dkr.ecr.us-east-1.amazonaws.com
repo: 991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer
tags:
name: push-to-ecr-tag
pull: default
image: plugins/ecr
settings:
create_repository: true
dockerfile: ./Dockerfile
region: us-east-1
registry: 991726456769.dkr.ecr.us-east-1.amazonaws.com
repo: 991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer
repository_policy: ecr_repository_policy.json
tags:
when:
event:
- tag
services:
volumes:
One thing that jumps out is I see you are mounting the docker socket from the host into the plugins/ecr step, however, this plugin uses docker-in-docker and starts its own docker daemon. Mounting a docker socket into this plugin, when the plugin tries to start its own docker-in-docker daemon and create its own docker socket, could be causing problems.
Have you tried using plugins/ecr without mounting the host machine docker socket?
if docker fails for some reason it crashes the Docker deamon on Kubernetes worker node!!
If running standard docker causes the host machine Docker daemon to crash on your Kubernetes node, you may consider opening an issue with the Docker project. This sounds like it would be a severe Docker bug.
While we generally recommend against mounting the host machine docker socket when using the docker plugin (since the plugin is meant to use docker-in-docker) you can disable the docker-in-docker daemon with the following settings:
image: plugins/ecr
settings:
daemon_off: true
purge: true
This is important, because if you do not disable the docker-in-docker daemon it may overwrite the host machine docker socket, or may cause the plugin to fail (this could even be the reason your Docker daemon on the node is crashing, because its socket is being overwritten). Also the purge setting prevents the plugin from running docker prune on the host and deleting your host machine docker cache.
However, I would recommend using this plugin without mounting the host machine docker socket if possible. Mounting the host machine docker socket effectively grants your pipelines root access to the host machine, which may be undesirable from a security perspective.
@bradrydzewski, @ashwilliams1 this is SUPER important information which clarify the symptom I had on my system and worth to publish it for anyone have this issue.
I mount the /var/run/docker.sock from the first place becasue when I building the Image on the Dockerfile the from is from private registry and I am getting the following error:
how can I tell drone/kubernetes to use secrets to pull the image from private registry both on Pipeline step and when building an image, this is critical and if I solved it I can skip mounting /var/run/docker.sock
@ihakimi the docker daemon uses auth credentials to push and pull images. However, one thing I noticed is that you have two different registries (991726456769.dkr.ecr.us-east-1.amazonaws.com and registry.strln.net). Do they have different authentication credentials? Or can they use the same credentials? Are both of these ECR registries? Do they both require generated (short lived) credentials?
Yes, for 991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com Kubernetes can pull it automatically with IAM roles to this ecr,
the problem is on http://registry.strln.net/ which uses temporary credentials and the login is using with CEC user+2MFA, and when do cut to dockerjson its hidden on mac chain so it quite complicated and not understand how to get this credential because of that I used this step before:
when do cut to dockerjson its hidden on mac chain so it quite complicated and not understand how to get this credential because of that I used this step before:
I have a mac and have run into this issue. To get the credentials I usually run docker login inside a container on my mac and login and then copy the credentials:
$ docker run -t -i docker /bin/sh
# docker login
# cat ~/.docker/config.json
You can actually store the entire contents of the config.json file as a secret, and then you can use this config file in your plugin, like this:
image: plugins/ecr
settings:
config:
from_secret: ...
Perhaps the combination of using the config.json for strln.net and IAM for 991726456769.dkr.ecr.us-east-1.amazonaws.com would solve the issue?
@bradrydzewski BINGO
image: plugins/ecr
settings:
config:
from_secret: auth
Step 1/25 : FROM registry.strln.net/thor/alpine_hardened:latest
85 latest: Pulling from thor/alpine_hardened
86
the only probelm is this is short live credentails, and this not understand how to solve, I can do step before login but how can I pass this credentail to next step for example config: from_file: XZZ
@ihakimi I feel like there are a few different options that could be employed, each with different pros and cons. One option would be to mount the .docker home path as a temporary volume, run docker login to generate the config.json file which would be saved to this volume, making it available to subsequent steps:
steps:
- name: login
image: docker
commands:
- curl https://downloads.strln.net/install | bash
- sl login
- $(sl container registry auth generate)
volumes:
- name: dockerconfig
path: /root/.docker
- name: build
image: plugins/ecr
settings:
...
volumes:
- name: dockerconfig
path: /root/.docker
volumes:
- name: dockerconfig
temp: {}
Another option could be to provide these credentials automatically using a custom extension. You would have to write the custom extension (we provide starter templates to help simplify this process). Another option would be to fork the ecr plugin and add some extra parameters and logic to generate the credentials for registry.strln.net (I would probably consider this option). Happy to discuss these other options in greater depth if you are intersted.
@bradrydzewski thank you very much! the following pipeline work!!!
my question can I take the volume I create and leverage it for pull private images from steps
for example
- name: login
image: docker
commands:
- apk update
- apk add curl bash
- curl https://downloads.strln.net/install | bash
- sl login
- $(sl container registry auth generate)
volumes:
- name: dockerconfig
path: /root/.docker
- name: private_image
image: registry.strln.net/XXX/alpine_hardened:latest
image_pull_secrets:
from_secret_file: /root/.docker
volumes:
- name: dockerconfig
path: /root/.docker
volumes:
- name: dockerconfig
temp: {}
does it possible to get the image_pull_secret from file rather then secrets because of short live session credentials.
@ihakimi Drone can pull pipeline images (defined by the image: attribute) using a static username and password, but it cannot generate temporary usernames and passwords (for example, like what you are doing with sl container registry auth generate). Unfortunately there is no standard for generating temporary credentials, and every registry implements its own custom interface, which means there is no way for Drone to support this feature in a generic or universal manner.
So to solve this problem we introduced registry extensions:
https://docs.drone.io/extensions/registry/
https://docs.drone.io/extensions/registry/#starter-project
You can create a registry extension that generates the credentials (by interface with your private registry) and return the credentials back to Drone. This is going to be your best option because it will simplify everyone’s pipeline since it will automatically handle credential configuration. Extensions are simple REST microservices and we even provide starter projects that you can use to accelerate development.