Kubernetes drone push to local registry, DNS Issues

Drone Support
1

Hi, I wanted to share my nightmare (and eventual solution) of wanting to docker push from our kubernetes’ hosted drone to our kubernetes’ hosted local secure registry:

This is imo NOT POSSIBLE with plugin/docker, but I’d like to hear otherwise :wink:

After reading a lot about plugin/docker’s custom network and it’s limitations on dns resolution, I figured out it’s simply not possible to instrument the plugin to correctly use custom_dns and custom_dns_search. The dind container’s /etc/resolv.conf simply never got injected these values, and resolving cluster domains to our cluster ips just never works. I tried all possible combinations to get resolution working, but to no avail.

The solution, which should only be used when your project is trusted and the environment under your control:

  • as Drone admin set the repo to ‘Trusted project’, which allows the mounting of volumes.
  • mount the host docker socket
  • use a plain ‘docker’ image and do all the tagging and pushing manually

Example yaml:

pipeline:

  build:
    image: docker
    environment:
      - IMG=kube-registry.kube-system.svc.cluster.local:31000/your-app
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    commands:
      - echo building image $${IMG}
      - docker build --rm -t $${IMG} .
      - docker push $${IMG}
      - docker tag $${IMG} $${IMG}:${DRONE_COMMIT_SHA}
      - docker push $${IMG}:${DRONE_COMMIT_SHA}

I would like to thank @bradrydzewski and helping me out with this :slight_smile:

2

What if you use public DNS entry for your registry? or the direct IP address? I would usually expect that an organization using Drone would use public DNS for their registry so their developers can access the registry from their laptops.

3

Personally I don’t think that it is needed (and possible unwanted even) to make those repos publicly available. Git is the ssot, and all drone build artifacts may reside in the cluster only. If you want to publish images you can alter the pipeline and push them to a public repo as well.

Direct ip is not known. 12 factor, minikube, dev, prod, etc etc

4

The majority of organizations I work with configure their internal registries with DNS so that their employees can access from their personal workstations. Drone is optimized for these common configurations. So this does not seem like an unreasonable solution to me.

If your configuration is inflexible, and you are not able to try workarounds, I am not sure we will be able to find a solution for you. I do not want to waste your time or my time.

5

I completely understand and do not want to come across as unflexible. I have set up a publicly accessible registry before, and am just curious how to limit access this time and keep everything in the cluster. Sorry if you feel I am wasting your time. I do however feel there is work to be done for plugin/docker to be able to resolve cluster local domains.

Also, I am modeling a stack and didn’t want to configure anything I didn’t need at this stage. It’s more work to expose the registry to the outside world, and like I said, I hope to avoid that alltogether.

Thanks a lot for being so responsive and thinking along.

1 Like
6

Using registry-docker-registry:5000, registry-docker-regisrty.registry:5000, or registry-docker-regisrty.registry.svc.cluster.local:5000 fails for me on Drone 1.0, on kubernetes.
I’m forced to use NodePort and to use private ip (127.0.0.1 doesn’t work)

But, the weird case is that using service IP is ok. That means that kube-dns is not requested ?

Actually, “nslookup” in is working inside image, I mean that I cannot use that address for “image” in steps…