Drone runner can not connect with private registry

Drone Support
1

Hi . I’m trying to use private registry to run images in the drone pipeline.( I did as in Harness Community | Harness Developer Hub)

This is the pipeline

ind: pipeline
type: docker
name: greeting

steps:
- name: en
  image: alpine
  commands:
  - echo hello
  - echo world
  - echo hola

- name: es
  image: nexus:8082/toni_alpine
  commands:
  - echo hola
  - echo mundo

- name: fr
  image: alpine
  commands:
  - echo bonjour
  - echo monde

image_pull_secrets:
- nexus_credentials

This is the output when trying to connect with the nexus:8082 registry

image
image1005×458 31.6 KB

But drone-runner is running on one worker and I’ve tested connectivity on this host as you can see in the next test. (running netcat inside a docker image on the same host)

docker@worker3:~$ docker ps                                                                                                                                                                        
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
00e263f034e9        drone/drone-runner-docker:latest   "/bin/drone-runner-d…"   About an hour ago   Up About an hour    3000/tcp            web-cicd_drone-runner.1.z1cxihkgpgisoy4ybp1rxt7lj
967ef99bd06f        portainer/agent:latest             "./agent"                2 hours ago         Up 2 hours                              web-base_agent.wr7akdyft1cwj55mtcnasqwmc.tkmmzfi3c1wz78qi6s8zwbkyd
docker@worker3:~$  docker run -t --rm --network=net-public  subfuzion/netcat -vz nexus 8082                                                                                                  
Connection to nexus 8082 port [tcp/*] succeeded!

How can I debug this error?

2

2 things come to mind …

First it looks like your registry is using http (not https) which means you need to configure your Docker daemon to allow insecure registry connections.

Second I can see you are using custom dns to connect to the registry, which usually requires special configuration (attaching containers to the same network, etc). Have you confirmed you can connect to the nexus hostname from inside the Runner container?

Lastly see the Troubleshooting section of How to pull private images with 1.0, which enumerates a list of information we need in order to help triage. All information in this list should be provided; it makes our job way easier and saves us time :slight_smile:

3

Hi @bradrydzewski , I will try to answer you in order.

  1. As you can see in the next docker info command I have configured nexus:8082 as insecure
docker@worker3:~$ docker info                                                                                                                                                                      
Client:
 Debug Mode: false

Server:
 Containers: 8
  Running: 3
  Paused: 0
  Stopped: 5
 Images: 9
 Server Version: 19.03.4
 Storage Driver: overlay2
 
...
....
 Labels:
 Experimental: false
 Insecure Registries:
  nexus:8082
  nexus:8083
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
  1. About dns resolution, I’ve added a ping -c 4 nexus to the alpine image as shown in the pipeline
steps:
- name: en
  image: alpine
  commands:
  - ping -c 4 nexus
  - echo hello
  - echo world
  - env

- name: es
  image: nexus:8082/toni_alpine
  commands:
  - echo hola
  - echo mundo

- name: fr
  image: alpine
  commands:
  - echo bonjour
  - echo monde
  - env

image_pull_secrets:
- nexus_credentials

As you can see in the image the resolution inside the image is correct with IP 192.168.99.106 ( , but , the error message is resolving 10.71.208.193 .

image
image1018×590 58.6 KB

I’ve reviewed the host VM’s /etc/resolv.conf and they had a search otherdomain.org which has an existing service nexus.otherdomain.org in the IP 10.71.208.193, after removed the “search” entry the resolution is being done ok.

After this fix and with correct credentials now I’m able to use my private registry images.

4

Hi @bradrydzewski, I’m still confused.

I realized that drone-runner creates containers which are resolving names with my external laptop /etc/hosts ( I added to browse through traefik) , and not by its internal IP as I expect

This is my

vant@mylaptop:~/t$ cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	mylaptop
192.168.99.106	portainer portainer.mydomain.org
192.168.99.106	traefik  traefik.mydomain.org
192.168.99.106  nexus nexus.mydomain.org
192.168.99.106  git git.mydomain.org
192.168.99.106  drone drone.mydomain.org
#192.168.99.106	minio.mydomain.org

When I have commented the minio entry , the runner containers can not resolve, as you can see in the following image , for this pipeline.

kind: pipeline
type: docker
name: greeting

steps:
- name: test_connectivity
  image: alpine
  commands:
  - cat /etc/hosts
  - cat /etc/resolv.conf
  - ping -c 4 nexus
  - ping -c 4 minio
  
- name: create_file
  image: nexus:8082/toni_alpine
  commands:
  - env > environtment-${CI_BUILD_NUMBER}-${CI_BUILD_CREATED}.txt
  
- name: upload_minio
  image: plugins/s3
  settings:
    bucket: minio-logs
    access_key: admin
    secret_key: $apr1$r.lt7Y9U$G/KLt6iG5uqv6UZQd25KE.
    source: environtment-${CI_BUILD_NUMBER}-${CI_BUILD_CREATED}.txt
    target: /
    path_style: true
    endpoint: http://minio:9000

image_pull_secrets:
- nexus_credentials

image
image985×605 58.1 KB

If I ping inside other container attached to the same network it resolves with the internal

docker@manager1:~$ docker run --rm -it  --network=net-public  alpine ping -c 2 nexus                                                                                                         
PING nexus (10.0.0.207): 56 data bytes
64 bytes from 10.0.0.207: seq=0 ttl=64 time=0.187 ms
64 bytes from 10.0.0.207: seq=1 ttl=64 time=0.192 ms
--- nexus ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.187/0.189/0.192 ms
docker@manager1:~$ docker run --rm -it  --network=net-public  alpine ping -c 2 minio                                                                                                         
PING minio (10.0.0.151): 56 data bytes
64 bytes from 10.0.0.151: seq=0 ttl=64 time=0.070 ms
64 bytes from 10.0.0.151: seq=1 ttl=64 time=0.188 ms
--- minio ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.070/0.129/0.188 ms

It seems like drone-runner creates containers but is not attaching containers to the same network that it is (net-public) as you can see in the service definition.

  drone-runner:
    image: drone/drone-runner-docker:latest
    networks:
      - net-public
    environment:
      DRONE_RPC_HOST: drone-server
      DRONE_RPC_PROTO: http
      DRONE_RPC_SECRET: ${WEB_ADMIN_PASSWORD}
      DRONE_RUNNER_CAPACTIY: 8
      DRONE_DEBUG: "true"
      DRONE_RUNNER_ENVIRON:  GIT_SSL_NO_VERIFY:"true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints:
          - node.role==worker
      replicas: 3

How can I attach all the containers created by the drone-runner to the “net-public” network?

5

drone runner creates a user-defined network for each pipeline. If you want to attach pipeline containers to existing networks, you need to use https://docs.drone.io/runner/docker/configuration/reference/drone-runner-networks/

6

Hi @bradrydzewski, Thank you again!!

Now everything is ok.!!

image
image967×686 68 KB

7

Hi @toni-moreno
Can you please share the latest config which you succeeded?

8

Hi @nyamane, sorry for the late response. I added as Environment variable in the runner config.

DRONE_RUNNER_NETWORKS: myinternal-network

in order to attach docker images to the same network where my internal nexus is deployed. I hope this can help you.