DNS issues while running ECR plugin

Drone Support
1

I have not been able to use the ECR plugin to build a Dockerfile that pulls from debian sources. The Dockerfile is as follows:

FROM python:2.7-buster

EXPOSE 9000

RUN apt-get update && \
  apt-get install -y \
     ca-certificates \
     bash \
     collectd \
     collectd-utils \
     cron \
     logrotate \
     python-pip && \
  rm -rf /var/lib/apt/lists/* && \
  pip install collectd signalfx

COPY entrypoint.sh entrypoint.sh

ENV TOKEN=''
ENV REGION=''
ENV ENVIRONMENT=''

ENTRYPOINT [ "./entrypoint.sh" ]

My .drone.yml is as follows:

kind: pipeline
type: docker
name: default

steps:
  - name: publish
    image: plugins/ecr
    settings:
      access_key:
        from_secret: aws_access_key_id
      secret_key:
        from_secret: aws_secret_access_key
      repo: adserver-collectd-proxy
      tags:
        - latest
        - staging
      region: us-west-2
      registry: 000000000.dkr.ecr.us-west-2.amazonaws.com

trigger:
  branch:
    - staging
  event:
    include:
    - push

In the output of the build, I get errors like the following:

Step 4/23 : RUN apt-get update &&   apt-get install -y      ca-certificates      bash      collectd      collectd-utils      cron      logrotate      python-pip &&   rm -rf /var/lib/apt/lists/* &&   pip install collectd signalfx
 ---> Running in 48851e1013fe
Err:1 http://security.debian.org/debian-security buster/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian buster InRelease
  Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian buster-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists...
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package collectd
E: Unable to locate package collectd-utils
E: Unable to locate package cron
E: Unable to locate package logrotate
E: Unable to locate package python-pip
The command '/bin/sh -c apt-get update &&   apt-get install -y      ca-certificates      bash      collectd      collectd-utils      cron      logrotate      python-pip &&   rm -rf /var/lib/apt/lists/* &&   pip install collectd signalfx' returned a non-zero code: 100

I’ve been able to ssh into the instance that has my drone runner, and executed a build with the following:

docker run --rm \
  -e PLUGIN_TAG=latest \
  -e PLUGIN_REPO=octocat/hello-world \
  -e DRONE_COMMIT_SHA=d8dbe4d94f15fe89232e0402c6e8a0ddf21af3ab \
  -v $(pwd):$(pwd) \
  -w $(pwd) \
  --privileged \
  plugins/ecr --dry-run

This builds correctly and does not have any issues resolving debian.org. Can anybody offer some insight as to why this is happening, and/or how to solve this issue?

I’ve also provided a custom_dns line in the settings portion of the ECR with 8.8.8.8 and it also hasn’t worked.

Thank you!

2

I’ve been able to ssh into the instance that has my drone runner, and executed a build with the following […] This builds correctly and does not have any issues resolving debian.org. Can anybody offer some insight as to why this is happening

This is not quite an apples to apples comparison because it uses the default bridge network. Drone does not use the bridge network. Instead it creates a new user-defined network for each pipeline (similar to docker-compose). Docker also handles DNS differently for user-defined networks [1]. The following commands more closely emulate how Drone is running your container:

$ docker network create my-network
$ docker run --network=my-network ...

You can probably even further isolate the problem and reproduce without any Drone-specific code. In the below example we launch the dind container which drops us into a shell, and we then run the docker build command manually.

$ docker network create my-network
$ docker run -t -i -v $(pwd):$(pwd) -w $(pwd) --privileged --network=my-network docker:dind /bin/sh
# docker build .

Please note the above example should be considered pseudocode. I wrote this example from memory which means it will probably require some tweaking to run properly.

[1] Networking overview | Docker Docs

3

Hey thanks so much for this hint! It sent me down a rabbit hole, and I found a few links that were related to my issue by searching for some of the keywords in your post. For posterity, and possibly others that come across this post:

Following your instructions on creating the network, I was able to recreate my issue. It’s definitely a DNS issue from a container inside a container. For context, I’m running this on an EC2 instance, and found this documentation on How can I avoid DNS resolutions failures with EC2 Linux.

I’m currently struggling against systemd and getting dnsmasq up and running, I will update if I’m able to successfully solve the issue!

4

a quick workaround may be to change the network mode to bridge or host. I believe this requires trusted mode enabled in your repository settings, however, there are some ongoing discussions to have a global setting to always attach docker|ecr|gcr to the bridge network which would not require trusted mode.

kind: pipeline
type: docker
name: default

steps:
  - name: publish
    image: plugins/ecr
    network_mode: bridge
    settings:
    ...
5

Thanks so much for this piece of information! This is an acceptable solution for me, as this is a private repository and is trusted anyway.

6

great, if you don’t mind please follow up and let me know if this works.

7

I can’t seem to edit the title, but yes, this issue was solved by adding network_mode: bridge to the configuration.

Thanks again @bradrydzewski and @ashwilliams1!

8

Facing same issue. But I am running in kubernetes. What is the solution for it