Docker Hub rate limits and Drone Cloud exemption?

Drone Support
1

UPDATE the contents of this discussion have been consolidated in this FAQ entry:

Today Docker Hub tested their new rate limit of 100 pulls per 6 hours from anonymous users. This led to the first “down time” that I can remember since we started using Drone Cloud (really amazing service) when one of our first builds today got this error:

arm: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I’m guessing this is due to a limited pool of shared Drone Cloud machines used for running jobs easily hitting that limit.

In their blog post I noticed they’ve mentioned working with CI/CD services on exemptions: Updates on Hub Rate Limits, Partners and Customer Exemptions | Docker

It seems like GitHub Actions is one of the services that has gotten an exemption: https://github.com/actions/virtual-environments/issues/1445#issuecomment-713861495

I think it would be awesome if someone at Drone Cloud could reach out to them about setting up an exemption for your build machines so that we don’t encounter disruptions as the rate limit gradually goes into full effect. I’ve reached out to Docker support as well and suggested it to them, so hopefully one of you will take the initiative and help us out – Drone Cloud has been a really valuable service, and I don’t think there is any other way that we’d have been able to make sure our open source library works on ARM processors.

1 Like
2

I saw your message in Gitter and responded, but wanted to repost in case others come across this thread:

hey all. I am going to create a readonly dockerhub service account and purchase a plan, and then rollout global credentials to our fleet of servers. This will give us unlimited pulls, and seems well worth the $5 per month cost. I expect this to be resolved within the next 24 hours.

3 Likes
3

I’ve come across the same issue for our on-premise Drone CI and purchased a DockerHub license, using credentials in the drone registry plugin which is running alongside drone-runner-kube. However this doesn’t cover DIND use cases where a user may be using such an image and then performing a docker build, referencing a DockerHub base image. I’ve thought of a couple approaches around this which are not so elegant:

  • Custom private DIND image with read-only credentials so it is auto-authenticated
  • Mount dockerconfig or auth step using a secret in the DIND Drone Service step

Neither of these are ideal and impacting for everyone as they’d need to modify their pipeline steps (several hundred users). I was wondering if you had any suggestions on how this could potentially be handled in the runner to make it non-impacting for users? Possibly via a policy, match on an image (or trusted images) and mounts in readonly credentials?

1 Like
4

@Kash there are two different scenarios that Drone users need to account for when it comes to the docker plugin and rate limiting.

Scenario 1: Build and Publish to Dockerhub

If you are building and publishing to Dockerhub, you are already providing a username and password to the plugin. The plugins always executes a docker login before it builds and publishes the image. This means that image pulls are going to be authenticated and will receive increased rate limits. If the username and password are associated with a paid dockerhub account, there will be no pull limit.

The recommended solution to rate limiting in this scenario is to ensure the username and password provided to the plugin are associated with a paid account. You could even use the organization secrets feature to provide this at an organization level. However, this may require changing yaml files or changing existing secrets.

Scenario 2: Build and Publish to other Registry, but Dockerfile references Dockerhub Images

If you are supplying a username and password to the dind plugin, but the username and password is for another registry (for example, quay) and if your dockerfile pulls from dockerhub, the pulls would not be authenticated and would be subject to rate limits.

The recommended solution in this scenario is to provide dockerhub credentials using the config parameter. Note that these credentials should be associated with a paid account to avoid rate limiting. The config parameter should contain the contents of a docker/config.json file.

- name: build
  image: plugins/docker
  settings:
    repo: quay.io/foo/bar
    username:
    password:
    config: |
      {
          "auths": {
              "https://index.docker.io/v1/": {
                  "auth": "c3R...zE2"
              }
          }
      }

The config file can also be sourced from a secret:

config:
  from_secret: ...

Global Solution: Using a Global Mirror

Finally, a global solution to this problem is to setup a registry mirror. We have some customers that setup and mirror to proxy and authenticate requests to dockerhub. I am not sure how to configure registry mirrors with kubernetes, but with docker you can configure mirrors by editing the docker daemon.json file.

The docker plugin also provides an option to configure a mirronr in the yaml. You could globally configure this mirror for all pipelines by configuring the following a global environment variable with your runner. This would be the equivalent of setting the mirror in every yaml.

PLUGIN_MIRROR=https://docker.company.com
5

Thanks Brad for the response.

For a lot of our users they are performing docker image builds detached from the push stage. i.e. build, run and test, and then publishing to a repository is separate.

Based on what you’ve mentioned it looks like we’d have to go with organization secrets, which unfortunately means:

  • Org secret added to each “organization” (we use Gitlab and there are over 100 groups), including any new ones created in the future by users. We could script and automate this.
  • All users update their Drone pipeline dind steps to use credentials from a secret.
6

I edited my previous comment to discuss registry mirrors as an option. This is the approach most of our customers are using. It requires zero end user changes, but requires setting up a mirror and reverse proxy.

7

Many thanks Brad, I’ll look at setting up something similar as this seems like the cleanest approach!

8

@bradrydzewski we solved this in a different way with https://github.com/drone/drone-registry-plugin (which we were already using)

We added an entry for docker.io:

- address: docker.io
  username: dockerhub-user
  password: dockerhub-token

We really needed a solution that doesn’t require all teams to update their .drone.yml files and this seems to be working. Can you see any issues with this fix?

dockerhub-user is a user that we have signed up with a “pro” dockerhub account.

I should mention we are not using Drone cloud, we are running our own Drone instance.

9

@jimsheldon yes, your solution should work. The only edge case is using our docker-in-docker plugins (plugins/ecr, plugins/gcr) to build and publish to a third party registry (ecr, gcr, etc) but where the Dockerfile pulls images from dockerhub. Does this apply to your setup?

10

UPDATE, NOV 5, 6:00 PM EST

We have started rolling out changes to our Drone Cloud infrastructure to authenticate image pulls. These changes have been rolled out to our amd64 servers (as of 6pm est). The arm servers and arm64 servers are not updated yet. We will post status updates to this thread when we have more progress to share.

11

What’s the fix for those of us using the drone autoscaler?

12

Second question, does DRONE_DOCKER_CONFIG have to be /root/.docker ? We had assumed no.

13

Thanks for verifying.

We are aware of the issue you’re describing, we are working on a communication for our users around that.

14

  1. configure the autoscaler to use a custom cloud-init file that writes the config file to the host os

  2. configure the autoscaler to create runners that mount the config file using the DRONE_AGENT_VOLUMES variable

  3. configure the autoscaler to set the DRONE_DOCKER_CONFIG variable using the DRONE_AGENT_ENVIRON variable.

    DRONE_AGENT_ENVIRON=DRONE_DOCKER_CONFIG=/root/.docker/config.json

I will plan a minor release for next week that makes this configuration a little easier. This was previously an uncommon configuration, but now that it is going to be more common, we can optimize and make it easier for everyone. Stay tuned.

correct, it can be any path as long as it points to a file. Note that this file has to be mounted into the runner container in order for the runner to read and parse.

1 Like
15

We’re seeing an issue with launching new docker runners with the drone-autoscaler. The autoscaler pulls the drone-docker-runner image with an unauthenticated login, and there are no mechanisms to inject credentials to the docker daemon on the host that starts up drone-docker-runner.

I think it would be possible to create an AMI with the credentials pre-baked and signed in, but that seems a very complicated way to solve this.

Would it be possible to add a flag(s) to allow a user to add docker credentials for a docker login?

1 Like
16

You could use a customized cloud-init file for autoscaler, no need for custom AMI.

17

the recommended approach would be to customize the cloud-init file to install a ~/.docker/config.json with your credentials. See https://autoscale.drone.io/configure/cloud-init/

EDIT looks like @techknowlogick beat me to this answer :slight_smile:

18

We have a custom AMI built with /root/.docker/config.json

We have the autoscaler set up to insert this env var (using DRONE_AGENT_ENV_FILE) DRONE_DOCKER_CONFIG=/root/.docker/config.json

The autoscaler has this configuration
DRONE_AGENT_VOLUMES=/root/.docker:/root/.docker

I can check a working agent and see that the runner is working:

56af051cd495 drone/drone-runner-docker:1 "/bin/drone-runner-d…" 32 hours ago Up 32 hours 3000/tcp agent

I can do a docker exec -it /bin/sh into a working drone runner and see DRONE_DOCKER_CONFIG is set correctly and I can read from /root/.docker/config.json

When I use the CLI and type drone server info on a new agent I see:

Error: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

We have validated, today and several times, that the credentials we’re passing to docker works.

Our next step is to set DRONE_RUNNER_VOLUMES to /root/.docker, but we’re running out of options, and the rate limit keeps hitting us. Are we in some edge case with the autoscaler here?

19

An update on this.

After a lot of frustrated research, wondering why pipelines can pull private repos from docker hub but we’re still hitting a rate limit, we’ve narrowed down why we’re hitting the rate limit to the drone-docker plugin. Because we publish images to another registry (quay) we pull from docker hub and publish to quay - and it looks like drone-docker’s pulls are unauthenticated. We haven’t looked into a fix yet but one might be available. So all our dockerfiles with FROM python/blah for example are being pulled in an unauthenticated way.

We tested this by a multi-stage pipeline where we tried to trick drone into unauthenticated pulls of a private repo, all of which succeeded (meaning the drone step’s docker login worked, which is good) and a drone-docker step that built a container image with a FROM step pointing to a private repo the docker hub credentials we provided should have access to. That failed.

There’s still the lingering worry why we see the autoscaler was making unauthenticated docker hub pulls, even though the credential steps should be satisfied by our build. But maybe we’re doing something wrong there, hard to say.

20

I discuss this scenario in this comment, as well as possible solutions:
http://discuss.harness.io/t/docker-hub-rate-limits-and-drone-cloud-exemption/8293/4?u=bradrydzewski