How to prevent DockerHub pull rate limit errors

Drone FAQ
1

DockerHub recently implement rate limits that may cause pipeline to fail when Docker attempts to pull images. This failure can be identified by the following error message:

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: Understanding Your Docker Hub Rate Limit | Docker

Pulling Pipeline Images

The docker daemon pulls images defined in your pipeline with no authentication, by default. We recommend creating a service account at DockerHub and purchasing a plan, which costs $5 per month. Then provide the service account credentials to your runner

First, create the credentials json file and save to your machine
The Kubernetes team provides a nice tutorial for creating this file.

Next, mount this file as a volume into your runner container. Note that this is the path to the file that was mounted inside the container, not the path on the host machine.

-v /root/.docker/config.json:/root/.docker/config.json

Last, provide the runner the path to the credentials json file. Note that this is the path to the file that was mounted inside the container, not the path on the host machine.

-e DRONE_DOCKER_CONFIG=/root/.docker/config.json

Alternatively, you can install a registry credential extension. Registry credential extensions dynamically serve registry credentials to runners over http(s) and are especially useful if you need to generate credentials on-the-fly, for example, when working with ecr.

Caching Pipeline Image

The docker daemon automatically caches images defined in your pipeline. You can force your pipeline to use the cache by always using versioned images (not latest).

image: golang:1.15

Using untagged images or the latest image tag will result in docker pulling the image every time, even if it exists in your cache. You can override the behavior by customizing the pull policy as shown here:

image: golang:latest
pull: if-not-exists

Docker-in-Docker Plugins

There are two different scenarios that Drone users need to account for when it comes to the docker plugin and rate limiting (these may also applies to ecr and gcr plugins).

Scenario 1: Build and Publish to Dockerhub

If you are building and publishing to Dockerhub, you provide username and password to the plugin via the settings section in the yaml (see below example). The plugins always executes a docker login using these credentials before it builds and publishes the image. This means that image pulls are going to be authenticated and will receive increased rate limits. If the username and password are associated with a paid dockerhub account, there will be no pull limit.

The recommended solution to rate limiting in this scenario is to ensure the username and password provided to the plugin are associated with a paid account. You could even use the organization secrets feature to provide this at an organization level. However, this may require changing yaml files or changing existing secrets.

- name: build
  image: plugins/docker
  settings:
    repo: foo/bar
    username: ...
    password: ...

Scenario 2: Build and Publish to other Registry, but Dockerfile references Dockerhub Images

If you are supplying a username and password to the dind plugin, but the username and password is for another registry (for example, quay) and if your dockerfile pulls from dockerhub, the pulls would not be authenticated and would be subject to rate limits.

The recommended solution in this scenario is to provide dockerhub credentials using the config parameter. Note that these credentials should be associated with a paid account to avoid rate limiting. The config parameter should contain the contents of a docker/config.json file.

- name: build
  image: plugins/docker
  settings:
    repo: quay.io/foo/bar
    username: ...
    password: ...
    config: |
      {
          "auths": {
              "https://index.docker.io/v1/": {
                  "auth": "c3R...zE2"
              }
          }
      }

The config file can also be sourced from a secret:

config:
  from_secret: ...

This variable can also be configured globally by setting a global environment variable with your runner. This would be the equivalent of setting the config in every yaml.

PLUGIN_CONFIG={ "auths" .... }

Global Solution: Using a Global Mirror

Finally, a global solution to this problem is to setup a registry mirror. We have some customers that setup and mirror to proxy and authenticate requests to dockerhub. I am not sure how to configure registry mirrors with kubernetes, but with docker you can configure mirrors by editing the docker daemon.json file.

The docker plugin also provides an option to configure a mirror in the yaml. You could globally configure this mirror by setting a global environment variable with your runner. This would be the equivalent of setting the mirror in every yaml.

PLUGIN_MIRROR=https://docker.company.com
3 Likes