Drone 1.0 and AWS ECR going forward

davidbyttow · November 26, 2018, 7:47pm

This has discussed before and there are several solutions out there, but they all seem hacky/brittle.

Because AWS ECR requires login via a short-lived token, it’s not possible to bake in long-term credentials. awslabs created a Docker credential store that can be used with docker config.json, however it seems that while drone agent supports config.json location, it doesn’t seem to play well with credHelpers.

So, to solve this, I’ve had to boot EC2 worker instances with a cron that periodically invokes “aws ecr get-login” to update the /.docker/config.json with the latest token, and mount that directory in order for the docker agent to get access to it, as described here: How do you authenticate with and pull from an ECR registry?

This feels quite brittle. Is there some sort of configuration plugin/service we can devise for drone agents prior to pulling an image? Or give agents support for credHelpers and a way to mount the executable that it will invoke for specific repos? E.g.,

{
  "credHelpers": {
    "xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/repo:tag": "ecr-login",
  }
}

This would invoke docker-credential-ecr-login, which can be built from this: https://github.com/awslabs/amazon-ecr-credential-helper

bradrydzewski · November 26, 2018, 8:10pm

yes, 1.0 supports registry credentials extensions although I haven’t had the chance to document or provide a starter project yet. I was planning to provide an official ecr extension at some point.

extensions are microservices (communicate via rest) so you would pass the following variables to your drone agents, which would instruct them to fetch credentials from the plugin.

DRONE_REGISTRY_ENDPOINT=https://your.drone.microservice
DRONE_REGISTRY_SECRET=some-shared-secret

the api request would then return the following response, in json format:

[{
  "address": "docker.io",
  "username": "octocat",
  "password": "correct-horse-battery-staple"
}]

there is actually a helper library for this: https://github.com/drone/drone-go/tree/master/plugin/registry

I will try and see if I can put together a sample plugin early next week. These extensions tend to be quite easy to implement, especially when using the helper library. I’m currently at re-invent, but would be happy to touch base next week regarding extensions, and help you get setup / answer any questions you have.

davidbyttow · November 26, 2018, 8:20pm

Oh nice, and this should work today? If so, I can just switch to that. I already am successfully using the Secret and Jsonnet plugins in a similar way.

davidbyttow · November 26, 2018, 8:20pm

Also, is the source code for 1.0 anywhere? I couldn’t find it, and that would be super helpful with respect to digging on these sorts of issues. I’ve been looking at older code.

bradrydzewski · November 26, 2018, 8:22pm

awesome, yes it should work, with the caveat that I have only tested registry plugins with unit tests. Usually I write a sample / starter plugin so that I can test in my integration environment, but that is still a pending task in my todo

jccguimaraes · November 27, 2018, 2:24pm

Hi, I am really keen on understanding what changes is needed for 1.0.0 to create an image in ECR. Any help? Thanks

jccguimaraes · November 27, 2018, 2:40pm

doing this

steps:
- name: ecr
  image: plugins/docker
  environment:
    AWS_ACCOUNT_ID:
      from_secret: aws_account_id
    AWS_REGION:
      from_secret: aws_region
  settings:
    repo: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${DRONE_REPO_NAME}
    registry: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com

gives me

+ /usr/local/bin/dockerd -g /var/lib/docker
Registry credentials not provided. Guest mode enabled.
+ /usr/local/bin/docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:05:38 2017
 OS/Arch:	linux/amd64
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
time="2018-11-27T14:33:37Z" level=fatal msg="exit status 1"

davidbyttow · November 27, 2018, 4:25pm

You need to mount the docker volume still here, as per the docs.

bradrydzewski · November 27, 2018, 4:25pm

@jccguimaraes for reference, you can look at the ecr documentation which shows examples for using the plugin with the 1.0 format. There are a few different issues that I see that I will walk through below, which will hopefully clear things up

First is that you are using the wrong docker image:

steps:
  - name: ecr
-   image: plugins/docker
+   image: plugins/ecr

Second is that you should be providing the access key and access key secret:

  settings:
    access_key:
      from_secret: aws_access_key_id
    secret_key:
      from_secret: aws_secret_access_key

The third issue is that the secrets should passed via the settings section. Normally this syntax would be allowed (for legacy reasons) but the docker and ecr plugins restrict passing environment variables directly.

steps:
  - name: publish  
    image: plugins/ecr
-   environment:
-     AWS_ACCOUNT_ID:
-       from_secret: aws_account_id
-     AWS_REGION:
-       from_secret: aws_region
    settings:
      access_key:
        from_secret: aws_access_key_id
      secret_key:
        from_secret: aws_secret_access_key

The fourth issue is that you are trying to use bash variable substitution (e.g. ${AWS_ACCOUNT_ID}.dkr.ecr... in your yaml file. The repo and registry attributes are unmarshaled into Go string literals. They are not processed by bash a bash interpreter, which means substitution will not work here.

  settings:
    repo: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${DRONE_REPO_NAME}
    registry: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com

davidbyttow · November 27, 2018, 5:49pm

FYI, I started a little plugin here, not fully tested yet – doing so in real time now. https://github.com/davidbyttow/drone-ecr-registry-plugin

bradrydzewski · November 27, 2018, 6:20pm

awesome! one thing you might find super helpful is that we’ve been adding commands to the CLI that allow you to test plugins. For example:

$ drone plugins secret get secret/data/docker username
$ drone plugins secret get secret/data/docker password

If you are looking for a good way to test registry credentials plugins, the CLI would be a good option. Unfortunately I’m flying out to aws:reinvent in a couple of hours and won’t have time to add a command today, but if you look at the drone secret get implementation, it is a small amount of code and could really save you some time if you end up needing to debug.

bradrydzewski · November 27, 2018, 6:58pm

update: I published a drone plugins registry list command that can be used for testing registry plugins. I tagged a v1.0.3 binary that can be downloaded from GitHub or can be built from source.

jccguimaraes · November 27, 2018, 7:17pm

hey!
thank you both for the help. turns out that the only way I could make the plugin work is with

image: plugins/ecr
  settings:
    access_key:
      from_secret: aws_access_key_id
    secret_key:
      from_secret: aws_secret_acess_key
    region:
      from_secret: aws_region
    repo: name-of-repository-in-ecs

c2h5oh · May 7, 2019, 7:16pm

This approach is a fix if you have access to both ECR and hub.docker.com/another registry that uses login/pass that can be stored in Drone secrets.

It doesn’t help when you’re limited to ECR only - any idea how to solve that case?

bradrydzewski · May 7, 2019, 7:57pm

This is handled by registry credential extensions. See the following:

ScOut3R · December 9, 2019, 3:10am

@bradrydzewski @davidbyttow

The https://github.com/davidbyttow/drone-ecr-registry-plugin seems to have stopped working with the docker runner.

I have just upgraded my autoscalers from 1.4.4 to 1.5.0 and though the new docker runner receives the credentials from the plugin (based on network tracing) it fails to pull images from ECR. The runner container does not log any error.

Is there any information I could provide to enable you assisting me in resolving this issue?

staffanselander · May 18, 2020, 4:48pm

@bradrydzewski

I was looking into creating a registry plugin for gcr like the one for ecr.
There is a deprecation notice on the Registry struct here https://github.com/drone/drone-go/blob/master/drone/types.go#L175.
Is the registry struct something that i should avoid?

bradrydzewski · May 18, 2020, 5:31pm

Thanks for brining this to my attention, I will remove the comment from the Registry structure to avoid causing confusion . You can continue to use registry extensions as documented here. We recommend using our starter project when creating a new extension.

However, you may not need to create an extension for GCR. The reason an extension was required for ECR is AWS does not provide long-lived credentials which means they need to be constantly refreshed. With GCR, you can create long lived credentials using JSON key authentication. You can provide these long-lived credentials to Drone by storing them as secrets [1][2] and referencing in your yaml.

We also provide a registry credentials plugin [3] that works with standard docker username and password authentication, which would also work with JSON key authentication. The main difference between using secrets vs extensions is that extensions do not require any additional yaml configuration.

With that being said, if you want to use your access token to generate short lived GCR credentials, an extensions is your best option. Hope that helps!

[1] https://docs.drone.io/secret/repository/
[2] https://docs.drone.io/secret/organization/
[3] https://github.com/drone/drone-registry-plugin

Gowiem · October 21, 2020, 8:17pm

@bradrydzewski @davidbyttow is drone-ecr-registry-plugin still the best and proper way to setup using / pulling private repositories from ECR? I’m looking to add this without needing to do manual ECR logins as I have 20+ drone files and a handful of plugins I’m managing for a client.

Bear with me as I’m a bit confused on this subject, but if it is still the best way to handle access to ECR, is the below the correct approach to get it working in a Kubernetes cluster?

Setup amazon-ecr-credential-helper on all k8s nodes so they have access to ECR.
Or should I possibly just be ensuring my cluster has access to ECR via EKS IAM service accounts role?
Run drone-ecr-registry-plugin as a simple, single Deployment / Pod.
Configure the drone agents to use the DRONE_REGISTRY_ENDPOINT, DRONE_REGISTRY_SECRET, and DRONE_REGISTRY_VERIFY="false" parameters to point to the new ecr-registry-plugin address.

Is it that simple? In the above, is DRONE_REGISTRY_SECRET any random value and it doesn’t matter what it is as long as the ecr-registry-plugins’ value matches the agents’ configured value? Or does that need to be something like the DRONE_RPC_SECRET?

Also, the Drone docs on registry plugins seem to mention DRONE_REGISTRY_PLUGIN_ENDPOINT and DRONE_REGISTRY_PLUGIN_TOKEN. Are these needed?

Thanks for the direction gents – Appreciate any help!

bradrydzewski · October 21, 2020, 9:20pm

I recommend installing the following registry extension:

This extension handles generating the docker username and password for ECR registries, as well as refreshing the username and password which otherwise expire every few hours.
You can install this extension on a single machine, and then you configure each of your runners to point to this extension.

Topic		Replies	Views
New Docker Runner breaks the registry plugin Drone Bugs	10	1393	February 1, 2020
AWS ECR Example auth Drone Support	2	521	March 19, 2018
Trouble with 1.0.0-rc.2 and plugins/ecr Drone Support	6	790	January 19, 2021
Using secrets from the drone registry plugin Drone Support	5	902	November 15, 2019
Questions on ECR Drone Support	3	933	August 14, 2019

Drone 1.0 and AWS ECR going forward

Related topics