Drone on GKE with Workload Identity

Drone Support
1

Hello,

I run drone on GKE on a cluster with Workload Identity enabled: it’s a very neat way to avoid having to handle service account keys and would make drone just work without configuring secrets (at least from the gcloud side of things)

So far however, I have not been successful in configuring it, so I wonder if you intend to support it.

What I have done:

  • Enabled Workload Identity as in the documentation
  • Assign my GKE service account to the K8S “default” service account in the default namespace (used by the drone jobs) (and test it works)
  • On drone, test that it works by running:
---
kind: pipeline
type: kubernetes
name: Kapitan Compile

steps:
- name: Configure Docker
  image: google/cloud-sdk:slim
  commands:
  - gcloud auth configure-docker

- name: pull private image
  image: eu.gcr.io/privaterepo/image
  commands:
  - echo "works"

# Doesn't work
- name: publish  
  image: plugins/gcr
  settings:
    registry: eu.gcr.io
    repo: eu.gcr.io/private_repo/kapitan
    dockerfile: kapitan/Dockerfile
    tags: latest
    build_args:
    - KAPITAN_RELEASE=0.27.4-ci

As you can see, already at this step I am able to configure docker with no need to share any secret.

However, when I try to build and push an image, the job fails. I have tried with both docker and gcr plugins.

unauthorized: You don’t have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
224 time=“2020-06-10T15:46:20Z” level=fatal msg=“exit status 1”

Just to confirm, if I run the exact same command with a traditional secret, it works:

steps:
- name: publish  
  image: plugins/gcr
  settings:
    registry: eu.gcr.io
    repo: eu.gcr.io/private_repo/kapitan
    dockerfile: kapitan/Dockerfile
    tags: latest
    build_args:
    - KAPITAN_RELEASE=0.27.4-ci
    json_key:
      from_secret: google_credentials

---
kind: secret
name: google_credentials
get:
  path: drone-runner
  name: google_credentials

Would you be interested in supporting workload identity? It would make deploying drone to a GKE enabled cluster so much quicker and secure.

Regards
Alessandro

1 Like
2

Hi Allessandro,
I am not familiar with Workload Identity and I don’t personally use Kubernetes, so I cannot really answer specific questions. But I can probably provide some details that might explain why pulling an image vs pushing and image works differently.

When your pipeline step uses a private image, Drone creates a Kubernetes Pod where your pipeline step is represented as a container in the Pod. Kubernetes is therefore responsible for pulling all images.

When you use the Docker plugin to push an image, it is important to remember that this particular step is a container running inside a Pod, and this particular step plugin uses docker-in-docker. This means the docker commands are running inside a Pod using a docker-in-docker daemon, which are isolated from the host machine configuration.

3

Thanks, that explains the odd behaviour.

I believe in the case of workload identity the secrets are made available using a special metadata endpoint:

http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/drone-agent@gcp-projectiam.gserviceaccount.com/token

Which I think means that it could easily be supported even in the case of docker-in-docker as long as you have connectivity.

I was hoping that the libraries you use would support fetching this service account transparently, which would have been neat. Do you mind if I open a feature request for it on the project Github?

4

I was hoping that the libraries you use would support fetching this service account transparently, which would have been neat.

In this case we do not really use any libraries. A pipeline step is just a container that runs inside a Pod. A plugin is just a container that runs a pre-defined script. And the Docker plugin is really just running plain old docker commands against a docker-in-docker daemon. Drone does not have any special knowledge of what a plugin is or does, so it would not know that the Docker plugin publishes Docker images or might require some special authentication.

Do you mind if I open a feature request for it on the project Github?

I would recommend starting out by creating an extension to inject custom secrets or tokens into your pipeline steps: Overview | Drone

In general we prefer implementing these features as third party extensions (we have done similar for Vault secrets, for example). Once the feature is implemented as an extension it gives us the opportunity to gauge community demand to determine whether or not it should remain a third party extension, or integrated in Drone core.

1 Like
5

Hi @ademaria, did you ever get this working or created that feature request?
We’re faced with exactly the same problem, which is super annoying since we’ve managed to avoid the whole hassle of service accounts tokens this far by relying on workload identity.

1 Like
6

Unfortunately not, especially because I moved to use “docker” nodes instead of kubernetes ones (using drone-autoscaler). But I believe it should be pretty straightforward to adapt the plugin/gcr to use workload identity if you fancy a bit of golang.

7

That was my conclusion as well, unfortunately golang isn’t my language of choice. I’m guessing/hoping this will eventually get picked up anyhow, because who would be using grc.io if not running on google cloud, and if running on google cloud these days workload identify is the way to go.
So for now we’ll probably have to pause our evaluation of drone.io and stick to our old system :cry:

8

unfortunately golang isn’t my language of choice. I’m guessing/hoping this will eventually get picked up anyhow

Plugins are just docker container that invoke a script (or program), and the plugin parameters in the yaml are passed in a plugin-agnostic manner (environment variables). This means a plugin can be written in any programming language, including bash. Remember that you are never required to use a plugin, since they are just wrappers around scripts, and they exist for convenience and re-use. You can opt to write shell commands directly, or even create your own plugin in your language of choice.

I am not familiar with workload identity, but if you cannot use the gcr plugin as-is, you can definitely build and publish images by defining shell commands [1][2]. You would just need to figure out how the shell commands use workload identity.

[1] Docker (dind) | Drone
[2] https://docs.drone.io//kubernetes/examples/service/docker/

9

workload identity basically is an association between a kubernets service account and a gcloud service account, so all the google command line utilities will automatically be authenticated as the correct service account, and publishing to gcr.io is as easy as first doing gcloud auth configure-docker as I understand it. (it works outside the docker in docker as expected for me)
But figuring out where to inject that extra line in the existing gcr plugin is a major undertaking when not accustomed to the codebase nor language, otherwise I would have tried it already :slight_smile:
It should be possible to get the needed tokens directly as well, but piggy backing on the gcloud cli tools seems like a much simpler approach.
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#using_from_your_code

10

For now we solved with just doing the docker things directly

kind: pipeline
name: default
type: kubernetes

steps:
  - name: docker-build-gcr
    image: google/cloud-sdk
    volumes:
      - name: dockersock
        path: /var/run
    commands:
      - gcloud auth configure-docker
      - docker build -t gcr.io/yourproject/yourimage:tag .
      - docker push gcr.io/yourproject/yourimage:tag
services:
  - name: docker
    image: docker:dind
    privileged: true
    volumes:
      - name: dockersock
        path: /var/run
volumes:
  - name: dockersock
    temp: {}

this correctly picks up the credentials from the environment using workload identity, so no secrets or tokens of any kind needed