Drone-vault plugin not quite working

atugade · February 1, 2019, 2:21am

So me and colleague are trying to get the vault feature going with drone. Looks like we’re almost there but secrets aren’t being fetched as expected. This is our docker-compose file that we’re using to prototype this which includes a vault instance as well.

version: '3'

services:
  drone-server:
    image: drone/drone:1.0.0-rc.3
    ports:
      - 80:80
    volumes:
      - /var/lib/drone:/var/lib/drone
      - /etc/drone:/etc/drone
    restart: always
    environment:
      - DRONE_RPC_SECRET=XXXXXXXXXX
      - DRONE_SERVER_HOST=<YOUR_IP_HERE>
      - DRONE_SERVER_PROTO=http
      - DRONE_TLS_AUTOCERT=false
      - DRONE_GITHUB_CLIENT_ID=0000000
      - DRONE_GITHUB_CLIENT_SECRET=11111111
      - DRONE_GITHUB_SERVER=https://github.com
      - DRONE_LOGS_PRETTY=true
      - DRONE_LOGS_COLOR=true
      - DRONE_LICENSE=/etc/drone/drone.key
      - DRONE_SECRET_ENDPOINT=http://drone-vault
      - DRONE_DEBUG=true
  drone-agent:
    image: drone/agent:1.0.0-rc.3
    command: agent
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/drone:/etc/drone
    restart: always
    environment:
     - DRONE_RPC_SECRET=XXXXXXXXXX
     - DRONE_RPC_SERVER=http://drone-server 
     - DRONE_RUNNER_CAPACITY=3
     - DRONE_SECRET_SECRET=YYYYYYYYYYY
     - DRONE_SECRET_ENDPOINT=http://drone-vault
  drone-vault:
    image: drone/vault:latest
    ports:
      - 3000:3000
    command: agent
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/drone:/etc/drone
    restart: always
    environment:
     - DRONE_RPC_SECRET=XXXXXXXXXX
     - DRONE_RPC_SERVER=http://drone-server
     - DRONE_RUNNER_CAPACITY=3
     - VAULT_ADDR=http://127.0.0.1:8200
     - VAULT_TOKEN=<YOUR_VAULT_TOKEN>
     - SECRET_KEY=YYYYYYYYYYY
     - DEBUG=true
  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

We unseal vault, and create the inital secret as described in the document. We then verify from the host system we’re running docker-compose to validate we can fetch the secret:

$ curl --header "X-Vault-Token: $VAULT_TOKEN" http://127.0.0.1:8200/v1/secret/docker

{"request_id":"6fb16735-7b30-018c-b68c-35779c765f59","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"password":"correct-horse-battery-staple","username":"octocat","x-drone-events":"push,tag","x-drone-repos":"octocat/*,spaceghost/*"},"wrap_info":null,"warnings":null,"auth":null}

Despite this working, when we try to use the the following command, it is unable to find a secret:

export DRONE_SECRET_ENDPOINT=http://127.0.0.1:3000
export DRONE_SECRET_SECRET=YYYYYYYYYYY

arwin.tugade@john-drone-2:~/docker$ drone plugins secret get secret/docker --repo=octocat/stank
secret not found

$ docker logs 8da8692a0a0f
time="2019-02-01T01:54:20Z" level=info msg="server listening on address :3000"
time="2019-02-01T01:54:20Z" level=debug msg="vault: token rereshing disabled"
time="2019-02-01T01:54:27Z" level=debug msg="secrets: cannot find secret : secret not found"
time="2019-02-01T01:55:24Z" level=debug msg="secrets: cannot find secret : secret not found"
time="2019-02-01T02:17:51Z" level=debug msg="secrets: cannot find secret : secret not found"
time="2019-02-01T02:18:48Z" level=debug msg="secrets: cannot find secret : secret not found"

Might you have any insight into this?

bradrydzewski · February 1, 2019, 2:34am

Can you provide a sample of the yaml configuration file that you are using? Also, just an FYI the latest version is rc.5

atugade · February 1, 2019, 4:07am

I’m not sure what yaml configuration you are referring to, can you please clarify?

bradrydzewski · February 1, 2019, 6:40am

yes, it would be helpful to see the .drone.yml configuration file that you are using. The .drone.yml file should define the secrets that are fetched from the vault plugin. Seeing the yaml configuration should help triage this further.

VaultVulp · February 1, 2019, 6:45pm

I second the described problem. I’ve tried to setup Vault plugin, and was unsuccessful as well. I spend a little bit of time looking for the environment variables specified in the documentation: DRONE_SECRET_ENDPOINT and DRONE_SECRET_SECRE, but I wasn’t able to find all of them - only the DRONE_SECRET_ENDPOINT is used to init secret-service config parameter here. And, as far as I understand, after the initialization, this config parameter is ignored.

I’m not a Go developer, so I definitely may miss some Go-magic, but for me, it looks like that you cannot use this plugin, due to the fact that Drone Agent never tries to connect to the Drone-Vault plugin instance.

bradrydzewski · February 1, 2019, 6:52pm

the source code for the Drone 1.0 has not been released yet, which is why you are unable to find the corresponding code. In terms of difficulty getting things working, we are at the point where we need sample agent logs, sample agent configurations, and sample yaml files to proceed (the first two items have been provided in this thread, but I am waiting on the latter).

It is also possible the vault plugin’s Docker image is outdated and needs to be re-built to work with the latest release candidate due to some internal changes. I will force-publish a new image over the weekend.

atugade · February 1, 2019, 7:12pm

Ok here is a the .drone.yml that i’ve been testing with:

kind: pipeline
name: default

steps:
- name: build
  image: alpine
  environment:
    USERNAME:
      from_secret: username
  commands:
  - echo "hi there"
  - echo ${USERNAME}

---
kind: secret

external_data:
  username:
    path: secret/docker
    name: username

I just wanted to point out though, my test that I listed above is using the “drone plugin secret get” cli command, shouldn’t I be able to do it with the cli before trying it in a build job?

bradrydzewski · February 1, 2019, 8:00pm

I just wanted to point out though, my test that I listed above is using the “drone plugin secret get” cli command, shouldn’t I be able to do it with the cli before trying it in a build job?

yes you should. I have a feeling the drone/vault image is outdated which is the root cause of the problem. I seem to remember testing locally with the raw binary compiled from source, which also points to a problem with the docker image. I will build and publish the image and then test locally and let you know what I find.

bradrydzewski · February 6, 2019, 4:33am

I published an updated Vault image for testing.

wwcwang · February 12, 2019, 1:52pm

The new image seems still not work, but the main problem is lack of debug messages and method for trouble shooting. drone-vault and drone-agent print nothing. I think it’s useful if drone-vault and drone-agent shoud print any message when encountered problems.

zdk123 · February 14, 2019, 8:02pm

Can confirm that I’m also seeing this bug, with the latest drone/vault

level=debug msg="secrets: cannot find secret : secret key not found"

bradrydzewski · February 14, 2019, 9:06pm

but the main problem is lack of debug messages and method for trouble shooting

The vault plugin supports debug logging with DEBUG=true

The agent supports additional logging with DRONE_LOGS_TRACE=true. Note that this should set temporarily, and should be unset when you are finished debugging.

Can confirm that I’m also seeing this bug, with the latest drone/vault

I downloaded the latest vault image and I was able to confirm that it is working using the command line tools, and I was also able to confirm it is working with Drone server.

download the vault source code and start the server
download the CLI
use the drone plugins secret get command to confirm the connection works

$ drone plugins secret get secret/data/docker username --repo octocat/hello-world
octocat
$ drone plugins secret get secret/data/docker password --repo octocat/hello-world
correct-horse-battery-staple

I then tested with the following yaml configuration and it worked as expected:

kind: pipeline
name: greeting

steps:
- name: en
  image: alpine:3.8
  environment:
    USERNAME:
      from_secret: username
    PASSWORD:
      from_secret: password
  commands:
  - env
---
kind: secret
name: username
get:
  path: secret/data/docker
  name: username

So at this point if anyone is having trouble I recommend:

download the Vault plugin from source, build, run locally
download the CLI
test with the CLI and verify everything is working for you

zdk123 · February 14, 2019, 9:16pm

Thanks for the followup @bradrydzewski. It turns out the command in the OP didn’t actually specify the secret, and I followed it without thinking.

For posterity:

drone plugins secret get secret/docker --repo=octocat/stank

should be:

drone plugins secret get secret/docker username --repo=octocat/stank

For example. I updated vault for good measure and the command line is working.

Topic		Replies	Views
[solved] External secret empty, unreliable Drone Support	1	886	October 28, 2019
Drone-vault: Missing vault address Drone Support	14	1648	February 4, 2019
Can access Vault secrets over the CLI but not during builds Drone Support	9	1047	June 23, 2019
`image_pull_secrets` from vault secrets plugin not working Drone Support	0	754	September 21, 2019
Secret not working/empty Drone Support	2	1823	June 8, 2017

Drone-vault plugin not quite working

Related topics