Can access Vault secrets over the CLI but not during builds

Drone Support
1

I’ve got an issue running Drone as a single machine instance, trying to store secrets in Vault and access them during builds with the Vault plugin. They can be retrieved successfully on the CLI, but not during builds.

Drone configuration:

  • Host hxxps://drone.mydomain.com
  • Running at 127.0.0.1.10280, forwarded by Nginx

Vault configuration:

  • Host hxxps://drone-vault.mydomain.com
  • Running at 127.0.0.1:8200, forwarded by Nginx

Vault plugin:

  • Running at :10281

Drone startup variables include:

  • DRONE_TOKEN=6Psxxx
  • VAULT_ADDR=hxxps://drone-vault.mydomain.com
  • VAULT_ROOT_TOKEN=s.WOCxxx
  • VAULT_TOKEN=s.WOCxxx
  • DRONE_SECRET_ENDPOINT=hxxp://127.0.0.1:10281
  • DRONE_SECRET_SECRET=4d4xxx

Vault plugin variables include:

  • VAULT_TLS_SERVER_NAME=https://drone-vault.mydomain.com
  • VAULT_ADDRESS=hxxp://127.0.0.1:8200
  • VAULT_ADDR=hxxp://127.0.0.1:8200
  • VAULT_API_ADDR=hxxp://127.0.0.1:8200

Some of those variable settings are redundant but at this point I’m not sure
what is and isn’t necessary :slight_smile: Have also tried switching a lot between
referring to the Vault address as its domain name and its internal address, but
no luck either way.

Storing a key ‘mykey’ at kv/test.

A successful Drone CLI request:

VAULT_ADDR=hxxps://drone-vault.mydomain.com DRONE_SECRET_ENDPOINT=hxxp://127.0.0.1:10281 DRONE_SECRET_SECRET=4d4xxx ./drone plugins secret get --repo=me/testrepo kv/test mykey

Usage in .drone.yml

  ---
  kind: pipeline
  name: linux-amd64

  platform:
    os: linux
    arch: amd64

  workspace:
    base: /build

  steps:
  - name: check
    image: bash
    environment:
      MYKEY:
        from_secret: mykey
    commands:
      - echo $MYKEY | base64

  trigger:
    branch:
    - master
    event:
    - push
    - promote
    - rollback

  ---
  kind: secret
  name: mykey
  get:
    path: kv/test
    name: mykey

Build output:

+ echo $MYKEY | base64
Cg==

I’ve tried using the internal URL to be the Vault endpoint (hxxp://127.0.0.1:8200) but that stops it working with CLI retrieval too.

Any help greatly appreciated!

2

I seem to be having similar issues. My config looks similar, but I have a drone server and a separate agent configured to point at the vault plugin.

If I run the drone plugin secret get command like you do, I can get the secret.

When running a build with a very similar config to yours, I get no secrets.

I started the agent with DRONE_LOGS_TRACE=true, and I get this output when the build runs:

{"kind":"secret","level":"trace","msg":"secret: encrypted: no matching secret","name":"test123","time":"2019-06-22T22:35:48Z"}
{"kind":"secret","level":"trace","msg":"secret: external: cannot get secret","name":"test123","time":"2019-06-22T22:35:48Z"}

The plugin container offers no useful logs.

Also not sure how to proceed debugging further, but strangely hopeful since I found somebody else with the same symptoms.

3

I pushed an update to log the error message. It is available in the :latest tag. Hopefully this helps you triage further.

4

That is very helpful, thank you. In my case it is a certificate issue talking to the vault plugin container through an ingress using internally signed certs.

5

Now I am getting to real “cannot find secret” errors from the plugin container itself.

I am definitely a bit confused by the 3 different config parameters for a secret. What is the distinction between path and name?

6

I finally made it work after reading the drone-vault code by using:

kind: secret
name: test123
get:
  path: /secret/drone/test
  name: mykey

I feel like those semantics feel kinda off with the presence of the key field as well. Is that the intended usage, or a bug somewhere?

edit- well it looks like all the examples do it that way, so did OP, so maybe I just got confused.

7

What is the distinction between path and name ?

consider the following vault command:

$ vault kv put secret/docker \
    username=octocat \
    password=correct-horse-battery-staple \
    x-drone-events=push,tag

It creates two secrets named username and password at path secret/docker. This document might be helpful. https://docs.drone.io/runner/extensions/vault/

8

That makes sense. I think we have slightly different mental models of vault secrets. I would consider the path secret/docker a single secret with multiple key/value pairs inside it. Each key in the secret can be mapped to a single drone secret. Thanks for clarifying.

9

path and name were chosen as generic attribute names to support multiple providers (vault, aws secret manager, kubernetes, etc). I recognize that key would be more appropriate than name for vault, which is why we do plan to accept key [1] as an alias to name in a future release. Some of the code is already in place.

[1] https://github.com/drone/drone-yaml/blob/master/yaml/secret.go#L41

10

That logging is really helpful! Here’s the culprit:

{"error":"Post http://127.0.0.1:10281: dial tcp 127.0.0.1:10281: connect: connection refused","kind":"secret","level":"trace","msg":"secret: external: cannot get secret","name":"server_user","time":"2019-06-23T17:23:54Z"}

Using --network="host" solves this - don’t know if that’s the best way though!

Thanks for your help!