Drone-vault plugin x509: certificate signed by unknown authority

Drone Support
1

I’ve been trying to get the drone-vaults plugin working all day and been struggling against the “x509: certificate signed by unknown authority” error. Specifically, this is the error every time the drone-vault container comes up:

time="2019-10-01T03:33:43Z" level=fatal msg="Post https://vault.our.domain.com/v1/auth/kubernetes/login: x509: certificate signed by unknown authority"

This is common when using self-signed certs, and we use self-signed certs with our Vault server. The common way around using this issue in vault is the VAULT_CACERT env variable, which should point to the CA’s cert file. This works for the vault CLI or golang API library, but after digging through the drone-vault code, it appears drone-vault doesn’t use the vault API library.

It appears this error is coming from here: https://github.com/drone/drone-vault/blob/master/plugin/token/kubernetes/http.go#L20-L23
This is doing a straight HTTP POST request to the vault server as opposed to using Hashicorps vault API library to create a vault client. This means the VAULT_CACERT variable will be ignored unless the drone-vault code deals with it itself (i.e. adds the CA certificate file to the trusted store of the httpClient before making the calls), which it doesn’t. This also means we can’t use VAULT_SKIP_VERIFY to get around it by ignoring the certs in the response.

Basically, this means any Vault server using self-signed certs will be unsupported. The only way I’ve found around this is to mount the Vault CA certificate into the drone-vault container and manually add the certificate to the container’s trusted store by appending the certificate to the end of the /etc/ssl/certs/ca-certificates.crt file before calling /bin/drone-vault. Something like this (in Kubernetes):

    spec:
      containers:
        - name: drone-vault
          image: drone/vault:latest
          command: ["sh", "-c", "cat /usr/local/share/ca-certificates/vault.crt >> /etc/ssl/certs/ca-certificates.crt && /bin/drone-vault"]
          ...
          volumeMounts:
          - mountPath: /usr/local/share/ca-certificates/vault.crt
            name: vault-tls
            subPath: ca.crt
      volumes:
      - name: vault-tls
        secret:
          secretName: vault-ca-cert

This is a short-term hack. I think the best way to address this would be to use the Vault API client library to authenticate with the Vault server instead of using raw HTTP requests. That way Vault ENV variables like VAULT_CACERT, VAULT_CAPATH, VAULT_SKIP_VERIFY would be honored correctly.

@bradrydzewski let me know if there’s an easy work-around for this issue that I’ve missed. Otherwise, is this something that should be tracked as a GitHub issue (I wanted to post here first)?

2

The vault plugin does use the official Vault API client [1] and therefore does respect the mentioned environment variables. Perhaps you can clarify further if I am misunderstanding?

[1] https://github.com/drone/drone-vault/blob/master/main.go#L70:L74

3

ok, I think I see what you mean. The kubernetes token refresher makes a raw http request. This was a third party contribution so I’m not too familiar with how third party authentication mechanisms work. Is there an endpoint in the official client that can be used for this? https://godoc.org/github.com/hashicorp/vault/api

4

Yeah, I noticed that the Vault API client is created (just not used everywhere) right after I made this topic. This issue is specifically with the Vault Kubernetes auth token fetching and renewing.

A Vault API client is created first (https://github.com/drone/drone-vault/blob/master/main.go#L70-L75) - and then an attempt is made to authenticate to Vault using Kubernetes auth, but this doesn’t use the vault client. It uses raw HTTP requests (https://github.com/drone/drone-vault/blob/master/main.go#L77-L91).

Therefore this issue specifically affects drone-vault implementations where the Vault server is using self-signed certs, AND Vault Kubernetes auth is used for authenticating.

I dug around to try to find an example of another application doing Vault Kubernetes Auth using the Vault API client, and I found this: https://github.com/banzaicloud/bank-vaults/blob/master/pkg/sdk/vault/client.go#L185-L305

Bank-vaults is BanzaiCloud’s Vault operator/ecosystem. We actually use their mutating webhook to inject secrets into our Kubernetes containers. It looks like what they do is:

  1. Create a “logical” client (not sure what this is - maybe like a raw HTTP client?) from a Vault Client object (which already exists in drone-vault): https://github.com/banzaicloud/bank-vaults/blob/master/pkg/sdk/vault/client.go#L186
  2. Use the “logical” client to hit the vault Auth endpoint: https://github.com/banzaicloud/bank-vaults/blob/master/pkg/sdk/vault/client.go#L250-L257
  3. Extract the Token and add it to the client: https://github.com/banzaicloud/bank-vaults/blob/master/pkg/sdk/vault/client.go#L267-L268

They also have all the token renewing code in there as well. Note that drone-vault already does steps 2 and 3 here: https://github.com/drone/drone-vault/blob/master/plugin/token/kubernetes/kube.go#L43-L70 It just does it with a raw HTTP client which doesn’t honor VAULT_*** env variables, as opposed to the Vault API client, which does.

So, to answer your specific question, I don’t see an “endpoint” in the Vault API client, but I do see a way to use the Vault API client to send the POST request to the Kubernetes Auth endpoint which will still honor the VAULT_*** env variables.

5

I also found this example from Seth Vargo’s Kubernetes Authenticator. Instead of using Vault’s API client, he uses a raw HTTP client, but respects the Vault env variables: https://github.com/sethvargo/vault-kubernetes-authenticator

You can see that when authenticating, he loads the root CAs, which are found using VAULT_CACERT and/or VAULT_CAPATH: https://github.com/sethvargo/vault-kubernetes-authenticator/blob/master/main.go#L206-L233
And adds them to a tlsClientConfig: https://github.com/sethvargo/vault-kubernetes-authenticator/blob/master/main.go#L122-L125
Applies that to the HTTP client: https://github.com/sethvargo/vault-kubernetes-authenticator/blob/master/main.go#L135-L137
Then makes the standard HTTP POST calls, with the HTTP client now respecting the private CA: https://github.com/sethvargo/vault-kubernetes-authenticator/blob/master/main.go#L149-L161

This seems like a more straightforward approach, but it makes drone-vault responsible for handling VAULT_*** env variables.