alonsodomin
April 15, 2021, 11:58am
1
Hi all,
We have a setup of Drone deployed into an Azure AKS cluster using the K8S runner and we’d like to attach external volumes into the pipelines so we can offload the caching of build artifacts (i.e. dependencies) into an Azure FileShare.
We have already configured the appropriate persistent volume claims to allow Kubernetes to mount the Azure File Share as a volume into the pods that request it, so, for example, we can mount that storage into a /opt/drone/cache folder in the Drone Runner using extraVolumes and extraVolumeMounts.
Then we configure the pipelines to use the previously mentioned folder from the host using
kind: pipeline
type: kubernetes
name: default
steps:
- name: build
image: ...
volumes:
- name: cache
path: /root/.ivy2 # This is where Scala's SBT stores the dependencies
volumes:
- name: cache
host:
path: /opt/drone/cache
When the build kicks in and completes, we expect being able to see the cached artifacts in the Azure Storage, however we see the storage being empty. Which makes us think that Drone is not mounting the mentioned volume into the pipeline pod.
If we configure the extra volume mounts in the drone server deployment, notice the same thing.
How should we configure the Drone deployment so we can do that? Does this need to go in the runner policy instead or is not possible to do right now?
@alonsodomin ,
Could you please enable trace logging and share the logs so that we can review and suggest accordingly.
Regards,
alonsodomin
April 15, 2021, 1:13pm
3
@csgit Just enabled debug and tracing logs, this is what can be seen in the runner logs:
time="2021-04-15T13:04:24Z" level=debug msg="stage received" stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:24Z" level=debug msg="stage received" stage.id=9487 stage.name=validate stage.number=1 thread=6
time="2021-04-15T13:04:24Z" level=debug msg="stage accepted" stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:24Z" level=debug msg="stage accepted by another runner" stage.id=9487 stage.name=validate stage.number=1 thread=6
time="2021-04-15T13:04:24Z" level=debug msg="poller: request stage from remote server" thread=6
time="2021-04-15T13:04:24Z" level=debug msg="stage details fetched" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=bankifibot_ghtoken thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=artifactory_password thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=slack_webhook thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=dockerconfig thread=4
time="2021-04-15T13:04:24Z" level=debug msg="updated stage to running" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:32Z" level=debug msg="received exit code 0" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name=clone thread=4
time="2021-04-15T13:04:36Z" level=debug msg="received exit code 0" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name="fetch tags" thread=4
time="2021-04-15T13:04:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:04:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:05:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:05:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:07:09Z" level=debug msg="stage received" stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:09Z" level=debug msg="stage received" stage.id=9488 stage.name=build stage.number=1 thread=8
time="2021-04-15T13:07:09Z" level=debug msg="stage accepted by another runner" stage.id=9488 stage.name=build stage.number=1 thread=8
time="2021-04-15T13:07:09Z" level=debug msg="poller: request stage from remote server" thread=8
time="2021-04-15T13:07:09Z" level=debug msg="stage accepted" stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:10Z" level=debug msg="stage details fetched" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=artifactory_password thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=bankifibot_ghtoken thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_subscription_id thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_username thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_password thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_tenant_id thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_id thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_secret thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=slack_webhook thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=dockerconfig thread=7
time="2021-04-15T13:07:10Z" level=debug msg="updated stage to running" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:20Z" level=debug msg="received exit code 0" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name=clone thread=7
time="2021-04-15T13:07:24Z" level=debug msg="received exit code 0" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name="fetch tags" thread=7
time="2021-04-15T13:07:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:07:39Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:07:40Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:08:06Z" level=debug msg="received exit code 1" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name="Build Pull Request" thread=7
time="2021-04-15T13:08:08Z" level=debug msg="received exit code 0" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name=notify thread=7
time="2021-04-15T13:08:08Z" level=debug msg="destroying the pipeline environment" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:13Z" level=debug msg="successfully destroyed the pipeline environment" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:13Z" level=debug msg="updated stage to complete" build.id=6754 build.number=85 duration=58 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:13Z" level=debug msg="poller: request stage from remote server" thread=7
time="2021-04-15T13:08:13Z" level=trace msg="http: context canceled"
time="2021-04-15T13:08:13Z" level=debug msg="done listening for cancellations" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:08:19Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:08:39Z" level=debug msg="stage received" stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:39Z" level=debug msg="stage received" stage.id=9489 stage.name=build stage.number=1 thread=10
time="2021-04-15T13:08:39Z" level=debug msg="stage received" stage.id=9489 stage.name=build stage.number=1 thread=11
time="2021-04-15T13:08:39Z" level=debug msg="stage accepted" stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:39Z" level=debug msg="stage accepted by another runner" stage.id=9489 stage.name=build stage.number=1 thread=11
time="2021-04-15T13:08:39Z" level=debug msg="poller: request stage from remote server" thread=11
time="2021-04-15T13:08:39Z" level=debug msg="stage details fetched" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=artifactory_password thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=bankifibot_ghtoken thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_username thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_password thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_tenant_id thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_secret thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_subscription_id thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_id thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=slack_webhook thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=dockerconfig thread=9
time="2021-04-15T13:08:39Z" level=debug msg="stage accepted by another runner" stage.id=9489 stage.name=build stage.number=1 thread=10
time="2021-04-15T13:08:39Z" level=debug msg="poller: request stage from remote server" thread=10
time="2021-04-15T13:08:39Z" level=debug msg="updated stage to running" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:54Z" level=debug msg="received exit code 0" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name=clone thread=9
time="2021-04-15T13:08:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:00Z" level=debug msg="received exit code 0" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name="fetch tags" thread=9
time="2021-04-15T13:09:09Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:09Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:39Z" level=debug msg="received exit code 1" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name="Build Pull Request" thread=9
time="2021-04-15T13:09:41Z" level=debug msg="received exit code 0" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name=notify thread=9
time="2021-04-15T13:09:41Z" level=debug msg="destroying the pipeline environment" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:46Z" level=debug msg="successfully destroyed the pipeline environment" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:46Z" level=debug msg="updated stage to complete" build.id=6755 build.number=86 duration=62 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:46Z" level=debug msg="poller: request stage from remote server" thread=9
time="2021-04-15T13:09:46Z" level=trace msg="http: context canceled"
time="2021-04-15T13:09:46Z" level=debug msg="done listening for cancellations" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:49Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:10:09Z" level=debug msg="received exit code 1" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name=validate thread=4
time="2021-04-15T13:10:11Z" level=debug msg="received exit code 0" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name=notify thread=4
time="2021-04-15T13:10:12Z" level=debug msg="destroying the pipeline environment" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:10:17Z" level=debug msg="successfully destroyed the pipeline environment" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:17Z" level=debug msg="updated stage to complete" build.id=6753 build.number=953 duration=347 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:17Z" level=debug msg="poller: request stage from remote server" thread=4
time="2021-04-15T13:10:24Z" level=trace msg="http: context canceled"
time="2021-04-15T13:10:24Z" level=debug msg="done listening for cancellations" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:29Z" level=trace msg="http: no content returned: re-connect and re-try"
For sake of completion, this is the output from kubectl when describing the runner pod:
Name: default-drone-runner-kube-6b76d5f556-6xgv6
Namespace: drone
Priority: 0
Node: aks-overspill-73472730-vmss000008/10.1.1.58
Start Time: Thu, 15 Apr 2021 14:02:50 +0100
Labels: app.kubernetes.io/component=drone-runner-kube
app.kubernetes.io/instance=default
app.kubernetes.io/name=drone-runner-kube
pod-template-hash=6b76d5f556
Annotations: checksum/secrets: bad092f483a39ab149c361b3705111e229d7bd973055d38213a9b7fb3dcdcafa
Status: Running
IP: 10.1.1.66
IPs:
IP: 10.1.1.66
Controlled By: ReplicaSet/default-drone-runner-kube-6b76d5f556
Containers:
server:
Container ID: containerd://76fe79f57dbf83538ae9a4c2a17ddcb8cdaaca39a71dc392bfc7e8dd8006cb4d
Image: drone/drone-runner-kube:1.0.0-beta.6
Image ID: docker.io/drone/drone-runner-kube@sha256:170aa6766441df9e76a44b529dedeb5ac84a2edb81b7bc0a527cbbfabf8c35f2
Port: 3000/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 15 Apr 2021 14:02:52 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 3
memory: 500Mi
Requests:
cpu: 2
memory: 250Mi
Environment Variables from:
default-drone-runner-kube ConfigMap Optional: false
drone-runner-config Secret Optional: false
Environment: <none>
Mounts:
/etc/runner-config from config-files (ro)
/opt/cache from cache (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-drone-runner-kube-token-49rcj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config-files:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: drone-runner-configfiles
Optional: false
cache:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: cache
ReadOnly: false
default-drone-runner-kube-token-49rcj:
Type: Secret (a volume populated by a Secret)
SecretName: default-drone-runner-kube-token-49rcj
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
The yaml configuration you posted for mounting volumes looks correct to me. For reference, you can see a sample yaml configuration that we use in our integration tests here [1]. I feel like you could definitively test whether or not drone is mounting the volume with the below pipeline:
kind: pipeline
type: kubernetes
name: default
steps:
- name: write
image: alpine
commands:
- echo "hello" > /root/.ivy2/greetings.txt
volumes:
- name: cache
path: /root/.ivy2
- name: read
image: alpine
commands:
- ls -la /root/.ivy2
- cat /root/.ivy2/greetings.txt
volumes:
- name: cache
path: /root/.ivy2
volumes:
- name: cache
host:
path: /opt/drone/cache
However, please note that the above mounts a host volume, not a persistent volume claim. I believe configuring a persistent volume claim requires different syntax than mounting a host volume.
[1] https://github.com/drone-runners/drone-runner-kube/blob/master/samples/volume_host.yml
If we configure the extra volume mounts in the drone server deployment, notice the same thing.
I’m not sure I fully understand, since the Drone server does not have any configuration options that relate to kubernetes or volume mounting.
alonsodomin
April 15, 2021, 2:36pm
5
@bradrydzewski thanks for checking that in your side.
Regarding your comment:
I’m not sure I fully understand, since the Drone server does not have any configuration options that relate to kubernetes or volume mounting.
The chart in here seems to have the options for extraVolumes and extraVolumeMounts. Regardless, we used those options but the drone runner instead: https://github.com/drone/charts/blob/master/charts/drone/values.yaml
We may have misunderstood how to take advantage of those settings. Our understanding was that we could mount a Persistent Volume Claim into the Drone Runner (using extraVolumes and extraVolumeMounts) and then in the pipeline, as you showed before, we could configure the pipeline to mount a host volume that just matches the same folder as where there PVC has been mounted in the Runner. The assumption is that when a new pipeline gets triggered, the host volume mounted from Drone will happen to be the PVC previously mounted.
It seems that our assumption is clearly not how the runner works and wondering if such a thing could be done. It would be very beneficial for us being able to share the caches across several pipelines running in different nodes as this kind of caches can get very big over time and being able to share them can save a lot of time out of our builds.
Yes it works a little differently. The kubernetes runner launches pipelines as pods on your clusters using the kubernetes API. This means mounting volumes into the runner’s Pod will have no impact on the pipeline Pods that the runner schedules on your cluster, since they are separate.
We merged a pull request that allows mounting a persistent volume claim that you may want to explore https://github.com/drone-runners/drone-runner-kube/pull/28 .
alonsodomin
April 16, 2021, 9:37am
7
Thanks for that tip, we just had the chance to give it a go right now and it does provide what we were expecting.
Many thanks again, worth adding that to the public documentation.