Kubernetes runner: operation cannot be fulfilled; the object has been modified

Drone Support
1

We have been having a lot of issues with pipeline pods terminating and messages similar to:

Operation cannot be fulfilled on pods "drone-pfrankdbb7xsyn18sraw": the object has been modified; please apply your changes to the latest version and try again

I am not sure if this is related to not setting any limits on containers/steps in the pipeline?
I have integration tests running using drone pipeline services, and want to run 5-10 services to run tests against, with maybe 10 of these running in parallel.
We are currently not setting any limits inside the .drone.yaml, unsure if these are needed if we have lots of resources?

Is there any performance or information you can give on how to resolve this issue or best practices for large work loads?

drone-server: 1.6.5
kube-runner: latest
running on a 3 node cluster with 16VCPU and 16Gig RAM with autoscaling (for example but we have tried other setups)

Thanks!

1 Like
2

Unfortunately I do not have answers to these questions. This is one reason the Kubernetes runner is still in Beta. It has not seen extensive real world usage (relative to the Docker runner which has been used in production for 6 years) and we have limited experience debugging. We encourage teams take an active role in the Beta including evaluating the code and sending patches when possible. We put together this guide to help: Contributing to Drone for Kubernetes.

3

Thanks for your response.

Is there other runners which have known good performance for something similar to what I described? the autoscaler in aws/gcp for example?

4

We recently merged a pull request (available in :latest) that attempts to solve this error