Drone-runner-kube slow to process spike in pipeline requests

[bradrydzewski]

@colinhoglund

When running with a single kube-runner replica, start times balloon pretty quickly to 10x+ their normal start times and build containers are scheduled in what looks to be something of a sequential process.

Thanks for testing this out. We will carve out some time to stress test the runner and see if we can reproduce. Our of curiosity, what is the start time you are seeing?

@colinhoglund

If scaling the kube-runner requires no special consideration, then we will probably just look at throwing more replicas at it for now. I suppose we could also look into setting up an HPA with custom metrics to scale using the drone_running_jobs metric. Do you think that would be the appropriate metric to determine if the kube-runner needs more capacity?

I would recommend using a fixed number for now. The problem with autoscaling is that kubernetes could end up terminating a runner while pipelines are executing, which would result in those pipelines getting stuck in a zombie state.