# Drone EKS autoscaling - controller needs CPU requests? **URL:** \ **Category:** Drone Support\ **Created:** [June 13, 2019, 2:26am UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600 "2019-06-13T02:26:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jakswa](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/jakswa/32/4532_2.png) [@jakswa](https://drone.discourse.group/u/jakswa)\ **Post date:** [June 13, 2019, 2:26am UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600/1 "2019-06-13T02:26:58Z")
I’m running Drone (kubernetes-native) on AWS EKS, with an autoscaler running in the cluster. The hope is that when CPU utilization rises, the autoscaler will trigger new nodes to be added, and jobs will run on the new nodes. At first glance adding `resources` with CPU requests would get me what I need. However, some interrelated things seem to be thwarting me: - pipelines are created with node affinity. Drone ‘sticks’ the pipeline steps to the same node as their services, which I read as the same node as their ‘drone-job-\*’ pod - job controllers are created _without_ resources. The cluster doesn’t have the CPU requests up front, so it places these seemingly anywhere, even on CPU-starved nodes. So I have autoscaling set up to recognize that a step requires more CPU than available, but it can’t scale up because it’s stuck on the same node due to node affinity (I think): ```auto Scale-up predicate failed: GeneralPredicates predicate mismatch, cannot put [...] on [...], reason: node(s) didn't match node selector ``` Any guidance here is welcome. I’m going to continue to experiment but I’m running out of ideas. Essentially, right now it feels like I want to get CPU requests on these `drone-job-*` entries: ![image](https://canada1.discourse-cdn.com/flex003/uploads/testdrone/original/2X/2/211e8466052fd1357a188cf741909ec9b7e2ee1a.png) ---
**Author:** ![tjterris](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/tjterris/32/4310_2.png) [@tjterris](https://drone.discourse.group/u/tjterris)\ **Post date:** [June 13, 2019, 6:50pm UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600/2 "2019-06-13T18:50:45Z")
Hey I’m having the same problem. Would love some input from other Drone on K8s users. ---
**Author:** ![combor](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/combor/32/5224_2.png) [@combor](https://drone.discourse.group/u/combor)\ **Post date:** [June 20, 2019, 9:20pm UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600/3 "2019-06-20T21:20:41Z")
I can confirm that this is exactly what is happening. I have a masivelly parallel build ( with depends\_on statements ) and all the steps are started on the same node hammering it’s CPU. Please check the node column in the listing below: ``` $ kubectl -n xvcp0k60xwr1r25eyvfnsu58j1ybf39p get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE 8k8tjypwid4adter866km8nzv7pqgd4q 1/1 Running 0 17s 100.96.32.14 ip-10-0-96-165.eu-west-1.compute.internal a28rar2fhj6t8qfbhzsm1qvuq6lltkp5 1/1 Running 0 17s 100.96.32.15 ip-10-0-96-165.eu-west-1.compute.internal ggls4fz6bes2i3wpz4hmqkwlan6j2u36 1/1 Running 0 17s 100.96.32.16 ip-10-0-96-165.eu-west-1.compute.internal k1yz80sz57dvbz5ra4cepevfgql9opzm 1/1 Running 0 16s 100.96.32.17 ip-10-0-96-165.eu-west-1.compute.internal kpax184m5dxrfd76w1glkrbu9ocrxqmx 1/1 Running 0 17s 100.96.32.9 ip-10-0-96-165.eu-west-1.compute.internal lhe297eg809tdi13rg7p1hdaq5x6urv6 1/1 Running 0 17s 100.96.32.10 ip-10-0-96-165.eu-west-1.compute.internal m9kkvmff629cgt201zg6qzcz9b5gca9f 0/1 Completed 0 5m37s 100.96.32.8 ip-10-0-96-165.eu-west-1.compute.internal mc2slr1hiqs482pqvcpeve9zc9wy7aj3 1/1 Running 0 17s 100.96.32.12 ip-10-0-96-165.eu-west-1.compute.internal oinn10m4w0s8af7f0z0vu1mormc38kzy 1/1 Running 0 17s 100.96.32.11 ip-10-0-96-165.eu-west-1.compute.internal qmmzt55iq5iuh30moa29wf3uwtj1q6vz 0/1 Completed 0 5m37s 100.96.32.7 ip-10-0-96-165.eu-west-1.compute.internal rl1rctgy3qrycus43dvcrh7muhm8vmwg 1/1 Running 0 17s 100.96.32.13 ip-10-0-96-165.eu-west-1.compute.internal xadcb3v14xjpdg3t0uc2w6zon8139ye1 0/1 Completed 0 5m46s 100.96.32.6 ip-10-0-96-165.eu-west-1.compute.internal ``` ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 20, 2019, 9:25pm UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600/4 "2019-06-20T21:25:12Z")
Just a reminder that native Kubernetes runtime is still experimental and is not recommended for production use. It may be deprecated and replaced by Tekton in the future, so just be careful if relying on this for a production deployment. With that being said, we will accept patches that fix bugs with the current implementation. ---
**Author:** ![jakswa](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/jakswa/32/4532_2.png) [@jakswa](https://drone.discourse.group/u/jakswa)\ **Post date:** [June 20, 2019, 10:22pm UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600/5 "2019-06-20T22:22:04Z")
Fair enough, and there _are_ labels in multiple spots specifying that it’s experimental. It could be just us confused early adopters. The only other ask I would have from this thread would be a general update since the “drone goes k8s” blog post 6 months ago: > The Kubernetes Runtime is still considered experiment, however, initial testing has been very positive. There are some known [issues](https://github.com/drone/drone-runtime/issues) and areas of improvement, however, **I expect rapid progress over the coming weeks.** ^ Dec 7th, 2018 - [Drone CI/CD Goes Kubernetes-Native](https://blog.drone.io/drone-goes-kubernetes-native/) Has testing remained very positive? Do we still expect rapid progress? Would an update somewhere help, to show a change in priorities? An update like this could save people from wasting their experimental time. ---
**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\ **Post date:** [June 20, 2019, 11:45pm UTC](https://drone.discourse.group/t/drone-eks-autoscaling-controller-needs-cpu-requests/3600/6 "2019-06-20T23:45:01Z")
> Has testing remained very positive? Do we still expect rapid progress? I do not believe so. The documentation has been updated to recommend against production use while we re-assess. We are tracking various issues related to Kubernetes where we have summarized our concerns, although no final decisions have been made. Some further reading: - [driver: knative / tekton · Issue #65 · drone/drone-runtime · GitHub](https://github.com/drone/drone-runtime/issues/65) - [Kubernetes: Planned Changes / Deprecation / Replacement · Issue #69 · drone/drone-runtime · GitHub](https://github.com/drone/drone-runtime/issues/69) - [kubernetes: allow skipping namespace creation · Issue #53 · drone/drone-runtime · GitHub](https://github.com/drone/drone-runtime/issues/53#issuecomment-482473370) Our current focus is on enabling custom stage definitions [1] and providing a runner framework (conceptually similar to Kubernetes operator framework). This will enable creation of custom runners, and will decouple runners from Drone core. I expect this will lead to a community-driven Kubernetes runtime that supersedes what we have today. I also expect the current Kubernetes runtime to remain active as a community-driven runtime, assuming there is interest in maintaining it despite its faults. [1] [[PROPOSAL] Drone Custom Stage Definitions · Issue #2680 · harness/gitness · GitHub](https://github.com/drone/drone/issues/2680)