Kubernetes runner stuck in running state / bringing the Kubernetes runner to GA

We have ran into an issue, this has been noticed after upgrading our kubernetes cluster from v1.18.18 to v1.19.12

This occurred when on runner version: v1.0.0-beta.6
and remained after upgrading to version: v1.0.0-beta.9

Error:

default: Pod "drone-rygfen8qb2yve8jmcxnh" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations) core.PodSpec{ ... // 10 identical fields AutomountServiceAccountToken: nil, NodeName: "ip-10-250-6-105.eu-west-2.compute.internal", SecurityContext: &core.PodSecurityContext{ ... // 11 identical fields
default – clone: Error

I can not see anything in the changelog from kubernetes that would suggest this issue with a new immutable field. Would you be able to advise? @marko-gacesa

Hi, @jim5252. Could you provide more details about this issue? Does this happen every time or it has happened only once? How does your YAML file look like? Do you use advanced Kubernetes features like secrets or tolerations?

I’m asking because the error comes from the Kubernetes and not the runner. The runner changes only the image of a container, nothing else. So, the error text sounds strange.

@jim5252 can you create a new thread for this discussion? In addition to the questions that Marko asked, could you please provide some additional detail in the new thread (we ask this of everyone, and it generally represents the minimum set of information we need to help triage).

  1. Enable debug and trace logging
  2. Publish your runner logs to a gist and provide a link
  3. Publish your yaml to a gist and provide a link (bonus points for providing a simplified yaml that we can use to easily reproduce)
  4. Publish your cluster details to a gist and provide a link; provide whatever information it takes for us to re-create your cluster in our environment

The reason the cluster details are important is because not all flavors of Kubernetes are created equal. For example, we have seen issues / regressions where openshift errors when attempting to update annotations. https://bugzilla.redhat.com/show_bug.cgi?id=1383408

Hi Both, thanks for you replies. I have created a new thread at: Immutable Field Change Error on Pod After Kubernetes v1.19.12 Upgrade