Juju injecting unwanted metadata in Resources deployed by K8s Charm

[Dominik Fleischmann]

We encountered this issue during the development of a charm for Rancher (https://github.com/DomFleischmann/rancher-operator).

This charm deploys the rancher-server service which will afterwards deploy several other Kubernetes Resources through Helm.

The unexpected part is that the Resources deployed through Helm also contain labels like this: "app.juju.is/created-by":"rancher".

This seems to be causing issues in the rancher-server pod which gives the following error message:

2021/01/05 14:01:21 [ERROR] failed to sync schemas: .authorization.k8s.io "" is invalid: metadata: Invalid value: v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.juju.is/created-by":"rancher"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"rancher", Operation:"Update", APIVersion:"authorization.k8s.io/v1", Time:(*v1.Time)(0xc01c0f2360), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc01c0f2380)}}}: must be empty

There should be a possibility of avoiding these labels being added.

To reproduce this follow the deployment instructions of the charm linked above. (Use a disposable cluster, as it will fill it with a lot of not easily removable resources). Afterwards after some minutes check the logs of the workload pod of the rancher charm where you will encounter the above mentioned error messages.

[Thomas Miller (tlmiller)]

Hey Dominik,

This is our model operator service adding these labels so we can make sure that Juju cleans up after itself correctly. On face value this doesn't appear as a Juju bug to me as we should be free to add labels to help us manage. For example's our labels are scoped off to our domain.

The rancher error doesn't 100% indicate the labels are the problem as well. In any event I am very curious to dig into this one and work on a solution with you whatever that may look like.

I'll do some testing on my end to get the lay of the land.

Cheers
tlm

[Dominik Fleischmann (dominik.f)]

Does this mean juju should be able to remove all the resources that contain these kind of labels? Because that is also not the case with this charm and might require an additional bug.

[Dominik Fleischmann (dominik.f)]

I'm hitting the same error in a different charm that I'm developing. Did you arrive at any conclusions during Triage? Or maybe a possible workaround?

[Thomas Miller (tlmiller)]

Hey Dominik,

Thanks for the latest update. I have spent some time looking into this today. Correctly we are adding our label to the resources created by Rancher and Rancher is doing a diff of these objects against an empty meta data struct.

I disabled our model admission controller to see how Rancher behaved and it bypassed that error onto another one related to Helm which is fine.

Before committing to any changes in Juju we would like to understand the why of Ranchers opinion that the label we are adding is causing it to fail. I had a look through their code base and I couldn't find any obvious reason for the error message as to it's purpose. We believe that at the moment Juju is operating correctly and performing it's job well as a good Kubernetes citizen.

Before we potentially make changes to Juju that could just adversely affect another aspect of Juju I am going to reach out Rancher to try and understand the why behind the decision.

Would you possibly be able to shed any light on the why and provide an example of the other case you have seen so that if we do need to change Juju's behavior we have all the information available to make an informed design decision?

Cheers
tlm

[Dominik Fleischmann (dominik.f)]

Thanks for that detailed response Thomas!

This issue has reocurred to me while updating the argo-controller charm from version 2.3.0 to 2.12.8.

To reproduce the issue you can deploy the charms that are located in the following branch: https://github.com/DomFleischmann/argo-operators/tree/2.12.8-reactive

With this being a completely different application from rancher I suspect that this might be caused by some Kubernetes library or API.

The upstream code of the argo controller is in this repository: https://github.com/argoproj/argo-workflows

[Thomas Miller (tlmiller)]

Hey Dominik,

Thanks for hanging in there on this one. Took a little bit to fully get to the bottom of what is happening. Happy to tell you this is a problem in Juju that we have fixed. Specifically we where adding labels to SelfSubjectAccessReview objects and Kubernetes doesn't want that.

PR: https://github.com/juju/juju/pull/12664 for this will be out in Juju 2.9 and 2.8.10 if we release that.

Cheers
tlm

[Dominik Fleischmann (dominik.f)]

That's great news! Thanks for working on this Thomas!