pod-spec-set: can't create custom resources in another namespace

[George Kraft]

When calling pod-spec-set with a customResource that has a namespace in its metadata, the custom resource does not get created. The pod spec looks something like:

kubernetesResources:
  customResourceDefinitions:
    name: network-attachment-definitions.k8s.cni.cncf.io
    spec: ...
  customResources:
    network-attachment-definitions.k8s.cni.cncf.io:
    - apiVersion: k8s.cni.cncf.io/v1
      kind: NetworkAttachmentDefinition
      metadata:
        name: test
        namespace: default
      spec: ...

The controller logs an error:

controller-0: 15:06:31 ERROR juju.worker.caasunitprovisioner creating or updating custom resources: ensuring custom resource "flannel": the namespace of the provided object does not match the namespace sent on the request

[George Kraft (cynerva)]

Attached juju debug-log output with log level at DEBUG. Output includes a full dump of the pod spec.

[George Kraft (cynerva)]

Juju client 2.8.0-focal-amd64 and server 2.8.0

[Ian Booth (wallyworld)]

On k8s, Juju treats namespaces like models. Just as all Juju resources are confined to the model in which they are created, he same applies to k8s namespaces. Otherwise you can have 2 Juju models interfering with each other's resources. We do allow global resources to be created, but where possible we prepend the model; name, again to prevent 2 different deployments from stomping on each other.

[George Kraft (cynerva)]

Is that set in stone? We were hoping to allow users to create NetworkAttachmentDefinitions for Multus though charm config - mainly so they can specify NetworkAttachmentDefinitions in bundles - but if we can't create resources in other namespaces, then that's a pretty serious limitation. NetworkAttachmentDefinitions can only be used by workload pods that are deployed in the same namespace.

If we can't create them in other namespaces, then we'll have to scrap the charm config and instruct users to create them via kubectl instead.

[Pen Gale (pengale)]

Triaging as wishlist for now. It sounds like there is a use case for using namespaces differently. For now, though, it's important for Juju to be able to map namespaces to models.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.