Charm does not inject `nodeSelector` terms to the `speaker` pods

[Wubin Li]

When using `--constraints` with `tags` for deployment, e.g.,

`juju deploy metallb --channel 1.28/stable --trust --constraints="tags=kubeflowserver=true"`

`nodeSelector` is only injected in the `operator` pod:

```
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: metallb
    uid: 9f937c5d-8a81-4225-a20c-ab2f14421450
  resourceVersion: "78864"
  uid: 0d69bc57-a171-4344-87c4-8fcce63c7f6f
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubeflowserver
            operator: In
            values:
            - "true"
```

for other pods spawned by the charm pod, it does not:
```
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: speaker
    uid: 1ad60a90-1bc4-4406-8fe5-fb993ebcb862
  resourceVersion: "79003"
  uid: d34dfaa9-4d4b-4c46-8ca5-5b149787de39
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - vm-0
  containers:
```

Note: `vm-0` is not the node with the label (`kubeflowserver=true`). In this case, each `speaker` pod has one `nodeAffinity` making sure they can be spread over each node (e.g., `vm0`, `vm-1`, etc).

juju version: 2.9.43
kubernetes: v1.24.17

[Paul Goins (vultaire)]

I tested this locally on microk8s.

Here's a breakdown of what I see:

* charm pod, metallb-0 in the app's model's namespace: Works. I see this as part of the spec:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubeflowserver
            operator: In
            values:
            - "true"

* metallb-system namespace, controller-$HASH-$PODSUFFIX pod: no affinity rules, only a node selector:

  nodeSelector:
    kubernetes.io/os: linux

* metallb-system namespace, speaker-$PODSUFFIX pod: contains a daemonset-generated selector:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - rog2023

(rog2023 is the hostname of my single microk8s node, so this is correct)

* metallb-system namespace, replicaset controller-$HASH: no affinity rules, although it contained the "kubernetes.io/os: linux" nodeSelector nested within the template.

* metallb-system namespace, deployment controller: no affinity rules, contains the above replicaset.

* metallb-system namespace, daemonset speaker: no affinity rules, also contains the contains the "kubernetes.io/os: linux" nodeSelector nested within the template.

For the record: I see that the older metallb-speaker charm does seem to apply the nodeselector to the daemonset, unlike this version. I'm not sure what's changed.

[Paul Goins (vultaire)]

Also, the above was tested today on a Juju 2.9.49 controller.

[Adam Dyess (addyess)]

The old metallb-speaker and metalb-controller charms were "pod-spec" charms -- a pre-cursor to the sidecar kubernetes charm framework.

This "new" metallb charm is not a "juju operator" charm because juju operators do not support DaemonSets. So, think of this metallb charm as if it's a helm chart installer. it has a ball of manifests that are applied to the cluster. You can constrain the CHARM as much as you wish and the pod that the unit is running on will behave appropriately, but in the end -- you're just deploying the metallb manifests with the charm.

If this "constraint" is something that metallb needs to apply, it will likely need to be charm config since the ops framework doesn't give the charm dev access to the application constraints so far as i can tell.

[Adam Dyess (addyess)]

Confirmed from LP#2068076 we should likely make this node-selector charm config