openstack integrator ignore-volume-az not working

[Patrizio Bassi]

Hi,

I have a Openstack env with 3 Nova AZs and 1 Cinder AZ (nova).

i set the flag to ignore this topology

juju config openstack-integrator ignore-volume-az
true

according to the documentation this should be enough but all the claims are failing

kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cdk-cinder (default) cinder.csi.openstack.org Delete Immediate false 41m

kubectl describe sc
Name: cdk-cinder
IsDefaultClass: Yes
Annotations: juju.io/workload-storage=true,kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"juju.io/workload-storage":"true"},"labels":{"cdk-addons":"true"},"name":"cdk-cinder"},"provisioner":"cinder.csi.openstack.org"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner: cinder.csi.openstack.org
Parameters: <none>
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>

kubectl describe pvc
Name: testclaim
Namespace: default
StorageClass: cdk-cinder
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
               volume.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal ExternalProvisioning 16s (x2 over 22s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
  Normal Provisioning 9s (x5 over 23s) cinder.csi.openstack.org_csi-cinder-controllerplugin-0_0dd65618-f67c-46f2-96ca-470ca803c248 External provisioner is provisioning volume for claim "default/testclaim"
  Warning ProvisioningFailed 9s (x5 over 22s) cinder.csi.openstack.org_csi-cinder-controllerplugin-0_0dd65618-f67c-46f2-96ca-470ca803c248 failed to provision volume with StorageClass "cdk-cinder": rpc error: code = Internal desc = CreateVolume failed with error Bad request with: [POST https://openstack.mydomain.com:13776/v3/0c2409af0cd849f887e79dee12af751e/volumes], error message: {"badRequest": {"code": 400, "message": "Availability zone 'zone1' is invalid."}}

infact i can see all k8s nodes have topology.cinder.csi.openstack.org/zone=zone[1,2,3] labels

i can play with nodes label but when adding/removing/modifying k8s-workers via juju labels are not kept

kubectl label node juju-906979-k8s-10 topology.cinder.csi.openstack.org/zone=nova --overwrite

it looks like something already reported in https://github.com/kubernetes/cloud-provider-openstack/issues/1300

i'm using charm ~containers/openstack-integrator-204, can you please check if the fix is properly included?

[Patrizio Bassi (patrizio-bassi)]

Hi, extra comment
that charm pulled
openstack-cloud-controller-manager:v1.22.0

I just saw upstream has v1.23 and v1.23.1, thus release notes do not mention anything related to AZ.

[George Kraft (cynerva)]

> it looks like something already reported in https://github.com/kubernetes/cloud-provider-openstack/issues/1300

Looks like this issue has been fixed since since the cloud-provider-openstack 1.20 release[1] which is included in cdk-addons 1.21 and later[2].

One thing I noticed in the linked issue is that their StorageClass has the "availability" parameter[3] set to their volume AZ. The default cdk-cinder StorageClass does not include this and there's no way to set it currently. I suspect that's the missing piece.

You might be able to work around this by creating a new StorageClass that has the availability parameter set to nova:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cdk-cinder-nova
  annotations:
    juju.io/workload-storage: "true"
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
parameters:
  availability: nova

[1]: https://github.com/kubernetes/cloud-provider-openstack/releases/tag/v1.20.0
[2]: https://github.com/charmed-kubernetes/cdk-addons/pull/202/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R14
[3]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#supported-parameters

[George Kraft (cynerva)]

Can you please also check that ignore-volume-az is set correctly in the cloud-config secret?

kubectl get secrets -n kube-system cloud-config -o jsonpath='{.data.cloud\.conf}' | base64 -d

You can also try restarting cinder-csi to ensure that it is using the latest cloud-config data:

kubectl rollout restart -n kube-system statefulset/csi-cinder-controllerplugin
kubectl rollout restart -n kube-system daemonset/csi-cinder-nodeplugin

[Brian Holmes (holmesb5)]

Many thanks for this guidance, @cynerva. I'm not the original bug submitter, but creating a new StorageClass as you've suggested yielded a major improvement and allowed my environment (three compute AZs, single 'nova' volume AZ, ignore-volume-az = true) to properly bind a Cinder PVC and then schedule a pod using that PVC successfully for the first time.

Do you happen to know why explicitly setting parameters.availability = 'nova' has this effect when the linked plugin documentation[3] states that 'nova' should be the default value?

Also, the plugin documentation for the topology feature[4] talks about allowedTopologies replacing parameters.availability -- is it expected that both approaches would work equally well? I was previously experimenting with the former option (based in part on the comments I found in a very similar OpenShift bug[5]) and found that this allowed my PVCs to bind but left associated pods unschedulable:

$ cat test.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    juju.io/workload-storage: "true"
  labels:
    cdk-addons: "true"
  name: cdk-cinder
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - nova
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: cdk-cinder
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  volumes:
  - name: test-pvc
    persistentVolumeClaim:
      claimName: test-pvc
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
      name: "http-server"
    volumeMounts:
    - mountPath: "/usr/share/nginx/html"
      name: test-pvc
$ kubectl apply -f test.yaml
storageclass.storage.k8s.io/cdk-cinder configured
persistentvolumeclaim/test-pvc created
pod/test-pod created
$ kubectl get pvc/test-pvc pod/test-pod -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/test-pvc Bound pvc-128e7fde-e488-4faf-88f4-591028c1eb36 1Gi RWO cdk-cinder 2m56s Filesystem

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/test-pod 0/1 Pending 0 2m47s <none> <none> <none> <none>
$ kubectl describe pod/test-pod
Name: test-pod
Namespace: default
Priority: 0
Node: <none>
Labels: <none>
Annotations: kubernetes.io/psp: privileged
Status: Pending
IP:
IPs: <none>
Containers:
  nginx:
    Image: nginx
    Port: 80/TCP
    Host Port: 0/TCP
    Environment: <none>
    Mounts:
      /usr/share/nginx/html from test-pvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-szgrp (ro)
Conditions:
  Type Status
  PodScheduled False
Volumes:
  test-pvc:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeC...

[George Kraft (cynerva)]

> Do you happen to know why explicitly setting parameters.availability = 'nova' has this effect when the linked plugin documentation states that 'nova' should be the default value?

I think it's just poorly documented. In the CSI cinder code that handles this[1] I can't find any reference to the "nova" default. It seems like the default behavior is automatic selection based on topology requirements.

> Also, the plugin documentation for the topology feature talks about allowedTopologies replacing parameters.availability -- is it expected that both approaches would work equally well?

I'm not really familiar with the topology feature but it looks like this would require the nodes to be labeled with topology.cinder.csi.openstack.org/zone=nova. This label comes from csi-cinder-nodeplugin[2] but the value is always equal to the node's AZ. I don't see any config options to change this behavior.

So no, as it is now, I don't think the allowedTopologies approach will work for this case. But I might consider it a bug in csi-cinder that it doesn't give you a way to set topology.cinder.csi.openstack.org/zone correctly on the nodes.

[1]: https://github.com/kubernetes/cloud-provider-openstack/blob/6a574baca1dd2d2eee1b69084f245f5bcea51e31/pkg/csi/cinder/controllerserver.go#L75-L84
[2]: https://github.com/kubernetes/cloud-provider-openstack/blob/6a574baca1dd2d2eee1b69084f245f5bcea51e31/pkg/csi/cinder/nodeserver.go#L474-L478

[Patrizio Bassi (patrizio-bassi)]

Hi all,
i can confirm allowedTopologies doesn't work, while, creating a new SC as @cynerva suggested in comment #2 works without having to deal with workers labels.

[Patrizio Bassi (patrizio-bassi)]

As the cloud controller keeps on recreating the cdk-cinder SC i wanted to patch it and keep it as default class in order to have a single, clean, sc for users.

unfortunately i cannot

kubectl patch storageclass cdk-cinder -p '{"parameters": {"availability":"nova"}}'
The StorageClass "cdk-cinder" is invalid: parameters: Forbidden: updates to parameters are forbidden.

is there a wait the inject such a config (or stop the re-creation?)

[George Kraft (cynerva)]

Unfortunately, there's currently no way to to inject configuration into the cdk-cinder StorageClass, and no way to stop cdk-addons from recreating it. That would require code changes to add new config options to cdk-addons, and changes to kubernetes-control-plane to expose those options to users.

We're aiming to deprecate cdk-addons, so I suspect the long-term solution will be for us to break csi-cinder out into a dedicated charm and add config options for the StorageClass there.

[Patrizio Bassi (patrizio-bassi)]

Hi any news here?

i saw some changes in the cdk-addons repos but i still fail to set the parameters after the cloud-control created the sc

[George Kraft (cynerva)]

I'm working on adding a config option to kubernetes-control-plane and cdk-addons to make this configurable.

[George Kraft (cynerva)]

PRs:
https://github.com/charmed-kubernetes/cdk-addons/pull/225
https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/289

These add a config option to kubernetes-control-plane to make the Cinder AZ configurable. For example:

juju config kubernetes-control-plane cinder-availability-zone=nova

[George Kraft (cynerva)]

Cherry-picked to release branches:

cdk-addons release-1.27: https://github.com/charmed-kubernetes/cdk-addons/commit/b6548165911e094d0db8f92a89fd522158e3694c
cdk-addons release-1.26: https://github.com/charmed-kubernetes/cdk-addons/commit/ab82ab3cdbc0d859dacda5d350867b9e26aece90
cdk-addons release-1.25: https://github.com/charmed-kubernetes/cdk-addons/commit/741e03d754d004a5a285d366e85a22841b7818d1
kubernetes-control-plane release_1.27: https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/commit/54adcca0dd8e34a7b108d9cd1aaa44e5d606684a

[Alejandro Santoyo Gonzalez (al3jandrosg)]

any updates on this? has this been fixed in 1.28?

[Adam Dyess (addyess)]

It seems this was resolved in 1.28 releases of the charms and beyond
https://charmhub.io/kubernetes-control-plane/configure?channel=1.28/stable#cinder-availability-zone

[Alan Baghumian (alanbach)]

Confirmed it is working with 1.28 after setting a proper volume AZ:

$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-b80925df-ed6d-42a0-a619-b9a00ac12429 20Gi RWO Delete Bound controller-k8s-29-controller/storage-controller-0 cdk-cinder 10m