Comment 0 for bug 2054824

Brief Description

Vault-manager is supposed to remove the PersistentVolumeClaim resources after moving key shards to kubernetes secrets. A test of the existence of PVC with warning log was added in case the deletion did not succeed. Immediately after the conversion vault-manager is issuing the warning, perhaps because the PVC deletion is in progress at that time:

2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion

Severity

Minor: in this observation the warning log is a false report

Steps to Reproduce

application-update from vault app before the TCPG-1718, to the latest code

Expected Behavior

The vault manager doesn't report PVC storage deletion failure if the PVC is still being deleted.

Actual Behavior

A warning log is issued, but when you go look you will find the PVC isn't there.

Reproducibility

100% each test when the conversion code runs and the PVC is deleted

System Configuration

any configuration with ceph, and where vault application can be updated to TCPG-1718 feature code.

Load info (eg: 2022-03-10_20-00-07)

master branch developer loads, or 22.12 load patched to unreleased 22.12 code
(we want the fix on starlingx master, with low priority cherry-pick for 22.12)

Last Pass

n/a

Timestamp/Logs

2024-01-11T18-20-40 INFO Switching to use kubectl version v1.24
2024-01-11T18-20-40 INFO Mode is VAULT_MANAGER
2024-01-11T18-20-40 INFO Using secrets provided in cluster-key-bootstrap
2024-01-11T18-20-42 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-20-42 INFO secret "cluster-key-bootstrap" deleted
2024-01-11T18-20-42 INFO Waiting for vault-manager pod to exit
2024-01-11T18-21-14 INFO Waiting for mount-helper pod to run
2024-01-11T18-21-19 INFO Data retrieved from PVC
2024-01-11T18-21-19 INFO Cluster secrets exist: validating
2024-01-11T18-21-20 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-21-20 INFO Shredding of PVC data verified
2024-01-11T18-21-21 INFO persistentvolumeclaim "manager-pvc-sva-vault-manager-0" deleted
2024-01-11T18-21-21 INFO Auto rekey enabled: [true]
2024-01-11T18-21-21 INFO Rekey requested: bd584b85-a62c-47fc-9380-52e7b5a7a5cd
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
2024-01-11T18-21-21 INFO Query server 172-16-166-142 for initialization status
2024-01-11T18-21-21 INFO Vault is initialized
2024-01-11T18-21-21 INFO Checking vault pods seal status in perpetuity...
2024-01-11T18-21-27 INFO Sealed status of 172-16-166-142 is now: false
2024-01-11T18-21-27 INFO Sealed status of 172-16-192-97 is now: false
<snip unrelated log>

Alarms

n/a

Test Activity

developer test of feature on 22.12

Workaround

check the pvc manually and ignore the log:
kubectl get persistentvolumeclaims -n vault | grep manager