vault-manager reports PVC not deleted immediately after conversion

[Tae Park]

Brief Description

Vault-manager is supposed to remove the PersistentVolumeClaim resources after moving key shards to kubernetes secrets. A test of the existence of PVC with warning log was added in case the deletion did not succeed. Immediately after the conversion vault-manager is issuing the warning, perhaps because the PVC deletion is in progress at that time:

2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion

Severity

Minor: in this observation the warning log is a false report

Steps to Reproduce

application-update from vault app before vault PVC conversion functionality, to the latest code

Expected Behavior

The vault manager doesn't report PVC storage deletion failure if the PVC is still being deleted.

Actual Behavior

A warning log is issued, but when you go look you will find the PVC isn't there.

Reproducibility

100% each test when the conversion code runs and the PVC is deleted

System Configuration

any configuration with ceph, and where vault application can be updated to current feature code.

Load info (eg: 2022-03-10_20-00-07)

master branch developer loads

Last Pass

n/a

Timestamp/Logs

2024-01-11T18-20-40 INFO Switching to use kubectl version v1.24
2024-01-11T18-20-40 INFO Mode is VAULT_MANAGER
2024-01-11T18-20-40 INFO Using secrets provided in cluster-key-bootstrap
2024-01-11T18-20-42 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-20-42 INFO secret "cluster-key-bootstrap" deleted
2024-01-11T18-20-42 INFO Waiting for vault-manager pod to exit
2024-01-11T18-21-14 INFO Waiting for mount-helper pod to run
2024-01-11T18-21-19 INFO Data retrieved from PVC
2024-01-11T18-21-19 INFO Cluster secrets exist: validating
2024-01-11T18-21-20 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-21-20 INFO Shredding of PVC data verified
2024-01-11T18-21-21 INFO persistentvolumeclaim "manager-pvc-sva-vault-manager-0" deleted
2024-01-11T18-21-21 INFO Auto rekey enabled: [true]
2024-01-11T18-21-21 INFO Rekey requested: bd584b85-a62c-47fc-9380-52e7b5a7a5cd
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
2024-01-11T18-21-21 INFO Query server 172-16-166-142 for initialization status
2024-01-11T18-21-21 INFO Vault is initialized
2024-01-11T18-21-21 INFO Checking vault pods seal status in perpetuity...
2024-01-11T18-21-27 INFO Sealed status of 172-16-166-142 is now: false
2024-01-11T18-21-27 INFO Sealed status of 172-16-192-97 is now: false
<snip unrelated log>

Alarms

n/a

Test Activity

developer test of feature

Workaround

check the pvc manually and ignore the log:
kubectl get persistentvolumeclaims -n vault | grep manager

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/vault-armada-app/+/910002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/vault-armada-app/+/910002
Committed: https://opendev.org/starlingx/vault-armada-app/commit/05ccd6fea52da96968134eda00ebe51debd7082a
Submitter: "Zuul (22348)"
Branch: master

commit 05ccd6fea52da96968134eda00ebe51debd7082a
Author: Tae Park <email address hidden>
Date: Thu Feb 22 23:43:41 2024 -0500

    Remove warning log for PVC currently terminating

    Adding an extra check in the post-convert PVC existence check. The old
    vault manager pod may exist beyond the set wait time in the conversion,
    preventing the PVC from finishing termination. This is intended
    behaviour, so a separate debug log indicating such is issued instead.
    Includes a 5 second wait time after PVC conversion is completed, so that
    the PVC termination process is started before verification

    Test Plan:
    PASS Bashate
    PASS AIO-SX vault sanity
    PASS During application update, the debug log is seen instead of the
    warning log if the PVC has status "Terminating"
    PASS No log is reported, if the PVC is correctly deleted before the
    verification

    Closes-bug: 2054824

    Change-Id: Ib9cd45a93550d22dee9d45b5994e89ea2191849a
    Signed-off-by: Tae Park <email address hidden>