| 2024-02-23 17:51:53 |
Tae Park |
description |
Brief Description
Vault-manager is supposed to remove the PersistentVolumeClaim resources after moving key shards to kubernetes secrets. A test of the existence of PVC with warning log was added in case the deletion did not succeed. Immediately after the conversion vault-manager is issuing the warning, perhaps because the PVC deletion is in progress at that time:
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
Severity
Minor: in this observation the warning log is a false report
Steps to Reproduce
application-update from vault app before the TCPG-1718, to the latest code
Expected Behavior
The vault manager doesn't report PVC storage deletion failure if the PVC is still being deleted.
Actual Behavior
A warning log is issued, but when you go look you will find the PVC isn't there.
Reproducibility
100% each test when the conversion code runs and the PVC is deleted
System Configuration
any configuration with ceph, and where vault application can be updated to TCPG-1718 feature code.
Load info (eg: 2022-03-10_20-00-07)
master branch developer loads, or 22.12 load patched to unreleased 22.12 code
(we want the fix on starlingx master, with low priority cherry-pick for 22.12)
Last Pass
n/a
Timestamp/Logs
2024-01-11T18-20-40 INFO Switching to use kubectl version v1.24
2024-01-11T18-20-40 INFO Mode is VAULT_MANAGER
2024-01-11T18-20-40 INFO Using secrets provided in cluster-key-bootstrap
2024-01-11T18-20-42 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-20-42 INFO secret "cluster-key-bootstrap" deleted
2024-01-11T18-20-42 INFO Waiting for vault-manager pod to exit
2024-01-11T18-21-14 INFO Waiting for mount-helper pod to run
2024-01-11T18-21-19 INFO Data retrieved from PVC
2024-01-11T18-21-19 INFO Cluster secrets exist: validating
2024-01-11T18-21-20 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-21-20 INFO Shredding of PVC data verified
2024-01-11T18-21-21 INFO persistentvolumeclaim "manager-pvc-sva-vault-manager-0" deleted
2024-01-11T18-21-21 INFO Auto rekey enabled: [true]
2024-01-11T18-21-21 INFO Rekey requested: bd584b85-a62c-47fc-9380-52e7b5a7a5cd
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
2024-01-11T18-21-21 INFO Query server 172-16-166-142 for initialization status
2024-01-11T18-21-21 INFO Vault is initialized
2024-01-11T18-21-21 INFO Checking vault pods seal status in perpetuity...
2024-01-11T18-21-27 INFO Sealed status of 172-16-166-142 is now: false
2024-01-11T18-21-27 INFO Sealed status of 172-16-192-97 is now: false
<snip unrelated log>
Alarms
n/a
Test Activity
developer test of feature on 22.12
Workaround
check the pvc manually and ignore the log:
kubectl get persistentvolumeclaims -n vault | grep manager |
Brief Description
Vault-manager is supposed to remove the PersistentVolumeClaim resources after moving key shards to kubernetes secrets. A test of the existence of PVC with warning log was added in case the deletion did not succeed. Immediately after the conversion vault-manager is issuing the warning, perhaps because the PVC deletion is in progress at that time:
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
Severity
Minor: in this observation the warning log is a false report
Steps to Reproduce
application-update from vault app before vault PVC conversion functionality, to the latest code
Expected Behavior
The vault manager doesn't report PVC storage deletion failure if the PVC is still being deleted.
Actual Behavior
A warning log is issued, but when you go look you will find the PVC isn't there.
Reproducibility
100% each test when the conversion code runs and the PVC is deleted
System Configuration
any configuration with ceph, and where vault application can be updated to TCPG-1718 feature code.
Load info (eg: 2022-03-10_20-00-07)
master branch developer loads, or 22.12 load patched to unreleased 22.12 code
(we want the fix on starlingx master, with low priority cherry-pick for 22.12)
Last Pass
n/a
Timestamp/Logs
2024-01-11T18-20-40 INFO Switching to use kubectl version v1.24
2024-01-11T18-20-40 INFO Mode is VAULT_MANAGER
2024-01-11T18-20-40 INFO Using secrets provided in cluster-key-bootstrap
2024-01-11T18-20-42 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-20-42 INFO secret "cluster-key-bootstrap" deleted
2024-01-11T18-20-42 INFO Waiting for vault-manager pod to exit
2024-01-11T18-21-14 INFO Waiting for mount-helper pod to run
2024-01-11T18-21-19 INFO Data retrieved from PVC
2024-01-11T18-21-19 INFO Cluster secrets exist: validating
2024-01-11T18-21-20 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-21-20 INFO Shredding of PVC data verified
2024-01-11T18-21-21 INFO persistentvolumeclaim "manager-pvc-sva-vault-manager-0" deleted
2024-01-11T18-21-21 INFO Auto rekey enabled: [true]
2024-01-11T18-21-21 INFO Rekey requested: bd584b85-a62c-47fc-9380-52e7b5a7a5cd
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
2024-01-11T18-21-21 INFO Query server 172-16-166-142 for initialization status
2024-01-11T18-21-21 INFO Vault is initialized
2024-01-11T18-21-21 INFO Checking vault pods seal status in perpetuity...
2024-01-11T18-21-27 INFO Sealed status of 172-16-166-142 is now: false
2024-01-11T18-21-27 INFO Sealed status of 172-16-192-97 is now: false
<snip unrelated log>
Alarms
n/a
Test Activity
developer test of feature on 22.12
Workaround
check the pvc manually and ignore the log:
kubectl get persistentvolumeclaims -n vault | grep manager |
|
| 2024-02-23 17:52:48 |
Tae Park |
description |
Brief Description
Vault-manager is supposed to remove the PersistentVolumeClaim resources after moving key shards to kubernetes secrets. A test of the existence of PVC with warning log was added in case the deletion did not succeed. Immediately after the conversion vault-manager is issuing the warning, perhaps because the PVC deletion is in progress at that time:
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
Severity
Minor: in this observation the warning log is a false report
Steps to Reproduce
application-update from vault app before vault PVC conversion functionality, to the latest code
Expected Behavior
The vault manager doesn't report PVC storage deletion failure if the PVC is still being deleted.
Actual Behavior
A warning log is issued, but when you go look you will find the PVC isn't there.
Reproducibility
100% each test when the conversion code runs and the PVC is deleted
System Configuration
any configuration with ceph, and where vault application can be updated to TCPG-1718 feature code.
Load info (eg: 2022-03-10_20-00-07)
master branch developer loads, or 22.12 load patched to unreleased 22.12 code
(we want the fix on starlingx master, with low priority cherry-pick for 22.12)
Last Pass
n/a
Timestamp/Logs
2024-01-11T18-20-40 INFO Switching to use kubectl version v1.24
2024-01-11T18-20-40 INFO Mode is VAULT_MANAGER
2024-01-11T18-20-40 INFO Using secrets provided in cluster-key-bootstrap
2024-01-11T18-20-42 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-20-42 INFO secret "cluster-key-bootstrap" deleted
2024-01-11T18-20-42 INFO Waiting for vault-manager pod to exit
2024-01-11T18-21-14 INFO Waiting for mount-helper pod to run
2024-01-11T18-21-19 INFO Data retrieved from PVC
2024-01-11T18-21-19 INFO Cluster secrets exist: validating
2024-01-11T18-21-20 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-21-20 INFO Shredding of PVC data verified
2024-01-11T18-21-21 INFO persistentvolumeclaim "manager-pvc-sva-vault-manager-0" deleted
2024-01-11T18-21-21 INFO Auto rekey enabled: [true]
2024-01-11T18-21-21 INFO Rekey requested: bd584b85-a62c-47fc-9380-52e7b5a7a5cd
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
2024-01-11T18-21-21 INFO Query server 172-16-166-142 for initialization status
2024-01-11T18-21-21 INFO Vault is initialized
2024-01-11T18-21-21 INFO Checking vault pods seal status in perpetuity...
2024-01-11T18-21-27 INFO Sealed status of 172-16-166-142 is now: false
2024-01-11T18-21-27 INFO Sealed status of 172-16-192-97 is now: false
<snip unrelated log>
Alarms
n/a
Test Activity
developer test of feature on 22.12
Workaround
check the pvc manually and ignore the log:
kubectl get persistentvolumeclaims -n vault | grep manager |
Brief Description
Vault-manager is supposed to remove the PersistentVolumeClaim resources after moving key shards to kubernetes secrets. A test of the existence of PVC with warning log was added in case the deletion did not succeed. Immediately after the conversion vault-manager is issuing the warning, perhaps because the PVC deletion is in progress at that time:
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
Severity
Minor: in this observation the warning log is a false report
Steps to Reproduce
application-update from vault app before vault PVC conversion functionality, to the latest code
Expected Behavior
The vault manager doesn't report PVC storage deletion failure if the PVC is still being deleted.
Actual Behavior
A warning log is issued, but when you go look you will find the PVC isn't there.
Reproducibility
100% each test when the conversion code runs and the PVC is deleted
System Configuration
any configuration with ceph, and where vault application can be updated to current feature code.
Load info (eg: 2022-03-10_20-00-07)
master branch developer loads
Last Pass
n/a
Timestamp/Logs
2024-01-11T18-20-40 INFO Switching to use kubectl version v1.24
2024-01-11T18-20-40 INFO Mode is VAULT_MANAGER
2024-01-11T18-20-40 INFO Using secrets provided in cluster-key-bootstrap
2024-01-11T18-20-42 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-20-42 INFO secret "cluster-key-bootstrap" deleted
2024-01-11T18-20-42 INFO Waiting for vault-manager pod to exit
2024-01-11T18-21-14 INFO Waiting for mount-helper pod to run
2024-01-11T18-21-19 INFO Data retrieved from PVC
2024-01-11T18-21-19 INFO Cluster secrets exist: validating
2024-01-11T18-21-20 INFO Verified stored secrets are the same as supplied data
2024-01-11T18-21-20 INFO Shredding of PVC data verified
2024-01-11T18-21-21 INFO persistentvolumeclaim "manager-pvc-sva-vault-manager-0" deleted
2024-01-11T18-21-21 INFO Auto rekey enabled: [true]
2024-01-11T18-21-21 INFO Rekey requested: bd584b85-a62c-47fc-9380-52e7b5a7a5cd
2024-01-11T18-21-21 WARNING PVC storage manager-pvc-sva-vault-manager-0 deletion has failed during conversion
2024-01-11T18-21-21 INFO Query server 172-16-166-142 for initialization status
2024-01-11T18-21-21 INFO Vault is initialized
2024-01-11T18-21-21 INFO Checking vault pods seal status in perpetuity...
2024-01-11T18-21-27 INFO Sealed status of 172-16-166-142 is now: false
2024-01-11T18-21-27 INFO Sealed status of 172-16-192-97 is now: false
<snip unrelated log>
Alarms
n/a
Test Activity
developer test of feature
Workaround
check the pvc manually and ignore the log:
kubectl get persistentvolumeclaims -n vault | grep manager |
|
