AIODX upgrade：cgts-vg partition space is decreased on controller-0

[Yuxing]

Brief Description
-----------------
During an upgrade of an AIODX system, the cgts-vg partition size on controller-0 will decrease, resulting in the cgts-vg partition size not equal on both controllers, further, it will result in puppet apply runtime manifest fail during unlocking the controller-0.

Severity
--------
Major

Steps to Reproduce
------------------
1 Upgrade an AIODX system from stx4.0 to stx5.0.
2 Upgrade controller-1, unlock controller-1
3 Swact controller-0, lock it, upgrade controller-0
4 Unlock controller-0

Expected Behavior
------------------
The cgts-vg size on controller-0 equals it on controller-1

Actual Behavior
----------------
The cgts-vg size on controller-1 decreased, puppet run time manifest fail

Reproducibility
---------------
2/2

System Configuration
--------------------
AIODX

Branch/Pull Time/Commit
-----------------------
Master, Jan 6th

Last Pass
---------
NA

Timestamp/Logs
--------------
[sysadmin@controller-1 ~(keystone_admin)]$ system host-lvg-list 1
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| bc24b5a1-6c44-4e34-86fc-33762214007c | cgts-vg | provisioned | wz--n- | 326.937 | 149.125 | 2 | 13 |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
[sysadmin@controller-1 ~(keystone_admin)]$ system host-lvg-list 2
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| 78483839-495c-4afc-b227-603658be2fec | cgts-vg | provisioned | wz--n- | 326.937 | 149.125 | 2 | 13 |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+

[sysadmin@controller-1 ~(keystone_admin)]$ system host-lvg-list 1
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| bc24b5a1-6c44-4e34-86fc-33762214007c | cgts-vg | provisioned | wz--n- | 163.968 | 140.531 | 1 | 2 |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
[sysadmin@controller-1 ~(keystone_admin)]$ system host-lvg-list 2
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| 78483839-495c-4afc-b227-603658be2fec | cgts-vg | provisioned | wz--n- | 326.937 | 149.125 | 2 | 13 |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
[sysadmin@controller-1 ~(keystone_admin)]$ system host-unlock controller-0
+-----------------------+--------------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------------+
| action | none |
| administrative | locked |
| availability | online |
| bm_ip | None |
| bm_type | none |
| bm_username | None |
| boot_device | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
| capabilities | {u'stor_function': u'monitor'} |
| clock_synchronization | ntp |
| config_applied | 21c9d2bc-7c4e-4668-95c7-44dbdf909eb9 |
| config_status | None |
| config_target | 21c9d2bc-7c4e-4668-95c7-44dbdf909eb9 |
| console | ttyS0,115200 |
| created_at | 2021-01-22T18:47:57.961891+00:00 |
| device_image_update | None |
| hostname | controller-0 |
| id | 1 |
| install_output | text |
| install_state | completed |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | provisioned |
| location | {} |
| mgmt_ip | 192.168.204.3 |
| mgmt_mac | 08:00:27:ab:57:80 |
| operational | disabled |
| personality | controller |
| reboot_needed | False |
| reserved | False |
| rootfs_device | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
| serialid | None |
| software_load | 20.12 |
| subfunction_avail | online |
| subfunction_oper | disabled |
| subfunctions | controller,worker |
| task | Unlocking |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2021-01-25T17:25:38.180143+00:00 |
| uptime | 150 |
| uuid | 32e6284e-3b5e-4838-ae0b-95b9fcba30c9 |
| vim_progress_status | services-disabled |
+-----------------------+--------------------------------------------+

[sysadmin@controller-1 ~(keystone_admin)]$ system host-lvg-list 1
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| bc24b5a1-6c44-4e34-86fc-33762214007c | cgts-vg | provisioned | wz--n- | 263.937 | 86.125 | 2 | 13 |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
[sysadmin@controller-1 ~(keystone_admin)]$ system host-lvg-list 2
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+
| 78483839-495c-4afc-b227-603658be2fec | cgts-vg | provisioned | wz--n- | 326.937 | 149.125 | 2 | 13 |
+--------------------------------------+----------+-------------+--------+------------------+------------------+-------------+-------------+

Test Activity
-------------
Developer test

Workaround
----------
Creat a new pv and append it on controller-0

[Ghada Khalil (gkhalil)]

stx.5.0 / medium - issue impacts upgrade framework testing, but there is a workaround

[Ghada Khalil (gkhalil)]

From David Sullivan:
Only seen once in vbox. Can not reproduce. Closing.