As of today the system fs commands, depending on the order, allow the user to run multiple commands in a sequence, even if there isn't enough space left on the host cgts-vg.
The following example shows a scenario where cgts-vg has 16GB available, but it allowed increasing docker host-fs in an AIO-SX deployment from 30 to 80GB (+50GB) in five chunks of 10GB:
In the end, only the first docker fs increment happened in fact, but the database remained with an inconsistent value of 80GB, which would cause an upgrade failure later, as observed when the bug was reported.
This fs increase is allowed because the cgts-vg is updated by the agent audit, which happens every 60s, so if another host-fs or controllerfs resize is executed before the agent reports back to conductor, it is allowed to run the resize command, the database is updated, but the resize may not be executed with success on the target host because of lacking vg space.
As of today the system fs commands, depending on the order, allow the user to run multiple commands in a sequence, even if there isn't enough space left on the host cgts-vg.
The following example shows a scenario where cgts-vg has 16GB available, but it allowed increasing docker host-fs in an AIO-SX deployment from 30 to 80GB (+50GB) in five chunks of 10GB:
[sysadmin@ controller- 0 ~(keystone_admin)]$ s controller- 0 ~(keystone_admin)]$ SIZE=30 controller- 0 ~(keystone_admin)]$ s host-lvg-list controller-0 ------- ------- ------- ------- ----+-- ------- ---+--- ------- ---+--- -----+- ------- ------- ---+--- ------- ------- -+----- ------- -+----- ------- -+ ------- ------- ------- ------- ----+-- ------- ---+--- ------- ---+--- -----+- ------- ------- ---+--- ------- ------- -+----- ------- -+----- ------- -+ ea19-415a- a0d0-7efcab837a ce | cgts-vg | provisioned | wz--n- | 178.968 | 16.156 | 1 | 12 | e8ef-4753- 8937-575e3ef4f1 6e | nova-local | provisioned | wz--n- | 10.996 | 0.0 | 1 | 1 | ------- ------- ------- ------- ----+-- ------- ---+--- ------- ---+--- -----+- ------- ------- ---+--- ------- ------- -+----- ------- -+----- ------- -+ controller- 0 ~(keystone_admin)]$ for I in $(seq 5); do date; s host-fs-modify controller-0 docker=$(( SIZE + (10 * I) )); sleep 3; done ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ e3d7-42d9- a441-2be9499aad b6 | backup | 25 | backup-lv | 1489-4fb5- 9119-cd588b3f06 49 | docker | 40 | docker-lv | b4af-48de- bfbf-be856e1c27 d7 | kubelet | 10 | kubelet-lv | 2135-48ff- bf01-da94182112 1e | scratch | 16 | scratch-lv | ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ e3d7-42d9- a441-2be9499aad b6 | backup | 25 | backup-lv | 1489-4fb5- 9119-cd588b3f06 49 | docker | 50 | docker-lv | b4af-48de- bfbf-be856e1c27 d7 | kubelet | 10 | kubelet-lv | 2135-48ff- bf01-da94182112 1e | scratch | 16 | scratch-lv | ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ e3d7-42d9- a441-2be9499aad b6 | backup | 25 | backup-lv | 1489-4fb5- 9119-cd588b3f06 49 | docker | 60 | docker-lv | b4af-48de- bfbf-be856e1c27 d7 | kubelet | 10 | kubelet-lv | 2135-48ff- bf01-da94182112 1e | scratch | 16 | scratch-lv | ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ e3d7-42d9- a441-2be9499aad b6 | backup | 25 | backup-lv | 1489-4fb5- 9119-cd588b3f06 49 | docker | 70 | docker-lv | b4af-48de- bfbf-be856e1c27 d7 | kubelet | 10 | kubelet-lv | 2135-48ff- bf01-da94182112 1e | scratch | 16 | scratch-lv | ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ e3d7-42d9- a441-2be9499aad b6 | backup | 25 | backup-lv | 1489-4fb5- 9119-cd588b3f06 49 | docker | 80 | docker-lv | b4af-48de- bfbf-be856e1c27 d7 | kubelet | 10 | kubelet-lv | 2135-48ff- bf01-da94182112 1e | scratch | 16 | scratch-lv | ------- ------- ------- ------- ----+-- ------- +------ ------- +------ ------- ---+ controller- 0 ~(keystone_admin)]$ s host-lvg-list controller-0 ------- ------- ------- ------- ----+-- ------- ---+--- ------- ---+--- -----+- ------- ----+-- ------- ------- --+---- ------- --+---- ------- --+ ------- ------- ------- ------- ----+-- ------- ---+--- ------- ---+--- -----+- ------- ----+-- ------- ------- --+---- ------- --+---- ------- --+ ea19-415a- a0d0-7efcab837a ce | cgts-vg | provisioned | wz--n- | 178.968 | 6.156 | 1 | 12 | e8ef-4753- 8937-575e3ef4f1 6e | nova-local | provisioned | wz--n- | 10.996 | 0.0 | 1 | 1 | ------- ------- ------- ------- ----+-- ------- ---+--- ------- ---+--- -----+- ------- ----+-- ------- ------- --+---- ------- --+---- ------- --+
[sysadmin@
s [sysadmin@
+------
| UUID | LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+------
| 14665345-
| 504332be-
+------
[sysadmin@
Qua Abr 6 20:13:11 UTC 2022
+------
| UUID | FS Name | Size in GiB | Logical Volume |
+------
| 535e4841-
| 38d2bf0a-
| 2ad5d50c-
| f536231f-
+------
Qua Abr 6 20:13:19 UTC 2022
+------
| UUID | FS Name | Size in GiB | Logical Volume |
+------
| 535e4841-
| 38d2bf0a-
| 2ad5d50c-
| f536231f-
+------
Qua Abr 6 20:13:26 UTC 2022
+------
| UUID | FS Name | Size in GiB | Logical Volume |
+------
| 535e4841-
| 38d2bf0a-
| 2ad5d50c-
| f536231f-
+------
Qua Abr 6 20:13:34 UTC 2022
+------
| UUID | FS Name | Size in GiB | Logical Volume |
+------
| 535e4841-
| 38d2bf0a-
| 2ad5d50c-
| f536231f-
+------
Qua Abr 6 20:13:42 UTC 2022
+------
| UUID | FS Name | Size in GiB | Logical Volume |
+------
| 535e4841-
| 38d2bf0a-
| 2ad5d50c-
| f536231f-
+------
[sysadmin@
+------
| UUID | LVG Name | State | Access | Total Size | Avail Size (GiB) | Current PVs | Current LVs |
| | | | | (GiB) | | | |
+------
| 14665345-
| 504332be-
+------
In the end, only the first docker fs increment happened in fact, but the database remained with an inconsistent value of 80GB, which would cause an upgrade failure later, as observed when the bug was reported.
This fs increase is allowed because the cgts-vg is updated by the agent audit, which happens every 60s, so if another host-fs or controllerfs resize is executed before the agent reports back to conductor, it is allowed to run the resize command, the database is updated, but the resize may not be executed with success on the target host because of lacking vg space.