STX-O Master failing to: Not possible launch VM due volume mapping

[Jonathan de Carvalho Silva]

Brief Description
--------------------

After the STX-O Master installation on DX, the Sanity tests failed due to an inability to launch the VM caused by volume mapping issues

FAIL 20240823 01:13:04 neutron/test_ping_vms.py::test_ping_between_two_vms[tis-centos-guest-virtio-virtio]
FAIL 20240823 01:16:55 nova/test_migrate_vms.py::test_migrate_vm[tis-centos-guest-live-None]
FAIL 20240823 01:18:53 nova/test_nova_actions.py::test_nova_actions[tis-centos-guest-dedicated-pause-unpause]
FAIL 20240823 01:20:25 nova/test_nova_actions.py::test_nova_actions[tis-centos-guest-dedicated-suspend-resume]
FAIL 20240823 01:21:53 test_evacuate.py::TestTisGuest::test_evacuate_vms

Severity
--------------------

Major: VM does not launch

Steps to Reproduce
--------------------

    Create Flavor
    Create Volume 2GB
    Check keypair
    Launch VM with flavor, volume , keypair and the networks

Expected Behavior
--------------------

VM launch with success.

Actual Behavior
--------------------

ERROR (BadRequest): Block Device Mapping is Invalid: failed to get volume bdc7018c-8a4f-409c-ac59-0a3f567af1f0. (HTTP 400) (Request-ID: req-9289d37e-af3f-483b-847d-3c0e9ae2b904)

Reproducibility
--------------------

Reproducible

System Configuration
--------------------

STX-O (DX)

Load info (eg: 2022-03-10_20-00-07)
--------------------

STX-O Master

Last Pass
--------------------

Timestamp/Logs
--------------------

 [sysadmin@controller-1 ~(keystone_admin)]$ openstack flavor list

+--------------------------------------+-------------+-------+------+-----------+-------+-----------+

| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |

+--------------------------------------+-------------+-------+------+-----------+-------+-----------+

| 2426be7e-a40a-4bc1-93e8-23be29d4d1b9 | small | 1024 | 2 | 0 | 1 | True |

| 28b33686-466c-48f3-8225-796eebe1f442 | dedicated | 1024 | 9 | 0 | 1 | True |

| 785ced4c-2ef8-452e-952d-804fdf989f0a | small.float | 1024 | 2 | 0 | 1 | True |

| 93019886-c03f-48e1-94be-02ef92982110 | m1.tiny | 512 | 1 | 0 | 1 | True |

| 9e116dcf-2455-490b-91f7-ef7033246032 | medium.dpdk | 1024 | 2 | 0 | 2 | True |

| b056a105-9e84-4b09-913b-94e7140b0896 | m1.small | 2048 | 20 | 0 | 1 | True |

| b81a881c-4a2c-454c-bfd8-01167058bb3b | m1.large | 8192 | 80 | 0 | 4 | True |

| bd7db92b-be7a-49e0-b28c-00bcaa746815 | m1.medium | 4096 | 40 | 0 | 2 | True |

| fc81475c-6f70-491e-b2bd-6b7142c26f50 | m1.xlarge | 16384 | 160 | 0 | 8 | True |

+--------------------------------------+-------------+-------+------+-----------+-------+-----------+

[sysadmin@controller-1 ~(keystone_admin)]$ openstack volume list

+--------------------------------------+------------------------+-----------+------+-------------+

| ID | Name | Status | Size | Attached to |

+--------------------------------------+------------------------+-----------+------+-------------+

| 88dcc593-52df-4197-8873-9fe26c3bc795 | vol-tis-centos-guest-3 | available | 2 | |

+--------------------------------------+------------------------+-----------+------+-------------+

[sysadmin@controller-1 ~(keystone_admin)]$ openstack keypair list

+-----------------+-------------------------------------------------+------+

| Name | Fingerprint | Type |

+-----------------+-------------------------------------------------+------+

| keypair-tenant1 | b8:86:a6:40:77:98:9a:f8:33:fa:6a:88:59:83:3e:c0 | ssh |

+-----------------+-------------------------------------------------+------+

[sysadmin@controller-1 ~(keystone_admin)]$ openstack network list

+--------------------------------------+------------------+------------------------------------------------------------------------------------------------------------------+

| ID | Name | Subnets |

+--------------------------------------+------------------+------------------------------------------------------------------------------------------------------------------+

| 13a97a7b-9704-4222-9761-99782eb1d873 | tenant1-net1 | 7ae82db3-2b4f-4af9-9471-d160c566f2e1 |

| 231ce6db-301a-47d2-9e5b-1fe5fa1f67b9 | internal0-net0-3 | 97901c34-faf5-4619-bb03-9dcaf4854e17 |

| 2e124eac-d494-4098-8165-d164a3123c3d | internal0-net0 | 7fe65ba5-d462-484f-add7-1cca41c77cb8 |

| 374c41b4-9204-443a-ad67-e30aff83c8e7 | tenant1-mgmt-net | 3a9bd97c-c61e-400e-b133-92502d57bc95, 4dd4e7ef-06bd-4929-9cdd-8d89742ffcc2, e23cec23-4e91-4711-a73f-54ccf5a72de3 |

| 3ac437b6-8082-4db2-90b6-c979de4b5239 | tenant1-net2 | 5d7a76ff-b422-4b28-9bd8-93422d71cf3d |

| 5723a85a-e43b-42a3-a84d-510fd9747542 | internal0-net1 | 58d6ac92-8421-450c-8e8c-e2280a47dc7b |

| 6c707187-39e1-4d58-ba22-8586c8ffa9df | tenant1-net3 | cc7d0871-9dc0-453b-b91b-cd6f63a1ac42 |

| 9df55465-6142-4178-9a6c-c590dc252695 | tenant1-net5 | 067127dd-fab5-43fe-ba09-e0c5535a8020 |

| 9f61d253-3f69-4065-ab12-1192c6d46452 | external-net0 | f37d389a-841f-4657-a515-d748eeeb45d7 |

| a4c0e816-3372-4ea2-a6a7-94a907f329af | internal0-net0-2 | ee8d7cca-bda9-456b-b087-7240a61b6a0f |

| b6e75af4-62e0-4312-b35b-9dfeeba13b9f | tenant1-net4 | 306e23ed-e6d1-4d05-9809-6492c1ce5c58 |

| dad05db7-354b-429c-b8c3-40f478c24e4b | tenant1-net0 | 901b624e-6039-44a3-bb82-d42111d5b2bc |

| e0174f5b-2ad8-414e-8702-16a538cbb0ee | internal0-net1-1 | 3daeb1bf-c894-400b-9cee-cd44ea866284 |

| e50fc362-d350-45ae-a279-005f37a2643f | internal0-net0-1 | 6ed228b6-d8d3-4f3d-ab02-b174f70b1c35 |

+--------------------------------------+------------------+------------------------------------------------------------------------------------------------------------------+

[sysadmin@controller-1 ~(keystone_admin)]$ openstack server create --flavor 28b33686-466c-48f3-8225-796eebe1f442 --volume 88dcc593-52df-4197-8873-9fe26c3bc795 --key-name keypair-tenant1 --nic net-id=374c41b4-9204-443a-ad67-e30aff83c8e7 --nic net-id=13a97a7b-9704-4222-9761-99782eb1d873 --wait tenant1-nova_actions-3

Block Device Mapping is Invalid: failed to get volume 88dcc593-52df-4197-8873-9fe26c3bc795. (HTTP 400) (Request-ID: req-85106faa-edcf-4ff6-bb25-fdfaf9c577b1)

command terminated with exit code 1

Or using nova CLI

.
[2024-08-23 01:19:37,449] 544 DEBUG MainThread ssh.exec_cmd:: Running command: nova - boot --flavor=d1d0b19d-b13b-4306-8239-c72fe83eafeb --boot-volume=bdc7018c-8a4f-409c-ac59-0a3f567af1f0 --key-name=keypair-tenant1 --poll --nic net-id=374c41b4-9204-443a-ad67-e30aff83c8e7 --nic net-id=13a97a7b-9704-4222-9761-99782eb1d873 tenant1-nova_actions-3

[2024-08-23 01:19:39,091] 471 DEBUG MainThread ssh.expect :: Output:
nova CLI is deprecated and will be a removed in a future release
ERROR (BadRequest): Block Device Mapping is Invalid: failed to get volume bdc7018c-8a4f-409c-ac59-0a3f567af1f0. (HTTP 400) (Request-ID: req-9289d37e-af3f-483b-847d-3c0e9ae2b904)
command terminated with exit code 1

Alarms
--------------------

[sysadmin@controller-1 ~(keystone_admin)]$ fm alarm-list
+----------+-------------------------------------------------------------+-----------------------------+----------+-------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+-------------------------------------------------------------+-----------------------------+----------+-------------------+
| 100.103 | Memory threshold exceeded ; threshold 90.00%, actual 90.36% | host=controller-0.memory= | critical | 2024-08-23T11:20: |
| | | platform | | 58.301499 |
| | | | | |
| 100.103 | Memory threshold exceeded ; threshold 90.00%, actual 99.89% | host=controller-1.memory= | critical | 2024-08-23T00:57: |
| | | platform | | 25.608206 |
| | | | | |
+----------+-------------------------------------------------------------+-----------------------------+----------+-------------------+

Test Activity
--------------------

Sanity

Workaround
--------------------

None

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/927632
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/693bf136bd88c9b508a28572d22c0021fa06a52d
Submitter: "Zuul (22348)"
Branch: master

commit 693bf136bd88c9b508a28572d22c0021fa06a52d
Author: Lucas de Ataides <email address hidden>
Date: Fri Aug 30 14:40:54 2024 -0300

    Refactor STX-O to support dynamic region names

    This change updates the STX-Openstack application to handle dynamic
    region names for both standalone and DC systems introduced by [1].
    Previously, the region name "RegionOne" was hardcoded, which is
    incompatible with the new deployment methodology where the region name
    is now a unique UUID generated dynamically during installation.

    This change addresses the issues identified in several launchpads,
    where the hardcoded region name caused compatibility problems in system
    operations and deployment processes. The following updates have been
    made:

    - Updated instances where the region name was hardcoded to dynamically
      retrieve and apply the region name based on the system’s
      configuration.
    - Ensured compatibility with both standalone and subcloud deployments
      by refactoring the application codebase.

    [1] https://review.opendev.org/c/starlingx/stx-puppet/+/914159

    Test Plan:
      - PASS: Build STX-Openstack tarball
      - PASS: Upload / apply STX-Openstack
      - PASS: Create a bootable volume
      - PASS: Launch a VM via volume
      - PASS: Launch a VM via image
      - PASS: Access the Horizon dashboard

    Closes-Bug: 2077551
    Closes-Bug: 2077756

    Change-Id: Ibf6c2fa84d71f31ddae9f12cfe2c076b905aa6e2
    Signed-off-by: Lucas de Ataides <email address hidden>