STX-O Master failing to: Not possible launch VM due exceeded maximum number of retries

[Jonathan de Carvalho Silva]

Brief Description
--------------------

After the STX-O Master installation on DX, the Sanity tests failed due to an inability to launch the VM caused by exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance

FAIL 20240910 22:05:11 test_swact_controllers
FAIL 20240910 22:09:44 test_ping_between_two_vms[tis-centos-guest-virtio-virtio]
FAIL 20240910 22:14:01 test_migrate_vm[tis-centos-guest-live-None]
FAIL 20240910 22:18:06 test_nova_actions[tis-centos-guest-dedicated-pause-unpause]
FAIL 20240910 22:21:45 test_nova_actions[tis-centos-guest-dedicated-suspend-resume]
FAIL 20240910 22:25:22 test_evacuate_vms

Severity
--------------------

Major: VM does not launch

Steps to Reproduce
--------------------

    Create Flavor
    Create Volume 2GB
    Check keypair
    Launch VM with flavor, volume , keypair and the networks

Expected Behavior
--------------------

VM launch with success.

Actual Behavior
--------------------

Error building server
ERROR (ResourceInErrorState): <Server: tenant1-tis-centos-guest-3>
command terminated with exit code 1

{'code': 500, 'created': '2024-09-10T22:15:37Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e80a5c2f-225e-4420-bacf-04fbac699b74.', 'details': 'Traceback (most recent call last):\n File "/var/lib/openstack/lib/python3.9/site-packages/nova/conductor/manager.py", line 705, in build_instances\n raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance e80a5c2f-225e-4420-bacf-04fbac699b74.\n'}

Reproducibility
--------------------

Reproducible

System Configuration
--------------------

STX-O (DX)

Load info (eg: 2022-03-10_20-00-07)
--------------------

STX-O Master

Timestamp/Logs
--------------------

Running command: nova boot --flavor=e5e208d7-df17-4b75-88e4-3273d883c201 --boot-volume=765dad7c-c163-41d3-95d0-73eeb949b92b --key-name=keypair-tenant1 --poll --nic net-id=3bc6a9d7-9141-4122-9423-e183ca64c38d --nic net-id=142f653a-98b5-4d32-a090-5db3246d0bd2 tenant1-tis-centos-guest-3

nova CLI is deprecated and will be a removed in a future release
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:hostname | tenant1-tis-centos-guest-3 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | CnRFUNkXkSt2 |
| config_drive | |
| created | 2024-09-10T22:15:14Z |
| description | - |
| flavor:disk | 9 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:mem_page_size": "large"} |
| flavor:original_name | live-mig |
| flavor:ram | 1024 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| id | e80a5c2f-225e-4420-bacf-04fbac699b74 |
| image | Attempt to boot from volume - no image supplied |
| key_name | keypair-tenant1 |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | tenant1-tis-centos-guest-3 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 6e0780c8eeaf4fda8fadbe9aa0d3f497 |
| trusted_image_certificates | - |
| updated | 2024-09-10T22:15:14Z |
| user_id | bcde8e8abf81490c9ae596ed4e52e42c |
+--------------------------------------+-------------------------------------------------+

Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Error building server
ERROR (ResourceInErrorState): <Server: tenant1-tis-centos-guest-3>

More details log in attached.

Alarms
--------------------

[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+-------+-----------------------------------------------------------+--------------------------------------+----------+--------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+-----------------------------------------------------------+--------------------------------------+----------+--------------+
| 700. | Instance test.vm2 owned by tenant1 has failed to schedule | tenant=6e0780c8-eeaf-4fda-8fad- | critical | 2024-09-10T1 |
| 001 | | be9aa0d3f497.instance=14c426d8-048a- | | 8:40:12. |
| | | 4ddb-8d6e-42871bc60321 | | 386888 |
| | | | | |
| 700. | Instance test.vm owned by tenant1 has failed to schedule | tenant=6e0780c8-eeaf-4fda-8fad- | critical | 2024-09-10T1 |
| 001 | | be9aa0d3f497.instance=d4ae31ae-c8ee- | | 8:38:19. |
| | | 4b0e-9d80-584ee55867a2 | | 800530 |
| | | | | |
+-------+-----------------------------------------------------------+--------------------------------------+----------+--------------+

Test Activity
--------------------

Sanity

Workaround
--------------------

None

[Daniel Marques Caires (dcaires)]

The problem was not a bug with the STX Openstack application but rather an error in the new lab network infrastructure, in which stx-openstack sanity is running. I will be closing this issue as the problem was resolved.