| 2022-03-16 18:32:20 |
Gabriel Silva Trevisan |
bug |
|
|
added bug |
| 2022-03-16 18:35:06 |
Gabriel Silva Trevisan |
description |
Brief Description
-----------------
While performing the unlock_simplex step during a subcloud upgrade, the subcloud platform endpoints became unavailable, causing the step to fail.
Retrying the requests on the host unlock step may be required to work around the flakiness of platform services post data migration.
Severity
--------
Major
Steps to Reproduce
------------------
Prepare a distributed cloud system for orchestrated subcloud upgrade
Create and apply an upgrade strategy for a large number of simplex subclouds in parallel
Expected Behavior
------------------
The upgrade strategy completes successfully.
Actual Behavior
----------------
Upgrade failed for one subcloud at the unlocking-host step as it was unable to retrieve host data from the subcloud.
Reproducibility
---------------
Intermittent. Seems to occur in the very first attempt to upgrade a large number of subclouds in parallel.
System Configuration
--------------------
Distributed cloud
Branch/Pull Time/Commit
-----------------------
2022-01-19 18:42:01
Last Pass
---------
Not sure if this test was previously performed for the stx/6.0 load.
Timestamp/Logs
--------------
2022-03-10 21:10:16.655 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Host: controller-0 is now: unlocked enabled
2022-03-10 21:10:16.667 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Data migration completed.
2022-03-10 21:10:17.167 3236316 ERROR dccommon.drivers.openstack.sdk_platform [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] keystone_client region subcloud2008 error: Service Unavailable (HTTP 503): ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.170 3236316 WARNING dcmanager.orchestrator.states.base [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] Failure initializing KeystoneClient for region: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] (upgrade) Failed! Stage: 1, State: unlocking controller-0, Subcloud: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread Traceback (most recent call last):
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/orch_thread.py", line 512, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread next_state = state_operator.perform_state_action(strategy_step)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/unlock_host.py", line 63, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread sysinv_client = self.get_sysinv_client(strategy_step.subcloud.name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 107, in get_sysinv_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread keystone_client = self.get_keystone_client(region_name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 96, in get_keystone_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread region_clients=['sysinv'])
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sdk_platform.py", line 101, in _init_
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread raise exception
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread
Test Activity
-------------
Feature Testing
Workaround
Delete the failed strategy, create a new one and reapply |
Brief Description
-----------------
While performing the unlock_simplex step during a subcloud upgrade, the subcloud platform endpoints became unavailable, causing the step to fail.
Retrying the requests on the host unlock step may be required to work around the flakiness of platform services post data migration.
Severity
--------
Major
Steps to Reproduce
------------------
Prepare a distributed cloud system for orchestrated subcloud upgrade
Create and apply an upgrade strategy for a large number of simplex subclouds in parallel
Expected Behavior
------------------
The upgrade strategy completes successfully.
Actual Behavior
----------------
Upgrade failed for one subcloud at the unlocking-host step as it was unable to retrieve host data from the subcloud.
Reproducibility
---------------
Seen once. Seems to occur in the very first attempt to upgrade a large number of subclouds in parallel.
System Configuration
--------------------
Distributed cloud
Branch/Pull Time/Commit
-----------------------
2022-01-19 18:42:01
Last Pass
---------
Not sure if this test was previously performed for the stx/6.0 load.
Timestamp/Logs
--------------
2022-03-10 21:10:16.655 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Host: controller-0 is now: unlocked enabled
2022-03-10 21:10:16.667 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Data migration completed.
2022-03-10 21:10:17.167 3236316 ERROR dccommon.drivers.openstack.sdk_platform [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] keystone_client region subcloud2008 error: Service Unavailable (HTTP 503): ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.170 3236316 WARNING dcmanager.orchestrator.states.base [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] Failure initializing KeystoneClient for region: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] (upgrade) Failed! Stage: 1, State: unlocking controller-0, Subcloud: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread Traceback (most recent call last):
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/orch_thread.py", line 512, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread next_state = state_operator.perform_state_action(strategy_step)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/unlock_host.py", line 63, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread sysinv_client = self.get_sysinv_client(strategy_step.subcloud.name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 107, in get_sysinv_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread keystone_client = self.get_keystone_client(region_name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 96, in get_keystone_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread region_clients=['sysinv'])
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sdk_platform.py", line 101, in _init_
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread raise exception
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread
Test Activity
-------------
Feature Testing
Workaround
Delete the failed strategy, create a new one and reapply |
|
| 2022-03-16 18:36:24 |
Gabriel Silva Trevisan |
description |
Brief Description
-----------------
While performing the unlock_simplex step during a subcloud upgrade, the subcloud platform endpoints became unavailable, causing the step to fail.
Retrying the requests on the host unlock step may be required to work around the flakiness of platform services post data migration.
Severity
--------
Major
Steps to Reproduce
------------------
Prepare a distributed cloud system for orchestrated subcloud upgrade
Create and apply an upgrade strategy for a large number of simplex subclouds in parallel
Expected Behavior
------------------
The upgrade strategy completes successfully.
Actual Behavior
----------------
Upgrade failed for one subcloud at the unlocking-host step as it was unable to retrieve host data from the subcloud.
Reproducibility
---------------
Seen once. Seems to occur in the very first attempt to upgrade a large number of subclouds in parallel.
System Configuration
--------------------
Distributed cloud
Branch/Pull Time/Commit
-----------------------
2022-01-19 18:42:01
Last Pass
---------
Not sure if this test was previously performed for the stx/6.0 load.
Timestamp/Logs
--------------
2022-03-10 21:10:16.655 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Host: controller-0 is now: unlocked enabled
2022-03-10 21:10:16.667 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Data migration completed.
2022-03-10 21:10:17.167 3236316 ERROR dccommon.drivers.openstack.sdk_platform [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] keystone_client region subcloud2008 error: Service Unavailable (HTTP 503): ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.170 3236316 WARNING dcmanager.orchestrator.states.base [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] Failure initializing KeystoneClient for region: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] (upgrade) Failed! Stage: 1, State: unlocking controller-0, Subcloud: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread Traceback (most recent call last):
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/orch_thread.py", line 512, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread next_state = state_operator.perform_state_action(strategy_step)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/unlock_host.py", line 63, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread sysinv_client = self.get_sysinv_client(strategy_step.subcloud.name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 107, in get_sysinv_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread keystone_client = self.get_keystone_client(region_name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 96, in get_keystone_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread region_clients=['sysinv'])
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sdk_platform.py", line 101, in _init_
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread raise exception
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread
Test Activity
-------------
Feature Testing
Workaround
Delete the failed strategy, create a new one and reapply |
Brief Description
-----------------
While performing the unlock_simplex step during a subcloud upgrade, the subcloud platform endpoints became unavailable, causing the step to fail.
Retrying the requests on the host unlock step may be required to work around the flakiness of platform services post data migration.
Severity
--------
Major
Steps to Reproduce
------------------
Prepare a distributed cloud system for orchestrated subcloud upgrade
Create and apply an upgrade strategy for a large number of simplex subclouds in parallel
Expected Behavior
------------------
The upgrade strategy completes successfully.
Actual Behavior
----------------
Upgrade failed for one subcloud at the unlocking-host step as it was unable to retrieve host data from the subcloud.
Reproducibility
---------------
Seen once
System Configuration
--------------------
Distributed cloud
Branch/Pull Time/Commit
-----------------------
2022-01-19 18:42:01
Last Pass
---------
Not sure if this test was previously performed for the stx/6.0 load.
Timestamp/Logs
--------------
2022-03-10 21:10:16.655 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Host: controller-0 is now: unlocked enabled
2022-03-10 21:10:16.667 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Data migration completed.
2022-03-10 21:10:17.167 3236316 ERROR dccommon.drivers.openstack.sdk_platform [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] keystone_client region subcloud2008 error: Service Unavailable (HTTP 503): ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.170 3236316 WARNING dcmanager.orchestrator.states.base [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] Failure initializing KeystoneClient for region: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] (upgrade) Failed! Stage: 1, State: unlocking controller-0, Subcloud: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread Traceback (most recent call last):
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/orch_thread.py", line 512, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread next_state = state_operator.perform_state_action(strategy_step)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/unlock_host.py", line 63, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread sysinv_client = self.get_sysinv_client(strategy_step.subcloud.name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 107, in get_sysinv_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread keystone_client = self.get_keystone_client(region_name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 96, in get_keystone_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread region_clients=['sysinv'])
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sdk_platform.py", line 101, in _init_
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread raise exception
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread
Test Activity
-------------
Feature Testing
Workaround
Delete the failed strategy, create a new one and reapply |
|
| 2022-03-16 18:38:57 |
Gabriel Silva Trevisan |
description |
Brief Description
-----------------
While performing the unlock_simplex step during a subcloud upgrade, the subcloud platform endpoints became unavailable, causing the step to fail.
Retrying the requests on the host unlock step may be required to work around the flakiness of platform services post data migration.
Severity
--------
Major
Steps to Reproduce
------------------
Prepare a distributed cloud system for orchestrated subcloud upgrade
Create and apply an upgrade strategy for a large number of simplex subclouds in parallel
Expected Behavior
------------------
The upgrade strategy completes successfully.
Actual Behavior
----------------
Upgrade failed for one subcloud at the unlocking-host step as it was unable to retrieve host data from the subcloud.
Reproducibility
---------------
Seen once
System Configuration
--------------------
Distributed cloud
Branch/Pull Time/Commit
-----------------------
2022-01-19 18:42:01
Last Pass
---------
Not sure if this test was previously performed for the stx/6.0 load.
Timestamp/Logs
--------------
2022-03-10 21:10:16.655 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Host: controller-0 is now: unlocked enabled
2022-03-10 21:10:16.667 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Data migration completed.
2022-03-10 21:10:17.167 3236316 ERROR dccommon.drivers.openstack.sdk_platform [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] keystone_client region subcloud2008 error: Service Unavailable (HTTP 503): ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.170 3236316 WARNING dcmanager.orchestrator.states.base [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] Failure initializing KeystoneClient for region: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] (upgrade) Failed! Stage: 1, State: unlocking controller-0, Subcloud: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread Traceback (most recent call last):
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/orch_thread.py", line 512, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread next_state = state_operator.perform_state_action(strategy_step)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/unlock_host.py", line 63, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread sysinv_client = self.get_sysinv_client(strategy_step.subcloud.name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 107, in get_sysinv_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread keystone_client = self.get_keystone_client(region_name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 96, in get_keystone_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread region_clients=['sysinv'])
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sdk_platform.py", line 101, in _init_
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread raise exception
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread
Test Activity
-------------
Feature Testing
Workaround
Delete the failed strategy, create a new one and reapply |
Brief Description
-----------------
While performing the unlock_simplex step during a subcloud upgrade, the subcloud platform endpoints became unavailable, causing the step to fail.
Retrying the requests on the host unlock step may be required to work around the flakiness of platform services post data migration.
Severity
--------
Major
Steps to Reproduce
------------------
Prepare a distributed cloud system for orchestrated subcloud upgrade
Create and apply an upgrade strategy for a large number of simplex subclouds in parallel
Expected Behavior
------------------
The upgrade strategy completes successfully.
Actual Behavior
----------------
Upgrade failed for one subcloud at the unlocking-host step as it was unable to retrieve host data from the subcloud.
Reproducibility
---------------
Seen once
System Configuration
--------------------
Distributed cloud
Branch/Pull Time/Commit
-----------------------
2022-01-19 18:42:01
Last Pass
---------
Not sure if an upgrade test with a large number of subclouds was previously performed for the stx/6.0 load.
Timestamp/Logs
--------------
2022-03-10 21:10:16.655 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Host: controller-0 is now: unlocked enabled
2022-03-10 21:10:16.667 3236316 INFO dcmanager.orchestrator.states.base [req-925ded40-e63f-4167-9f2e-a2d05f85d810 - - - - -] Stage: 1, State: migrating data, Subcloud: subcloud2030, Details: Data migration completed.
2022-03-10 21:10:17.167 3236316 ERROR dccommon.drivers.openstack.sdk_platform [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] keystone_client region subcloud2008 error: Service Unavailable (HTTP 503): ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.170 3236316 WARNING dcmanager.orchestrator.states.base [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] Failure initializing KeystoneClient for region: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread [req-cbd71502-0865-4e10-8d19-a2469cb7d30c - - - - -] (upgrade) Failed! Stage: 1, State: unlocking controller-0, Subcloud: subcloud2008: ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread Traceback (most recent call last):
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/orch_thread.py", line 512, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread next_state = state_operator.perform_state_action(strategy_step)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/unlock_host.py", line 63, in perform_state_action
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread sysinv_client = self.get_sysinv_client(strategy_step.subcloud.name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 107, in get_sysinv_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread keystone_client = self.get_keystone_client(region_name)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dcmanager/orchestrator/states/base.py", line 96, in get_keystone_client
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread region_clients=['sysinv'])
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sdk_platform.py", line 101, in _init_
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread raise exception
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread ServiceUnavailable: Service Unavailable (HTTP 503)
2022-03-10 21:10:17.171 3236316 ERROR dcmanager.orchestrator.orch_thread
Test Activity
-------------
Feature Testing
Workaround
Delete the failed strategy, create a new one and reapply |
|
| 2022-03-16 19:37:38 |
OpenStack Infra |
starlingx: status |
New |
In Progress |
|
| 2022-03-16 19:53:26 |
Gabriel Silva Trevisan |
starlingx: assignee |
|
Gabriel Silva Trevisan (g-trevisan) |
|
| 2022-03-20 21:06:05 |
OpenStack Infra |
starlingx: status |
In Progress |
Fix Released |
|
| 2022-03-21 23:13:33 |
Ghada Khalil |
starlingx: importance |
Undecided |
Medium |
|
| 2022-03-21 23:13:48 |
Ghada Khalil |
tags |
stx.distcloud |
stx.7.0 stx.distcloud |
|
