Openstack cli command unexpectedly rejected - Bad Gateway 502

[Hugo Nicodemos Brito]

Brief Description
-----------------
cli command for openstack server show rejected unexpectedly - Bad Gateway HTTP 502

Severity
--------
Standard

Steps to Reproduce
------------------

***Note: Prior to running the testcase below, a volume (from previous testcase test_cpu_realtime_vm_actions[3-None-^0-flavor-None-None-prefer] ) failed to delete and is still in attaching state prior to launching the instance at 1:50:10
[2020-09-05 23:58:03,266] 186 ERROR MainThread resource_mgmt._delete_resources:: ERROR: Failed to delete resource(s).
Details: ["Failed to delete volume with name or ID 'ded36974-9b4a-46e3-b353-7b26b5394608': Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-edba81c0-25f9-4eee-888e-a9c73899f95d)\n1 of 1 volumes failed to delete."]
...
[2020-09-06 01:50:10,065] 436 DEBUG MainThread ssh.expect :: Output:
-------------------------------------------------------------------------------------------------------+

ID Name Status Size Type Bootable Attached to Properties
-------------------------------------------------------------------------------------------------------+

ded36974-9b4a-46e3-b353-7b26b5394608 vol-tenant1-8 attaching 2 ceph-store true
-------------------------------------------------------------------------------------------------------+

In nova regression, the following testcase was executed which does the following:
test_live_migrate_vm_positive[local_image-0-0-None-2-image_with_vol-False]

1. Boots the instance
2020-09-06 01:50:10,174] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --non-bootable --image=9b6c4c39-59b2-4198-bc95-9fc92ece9ca3 --size=2 vol-tenant1'

> The instance was created
OS-EXT-SRV-ATTR:host controller-0
OS-EXT-SRV-ATTR:hypervisor_hostname controller-0
OS-EXT-SRV-ATTR:instance_name instance-00000042
beb82c2f-bb09-4505-9860-42703c5fc212
2. Creates volume, attach (and mount operations) here:
[2020-09-06 01:50:10,174] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne volume create --non-bootable --image=9b6c4c39-59b2-4198-bc95-9fc92ece9ca3 --size=2 vol-tenant1'
volume id | cbd3b2dc-e939-4afa-ae62-d9f72fb3ee31

[2020-09-06 01:50:50,504] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server add volume beb82c2f-bb09-4505-9860-42703c5fc212 cbd3b2dc-e939-4afa-ae62-d9f72fb3ee31'

nova compute log reports attaching here
2020-09-06T01:50:53.45330374Z stdout F 2020-09-06 01:50:53.452 553470 INFO nova.compute.manager [req-343bec87-bcce-47dd-bbbb-28a108affa55 39cae71c1be94e85b501c17fa895885a 1c9fd9588bf943d7a8176729f71d501e - default default] [instance: beb82c2f-bb09-4505-9860-42703c5fc212] Attaching volume cbd3b2dc-e939-4afa-ae62-d9f72fb3ee31 to /dev/vdb

3. The server show command was issued here successfully
[2020-09-06 01:51:50,470] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-09-06 01:51:50,471] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show beb82c2f-bb09-4505-9860-42703c5fc212'
[2020-09-06 01:51:52,540] 436 DEBUG MainThread ssh.expect :: Output:
----------------------------------------------------------------------------------------------+

Field Value
----------------------------------------------------------------------------------------------+

OS-DCF:diskConfig MANUAL
OS-EXT-AZ:availability_zone nova
OS-EXT-SRV-ATTR:host controller-0
OS-EXT-SRV-ATTR:hypervisor_hostname controller-0
OS-EXT-SRV-ATTR:instance_name instance-00000042
OS-EXT-STS:power_state Running
OS-EXT-STS:task_state None
OS-EXT-STS:vm_state active
OS-SRV-USG:launched_at 2020-09-06T01:50:02.000000
OS-SRV-USG:terminated_at None
accessIPv4
accessIPv6
addresses tenant1-mgmt-net=192.168.99.14; tenant1-net4=172.16.4.206
config_drive
created 2020-09-06T01:49:52Z
flavor migration_test (e2891b34-095f-45ec-bc18-229e05949fa2)
hostId 824b560a1548fe4205a118381d0b6fa642430e0c89b67e2a5ae3b3e9
id beb82c2f-bb09-4505-9860-42703c5fc212
image tis-centos-guest (9b6c4c39-59b2-4198-bc95-9fc92ece9ca3)
key_name keypair-tenant1
name tenant1-migration_test-34
progress 0
project_id 1c9fd9588bf943d7a8176729f71d501e
properties
security_groups name='default'
  name='default'
status ACTIVE
updated 2020-09-06T01:50:02Z
user_id 39cae71c1be94e85b501c17fa895885a
volumes_attached id='cbd3b2dc-e939-4afa-ae62-d9f72fb3ee31'
----------------------------------------------------------------------------------------------+

4. The server show command was issued here again
[sysadmin@controller-0 ~(keystone_admin)]$
[2020-09-06 01:51:52,647] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show beb82c2f-bb09-4505-9860-42703c5fc212'

Expected Behavior
------------------
Did not expect rejected cli command in step 4

Actual Behavior
----------
d. The server show command issued here was rejected - returns Bad Gateway 502
[sysadmin@controller-0 ~(keystone_admin)]$
[2020-09-06 01:51:52,645] 205 DEBUG MainThread table_parser.__convert_multilines_values:: Multi-row entry found for: security_groups
[2020-09-06 01:51:52,646] 1649 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for wcp_69_70
[2020-09-06 01:51:52,646] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-09-06 01:51:52,647] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show beb82c2f-bb09-4505-9860-42703c5fc212'
[2020-09-06 01:51:53,637] 436 DEBUG MainThread ssh.expect :: Output:
Bad Gateway (HTTP 502)

[Ghada Khalil (gkhalil)]

Review in progress: https://review.opendev.org/c/starlingx/openstack-armada-app/+/767738

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/openstack-armada-app/+/792235

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/openstack-armada-app/+/792235
Committed: https://opendev.org/starlingx/openstack-armada-app/commit/1bf694661282a019bf79f253fc148baede65db64
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 963e63cd55d5be4f5ddfc148ae00b6a46e071295
Author: Thiago Brito <email address hidden>
Date: Fri May 14 15:36:07 2021 -0300

    Fix cpu_shared/dedicated_set config location

    Change I61514389b616db754b0d2f35deb0101f90dbdd02 removed the deprecated
    property vcpu_pin_set in favor of the newer cpu_shared_set and
    cpu_dedicated_set, but those new configs are placed under the [compute]
    section of nova.conf instead of [DEFAULT]. This is causing VMs to be
    scheduled on platform reserved cores. This commit will fix it.

    Closes-Bug: #1928683

    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: I541760619f4c79c66a2bf22715afdc873b8343ce

commit 58f4d9ffcaf47fe969267149135201aec01624a8
Author: Gustavo Santos <email address hidden>
Date: Mon Mar 8 14:56:55 2021 -0300

    Add k8s proxy-body-size to horizon overrides

    The current network.dashboard.ingress.annotations in horizon's
    values.yaml helm charts do not include the kubernetes property
    'proxy-body-size'. This makes the resulting nginx.conf file in ingress
    add the default rule 'max_body_size 1m' to the horizon servers,
    which limits all http requests' size inside horizon to 1MiB, making it
    impossible to upload images larger than that to glance using the
    horizon GUI, for example.

    This change adds said property to the horizon overrides, making
    horizon's servers in nginx.conf include a 'max_body_size' of 2500MiB,
    which makes uploading images up to that size possible again.

    Story: 2008692
    Task: 41996
    Change-Id: I91888ce238d5304c08eb1e97918989b8f93ee34f

commit b5c1f62088778287e4b50aeac1f17d166a7a177a
Author: Dan Voiculeasa <email address hidden>
Date: Wed Feb 3 16:00:47 2021 +0200

    Introduce metadata for app behavior control

    Keep existing behavior when evaluating app reapplies.

    Story: 2007960
    Task: 41755
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: Ie02743cdf056dda3feb66911c74f9dabe69d98dd

commit eab750b7ff03808002acf35deebdf762e687b332
Author: Martin, Chen <email address hidden>
Date: Sat May 30 09:11:05 2020 +0800

    Add override setting in openstack helm plugin for rook-ceph

    Deploy with rook-ceph, without "system storage-backend-add ceph"
    there is no object storage-ceph in database. As current openstack
    helm plugin fixed on object storage-ceph, in rook-ceph case
    use a fixed override setting

    Story: 2005527
    Task: 39914

    Depends-On: https://review.opendev.org/#/c/713084/

    Change-Id: Ied852d60e8b15d55865747e0b6f4b54f2392d6df
    Signed-off-by: Martin, Chen <email address hidden>

commit 852d8d61dbfc4f9f29afe8da10924731a58028ea
Author: Dan Voiculeasa <email address hidden>
Date: Mon Nov 16 12:41:55 2020 +0200

    Introduce lifecycle operator to openstack app

    A big chunk...