Pike to queen failed to cleanup nova container

[Satish Patel]

On CentOS7 after upgrade from pike to queens got following error during clean up

openstack-ansible /var/opt/openstack-ansible/scripts/upgrade-utilities/playbooks/cleanup-nova.yml -e force_containers_destroy=yes -e force_containers_data_destroy=yes

http://paste.openstack.org/show/726826/

TASK [Collect the ID's for the nova services to delete] **********************************************************************************************************************************************************
Monday 30 July 2018 07:30:52 -0400 (0:00:09.421) 0:00:54.338 ***********
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-02_nova_scheduler_container-907e7b2f)
failed: [ostack-infra-02_utility_container-c39f9322] (item=ostack-infra-03_nova_scheduler_container-69376e50) => {"changed": true, "cmd": ["openstack", "--os-cloud", "default", "compute", "service", "list", "--format", "value", "--column", "ID", "--host", "ostack-infra-03-nova-scheduler-container-69376e50", "--service", "nova-scheduler"], "delta": "0:00:15.850361", "end": "2018-07-30 07:31:14.014395", "failed": true, "item": "ostack-infra-03_nova_scheduler_container-69376e50", "msg": "non-zero return code", "rc": 1, "start": "2018-07-30 07:30:58.164034", "stderr": "Unknown Error (HTTP 503)", "stderr_lines": ["Unknown Error (HTTP 503)"], "stdout": "", "stdout_lines": []}
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-01_nova_scheduler_container-284ddde8)
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-02_nova_conductor_container-695cdd36)
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-03_nova_conductor_container-e5908a88)
failed: [ostack-infra-02_utility_container-c39f9322] (item=ostack-infra-01_nova_conductor_container-c9b8bd17) => {"changed": true, "cmd": ["openstack", "--os-cloud", "default", "compute", "service", "list", "--format", "value", "--column", "ID", "--host", "ostack-infra-01-nova-conductor-container-c9b8bd17", "--service", "nova-conductor"], "delta": "0:00:12.593404", "end": "2018-07-30 07:31:47.016373", "failed": true, "item": "ostack-infra-01_nova_conductor_container-c9b8bd17", "msg": "non-zero return code", "rc": 1, "start": "2018-07-30 07:31:34.422969", "stderr": "Unknown Error (HTTP 503)", "stderr_lines": ["Unknown Error (HTTP 503)"], "stdout": "", "stdout_lines": []}
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-02_nova_console_container-02ba5d76)
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-03_nova_console_container-19da1744)
changed: [ostack-infra-02_utility_container-c39f9322] => (item=ostack-infra-01_nova_console_container-52ec2424)

PLAY RECAP *******************************************************************************************************************************************************************************************************
ostack-infra-01_nova_api_metadata_container-42169131 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-01_nova_api_os_compute_container-319cebbf : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-01_nova_api_placement_container-fcae5673 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-01_nova_conductor_container-c9b8bd17 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-01_nova_console_container-52ec2424 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-01_nova_scheduler_container-284ddde8 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-02_nova_api_metadata_container-8f0dcb96 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-02_nova_api_os_compute_container-5723e379 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-02_nova_api_placement_container-888b2fe0 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-02_nova_conductor_container-695cdd36 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-02_nova_console_container-02ba5d76 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-02_nova_scheduler_container-907e7b2f : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-02_utility_container-c39f9322 : ok=0 changed=0 unreachable=0 failed=1
ostack-infra-03_nova_api_metadata_container-edf3c805 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-03_nova_api_os_compute_container-03bb8ce3 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-03_nova_api_placement_container-1469a861 : ok=3 changed=2 unreachable=0 failed=0
ostack-infra-03_nova_conductor_container-e5908a88 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-03_nova_console_container-19da1744 : ok=5 changed=3 unreachable=0 failed=0
ostack-infra-03_nova_scheduler_container-69376e50 : ok=5 changed=3 unreachable=0 failed=0

Monday 30 July 2018 07:31:59 -0400 (0:01:07.364) 0:02:01.703 ***********
===============================================================================
Collect the ID's for the nova services to delete --------------------------------------------------------------------------------------------------------------------------------------------------------- 67.36s
Destroy a container -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 25.85s
Shut down nova services in containers which will be deleted ---------------------------------------------------------------------------------------------------------------------------------------------- 14.85s
Destroy container data ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 9.42s
Destroy container service directories --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.55s
Gather the necessary facts -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.53s

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

[Satish Patel (satish-txt)]

If i run same command on CLI i got following

[root@ostack-infra-02-utility-container-c39f9322 ~]# openstack --os-cloud default compute service list --format value --column ID --host ostack-infra-03-nova-scheduler-container-69376e50 --service nova-scheduler
36

[Mohammed Naser (mnaser)]

Adding retries would be a good solution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/587563

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/587563
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=122b75b033f89d15fef0620002223707a3f98c3d
Submitter: Zuul
Branch: stable/queens

commit 122b75b033f89d15fef0620002223707a3f98c3d
Author: Guilherme Steinmüller <email address hidden>
Date: Tue Jul 31 14:39:26 2018 -0300

    Add retries to collect ID's for nova services

    Due to a report of a transient error while collecting
    the Id's for the nova services to delete, this patch aims
    to add retries in the task referred to avoid eventual errors
    during its execution.

    Change-Id: I853b1adfaf978c103105245287ab42bc8ec007d5
    Closes-Bug: 1784369

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 17.0.8 release.