[victoria] [ERROR]: Container(s) which failed to be created by podman_container module: ['nova_wait_for_compute_service'] failing on fs039

[Bhagyashri Shewale]

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria is failing with below error.

Error logs:

2021-12-30 13:09:44 | 2021-12-30 13:09:44.798746 | fa163e7e-13fe-30d5-bcdb-00000000ec65 | TASK | Check containers status
2021-12-30 13:09:46 | [ERROR]: Container(s) which finished with wrong return code:
2021-12-30 13:09:46 | ['nova_wait_for_compute_service']
2021-12-30 13:09:46 | [ERROR]: Container(s) which failed to be created by podman_container module:
2021-12-30 13:09:46 | ['nova_wait_for_compute_service']
2021-12-30 13:09:46 | 2021-12-30 13:09:46.575435 | fa163e7e-13fe-30d5-bcdb-00000000ec65 | FATAL | Check containers status | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_compute_service'], check logs in /var/log/containers/stdouts/"}
2021-12-30 13:09:46 | 2021-12-30 13:09:46.577093 | fa163e7e-13fe-30d5-bcdb-00000000ec65 | TIMING | tripleo_container_manage : Check containers status | overcloud-novacompute-0 | 0:53:16.551035 | 1.78s
2021-12-30 13:09:46 |
2021-12-30 13:09:46 | PLAY RECAP *********************************************************************
2021-12-30 13:09:46 | overcloud-controller-0 : ok=667 changed=311 unreachable=0 failed=0 skipped=268 rescued=0 ignored=0
2021-12-30 13:09:46 | overcloud-controller-1 : ok=652 changed=299 unreachable=0 failed=0 skipped=267 rescued=0 ignored=0
2021-12-30 13:09:46 | overcloud-controller-2 : ok=651 changed=299 unreachable=0 failed=0 skipped=267 rescued=0 ignored=0
2021-12-30 13:09:46 | overcloud-novacompute-0 : ok=494 changed=199 unreachable=0 failed=1 skipped=233 rescued=0 ignored=0
2021-12-30 13:09:46 | undercloud : ok=823 changed=291 unreachable=0 failed=0 skipped=107 rescued=0 ignored=0
2021-12-30 13:09:46 | 2021-12-30 13:09:46.606647 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-30 13:09:46 | 2021-12-30 13:09:46.607216 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 3138 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-30 13:09:46 | 2021-12-30 13:09:46.607654 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:53:16.581610 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-30 13:09:46 | 2021-12-30 13:09:46.608125 | UUID | Info | Host | Task Name | Run Time
2021-12-30 13:09:46 | 2021-12-30 13:09:46.608551 | fa163e7e-13fe-30d5-bcdb-00000000ec63 | SUMMARY | overcloud-novacompute-0 | tripleo_container_manage : Check podman create status | 640.64s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.609069 | fa163e7e-13fe-30d5-bcdb-00000000359f | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 301.57s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.609551 | fa163e7e-13fe-30d5-bcdb-000000006c58 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for haproxy | 154.31s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.609969 | fa163e7e-13fe-30d5-bcdb-000000004646 | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 124.71s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.610412 | fa163e7e-13fe-30d5-bcdb-000000004781 | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 114.07s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.610910 | fa163e7e-13fe-30d5-bcdb-0000000046e7 | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 112.70s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.611325 | fa163e7e-13fe-30d5-bcdb-00000000bd3c | SUMMARY | overcloud-controller-2 | tripleo_container_manage : Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | 108.39s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.611809 | fa163e7e-13fe-30d5-bcdb-00000000be5f | SUMMARY | overcloud-controller-1 | tripleo_container_manage : Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | 107.71s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.612227 | fa163e7e-13fe-30d5-bcdb-00000000c3ac | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | 106.53s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.612680 | fa163e7e-13fe-30d5-bcdb-0000000047e2 | SUMMARY | overcloud-novacompute-0 | Pre-fetch all the containers | 100.91s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.613068 | fa163e7e-13fe-30d5-bcdb-000000005322 | SUMMARY | overcloud-controller-2 | Wait for puppet host configuration to finish | 100.55s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.613514 | fa163e7e-13fe-30d5-bcdb-000000005388 | SUMMARY | overcloud-controller-1 | Wait for puppet host configuration to finish | 100.22s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.613928 | fa163e7e-13fe-30d5-bcdb-00000000534f | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 96.91s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.614326 | fa163e7e-13fe-30d5-bcdb-00000000875b | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for ovn_dbs | 62.41s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.614732 | fa163e7e-13fe-30d5-bcdb-0000000019cb | SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 56.70s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.615080 | fa163e7e-13fe-30d5-bcdb-0000000019cb | SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 50.10s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.615676 | fa163e7e-13fe-30d5-bcdb-0000000019cb | SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 49.73s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.616075 | fa163e7e-13fe-30d5-bcdb-000000006c68 | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for mysql | 42.23s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.616485 | fa163e7e-13fe-30d5-bcdb-000000006c8b | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for redis | 42.21s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.616857 | fa163e7e-13fe-30d5-bcdb-000000006c7b | SUMMARY | overcloud-controller-0 | tripleo_ha_wrapper : Run init bundle puppet on the host for oslo_messaging_rpc | 41.74s
2021-12-30 13:09:46 | 2021-12-30 13:09:46.617256 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-30 13:09:46 | 2021-12-30 13:09:46.617737 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-30 13:09:46 | 2021-12-30 13:09:46.618145 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2021-12-30 13:09:46 | 2021-12-30 13:09:46.618614 | The following node(s) had failures: overcloud-novacompute-0
2021-12-30 13:09:46 | 2021-12-30 13:09:46.618989 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-12-30 13:09:48 | 2021-12-30 13:09:48.066 192432 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleon1aq19lm ] cleaned up
2021-12-30 13:09:48 | 2021-12-30 13:09:48.126 192432 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/zuul/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/config-download/overcloud/ansible-playbook-command.sh
2021-12-30 13:09:48 | 2021-12-30 13:09:48.139 192432 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo6zf3goji ] cleaned up
2021-12-30 13:09:50 | Host overcloud.ooo.test not found in /home/zuul/.ssh/known_hosts
2021-12-30 13:09:50 | Overcloud Endpoint: https://overcloud.ooo.test:13000
2021-12-30 13:09:50 | Overcloud Horizon Dashboard URL: https://overcloud.ooo.test:443/dashboard
2021-12-30 13:09:50 | Overcloud rc file: /home/zuul/overcloudrc
2021-12-30 13:09:50 | Overcloud Deployed with error
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/zuul/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/config-download/overcloud/ansible-playbook-command.sh
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1223, in take_action
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise(deploy_trace)
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1189, in take_action
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=parsed_args.ansible_forks
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 463, in config_download
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=forks
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 725, in run_ansible_playbook
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg)
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/zuul/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/config-download/overcloud/ansible-playbook-command.sh
2021-12-30 13:09:50 | 2021-12-30 13:09:50.426 192432 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2021-12-30 13:09:50 | 2021-12-30 13:09:50.486 192432 ERROR openstack [-] Ansible execution failed. playbook: /home/zuul/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/config-download/overcloud/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/zuul/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/config-download/overcloud/ansible-playbook-command.sh
2021-12-30 13:09:50 | 2021-12-30 13:09:50.496 192432 INFO osc_lib.shell [-] END return value: 1
2021-12-30 13:09:50 | /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
2021-12-30 13:09:50 | % (name, name, name))
2021-12-30 13:09:50 | /usr/lib/python3.6/site-packages/openstack/resource.py:351: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
2021-12-30 13:09:50 | len(inspect.getargspec(type_).args) > 1)
2021-12-30 13:09:50 | /usr/lib/python3.6/site-packages/openstack/resource.py:351: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
2021-12-30 13:09:50 | len(inspect.getargspec(type_).args) > 1)
2021-12-30 13:09:53 | + status_code=1
2021-12-30 13:09:53 | + openstack stack list
2021-12-30 13:09:53 | + grep -q overcloud
2021-12-30 13:09:56 | /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
2021-12-30 13:09:56 | % (name, name, name))
2021-12-30 13:09:56 | + openstack stack list
2021-12-30 13:09:56 | + grep -Eq '(CREATE|UPDATE)_COMPLETE'
2021-12-30 13:09:59 | /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
2021-12-30 13:09:59 | % (name, name, name))
2021-12-30 13:10:00 | + exit 1

# link:

[1]: https://logserver.rdoproject.org/60/31460/49/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/4c846dd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[2]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria

[3]: https://logserver.rdoproject.org/60/31460/49/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/4c846dd/job-output.txt

[4]: https://logserver.rdoproject.org/60/31460/49/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/4c846dd/logs/overcloud-novacompute-0/var/log/containers/stdouts/nova_wait_for_compute_service.log.txt.gz

[5]: https://logserver.rdoproject.org/60/31460/49/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/4c846dd/logs/overcloud-novacompute-0/var/log/extra/podman/containers/nova_wait_for_compute_service/stdout.log.txt.gz

[Rabi Mishra (rabi)]

side-effect of https://review.opendev.org/c/openstack/tripleo-heat-templates/+/821109

Container images have old oslo-cache that does not contain https://review.opendev.org/c/openstack/oslo.cache/+/773908.

https://logserver.rdoproject.org/60/31460/49/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/4c846dd/logs/overcloud-controller-0/var/log/containers/nova/nova-conductor.log.txt.gz

2021-12-30 13:23:26.605 8 CRITICAL nova [req-9a1bc7ef-e0e9-4ff5-8316-7738109c9ae2 - - - - -] Unhandled error: oslo_config.cfg.ConfigFileValueError: Value for option backend from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/nova/nova.conf') is not valid: Valid values are [oslo_cache.memcache_pool, oslo_cache.dict, oslo_cache.mongo, oslo_cache.etcd3gw, dogpile.cache.memcached, dogpile.cache.pylibmc, dogpile.cache.bmemcached, dogpile.cache.dbm, dogpile.cache.redis, dogpile.cache.memory, dogpile.cache.memory_pickle, dogpile.cache.null], but found 'dogpile.cache.pymemcache'
2021-12-30 13:23:26.605 8 ERROR nova Traceback (most recent call last):
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2720, in _do_get
2021-12-30 13:23:26.605 8 ERROR nova return (convert(val), alt_loc)
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2690, in convert
2021-12-30 13:23:26.605 8 ERROR nova self._substitute(value, group, namespace), opt)
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2817, in _convert_value
2021-12-30 13:23:26.605 8 ERROR nova return opt.type(value)
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 170, in __call__
2021-12-30 13:23:26.605 8 ERROR nova repr(value)))
2021-12-30 13:23:26.605 8 ERROR nova ValueError: Valid values are [oslo_cache.memcache_pool, oslo_cache.dict, oslo_cache.mongo, oslo_cache.etcd3gw, dogpile.cache.memcached, dogpile.cache.pylibmc, dogpile.cache.bmemcached, dogpile.cache.dbm, dogpile.cache.redis, dogpile.cache.memory, dogpile.cache.memory_pickle, dogpile.cache.null], but found 'dogpile.cache.pymemcache'
2021-12-30 13:23:26.605 8 ERROR nova
2021-12-30 13:23:26.605 8 ERROR nova During handling of the above exception, another exception occurred:
2021-12-30 13:23:26.605 8 ERROR nova
2021-12-30 13:23:26.605 8 ERROR nova Traceback (most recent call last):
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
2021-12-30 13:23:26.605 8 ERROR nova sys.exit(main())
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 49, in main
2021-12-30 13:23:26.605 8 ERROR nova service.wait()
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 489, in wait
2021-12-30 13:23:26.605 8 ERROR nova _launcher.wait()
2021-12-30 13:23:26.605 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 389, in wait
2021-12-30 13:23:26.605 8 ERROR nova status, signo = self._wait_for_exit_or_signal()
2021-12...

[Rabi Mishra (rabi)]

Also we need to bump https://github.com/redhat-openstack/rdoinfo/blob/master/buildsys-tags/cloud8-openstack-victoria-release.yml#L1270 if we want to use dogpile.cache.pymemcache backend with tls.

[Douglas Viroel (dviroel)]

Updating python-memcache and dogpile-cache [1] seems to fix the issue [2][3]

[1] https://review.rdoproject.org/r/c/rdoinfo/+/31955
[2] https://review.rdoproject.org/zuul/build/5d8dccd304f849598bee2d999bebbc74
[3] https://logserver.rdoproject.org/56/36356/8/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/5d8dccd/logs/undercloud/var/log/extra/package-list-installed.txt.gz

[Alfredo Moralejo (amoralej)]

Sending similar deps updated for train and ussuri in https://review.rdoproject.org/r/c/rdoinfo/+/37692 and https://review.rdoproject.org/r/c/rdoinfo/+/37694