A container puppet task fails without logs and traces explaining any details

[Bogdan Dobrelya]

Seen in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_462/770843/2/check/tripleo-ci-centos-8-standalone/46247b7/logs/undercloud/home/zuul/standalone_deploy.log

The failure points out container-puppet-cinder. But there is no logs, no container ID provided, nothing to figure out what went wrong:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_462/770843/2/check/tripleo-ci-centos-8-standalone/46247b7/logs/undercloud/home/zuul/ansible.log
2021-01-15 11:41:36,260 p=43995 u=root n=ansible | 2021-01-15 11:41:36.259981 | fa163e42-bac8-e4fa-8f02-000000001367 | OK | Check podman create status | standalone | item=container-puppet-cinder
2021-01-15 11:41:36,449 p=43995 u=root n=ansible | 2021-01-15 11:41:36.448988 | fa163e42-bac8-e4fa-8f02-000000001369 | TASK | Check containers status
2021-01-15 11:41:38,099 p=66717 u=root n=ansible | [ERROR]: Container(s) which failed to be created by podman_container module:
['container-puppet-cinder']

2021-01-15 11:41:38,113 p=43995 u=root n=ansible | 2021-01-15 11:41:38.112530 | fa163e42-bac8-e4fa-8f02-000000001369 | FATAL | Check containers status | standalone | error={"changed": false, "msg": "Failed container(s): ['container-puppet-cinder'], check logs in /var/log/containers/stdouts/"}

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_462/770843/2/check/tripleo-ci-centos-8-standalone/46247b7/logs/undercloud/var/log/extra/journal.txt
Jan 15 11:30:45 standalone.localdomain ansible-async_wrapper.py[64758]: error: [Errno 32] Broken pipe

[Cédric Jeanneret (cjeanner)]

Funky... 1-2 things:

- while it seems the deploy should have created the cinder log directories, they aren't present
- while it seems to run cinder container at some point, nothing shows it - no log, no presence in `podman ps -a'.
- no failed container at all here: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_462/770843/2/check/tripleo-ci-centos-8-standalone/46247b7/logs/undercloud/var/log/extra/podman/podman_allinfo.log

- One can also appreciate this kind of things in the journal.log:
Jan 15 11:27:15 standalone.localdomain rsyslogd[51064]: message too long (21963) with configured size 8096, begin of message is: Invoked with config_data={'/var/lib/kolla/config_files/cinder_api.json': {'comma [v8.1911.0-6.el8 try https://www.rsyslog.com/e/2445 ]

For the sake of CI, we might want to raise a bit that number of "8096" with something allowing us to actually get the full log for debugging purpose?

Also, not sure if the Broken pipe has anything to do with the actual error... We can also see pacemaker related issues, so maybe there's a link at some point? missing DB crashing some deploy steps in such a way we don't see anything?

[Bogdan Dobrelya (bogdando)]

probably related https://bugs.launchpad.net/tripleo/+bug/1906625

[Bogdan Dobrelya (bogdando)]

lowered to medium since the issue looks intermittent, but the described debug UX still applies

[Bogdan Dobrelya (bogdando)]

with https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/771851 we'll have stats for how often the issue happens in CI