On master noop check job, tripleo-ci-centos-7-containers-multinode job got failed at undercloud deploy step http://logs.openstack.org/45/560445/193/check/tripleo-ci-centos-7-containers-multinode/1f1369f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-11-15_01_41_29
2018-11-15 01:41:29 | "Completed $ podman create --name ironic_inspector_dnsmasq --label config_id=tripleo_step4 --label container_name=ironic_inspector_dnsmasq --label managed_by=paunch --label config_data={\"start_order\": 93, \"healthcheck\": {\"test\": \"/openstack/healthcheck\"}, \"image\": \"192.168.24.1:8787/tripleomaster/centos-binary-ironic-inspector:current-tripleo-updated-20181115001011\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=be9e84433a3f21e95685b27b0480716c\"], \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/config_files/ironic_inspector_dnsmasq.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/ironic_inspector/:/var/lib/kolla/config_files/src:ro\", \"/var/log/containers/ironic-inspector:/var/log/ironic-inspector\", \"/var/lib/ironic-inspector/dhcp-hostsdir:/var/lib/ironic-inspector/dhcp-hostsdir:shared\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"} --detach=true --env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env=TRIPLEO_CONFIG_HASH=be9e84433a3f21e95685b27b0480716c --net=host --privileged=true --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/lib/kolla/config_files/ironic_inspector_dnsmasq.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/ironic_inspector/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/ironic-inspector:/var/log/ironic-inspector --volume=/var/lib/ironic-inspector/dhcp-hostsdir:/var/lib/ironic-inspector/dhcp-hostsdir:shared 192.168.24.1:8787/tripleomaster/centos-binary-ironic-inspector:current-tripleo-updated-20181115001011",
2018-11-15 01:41:29 | "stdout: 21f6bf585613661b105572b54972e6dbb13ee4f539b0c7f130bedba1755f0fa3",
2018-11-15 01:41:29 | "Creating systemd unit file: /etc/systemd/system/ironic_inspector_dnsmasq.service",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_container_updater.service to /etc/systemd/system/swift_container_updater.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/mistral_executor.service to /etc/systemd/system/mistral_executor.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/nova_scheduler.service to /etc/systemd/system/nova_scheduler.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_object_server.service to /etc/systemd/system/swift_object_server.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/heat_api.service to /etc/systemd/system/heat_api.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/zaqar_websocket.service to /etc/systemd/system/zaqar_websocket.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/nova_api_cron.service to /etc/systemd/system/nova_api_cron.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/mistral_event_engine.service to /etc/systemd/system/mistral_event_engine.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_account_reaper.service to /etc/systemd/system/swift_account_reaper.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_object_expirer.service to /etc/systemd/system/swift_object_expirer.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_rsync.service to /etc/systemd/system/swift_rsync.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/heat_engine.service to /etc/systemd/system/heat_engine.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/heat_api_cron.service to /etc/systemd/system/heat_api_cron.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/nova_conductor.service to /etc/systemd/system/nova_conductor.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_account_server.service to /etc/systemd/system/swift_account_server.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/neutron_api.service to /etc/systemd/system/neutron_api.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_object_updater.service to /etc/systemd/system/swift_object_updater.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_container_server.service to /etc/systemd/system/swift_container_server.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/zaqar.service to /etc/systemd/system/zaqar.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/mistral_engine.service to /etc/systemd/system/mistral_engine.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/heat_api_cfn.service to /etc/systemd/system/heat_api_cfn.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/tripleo_ui.service to /etc/systemd/system/tripleo_ui.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/logrotate_crond.service to /etc/systemd/system/logrotate_crond.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/swift_proxy.service to /etc/systemd/system/swift_proxy.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/glance_api.service to /etc/systemd/system/glance_api.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/nova_api.service to /etc/systemd/system/nova_api.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/nova_metadata.service to /etc/systemd/system/nova_metadata.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_api.service to /etc/systemd/system/ironic_api.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/neutron_dhcp.service to /etc/systemd/system/neutron_dhcp.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/neutron_l3_agent.service to /etc/systemd/system/neutron_l3_agent.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/neutron_ovs_agent.service to /etc/systemd/system/neutron_ovs_agent.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/mistral_api.service to /etc/systemd/system/mistral_api.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_neutron_agent.service to /etc/systemd/system/ironic_neutron_agent.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_conductor.service to /etc/systemd/system/ironic_conductor.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_pxe_tftp.service to /etc/systemd/system/ironic_pxe_tftp.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_pxe_http.service to /etc/systemd/system/ironic_pxe_http.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_inspector.service to /etc/systemd/system/ironic_inspector.service.",
2018-11-15 01:41:29 | "Created symlink from /etc/systemd/system/multi-user.target.wants/ironic_inspector_dnsmasq.service to /etc/systemd/system/ironic_inspector_dnsmasq.service."
2018-11-15 01:41:29 | ]
2018-11-15 01:41:29 | }
2018-11-15 01:41:29 | ]
2018-11-15 01:41:29 | ]
From other logs: http://logs.openstack.org/45/560445/193/check/tripleo-ci-centos-7-containers-multinode/1f1369f/logs/undercloud/var/log/extra/errors.txt.gz#_2018-11-15_01_40_38_974
we found this 2018-11-15 01:40:38.973 ERROR /var/log/paunch.log: 38112 ERROR paunch [ ] Error running ['podman', 'exec', '--user=root', 'keystone', 'pkill', '--signal', 'USR1', 'httpd']. [125]
2018-11-15 01:40:38.973 ERROR /var/log/paunch.log: 38112 ERROR paunch [ ] stdout:
2018-11-15 01:40:38.974 ERROR /var/log/paunch.log: 38112 ERROR paunch [ ] stderr: timed out waiting for runtime to create pidfile for exec session in container 9d042e4b0fdd47efef90d902ae515c84a5377e82516416024338916b42379d4a: timed out waiting for file /var/run/containers/storage/overlay-containers/9d042e4b0fdd47efef90d902ae515c84a5377e82516416024338916b42379d4a/userdata/exec_pid_f7c0fa93a27fa552d4ab58929929fdef14c113796d6d488995f7baf59c9617b2: internal libpod error
The podman got timed out while executing keystone container.
For more info check these two issues:
https://github.com/containers/libpod/issues/1816
https://github.com/containers/libpod/issues/1495
It pops up randomly but good to investigated
Podman version used: http://logs.openstack.org/45/560445/193/check/tripleo-ci-centos-7-containers-multinode/1f1369f/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz
It sounds related to https:/
Maybe the code changed in the meantime and we hit it again.