[master][promotion] centos 7 container build job deploying undercloud with usr/lib/rabbitmq/bin/rabbitmq-server: No such file or directory

[chandan kumar]

Periodic master promotion centos7 container build job got failed while undercloud at step 5:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/f56417e/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-02-06_05_58_14

Error running ['podman', 'run', '--name', 'nova_cellv2_discover_hosts', '--label', 'config_id=tripleo_step5', '--label', 'container_name=nova_cellv2_discover_hosts', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 1, \"command\": \"/docker-config-scripts/pyshim.sh /docker-config-scripts/nova_cell_v2_discover_host.py\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/nova/etc/my.cnf.d/:/etc/my.cnf.d/:ro\", \"/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/docker-config-scripts/:/docker-config-scripts/\"], \"image\": \"192.168.24.1:8787/tripleomaster/centos-binary-nova-compute-ironic:tripleo-ci-testing-updated-20190206052058\", \"detach\": false, \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/docker-config-scripts/:/docker-config-scripts/', '192.168.24.1:8787/tripleomaster/centos-binary-nova-compute-ironic:tripleo-ci-testing-updated-20190206052058', '/docker-config-scripts/pyshim.sh', '/docker-config-scripts/nova_cell_v2_discover_host.py']. [1]",
2019-02-06 05:58:14 | "stdout: (cellv2) Waiting for service to register",
2019-02-06 05:58:14 | "(cellv2) Waiting for service to register",

While checking the rabbitmq podman container logs, we found that:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/f56417e/logs/undercloud/var/log/extra/podman/containers/rabbitmq/stdout.log.txt.gz

Running command: '/usr/lib/rabbitmq/bin/rabbitmq-server'
+ exec /usr/lib/rabbitmq/bin/rabbitmq-server
/usr/local/bin/kolla_start: line 18: /usr/lib/rabbitmq/bin/rabbitmq-server: No such file or directory

https://review.openstack.org/#/c/634405 adds the workaround for the rabbitmq-server might be causing the issue:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/635108

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635108
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=72041379dbabd08f2f56895df165760d25d980dd
Submitter: Zuul
Branch: master

commit 72041379dbabd08f2f56895df165760d25d980dd
Author: Quique Llorente <email address hidden>
Date: Wed Feb 6 08:38:59 2019 +0100

    Append rabbitmq_packages_append contents

    The review [1] override the fix later on the jinja template, this review
    replace the override by an append, and also check if there is nothing to
    append and add an empty list.

    Closes-Bug: #1814851

    [1] https://review.openstack.org/#/c/634405

    Change-Id: I35cbde7e3d44ce071a962cea278680bd5e453ec2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.4.0 release.