RabbitMQ fails to add guest user during bootstrap of Bifrost

[sayantani]

RabbitMQ fails to remove the guest user during bootstrapping of Bifrost Service. Please reference the following error:

http://paste.openstack.org/raw/594306/

This occurs when the hostname is not added or not added correctly to the /etc/hosts file. If the following entry exists in the /etc/hosts file, RabbitMQ fails. (ubuntu-14 is the hostname in the example below)

127.0.0.1 localhost ubuntu-14

The correct entry should be

127.0.0.1 ubuntu-14 localhost

This is handed by bifrost (https://github.com/openstack/bifrost/blob/master/playbooks/roles/bifrost-ironic-install/tasks/bootstrap.yml) but it fails when done inside a container. The reason it fails is because Docker bind mounts /etc/hosts into the container from a volume. This prevents atomic renames which will prevent ansible from fixing the /etc/hosts file automatically inside the container.

A task should be added to automatically add the hostname to the /etc/hosts file of the deployment host before it is mounted to the container from a volume.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/529543
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=19d4913c459aa59abde0a8c6ee92d8bd1e0fef8e
Submitter: Zuul
Branch: master

commit 19d4913c459aa59abde0a8c6ee92d8bd1e0fef8e
Author: Mark Goddard <email address hidden>
Date: Tue Mar 28 10:28:08 2017 +0100

    Workaround for Bifrost being unable to write to /etc/hosts

    Bifrost cannot modify /etc/hosts from within a container, so add a
    host entry during bootstrap.

    Change-Id: Ied378b4dd755788e75ad1814cecb5700732ba83e
    Closes-Bug: #1661009

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.0.0.0rc1 release candidate.

[Mark Goddard (mgoddard)]

Seeing this issue again with bifrost 5.0.0. The logic around handling /etc/hosts in bifrost was changed in [1], so kolla-ansible needs to handle this new logic. Ideally bifrost would be less eager to modify /etc/hosts.

[1] https://github.com/openstack/bifrost/commit/60b9a9917e987e321b97b0f611fc2c77813be8b0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/549790

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/549798

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/549790
Reason: Need to submit to master first.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549798
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=376c607f376f6102ad3d2d371226832fb759c386
Submitter: Zuul
Branch: master

commit 376c607f376f6102ad3d2d371226832fb759c386
Author: Mark Goddard <email address hidden>
Date: Fri Mar 2 12:20:03 2018 +0000

    Workaround for Bifrost being unable to write to /etc/hosts

    Bifrost cannot modify /etc/hosts from within a container, so add a
    host entry during bootstrap.

    This was previously fixed by Ied378b4dd755788e75ad1814cecb5700732ba83e
    but the logic in bifrost was changed [1], making it out of sync with
    kolla-ansible, and triggering a write to /etc/hosts. This change applies
    the new logic in bifrost to kolla-ansible, ensuring that the file will
    not need to change in the container.

    Longer term we should look at ways to make bifrost less eager to modify
    /etc/hosts, accepting any file that will work for RabbitMQ.

    [1]
    https://github.com/openstack/bifrost/commit/60b9a9917e987e321b97b0f611fc2c77813be8b0

    Change-Id: I0ee05feae3630435b2ec52cfeddf33647a974ee0
    Closes-Bug: #1661009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549790
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f56081ff53a6de7da5010b424efaa856e751f0d5
Submitter: Zuul
Branch: stable/queens

commit f56081ff53a6de7da5010b424efaa856e751f0d5
Author: Mark Goddard <email address hidden>
Date: Fri Mar 2 12:20:03 2018 +0000

    Workaround for Bifrost being unable to write to /etc/hosts

    Bifrost cannot modify /etc/hosts from within a container, so add a
    host entry during bootstrap.

    This was previously fixed by Ied378b4dd755788e75ad1814cecb5700732ba83e
    but the logic in bifrost was changed [1], making it out of sync with
    kolla-ansible, and triggering a write to /etc/hosts. This change applies
    the new logic in bifrost to kolla-ansible, ensuring that the file will
    not need to change in the container.

    Longer term we should look at ways to make bifrost less eager to modify
    /etc/hosts, accepting any file that will work for RabbitMQ.

    [1]
    https://github.com/openstack/bifrost/commit/60b9a9917e987e321b97b0f611fc2c77813be8b0

    Change-Id: I0ee05feae3630435b2ec52cfeddf33647a974ee0
    Closes-Bug: #1661009
    (cherry picked from commit 376c607f376f6102ad3d2d371226832fb759c386)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: master
Review: https://review.opendev.org/419196
Reason: Very old