[ubuntu host] Handle duplicated hostnames in /etc/hosts file (127.0.1.1 problem)

[Niklas Hagman]

As default in Debian and Ubuntu there is a line in /etc/hosts file that make sure the device is always answering on it's hostname.

127.0.1.1 control1.example.com control1

https://serverfault.com/questions/363095/why-does-my-hostname-appear-with-the-address-127-0-1-1-rather-than-127-0-0-1-in
This creates a problem for kolla-ansible because it adds lines like this in /etc/hosts file
# BEGIN ANSIBLE GENERATED HOSTS
192.168.38.14 control1
192.168.38.15 control2
192.168.38.18 control3
192.168.38.13 compute1
192.168.38.16 compute2
192.168.38.20 compute3
192.168.38.12 storage1
192.168.38.17 storage2
192.168.38.19 storage3
# END ANSIBLE GENERATED HOSTS

The problem here is that connection will be made against 127.0.1.1 instead of 192.168.38.14 where for example rabbitmq is listening.
The solution for me is not clear but it needs to be handled for a nicer user experience when installing kolla-ansible.

Currently am having problem with rabbitmq docker container restarting without telling me why.
RUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first node] says "non-zero return code" and fails.

[Radosław Piliszek (yoctozepto)]

We add this at the bottom and resolv seems to pick up the first occurence.

Hackish fix is to add it at the top of the file. :-)

Less hackish is to actually check lines for each host and fix them properly.

[Mark Goddard (mgoddard)]

We could do a lineinfile with state=absent.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/685233

[Dincer Celik (dincercelik)]

@Niklas: have you used MAAS to deploy the OS'es?

[Niklas Hagman (blinkiz)]

@Dincer: Yes I have used MAAS to deploy the hosts. Version 2.6.0.

[Dincer Celik (dincercelik)]

From Ubuntu 18.04 and above, /etc/hosts is managed via cloud-init if you use MAAS, and cloud-init will overwrite it at every boot, from /etc/cloud/templates/hosts.debian.tmpl. So you need to disable this too.

MAAS puts a file to /etc/cloud/cloud.cfg.d/90_dpkg_local_cloud_config.cfg, which includes "manage_etc_hosts: true", and needs to be changed to false.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685233
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=0b24a0f2f0880c35368581e3acc529672da665f8
Submitter: Zuul
Branch: master

commit 0b24a0f2f0880c35368581e3acc529672da665f8
Author: Mark Goddard <email address hidden>
Date: Fri Sep 27 09:20:44 2019 +0100

    Remove /etc/hosts entries pointing hostname to localhost and prevent
    cloud-init to manage /etc/hosts

    1) Ubuntu includes a line in /etc/hosts that makes the local hostname and
    nodename (if different) point to 127.0.1.1. This can break RabbitMQ,
    which expects the hostname to resolve to the API network address.

    2) The distribution might come with cloud-init installed, and manage_etc_hosts
    configuration enabled. If so, it will override the file /etc/hosts from cloud-init
    templates at every boot, which will break RabbitMQ.

    This change fixes these issues.

    Change-Id: I53261d0403b983ab419bd44e705b89f7b7a1c316
    Closes-Bug: #1837699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/688329

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/688329
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=2349364a210d5b97a442c0c0a3925fb7f820e42e
Submitter: Zuul
Branch: stable/stein

commit 2349364a210d5b97a442c0c0a3925fb7f820e42e
Author: Mark Goddard <email address hidden>
Date: Fri Sep 27 09:20:44 2019 +0100

    Remove /etc/hosts entries pointing hostname to localhost and prevent
    cloud-init to manage /etc/hosts

    1) Ubuntu includes a line in /etc/hosts that makes the local hostname and
    nodename (if different) point to 127.0.1.1. This can break RabbitMQ,
    which expects the hostname to resolve to the API network address.

    2) The distribution might come with cloud-init installed, and manage_etc_hosts
    configuration enabled. If so, it will override the file /etc/hosts from cloud-init
    templates at every boot, which will break RabbitMQ.

    This change fixes these issues.

    Change-Id: I53261d0403b983ab419bd44e705b89f7b7a1c316
    Closes-Bug: #1837699
    (cherry picked from commit 0b24a0f2f0880c35368581e3acc529672da665f8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/706192

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706192
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=3ca1b012f1972c38f75e91ff8d99d4f6a0a974d9
Submitter: Zuul
Branch: stable/rocky

commit 3ca1b012f1972c38f75e91ff8d99d4f6a0a974d9
Author: Mark Goddard <email address hidden>
Date: Fri Sep 27 09:20:44 2019 +0100

    Remove /etc/hosts entries pointing hostname to localhost and prevent
    cloud-init to manage /etc/hosts

    1) Ubuntu includes a line in /etc/hosts that makes the local hostname and
    nodename (if different) point to 127.0.1.1. This can break RabbitMQ,
    which expects the hostname to resolve to the API network address.

    2) The distribution might come with cloud-init installed, and manage_etc_hosts
    configuration enabled. If so, it will override the file /etc/hosts from cloud-init
    templates at every boot, which will break RabbitMQ.

    This change fixes these issues.

    Change-Id: I53261d0403b983ab419bd44e705b89f7b7a1c316
    Closes-Bug: #1837699
    (cherry picked from commit 0b24a0f2f0880c35368581e3acc529672da665f8)

[Eddie Yen (aksn74)]

Hi, we tested the fix today and found there's little thing may going to correct.

In MAAS deployment, for example, will write the host's own hostname into 127.0.1.1
just like the beginning.

127.0.1.1 controller1.maas controller1

Problem is, Ansible will mark 'controller1.maas' as ansible_fqdn value, not
ansible_nodename sometimes.

# ansible -i multinode -m setup controller1 | grep 'controller1.maas'
    "ansible_fqdn": "controller1.maas"

If search to ansible_nodename, it will become same as ansible_hostname.

# ansible -i multinode -m setup controller | grep 'ansible_nodename'
    "ansible_nodename": "controller1"

This may cause kolla-ansible to keep 127.0.1.1 setting after bootstrap-servers.
Should we change the variable? Or add another absent task to "double check"?

[Radosław Piliszek (yoctozepto)]

Eddie -> https://bugs.launchpad.net/kolla-ansible/+bug/1862739

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.2.1 release.