Nova compute needlessly depends on hostname for live migration url

[Kevin Tibi]

when you add a new compute to an already existing platform, kolla does not update the /etc/hosts used by old compute and libvirt containers.

So live migration fail because it can't resolve the name of the new compute.

Additionally, live migration interface is ignored even in non-TLS setup, and may fail live migration if hostname does not point to migration interface address.

[Michal Nasiadka (mnasiadka)]

kolla-ansible baremetal role (bootstrap-servers) is generating /etc/hosts entries, therefore it would need some refactor to generate /etc/hosts on all servers...

[Mark Goddard (mgoddard)]

Why not just re-run bootstrap-servers? Or do we need a more fine-grained command?

[Zijian Guo (zijianguo)]

After re-generating /etc/hosts. The most important thing is the file '/etc/hosts' of the container 'nova_compute' does not refresh until the container is restarted.

[Ross Martyn (rossmartyn04)]

We're also seeing this.

Is there an easy way to trigger the restart of specific containers through KA (or Kayobe) without risking other changes as noted on the KA documentation under bootstrap-servers (Subsequent bootstrap considerations)?

Michal, Zljian, how did you combat this when scaling up your environments?

[Zijian Guo (zijianguo)]

Hi, Ross.

I suggest splitting the Generate /etc/hosts action from bootstrap. Restart the necessary containers after each update, such as nova_compute,nova_libvirt,nova_ssh, and so on.

[Radosław Piliszek (yoctozepto)]

I think the not-updated-/etc/hosts-issue could be refactored here as we are in fact fixing a more general issue while leaving name resolution broken still.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/715494
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=628c27ce9e693902c5cabedc7f254f2a1229a195
Submitter: Zuul
Branch: master

commit 628c27ce9e693902c5cabedc7f254f2a1229a195
Author: John Garbutt <email address hidden>
Date: Fri Mar 27 17:37:51 2020 +0000

    Fix live migration to use migration int. address

    In kolla ansible we typically configure services to communicate via IP
    addresses rather than hostnames. One accidental exception to this was
    live migration, which used the hostname of the destination even when
    not required (i.e. TLS not being used for libvirt).

    To make such hostnames work, k-a adds entries to /etc/hosts in the
    bootstrap-servers command. Alternatively users may provide DNS.

    One problem with using /etc/hosts is that, if a new compute host is
    added to the cloud, or an IP address is changed, that will not be
    reflected in the /etc/hosts file of other hosts. This would cause live
    migration to the new host from an old host to fail, as the name cannot
    be resolved.

    The workaround for this was to update the /etc/hosts file (perhaps via
    bootstrap-servers) on all hosts after adding new compute hosts. Then the
    nova_libvirt container had to be restarted to pick up the change.

    Similarly, if user has overridden the migration_interface, the used
    hostname could point to a wrong address on which libvirt would not
    listen.

    This change adds the live_migration_inbound_addr option to nova.conf. If
    TLS is not in use for libvirt, this will be set to the IP address of the
    host on the migration network. If TLS is enabled for libvirt,
    live_migration_inbound_addr will be set to migration_hostname, since
    certificates will typically reference the hostname rather than the
    host's IP. With libvirt TLS enabled, DNS is recommended to avoid the
    /etc/hosts issue which is likely the case in production deployments.

    Change-Id: I0201b46a9fbab21433a9f53685131aeb461543a8
    Closes-Bug: #1729566

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/719416

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/719422

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/719422
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f174ec063676bea17e17c537282eb9eb08e89e70
Submitter: Zuul
Branch: stable/stein

commit f174ec063676bea17e17c537282eb9eb08e89e70
Author: John Garbutt <email address hidden>
Date: Fri Mar 27 17:37:51 2020 +0000

    Fix live migration to use migration int. address

    In kolla ansible we typically configure services to communicate via IP
    addresses rather than hostnames. One accidental exception to this was
    live migration, which used the hostname of the destination even when
    not required (i.e. TLS not being used for libvirt).

    To make such hostnames work, k-a adds entries to /etc/hosts in the
    bootstrap-servers command. Alternatively users may provide DNS.

    One problem with using /etc/hosts is that, if a new compute host is
    added to the cloud, or an IP address is changed, that will not be
    reflected in the /etc/hosts file of other hosts. This would cause live
    migration to the new host from an old host to fail, as the name cannot
    be resolved.

    The workaround for this was to update the /etc/hosts file (perhaps via
    bootstrap-servers) on all hosts after adding new compute hosts. Then the
    nova_libvirt container had to be restarted to pick up the change.

    Similarly, if user has overridden the migration_interface, the used
    hostname could point to a wrong address on which libvirt would not
    listen.

    This change adds the live_migration_inbound_addr option to nova.conf. If
    TLS is not in use for libvirt, this will be set to the IP address of the
    host on the migration network. If TLS is enabled for libvirt,
    live_migration_inbound_addr will be set to migration_hostname, since
    certificates will typically reference the hostname rather than the
    host's IP. With libvirt TLS enabled, DNS is recommended to avoid the
    /etc/hosts issue which is likely the case in production deployments.

    Change-Id: I0201b46a9fbab21433a9f53685131aeb461543a8
    Closes-Bug: #1729566
    (cherry picked from commit 628c27ce9e693902c5cabedc7f254f2a1229a195)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/719416
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=03784673557a0b2c5a2dba1ce06966dcd66473a8
Submitter: Zuul
Branch: stable/train

commit 03784673557a0b2c5a2dba1ce06966dcd66473a8
Author: John Garbutt <email address hidden>
Date: Fri Mar 27 17:37:51 2020 +0000

    Fix live migration to use migration int. address

    In kolla ansible we typically configure services to communicate via IP
    addresses rather than hostnames. One accidental exception to this was
    live migration, which used the hostname of the destination even when
    not required (i.e. TLS not being used for libvirt).

    To make such hostnames work, k-a adds entries to /etc/hosts in the
    bootstrap-servers command. Alternatively users may provide DNS.

    One problem with using /etc/hosts is that, if a new compute host is
    added to the cloud, or an IP address is changed, that will not be
    reflected in the /etc/hosts file of other hosts. This would cause live
    migration to the new host from an old host to fail, as the name cannot
    be resolved.

    The workaround for this was to update the /etc/hosts file (perhaps via
    bootstrap-servers) on all hosts after adding new compute hosts. Then the
    nova_libvirt container had to be restarted to pick up the change.

    Similarly, if user has overridden the migration_interface, the used
    hostname could point to a wrong address on which libvirt would not
    listen.

    This change adds the live_migration_inbound_addr option to nova.conf. If
    TLS is not in use for libvirt, this will be set to the IP address of the
    host on the migration network. If TLS is enabled for libvirt,
    live_migration_inbound_addr will be set to migration_hostname, since
    certificates will typically reference the hostname rather than the
    host's IP. With libvirt TLS enabled, DNS is recommended to avoid the
    /etc/hosts issue which is likely the case in production deployments.

    Change-Id: I0201b46a9fbab21433a9f53685131aeb461543a8
    Closes-Bug: #1729566
    (cherry picked from commit 628c27ce9e693902c5cabedc7f254f2a1229a195)