Reconfigure of nova with changed DB or MQ hostname or password creates a duplicate cell

[Mark Goddard]

Description
===========

If the hostname/IP or password of the nova database or message queue is changed, kolla-ansible will create a new cell using the new connection strings during kayobe overcloud service reconfigure.

This has a few consequences:

* Duplicate entries returned in nova hypervisor-list, nova usage and nova service-list.
* The nova usage and nova usage-list commands do not terminate, and loops forever calling the same API request.
* The compute overview page in horizon does not load correctly, and the horizon server process spins.

This was seen when changing from using an IP to an FQDN for the external and internal API VIPs.

Steps to reproduce
==================

* Deploy a cloud with nova enabled and kolla_internal_fqdn == kolla_internal_vip_address.
* Create one or more instances.
* Change kolla_internal_fqdn to use a hostname that resolves to kolla_internal_vip_address (modify /etc/hosts if no DNS server is available).
* Reconfigure the cloud using kolla-ansible reconfigure.

Expected results
================

The nova cell should be updated with the new DB connection string.

Actual results
==============

A new cell is created with the new DB connection string.

The cells can be checked via 'docker exec -it nova_api nova-manage cell_v2 list_cells'. There should be two cells - cell0 and one other.

Environment
===========

Seen on pike, CentOS 7.3 on the hosts and containers. Ironic is used as the nova hypervisor.

[Mark Goddard (mgoddard)]

During deploy and reconfigure, kolla-ansible always attempts to create a cell [1]. The corresponding nova-manage command uses the DB and MQ connection strings as the uniqueness check for cells [2]. Therefore, if either of these changes, create_cell will create a new cell.

A better approach here would be to use create_cell only when no cell other than cell0 exists. Otherwise, we should use update_cell to set the DB and transport connection strings.

[1] https://github.com/openstack/kolla-ansible/blob/stable/pike/ansible/roles/nova/tasks/simple_cell_setup.yml#L17
[2] https://github.com/openstack/nova/blob/stable/pike/nova/cmd/manage.py#L1614

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/663398

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663398
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=19b8dbe460315c5da23c33aa57c967d75d7f2018
Submitter: Zuul
Branch: master

commit 19b8dbe460315c5da23c33aa57c967d75d7f2018
Author: Pierre Riteau <email address hidden>
Date: Thu Jun 6 18:10:06 2019 +0100

    Stop duplicating Nova cells

    Check if a base Nova cell already exists before calling `nova-manage
    cell_v2 create_cell`, which would otherwise create a duplicate cell when
    the transport URL or database connection change.

    If a base cell already exists but the connection values have changed, we
    now call `nova-manage cell_v2 update_cell` instead. This is only
    possible if a duplicate cell has not yet been created. If one already
    exists, we print a warning inviting the operator to perform a manual
    cleanup. We don't use a hard fail to avoid an abrupt change of behavior
    if this is backported to stable branches.

    Change-Id: I7841ce0cff08e315fd7761d84e1e681b1a00d43e
    Closes-Bug: #1734872

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/665212

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/665214

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/665618

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/665618
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f9948e0a31e02457db186c27e874175cb9685932
Submitter: Zuul
Branch: stable/stein

commit f9948e0a31e02457db186c27e874175cb9685932
Author: Pierre Riteau <email address hidden>
Date: Thu Jun 6 18:10:06 2019 +0100

    Stop duplicating Nova cells

    Check if a base Nova cell already exists before calling `nova-manage
    cell_v2 create_cell`, which would otherwise create a duplicate cell when
    the transport URL or database connection change.

    If a base cell already exists but the connection values have changed, we
    now call `nova-manage cell_v2 update_cell` instead. This is only
    possible if a duplicate cell has not yet been created. If one already
    exists, we print a warning inviting the operator to perform a manual
    cleanup. We don't use a hard fail to avoid an abrupt change of behavior
    if this is backported to stable branches.

    Change-Id: I7841ce0cff08e315fd7761d84e1e681b1a00d43e
    Closes-Bug: #1734872
    (cherry picked from commit 19b8dbe460315c5da23c33aa57c967d75d7f2018)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/665212
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=3458a7b28ac0cfec9f1106ab1e3a2b83873fe97a
Submitter: Zuul
Branch: stable/rocky

commit 3458a7b28ac0cfec9f1106ab1e3a2b83873fe97a
Author: Pierre Riteau <email address hidden>
Date: Thu Jun 6 18:10:06 2019 +0100

    Stop duplicating Nova cells

    Check if a base Nova cell already exists before calling `nova-manage
    cell_v2 create_cell`, which would otherwise create a duplicate cell when
    the transport URL or database connection change.

    If a base cell already exists but the connection values have changed, we
    now call `nova-manage cell_v2 update_cell` instead. This is only
    possible if a duplicate cell has not yet been created. If one already
    exists, we print a warning inviting the operator to perform a manual
    cleanup. We don't use a hard fail to avoid an abrupt change of behavior
    if this is backported to stable branches.

    Change-Id: I7841ce0cff08e315fd7761d84e1e681b1a00d43e
    Closes-Bug: #1734872
    (cherry picked from commit 19b8dbe460315c5da23c33aa57c967d75d7f2018)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/665214
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=f6cfee3201d0067ee500dd507924038315ac6efd
Submitter: Zuul
Branch: stable/queens

commit f6cfee3201d0067ee500dd507924038315ac6efd
Author: Pierre Riteau <email address hidden>
Date: Thu Jun 6 18:10:06 2019 +0100

    Stop duplicating Nova cells

    Check if a base Nova cell already exists before calling `nova-manage
    cell_v2 create_cell`, which would otherwise create a duplicate cell when
    the transport URL or database connection change.

    If a base cell already exists but the connection values have changed, we
    now call `nova-manage cell_v2 update_cell` instead. This is only
    possible if a duplicate cell has not yet been created. If one already
    exists, we print a warning inviting the operator to perform a manual
    cleanup. We don't use a hard fail to avoid an abrupt change of behavior
    if this is backported to stable branches.

    Change-Id: I7841ce0cff08e315fd7761d84e1e681b1a00d43e
    Closes-Bug: #1734872
    (cherry picked from commit 19b8dbe460315c5da23c33aa57c967d75d7f2018)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.2.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/731452

[OpenStack Infra (hudson-openstack)]

Change abandoned by Yongjun Bai ([e-mail]) on branch: stable/pike
Review: https://review.opendev.org/731452