Upgrade from bionic-rocky to bionic-stein fails - no available hosts

Bug #1846781 reported by Andrew McLeod
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Neutron Open vSwitch Charm
Fix Released
Critical
Frode Nordahl
OpenStack Nova Cloud Controller Charm
Invalid
Undecided
Unassigned
OpenStack Nova Compute Charm
Fix Released
Critical
Frode Nordahl

Bug Description

After the upgrade from bionic-rocky to bionic-stein, you cannot launch a new instance. This is due to the listed hypervisors. Deleting the 'old' aka 'down' hypervisors allows the new ones to be registered with the placement api and instances can be launched

| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |

| 1 | juju-284597-admcleod-16.project.serverstack | QEMU | 10.5.0.71 | down |
| 2 | juju-284597-admcleod-18.project.serverstack | QEMU | 10.5.0.41 | down |
| 3 | juju-284597-admcleod-17.project.serverstack | QEMU | 10.5.0.38 | down |
| 4 | juju-284597-admcleod-18.project.serverstack | QEMU | 10.5.0.41 | up |
| 5 | juju-284597-admcleod-17.project.serverstack | QEMU | 10.5.0.38 | up |
| 6 | juju-284597-admcleod-16.project.serverstack | QEMU | 10.5.0.71 | up |

| ID | Hypervisor hostname | State | Status |

| 75481283-6858-4627-8081-1909338e3e14 | juju-284597-admcleod-16.project.serverstack | down | enabled |
| 5d09e737-0218-48ca-bf66-f917e357caab | juju-284597-admcleod-18.project.serverstack | down | disabled |
| 3dbaf0d0-a2d3-4925-a68e-25a845f0d5c4 | juju-284597-admcleod-17.project.serverstack | down | disabled |
| 692318fc-7450-43cf-aa32-3c57fb534c6c | juju-284597-admcleod-18.project.serverstack | up | enabled |
| 66c8dfd5-3619-4fe2-9c2b-ee0c58209cbf | juju-284597-admcleod-17.project.serverstack | up | enabled |
| 506b8ae3-97d3-46cc-9dd1-36a4ca684a0a | juju-284597-admcleod-16.project.serverstack | up | enabled |

$ openstack compute service list

| ID | Binary | Host | Zone | Status | State | Updated At |

| 1 | nova-scheduler | juju-284597-admcleod-15 | internal | enabled | up | 2019-10-04T14:34:25.000000 |
| 2 | nova-conductor | juju-284597-admcleod-15 | internal | enabled | up | 2019-10-04T14:34:16.000000 |
| 4 | nova-compute | juju-284597-admcleod-16 | nova | enabled | down | 2019-10-04T13:03:12.000000 |
| 5 | nova-compute | juju-284597-admcleod-18 | nova | disabled | down | 2019-10-04T13:00:29.000000 |
| 6 | nova-compute | juju-284597-admcleod-17 | nova | disabled | down | 2019-10-04T13:00:29.000000 |
| 9 | nova-compute | juju-284597-admcleod-18.project.serverstack | nova | enabled | up | 2019-10-04T14:34:16.000000 |
| 10 | nova-compute | juju-284597-admcleod-17.project.serverstack | nova | enabled | up | 2019-10-04T14:34:22.000000 |
| 11 | nova-compute | juju-284597-admcleod-16.project.serverstack | nova | enabled | up | 2019-10-04T14:34:16.000000 |

Revision history for this message
Frode Nordahl (fnordahl) wrote : Re: [Bug 1846781] [NEW] Upgrade from bionic-rocky to bionic-stein fails - no available hosts

This was most likely introduced by
https://review.opendev.org/#/q/topic:bug/1839300+(status:open+OR+status:merged)

The assumption made there was that if you would upgrade, you would also
come from a previous charm version.

I guess that is an incorrect assumption.

We could fix it by setting the initial kv key based on both openstack and
charm version, not just charm version.

Frode Nordahl (fnordahl)
Changed in charm-nova-compute:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Frode Nordahl (fnordahl)
Changed in charm-neutron-openvswitch:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Frode Nordahl (fnordahl)
milestone: none → 19.10
Changed in charm-nova-compute:
milestone: none → 19.10
Changed in charm-nova-cloud-controller:
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-compute (master)

Fix proposed to branch: master
Review: https://review.opendev.org/687281

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-openvswitch (master)

Fix proposed to branch: master
Review: https://review.opendev.org/687284

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-openvswitch (master)

Reviewed: https://review.opendev.org/687284
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=ed4ca7b1c440e1be8c8d4ca02b43c5bdd2dccee7
Submitter: Zuul
Branch: master

commit ed4ca7b1c440e1be8c8d4ca02b43c5bdd2dccee7
Author: Frode Nordahl <email address hidden>
Date: Tue Oct 8 14:27:14 2019 +0200

    Fix gate for enabling FQDN agent registration

    If a new version of the charm is used to install a version of
    OpenStack prior to Stein, do not enable the FQDN registration.

    Change-Id: I64afa582cdfadafcd249ddfbeec267ba610d166a
    Closes-Bug: #1846781

Changed in charm-neutron-openvswitch:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-compute (master)

Reviewed: https://review.opendev.org/687281
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=2169de6fcd4396345d45e7315ef0c57e7b906fef
Submitter: Zuul
Branch: master

commit 2169de6fcd4396345d45e7315ef0c57e7b906fef
Author: Frode Nordahl <email address hidden>
Date: Tue Oct 8 14:13:51 2019 +0200

    Fix gate for enabling FQDN agent registration

    If a new version of the charm is used to install a version of
    OpenStack prior to Stein, do not enable the FQDN registration.

    Change-Id: Ib86d6a48ee34c9efc42b4455ac669ef0cb8dc3bb
    Closes-Bug: #1846781

Changed in charm-nova-compute:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-nova-compute:
status: Fix Committed → Fix Released
Changed in charm-neutron-openvswitch:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.