nova kilo->liberty ceph configdrive upgrade fails
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
melanie witt | ||
Liberty |
Fix Released
|
High
|
melanie witt | ||
Mitaka |
Fix Released
|
High
|
melanie witt |
Bug Description
Using CEPH RBD as our ephemeral drive led to an issue when upgrading from Kilo to Liberty. Our environment has "force_config_drive = True".
In Icehouse, Juno, and Kilo, this uses an ISO 9660 image created in /var/lib/
However, in Liberty, if using CEPH RBD for ephemeral, there is a switch to putting this in rbe like this:
rbd:instances/
While this works GREAT for new VMs, it is problematic with existing VMs as not all transition states were considered. In particular, if you do a
nova stop $UUID
followed by a
nova start $UUID
you will find your instance still in the stopped state. There is something in the start code that ASSUMES that the new rbd format will be in place (but it doesn't actually create it.)
There is a work around if you find instances in that state, simply cold migrate them with
nova migrate $UUID
which redoes the config.drive plumbing and creates the rbd:instances/
Our permanent work around has been to prepopulate the rbd via a script though getting this bug fixed would be much better.
Liberty is a stable release and this is a loss of service type of bug so should get fixed. Not clear if this is also an issue (likely so) in Mitaka/Newton as we haven't got an environment yet to test it, but presumably with long running VMs from early config drive, it would also exist in Mitaka.
Specifics:
Liberty Nova
nova:12.
CEPH:
0.94.6-1trusty
Host OS:
Ubuntu Trusty
summary: |
- nova kilo liberty ceph configdrive upgrade + nova kilo->liberty ceph configdrive upgrade fails |
Changed in nova: | |
assignee: | nobody → melanie witt (melwitt) |
tags: | added: liberty-backport-potential mitaka-backport-potential |
Possibly related:
https:/ /bugs.launchpad .net/nova/ +bug/1303714