Nova hard reboot fails to mount logical volume (LVM + libvirt-lxc)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
Discovered with the experimental libvirt-lxc tempest gate job initially, but pared down to an easier test using Devstack and a node like that which is used in our CI for devstack-gate tests. Here's an etherpad with many details: https:/
The gist of it is there appears to be a bug where trying to hard reboot a libvirtLXC instance in nova, when using LVM storage backend, when nova goes to try and mount the LV, and it will sometimes fail with:
```
Traceback (most recent call last):
File "/usr/local/
incoming.
File "/usr/local/
return self._do_
File "/usr/local/
result = func(ctxt, **new_args)
File "/opt/stack/
payload)
File "/usr/local/
self.
File "/usr/local/
six.
File "/opt/stack/
return f(self, context, *args, **kw)
File "/opt/stack/
LOG.
File "/usr/local/
self.
File "/usr/local/
six.
File "/opt/stack/
return function(self, context, *args, **kwargs)
File "/opt/stack/
return function(self, context, *args, **kwargs)
File "/opt/stack/
kwargs[
File "/usr/local/
self.
File "/usr/local/
six.
File "/opt/stack/
return function(self, context, *args, **kwargs)
File "/opt/stack/
self.
File "/usr/local/
self.
File "/usr/local/
six.
File "/opt/stack/
bad_
File "/opt/stack/
block_
File "/opt/stack/
vifs_
File "/opt/stack/
block_
File "/usr/lib/
return self.gen.next()
File "/opt/stack/
block_
File "/opt/stack/
container_
File "/opt/stack/
raise exception.
NovaException:
--
Failed to mount filesystem: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/
Exit code: 32
Stdout: u''
Stderr: u'mount: /dev/mapper/
```
I can recreate this fairly consistently in devstack using the same form factor as nodes in our CI devstack-gate:
local.conf:
```
[[local|localrc]]
LIBVIRT_TYPE=lxc
NOVA_BACKEND=LVM
```
```
$ ./stack.sh
```
...
```
$ source openrc
$ for i in `seq 1 1 10`; do ( nova boot --image "cirros-
$ for i in `seq 1 1 10`; do ( nova reboot --hard test$i & ); done
```
After doing so, some of the instances should go into ERROR with the Traceback above in the compute log. The volume of instances is meant to perturb the issue more reliably. This doesn't always happen, however it has happened several times when I've just spun up one instance and tried.
I am running HEAD in Devstack when I see this problem.
Note: On this set up, nova's soft reboot is falling through to hard reboot, I believe due to this bug: https:/
description: | updated |
description: | updated |
summary: |
- Nova reboot fails to mount logical volume (LVM + libvirt-lxc) + Nova hard reboot fails to mount logical volume (LVM + libvirt-lxc) |
description: | updated |
description: | updated |
tags: | added: lxc |
Changed in nova: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in nova: | |
assignee: | nobody → Thomas Maddox (thomas-maddox) |
Changed in nova: | |
assignee: | Thomas Maddox (thomas-maddox) → nobody |