Description
===========
Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart.
==================
Running Openstack Ocata release with Virtuozzo KVM hypervisor on Vstorage backed by Cinder. A restart of the openstack-nova-compute service causes vzstorage mounts to be unreachable. The restart keeps VMs running using the mount but the filesystem is unreachable.
Before restart:
[user@node ~]$ sudo systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
Main PID: 3932 (nova-compute)
CGroup: /system.slice/openstack-nova-compute.service
├─3932 /usr/bin/python2 /usr/bin/nova-compute
├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpe2LS_d/privsep.sock
├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpOXZxyO/privsep.sock
└─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/vstorage/nova-openstackpoc.log.gz /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
We restart the service :
sudo systemctl restart openstack-nova-compute.service
After this we see:
[user@node ~]$ sudo systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
Main PID: 199753 (nova-compute)
CGroup: /system.slice/openstack-nova-compute.service
├─199753 /usr/bin/python2 /usr/bin/nova-compute
└─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpjDaH7y/privsep.sock
The pstorage-mount is not reinitialised by nova-compute.
Expected result
===============
The pstorage-mount is available after nova-compute restart.
Actual result
=============
The pstorage-mount is not reinitialised by nova-compute.
Nova mounts do not disappear after restart but are unreachable:
[user@node ~]$ sudo lsof | grep /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
lsof: WARNING: can't stat() fuse.vstorage file system /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
Output information may be incomplete.
qemu-kvm 7366 root 18u unknown /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
qemu-kvm 7366 7370 root 18u unknown /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
We currently implemented a workaround by adding
KillMode=process
to the openstack nova compute unit file override. This leave open the vzstorage mounts and keeps the VMs running.
Description
===========
Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart.
================== nova-compute service causes vzstorage mounts to be unreachable. The restart keeps VMs running using the mount but the filesystem is unreachable.
Running Openstack Ocata release with Virtuozzo KVM hypervisor on Vstorage backed by Cinder. A restart of the openstack-
Before restart: nova-compute. service nova-compute. service - OpenStack Nova Compute Server systemd/ system/ openstack- nova-compute. service; enabled; vendor preset: disabled) slice/openstack -nova-compute. service nova-compute nova/nova- dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ linux_bridge. privsep. vif_plug --privsep_sock_path /tmp/tmpe2LS_ d/privsep. sock nova/nova- dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick. privileged. default --privsep_sock_path /tmp/tmpOXZxyO/ privsep. sock vstorage/ nova-openstackp oc.log. gz /var/lib/ nova/mnt/ b2b894afb0fb3e4 7348888c87ad01e ee
[user@node ~]$ sudo systemctl status -l openstack-
● openstack-
Loaded: loaded (/usr/lib/
Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
Main PID: 3932 (nova-compute)
CGroup: /system.
├─3932 /usr/bin/python2 /usr/bin/
├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/
├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/
└─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/
We restart the service : nova-compute. service
sudo systemctl restart openstack-
After this we see:
[user@node ~]$ sudo systemctl status -l openstack- nova-compute. service nova-compute. service - OpenStack Nova Compute Server systemd/ system/ openstack- nova-compute. service; enabled; vendor preset: disabled) slice/openstack -nova-compute. service nova-compute nova/nova- dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_ linux_bridge. privsep. vif_plug --privsep_sock_path /tmp/tmpjDaH7y/ privsep. sock
● openstack-
Loaded: loaded (/usr/lib/
Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
Main PID: 199753 (nova-compute)
CGroup: /system.
├─199753 /usr/bin/python2 /usr/bin/
└─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/
The pstorage-mount is not reinitialised by nova-compute.
Expected result
===============
The pstorage-mount is available after nova-compute restart.
Actual result
=============
The pstorage-mount is not reinitialised by nova-compute.
Nova mounts do not disappear after restart but are unreachable:
[user@node ~]$ sudo lsof | grep /var/lib/ nova/mnt/ b2b894afb0fb3e4 7348888c87ad01e ee nova/mnt/ b2b894afb0fb3e4 7348888c87ad01e ee nova/mnt/ b2b894afb0fb3e4 7348888c87ad01e ee/volume- 7d555095- f36e-4673- ad10-30786d1052 70 (stat: Transport endpoint is not connected) nova/mnt/ b2b894afb0fb3e4 7348888c87ad01e ee/volume- 7d555095- f36e-4673- ad10-30786d1052 70 (stat: Transport endpoint is not connected)
lsof: WARNING: can't stat() fuse.vstorage file system /var/lib/
Output information may be incomplete.
qemu-kvm 7366 root 18u unknown /var/lib/
qemu-kvm 7366 7370 root 18u unknown /var/lib/
We currently implemented a workaround by adding
KillMode=process
to the openstack nova compute unit file override. This leave open the vzstorage mounts and keeps the VMs running.
Environment
===========
1. Openstack Ocata release 15.0.3-2.el7
2. Which hypervisor did you use?
qemu-kvm-vz 2.6.0-28. 3.9.vz7. 56.1
2. Which storage type did you use?
Vstorage, 7.4.106-1
3. Which networking type did you use?
Neutron with Linuxbridge