Nova service restart disconnects vzstorage volumes

[Carlo Baijens]

Description
===========
Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart.

==================
Running Openstack Ocata release with Virtuozzo KVM hypervisor on Vstorage backed by Cinder. A restart of the openstack-nova-compute service causes vzstorage mounts to be unreachable. The restart keeps VMs running using the mount but the filesystem is unreachable.

Before restart:
[user@node ~]$ sudo systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
 Main PID: 3932 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           ├─3932 /usr/bin/python2 /usr/bin/nova-compute
           ├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpe2LS_d/privsep.sock
           ├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpOXZxyO/privsep.sock
           └─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/vstorage/nova-openstackpoc.log.gz /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee

We restart the service :
sudo systemctl restart openstack-nova-compute.service

After this we see:

[user@node ~]$ sudo systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
 Main PID: 199753 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           ├─199753 /usr/bin/python2 /usr/bin/nova-compute
           └─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpjDaH7y/privsep.sock

The vstorage-mount is not reinitialised by nova-compute.

Expected result
===============
The vstorage-mount is available after nova-compute restart.

Actual result
=============
The vstorage-mount is not reinitialised by nova-compute.

Nova mounts do not disappear after restart but are unreachable:

[user@node ~]$ sudo lsof | grep /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
lsof: WARNING: can't stat() fuse.vstorage file system /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
      Output information may be incomplete.
qemu-kvm 7366 root 18u unknown /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
qemu-kvm 7366 7370 root 18u unknown /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)

We currently implemented a workaround by adding

KillMode=process

to the openstack nova compute unit file override. This leave open the vzstorage mounts and keeps the VMs running.

Environment
===========
1. Openstack Ocata release 15.0.3-2.el7

2. Which hypervisor did you use?

qemu-kvm-vz 2.6.0-28.3.9.vz7.56.1

2. Which storage type did you use?

Vstorage, 7.4.106-1

3. Which networking type did you use?
Neutron with Linuxbridge