Nova-computes need nfs-common installed if NFS cinder backend

[Kevin Stevens]

If we have Cinder backed by NFS we need to install nfs-common package on the compute nodes or else "volume-attach" fails (silent) with big ugly stack trace in nova-compute log.

AUDIT nova.compute.manager [req-XXX None] [instance: XXX] Attaching volume XXX to /dev/vdb
ERROR nova.virt.block_device [req-XXX None] [instance: XXX] Driver failed to attach volume XXX at /dev/vdb
TRACE nova.virt.block_device [instance: XXX] Traceback (most recent call last):
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 252, in attach
TRACE nova.virt.block_device [instance: XXX] device_type=self['device_type'], encryption=encryption)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1423, in attach_volume
TRACE nova.virt.block_device [instance: XXX] conf = self._connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1370, in _connect_volume
TRACE nova.virt.block_device [instance: XXX] return driver.connect_volume(connection_info, disk_info)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 755, in connect_volume
TRACE nova.virt.block_device [instance: XXX] self._ensure_mounted(connection_info['data']['export'], options)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 782, in _ensure_mounted
TRACE nova.virt.block_device [instance: XXX] self._mount_nfs(mount_path, nfs_export, options, ensure=True)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 798, in _mount_nfs
TRACE nova.virt.block_device [instance: XXX] utils.execute(*nfs_cmd, run_as_root=True)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/utils.py", line 187, in execute
TRACE nova.virt.block_device [instance: XXX] return processutils.execute(*cmd, **kwargs)
TRACE nova.virt.block_device [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 203, in execute
TRACE nova.virt.block_device [instance: XXX] cmd=sanitized_cmd)
TRACE nova.virt.block_device [instance: XXX] ProcessExecutionError: Unexpected error while running command.
TRACE nova.virt.block_device [instance: XXX] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -t nfs 172.30.244.100:/vol/cinders_NFS /var/lib/nova/mnt/XXX
TRACE nova.virt.block_device [instance: XXX] Exit code: 32
TRACE nova.virt.block_device [instance: XXX] Stdout: u''
TRACE nova.virt.block_device [instance: XXX] Stderr: u'mount: wrong fs type, bad option, bad superblock on 172.30.244.100:/vol/cinders_NFS,\n missing codepage or helper program, or other error\n (for several filesystems (e.g. nfs, cifs) you might\n need a /sbin/mount.<type> helper program)\n In some cases useful info is found in syslog - try\n dmesg | tail or so\n\n'
TRACE nova.virt.block_device [instance: XXX]
ERROR nova.compute.manager [req-XXX None] [instance: XXX] Failed to attach XXX at /dev/vdb
TRACE nova.compute.manager [instance: XXX] Traceback (most recent call last):
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4612, in _attach_volume
TRACE nova.compute.manager [instance: XXX] do_check_attach=False, do_driver_attach=True)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 46, in wrapped
TRACE nova.compute.manager [instance: XXX] ret_val = method(obj, context, *args, **kwargs)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 261, in attach
TRACE nova.compute.manager [instance: XXX] connector)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE nova.compute.manager [instance: XXX] six.reraise(self.type_, self.value, self.tb)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 252, in attach
TRACE nova.compute.manager [instance: XXX] device_type=self['device_type'], encryption=encryption)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1423, in attach_volume
TRACE nova.compute.manager [instance: XXX] conf = self._connect_volume(connection_info, disk_info)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1370, in _connect_volume
TRACE nova.compute.manager [instance: XXX] return driver.connect_volume(connection_info, disk_info)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 755, in connect_volume
TRACE nova.compute.manager [instance: XXX] self._ensure_mounted(connection_info['data']['export'], options)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 782, in _ensure_mounted
TRACE nova.compute.manager [instance: XXX] self._mount_nfs(mount_path, nfs_export, options, ensure=True)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 798, in _mount_nfs
TRACE nova.compute.manager [instance: XXX] utils.execute(*nfs_cmd, run_as_root=True)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/utils.py", line 187, in execute
TRACE nova.compute.manager [instance: XXX] return processutils.execute(*cmd, **kwargs)
TRACE nova.compute.manager [instance: XXX] File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 203, in execute
TRACE nova.compute.manager [instance: XXX] cmd=sanitized_cmd)
TRACE nova.compute.manager [instance: XXX] ProcessExecutionError: Unexpected error while running command.
TRACE nova.compute.manager [instance: XXX] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -t nfs 172.30.244.100:/vol/s_NFS /var/lib/nova/mnt/XXX
TRACE nova.compute.manager [instance: XXX] Exit code: 32
TRACE nova.compute.manager [instance: XXX] Stdout: u''
TRACE nova.compute.manager [instance: XXX] Stderr: u'mount: wrong fs type, bad option, bad superblock on 172.30.244.100:/vol/cinders_NFS,\n missing codepage or helper program, or other error\n (for several filesystems (e.g. nfs, cifs) you might\n need a /sbin/mount.<type> helper program)\n In some cases useful info is found in syslog - try\n dmesg | tail or so\n\n'
TRACE nova.compute.manager [instance: XXX]
ERROR oslo.messaging.rpc.dispatcher [req-XXX ] Exception during message handling: Unexpected error while running command.
tc/nova/rootwrap.conf mount -t nfs 172.30.244.100:/vol/cinders_NFS /var/lib/nova/mnt/XXX

, bad option, bad superblock on 172.30.244.100:/vol/cinders_NFS,\n missing codepage or helper program, or other error\n (for several filesystems (e.g. nfs, cifs) you might\n need a /sbin/mount.<type> helper program)\n In some cases useful info is found in syslog - try\n dmesg | tail or so\n\n'
TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
TRACE oslo.messaging.rpc.dispatcher incoming.message))
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 430, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
TRACE oslo.messaging.rpc.dispatcher payload)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 314, in decorated_function
TRACE oslo.messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 286, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 342, in decorated_function
TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 330, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4602, in attach_volume
TRACE oslo.messaging.rpc.dispatcher do_attach_volume(context, instance, driver_bdm)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 272, in inner
TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4600, in do_attach_volume
TRACE oslo.messaging.rpc.dispatcher bdm.destroy(context)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4597, in do_attach_volume
TRACE oslo.messaging.rpc.dispatcher return self._attach_volume(context, instance, driver_bdm)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4620, in _attach_volume
TRACE oslo.messaging.rpc.dispatcher self.volume_api.unreserve_volume(context, bdm.volume_id)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4612, in _attach_volume
TRACE oslo.messaging.rpc.dispatcher do_check_attach=False, do_driver_attach=True)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 46, in wrapped
TRACE oslo.messaging.rpc.dispatcher ret_val = method(obj, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 261, in attach
TRACE oslo.messaging.rpc.dispatcher connector)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/block_device.py", line 252, in attach
TRACE oslo.messaging.rpc.dispatcher device_type=self['device_type'], encryption=encryption)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1423, in attach_volume
TRACE oslo.messaging.rpc.dispatcher conf = self._connect_volume(connection_info, disk_info)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1370, in _connect_volume
TRACE oslo.messaging.rpc.dispatcher return driver.connect_volume(connection_info, disk_info)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 755, in connect_volume
TRACE oslo.messaging.rpc.dispatcher self._ensure_mounted(connection_info['data']['export'], options)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 782, in _ensure_mounted
TRACE oslo.messaging.rpc.dispatcher self._mount_nfs(mount_path, nfs_export, options, ensure=True)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 798, in _mount_nfs
TRACE oslo.messaging.rpc.dispatcher utils.execute(*nfs_cmd, run_as_root=True)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/utils.py", line 187, in execute
TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 203, in execute
TRACE oslo.messaging.rpc.dispatcher cmd=sanitized_cmd)
TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -t nfs 172.30.244.100:/vol/cinders_NFS /var/lib/nova/mnt/XXX
TRACE oslo.messaging.rpc.dispatcher Exit code: 32
TRACE oslo.messaging.rpc.dispatcher Stdout: u''
TRACE oslo.messaging.rpc.dispatcher Stderr: u'mount: wrong fs type, bad option, bad superblock on 172.30.244.100:/vol/cinders_NFS,\n missing codepage or helper program, or other error\n (for several filesystems (e.g. nfs, cifs) you might\n need a /sbin/mount.<type> helper program)\n In some cases useful info is found in syslog - try\n dmesg | tail or so\n\n'
TRACE oslo.messaging.rpc.dispatcher

[Jesse Pretorius (jesse-pretorius)]

master review: https://review.openstack.org/221844

[Hugh Saunders (hughsaunders)]

I was able to reproduce this bug when cinder-volumes is in its own container. In an AIO, if cinder-volumes is on the host, then nfs-common is installed via the cinder role and the bug does not manifest.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/221844
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=24c56bf0b85a44f5a38290de400864bd80f3fa40
Submitter: Jenkins
Branch: master

commit 24c56bf0b85a44f5a38290de400864bd80f3fa40
Author: Hugh Saunders <email address hidden>
Date: Wed Sep 9 16:28:42 2015 +0100

    Install nfs-common with nova-compute

    Needed for cinder nfs backend.

    Change-Id: Ida4b49d463d0d90d954fc6c1ba94f0fc56cc0a41
    Closes-bug: #1482265

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/223604

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223604
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=5c7899ede9dfaf57b8bb0d9441066a4e5a81d73e
Submitter: Jenkins
Branch: kilo

commit 5c7899ede9dfaf57b8bb0d9441066a4e5a81d73e
Author: Hugh Saunders <email address hidden>
Date: Wed Sep 9 16:28:42 2015 +0100

    Install nfs-common with nova-compute

    Needed for cinder nfs backend.

    Change-Id: Ida4b49d463d0d90d954fc6c1ba94f0fc56cc0a41
    Closes-bug: #1482265
    (cherry picked from commit 24c56bf0b85a44f5a38290de400864bd80f3fa40)

[Jesse Pretorius (jesse-pretorius)]

juno review: https://review.openstack.org/227969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227969
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=cb8b11bb713d4d8f7538e05f38bdcb9aab6f344f
Submitter: Jenkins
Branch: juno

commit cb8b11bb713d4d8f7538e05f38bdcb9aab6f344f
Author: Jesse Pretorius <email address hidden>
Date: Fri Sep 25 18:10:33 2015 +0100

    Install nfs-common with nova-compute

    Needed for cinder nfs backend.

    Closes-bug: #1482265
    Change-Id: Ida4b49d463d0d90d954fc6c1ba94f0fc56cc0a41
    (cherry picked from commit 24c56bf0b85a44f5a38290de400864bd80f3fa40)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 11.2.14 release.