OpenStack Compute (nova)

[GPFS] nova volume-attach a gpfs volume with an error log in nova-compute

Bug #1405044 reported by Lan Qi song
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Lan Qi song
OpenStack Compute (nova) 2015.1.0 "kilo"
Juno
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.2.2

Bug Description

When I attached a gpfs volume to an instance, the volume has been successfully attached to the instance, but there were some error logs in nova-compute log file as below:

2014-12-22 21:52:10.863 13396 ERROR nova.openstack.common.threadgroup [-] Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /gpfs/volume-98520c4e-935d-43d8-9c8d-00fcb54bb335
Exit code: 1
Stdout: u''
Stderr: u'BLKGETSIZE64: Inappropriate ioctl for device\n'
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup x.wait()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 173, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 121, in wait
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 293, in switch
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 212, in main
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 490, in run_service
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup service.start()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/service.py", line 181, in start
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1159, in pre_start_hook
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 6037, in update_available_resource
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup nodenames = set(self.driver.get_available_nodes())
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 1237, in get_available_nodes
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup stats = self.get_host_stats(refresh=refresh)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5794, in get_host_stats
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return self.host_state.get_host_stats(refresh=refresh)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 473, in host_state
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup self._host_state = HostState(self)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6360, in __init__
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup self.update_status()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6411, in update_status
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup data['disk_available_least'] = _get_disk_available_least()
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 6384, in _get_disk_available_least
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup disk_over_committed = (self.driver.
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5744, in _get_disk_over_committed_size_total
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup self._get_instance_disk_info(dom.name(), xml))
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 5697, in _get_instance_disk_info
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup dk_size = lvm.get_volume_size(path)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/lvm.py", line 55, in decorated_function
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return function(path)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/lvm.py", line 186, in get_volume_size
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup run_as_root=True)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 53, in execute
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return utils.execute(*args, **kwargs)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/utils.py", line 163, in execute
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup return processutils.execute(*cmd, **kwargs)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 203, in execute
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup cmd=sanitized_cmd)
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /gpfs/volume-98520c4e-935d-43d8-9c8d-00fcb54bb335
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Exit code: 1
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Stdout: u''
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup Stderr: u'BLKGETSIZE64: Inappropriate ioctl for device\n'
2014-12-22 21:52:10.863 13396 TRACE nova.openstack.common.threadgroup

This error occurred when nova tried to get the gpfs volume size using "blockdev --getsize64" command.
Cinder configuration "gpfs_mount_point_base" was set to "/gpfs"
The blockdev filter in /usr/share/nova/rootwrap/compute.filters was set to
"blockdev: RegExpFilter, blockdev, root, blockdev, (--getsize64|--flushbufs), (/dev/.*|/gpfs/.*)"

It seems we couldn't use "blockdev --getsize64" command to calculate the gpfs volume size. This error wouldn't affect the volume attachment, but it affect the nova-compute service, nova-compute service couldn't restart successfully with this error.

Lan Qi song (lqslan)
summary: - [GPFS] nova volume-attach a gpfs volume with a error log
+ [GPFS] nova volume-attach a gpfs volume with an error log in nova-
+ compute
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/143614

Changed in nova:
assignee: nobody → Lan Qi song (lqslan)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/143614
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8c62b79610cb5680a052c7a0c779a3faf4568781
Submitter: Jenkins
Branch: master

commit 8c62b79610cb5680a052c7a0c779a3faf4568781
Author: lqslan <email address hidden>
Date: Tue Dec 23 15:35:11 2014 +0800

    Add LibvirtGPFSVolumeDriver class

    Currently, GPFS use LibvirtVolumeDriver to generate the libvirt
    xml configuration when attaching a gpfs volume. But the driver
    always set the disk_type to 'block' which should be 'file' for
    GPFS volume.

    This patch create a new volume driver class for GPFS, the class
    can generate the right xml configuration for GPFS volume.

    Change-Id: Ica1cf3558baeea12b519bda50c61cea446429364
    Closes-Bug:#1405044

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/147411

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/juno)

Reviewed: https://review.openstack.org/147411
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8c07d429a00ee35166820c61a67618430dbb83b8
Submitter: Jenkins
Branch: stable/juno

commit 8c07d429a00ee35166820c61a67618430dbb83b8
Author: lqslan <email address hidden>
Date: Tue Dec 23 15:35:11 2014 +0800

    Add LibvirtGPFSVolumeDriver class

    Currently, GPFS use LibvirtVolumeDriver to generate the libvirt
    xml configuration when attaching a gpfs volume. But the driver
    always set the disk_type to 'block' which should be 'file' for
    GPFS volume.

    This patch create a new volume driver class for GPFS, the class
    can generate the right xml configuration for GPFS volume.

    Change-Id: Ica1cf3558baeea12b519bda50c61cea446429364
    Closes-Bug:#1405044
    (cherry picked from commit 8c62b79610cb5680a052c7a0c779a3faf4568781)

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.