Fuel for OpenStack

[kilo] Test create instace with file injection failed with NovaException: nbd unavailable: module not loaded

Bug #1472520 reported by Tatyanka
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Ivan Berezovskiy
Fuel for OpenStack 7.0

Bug Description

Steps:
1. Deploy env on Ubuntu with neutron gre
 - 1 controller
- 2 nodes with roles compute + cinder
2. Deploy cluster
3. When cluster is ready run ostf

Expected result:
test are passed

Actual result:
Instance with file injection do not get active state with next error in compute:

2015-07-07T23:35:54.292444+00:00 err: 2015-07-07 23:35:54.290 1770 ERROR nova.virt.disk.mount.nbd [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] nbd module not loaded
2015-07-07T23:35:54.294097+00:00 debug: 2015-07-07 23:35:54.292 1770 DEBUG oslo_concurrency.lockutils [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] Lock "nbd-allocation-lock" released by "_inner_get_dev" :: held 0.005s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-07-07T23:35:54.295596+00:00 info: 2015-07-07 23:35:54.294 1770 INFO nova.virt.disk.mount.api [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] Device allocation failed. Will retry in 2 seconds.
2015-07-07T23:35:56.297061+00:00 warning: 2015-07-07 23:35:56.295 1770 WARNING nova.virt.disk.mount.api [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] Device allocation failed after repeated retries.
2015-07-07T23:35:56.298160+00:00 debug: 2015-07-07 23:35:56.297 1770 DEBUG nova.virt.disk.mount.api [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] Fail to mount, tearing back down do_mount /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:223
2015-07-07T23:35:56.299363+00:00 debug: 2015-07-07 23:35:56.298 1770 DEBUG nova.virt.disk.vfs.localfs [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] Failed to mount image: nbd unavailable: module not loaded setup /usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py:82
2015-07-07T23:35:56.300330+00:00 err: 2015-07-07 23:35:56.299 1770 ERROR nova.virt.libvirt.driver [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] [instance: ddc8c054-a50d-4260-8175-551d446e0a96] Error injecting data into image 3fc91453-3ca9-41b3-b3ea-09327eebdaa5 (nbd unavailable: module not loaded)
2015-07-07T23:35:56.304549+00:00 err: 2015-07-07 23:35:56.300 1770 ERROR nova.compute.manager [req-3f992fe2-11eb-4e1f-bea9-8fbb5214accc - - - - -] [instance: ddc8c054-a50d-4260-8175-551d446e0a96] Instance failed to spawn
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] Traceback (most recent call last):
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2442, in _build_resources
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] yield resources
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] block_device_info=block_device_info)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2347, in spawn
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] admin_pass=admin_password)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2839, in _create_image
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] instance, network_info, admin_pass, files, suffix)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2665, in _inject_data
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] instance=instance)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] six.reraise(self.type_, self.value, self.tb)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2659, in _inject_data
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] mandatory=('files',))
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 389, in inject_data
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] fs.setup()
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py", line 83, in setup
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] self.teardown()
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] six.reraise(self.type_, self.value, self.tb)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py", line 78, in setup
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] raise exception.NovaException(mnt.error)
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96] NovaException: nbd unavailable: module not loaded
2015-07-07 23:35:56.300 1770 TRACE nova.compute.manager [instance: ddc8c054-a50d-4260-8175-551d446e0a96]
2015-07-07T23:35:56.305548+00:00 info: 2015-07-07 23:35:56.304 1770 INFO nova.compute.manager [req-2b827306-cc89-4ab3-bcd0-54ea9924606f 54b363b318f048608a6e1f8c957459cb 7b021ac0fba24d209f3f8aa5a2a2830a - - -] [instance: ddc8c054-a50d-4260-8175-551d446e0a96] Terminating instance

So looks like we miss config data for this,
on node nova-compute.conf looks like
root@node-1:~# cat /etc/nova/nova-compute.conf
[DEFAULT]
compute_driver=libvirt.LibvirtDriver
[libvirt]
virt_type=qemu
root@node-1:~#
nova.conf section libvirt
###########
# LIBVIRT #
###########
[libvirt]
# Libvirt handlers for remote volumes. (list value)
#volume_drivers=iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,iser=nova.virt.libvirt.volume.LibvirtISERVolumeDriver,local=nova.virt.libvirt.volume.LibvirtVolumeDriver,fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver,rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver,sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver,nfs=nova.virt.libvirt.volume.LibvirtNFSVolumeDriver,aoe=nova.virt.libvirt.volume.LibvirtAOEVolumeDriver,glusterfs=nova.virt.libvirt.volume.LibvirtGlusterfsVolumeDriver,fibre_channel=nova.virt.libvirt.volume.LibvirtFibreChannelVolumeDriver,scality=nova.virt.libvirt.volume.LibvirtScalityVolumeDriver

# Inject the admin password at boot time, without an agent.
#inject_password=false
inject_password=False

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "239"
  build_id: "2015-07-07_18-02-36"
  nailgun_sha: "2f8d9be6bca0c32a83acda02e1689a44f0d8eefe"
  python-fuelclient_sha: "018c53561baa87397e50368c7f48674829c322c0"
  astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
  fuel-library_sha: "93a5fe7251f3eb932d588ba338920e9e7cfe7f0c"
  fuel-ostf_sha: "ba48d52474515bcd1ad2a9dfa5228bff36ee98d7"
  fuelmain_sha: "c53bb47881f0bb9ba14274a596923d9860140406"

Affected:
http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.thread_1/20/testReport/junit/(root)/deploy_neutron_gre/deploy_neutron_gre/

http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.thread_1/20/testReport/junit/(root)/deploy_neutron_vlan/deploy_neutron_vlan/

Tags: kilo
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
summary: - [killo support] Test create instace with file injection failed with
+ [kilo] Test create instace with file injection failed with
NovaException: nbd unavailable: module not loaded
Ivan Berezovskiy (iberezovskiy)
Changed in fuel:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/200143

Revision history for this message
Ivan Berezovskiy (iberezovskiy) wrote :

https://review.openstack.org/#/c/200143/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/200143
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5013a663571684ab76840f666d797da3feb745da
Submitter: Jenkins
Branch: master

commit 5013a663571684ab76840f666d797da3feb745da
Author: iberezovskiy <email address hidden>
Date: Thu Jul 9 18:00:17 2015 +0300

    Enable nbd kernel module by default on each compute node

      Now nbd kernel module is not enabled by default
      on Ubuntu. We need to enable it if we are going
      to use file injection for Nova. If compute node
      will be rebooted nbd module will be loaded anyway

      Closes-bug: #1472520

    Change-Id: I56c714971551b8fc235cc0fcd2933df42114297f

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

verified 31 iso

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.