Fuel for OpenStack

Nova can't launch a VM from an RBD backed volume

Bug #1386735 reported by Ivan Kolodyazhny
52
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Aleksandr Didenko
Fuel for OpenStack 6.0

Bug Description

Issue is reproducible on a scale lab:
[root@fuel ~]# cat /etc/nailgun/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "44"
  build_id: "2014-10-23_19-30-06"
  astute_sha: "97eea90efe0a1f17b4934919d6e459d270c10372"
  fuellib_sha: "fb088f2c77b00b75c9c894a06d5779fdeeb536ca"
  ostf_sha: "de177931b53fbe9655502b73d03910b8118e25f1"
  nailgun_sha: "52e777a8dee1bfa8ca3f69e42bb563696fdfd995"
  fuelmain_sha: "7b34d04edb929659dad5fa4ddb224346d4e74ec3"

config
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 5,
      "times": 97
    },
    "args": {
      "volume_size": 10,
      "image": {
        "name": "TestVM"
      },
      "flavor": {
        "name": "m1.nano"
      }
    },
    "context": {
      "users": {
        "project_domain": "default",
        "concurrent": 30,
        "users_per_tenant": 2,
        "tenants": 2,
        "user_domain": "default"
      }
    }
  },
  "name": "CinderVolumes.create_and_attach_volume",
  "pos": 0
}
results
+----------------------+--------+--------+---------+----------+-------+--------+
| Action | min | avg | max | failures | total | result |
+----------------------+--------+--------+---------+----------+-------+--------+
| cinder.create_volume | 2.310 | 2.464 | 4.489 | 77 | 97 | :( |
| nova.boot_server | 2.841 | 3.494 | 4.399 | 0 | 97 | :) |
| ~total | 2.947 | 45.669 | 210.425 | 97 | 97 | :( |
+----------------------+--------+--------+---------+----------+-------+--------+
stderr-0000
<class 'rally.exceptions.TimeoutException'>
Timeout exceeded.
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/volumes.py", line 112, in create_and_attach_volume
    self._attach_volume(server, volume)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 255, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/utils.py", line 482, in _attach_volume
    CONF.benchmark.nova_server_resize_revert_poll_interval)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 121, in wait_for
    raise exceptions.TimeoutException()
TimeoutException: Timeout exceeded.

No errors in cinder logs found

Tags: cinder nova scale
Ivan Kolodyazhny (e0ne)
tags: added: cinder scale
Dmitry Mescheryakov (dmitrymex)
Changed in mos:
importance: Undecided → Medium
assignee: nobody → MOS Cinder (mos-cinder)
milestone: none → 6.0
status: New → Confirmed
Revision history for this message
Aleksandr Shaposhnikov (alashai8) wrote :
Download full text (5.7 KiB)

Here is more descriptive logs from nova compute. I think that it could be related to ceph/libvirt/qemu.
Need to mention that this is CentOS.

2014-10-27 00:40:41.098 8894 TRACE nova.compute.manager [instance: 742a85c5-167d-4c36-ba2c-8efa211d3959]
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] Traceback (most recent call last):
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2231, in _build_resources
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] yield resources
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2101, in _build_and_run_instance
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] block_device_info=block_device_info)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2619, in spawn
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] block_device_info, disk_info=disk_info)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4412, in _create_domain_and_network
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] power_on=power_on)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4336, in _create_domain
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] LOG.error(err)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] six.reraise(self.type_, self.value, self.tb)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4327, in _create_domain
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] domain.createWithFlags(launch_flags)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 183, in doit
2014-10-27 21:56:49.427 8894 TRACE nova.compute.manager [instance: 44bcaa9b-33ca-4114-88ed-a1acab0458fa] result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-10-27 21:56:49.427 8894 TRACE nova.compute.mana...

Read more...

Revision history for this message
Aleksandr Shaposhnikov (alashai8) wrote :
Download full text (3.2 KiB)

Here is some qemu logs:

2014-10-27 21:56:49.221+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-0000224d -S -machine pc-i440fx-2.0,accel=kvm,usb=off -cpu SandyBridge,+erms,+smep,+fsgsbase,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 64 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 44bcaa9b-33ca-4114-88ed-a1acab0458fa -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2014.2-fuel6.0.mira8,serial=797027f5-c28d-4e2a-96dd-5f1f9127e4e6,uuid=44bcaa9b-33ca-4114-88ed-a1acab0458fa -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000224d.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:volumes/volume-d1d4013a-205c-4074-a1fc-36b6bdbe0509:id=volumes:key=AQBVt0pU+K2HExAAcbkk6M+RQhMm7unNOUVw5Q==:auth_supported=cephx\;none:mon_host=192.168.0.18\:6789\;192.168.0.22\:6789\;192.168.0.52\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=d1d4013a-205c-4074-a1fc-36b6bdbe0509,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:4a:77:11,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/44bcaa9b-33ca-4114-88ed-a1acab0458fa/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
Domain id=26 is tainted: high-privileges
char device redirected to /dev/pts/0 (label charserial1)
2014-10-27T21:56:49.267353Z qemu-system-x86_64: -drive file=rbd:volumes/volume-d1d4013a-205c-4074-a1fc-36b6bdbe0509:id=volumes:key=AQBVt0pU+K2HExAAcbkk6M+RQhMm7unNOUVw5Q==:auth_supported=cephx\;none:mon_host=192.168.0.18\:6789\;192.168.0.22\:6789\;192.168.0.52\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=d1d4013a-205c-4074-a1fc-36b6bdbe0509,cache=none: error connecting
2014-10-27T21:56:49.268671Z qemu-system-x86_64: -drive file=rbd:volumes/volume-d1d4013a-205c-4074-a1fc-36b6bdbe0509:id=volumes:key=AQBVt0pU+K2HExAAcbkk6M+RQhMm7unNOUVw5Q==:auth_supported=cephx\;none:mon_host=192.168.0.18\:6789\;192.168.0.22\:6789\;192.168.0.52\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=d1d4013a-205c-4074-a1fc-36b6bdbe0509,cache=none: could not open disk image rbd:volumes/volume-d1d4013a-205c-4074-a1fc-36b6bdbe0509:id=volumes:key=AQBVt0pU+K2HExAAcbkk6M+RQhMm7unNOUVw5Q==:auth_supported=cephx\;none:mon_host=192.168.0.18\:6789\;192.168.0.22\:6789\;192.168.0.52\:6789: Could not open 'rbd:volumes/volume-d1d4013a-205c-4074-a1fc-36b6bdbe0509:id=volumes:key=AQBVt0pU+K2HExAAcbkk6M+RQhMm7unNOUVw5Q==:auth_supported=cephx\;none:mon_host=192....

Read more...

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

I'm getting the same problem in a kvm/devops based test environment deployed from 6.0 iso #53.

Changed in mos:
importance: Medium → High
assignee: MOS Cinder (mos-cinder) → MOS Nova (mos-nova)
Dmitry Borodaenko (angdraug)
Changed in mos:
assignee: MOS Nova (mos-nova) → Dmitry Borodaenko (dborodaenko)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

One part of the problem is that in Juno all libvirt_* options for Nova have been deprecated and moved under [libvirt] section of nova.conf, so Nova in 6.0 isn't actually configured to use rbd. Affected options relevant for Ceph:

images_type
inject_key
inject_partition
images_rbd_pool

There is also use_virtio_for_bridges option set by upstream Nova manifests that is impacted by this.

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

The commit that removed from Nova the options that were deprecated in Icehouse:
https://review.openstack.org/#/c/88456/

Dmitry Borodaenko (angdraug)
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Dmitry Borodaenko (dborodaenko)
milestone: none → 6.0
Dmitry Borodaenko (angdraug)
Changed in fuel:
status: Confirmed → In Progress
Dmitry Borodaenko (angdraug)
Changed in mos:
assignee: Dmitry Borodaenko (dborodaenko) → MOS Nova (mos-nova)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Review: https://review.openstack.org/131659

This commit fixes the rbd side of the problem, and we don't need to bother with use_virtio_for_bridges: it's already True by default in Nova, so not having it in the right place has zero impact.

I'm still not sure that Nova does the right thing when it's not configured to use rbd image backend, and is asked to launch a VM from a Cinder volume in rbd (which is a supported use case when Ceph RBD is not enabled for Nova ephemeral). MOS Nova team: please confirm that this code path is doing the right thing.

summary: - create-and-attach-volume rally scenario fails
+ Nova can't launch a VM from an RBD backed volume
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/131659
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=967787437bdc96654c6ea14072a4561bf7ef91de
Submitter: Jenkins
Branch: master

commit 967787437bdc96654c6ea14072a4561bf7ef91de
Author: Dmitry Borodaenko <email address hidden>
Date: Wed Oct 29 00:28:10 2014 -0700

    replace deprecated libvirt_* options for rbd

    All libvirt_* options for Nova were deprecated in Icehouse and removed
    in Juno, this commit replaces the rbd related deprecated options with
    corresponding options in the [libvirt] section of nova.conf.

    Change-Id: If9670e6df8dccbbb659d48d59e6a60b5583d68ac
    Partial-Bug: #1386735

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/131805

Changed in fuel:
assignee: Dmitry Borodaenko (dborodaenko) → Aleksandr Didenko (adidenko)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/131805
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3e0ed5a39945b16628d11b89f31af39f29c7ed65
Submitter: Jenkins
Branch: master

commit 3e0ed5a39945b16628d11b89f31af39f29c7ed65
Author: Aleksandr Didenko <email address hidden>
Date: Wed Oct 29 17:51:47 2014 +0200

    Fix deprecated options for nova.conf

    rbd_user and rbd_secret_uuid should be declared in [libvirt] block
    instead of [DEFAULT] since Juno.

    Change-Id: I699b76066a0940f14b279b89c46b2013794c1f81
    Partial-bug: 1386735

Dmitry Borodaenko (angdraug)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

{u'build_id': u'2014-10-30_04-21-22', u'ostf_sha': u'f47fd1d66a7255213ee075d5c11b8f11
1f922000', u'build_number': u'63', u'auth_required': True, u'nailgun_sha': u'02c6bb2e54bbec76da33167eaf5f2e0b3e2e50a7', u'production': u'docker', u'api': u'
1.0', u'fuelmain_sha': u'2ade7c571380a091048d103a6affff634b5b2520', u'astute_sha': u'97eea90efe0a1f17b4934919d6e459d270c10372', u'feature_groups': [u'mirant
is', u'techpreview'], u'release': u'6.0-techpreview', u'release_versions': {u'2014.2-6.0-techpreview': {u'VERSION': {u'build_id': u'2014-10-30_04-21-22', u'
ostf_sha': u'f47fd1d66a7255213ee075d5c11b8f111f922000', u'build_number': u'63', u'api': u'1.0', u'nailgun_sha': u'02c6bb2e54bbec76da33167eaf5f2e0b3e2e50a7',
 u'production': u'docker', u'fuelmain_sha': u'2ade7c571380a091048d103a6affff634b5b2520', u'astute_sha': u'97eea90efe0a1f17b4934919d6e459d270c10372', u'featu
re_groups': [u'mirantis', u'techpreview'], u'release': u'6.0-techpreview', u'fuellib_sha': u'45b6fc42091a0a33d3e48fbe78b782ce743aedc1'}}}, u'fuellib_sha': u
'45b6fc42091a0a33d3e48fbe78b782ce743aedc1'}

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/132596

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/132596
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=72d96597752277521c60e4a27c356a0d86329dda
Submitter: Jenkins
Branch: master

commit 72d96597752277521c60e4a27c356a0d86329dda
Author: Dmitry Borodaenko <email address hidden>
Date: Mon Nov 3 14:05:31 2014 -0800

    move live_migration_flag under [libvirt]

    One more libvirt option was moved from [DEFAULT] to [libvirt] in
    nova.conf in Juno.

    Change-Id: Ibbf63c330750dd3befca52605ef8b46e0a593014
    Related-Bug: #1386735

Roman Podoliaka (rpodolyaka)
tags: added: nova
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Checked the latest 6.0 ISO - it works properly now (I only added MOS nova before, so that we wouldn't forget to check it)

no longer affects: mos
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.