VFS blkid calls need to handle 0 or 2 return codes

Bug #1426324 reported by James Page on 2015-02-27
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
James Page
Kilo
Undecided
Unassigned
Ubuntu Cloud Archive
Undecided
Unassigned
nova (Ubuntu)
High
James Page

Bug Description

kilo-2 introduce blkid calls for fs detection on all new instances; if the specified key is not found on the block device, blkid will return 2 instead of 0 - nova needs to deal with this:

2015-02-27 10:48:51.270 3062 INFO nova.virt.disk.vfs.api [-] Unable to import guestfs, falling back to VFSLocalFS
2015-02-27 10:48:51.476 3062 ERROR nova.compute.manager [-] [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Instance failed to spawn
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Traceback (most recent call last):
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2328, in _build_resources
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] yield resources
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2198, in _build_and_run_instance
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] flavor=flavor)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2329, in spawn
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] admin_pass=admin_password)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2728, in _create_image
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] project_id=instance['project_id'])
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 230, in cache
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] *args, **kwargs)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 507, in create_image
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] copy_qcow2_image(base, self.path, size)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] return f(*args, **kwargs)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 473, in copy_qcow2_image
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] disk.extend(target, size, use_cow=True)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 183, in extend
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] if not is_image_extendable(image, use_cow):
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 235, in is_image_extendable
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] if fs.get_image_fs() in SUPPORTED_FS_TO_EXTEND:
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py", line 167, in get_image_fs
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] run_as_root=True)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 190, in execute
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] return processutils.execute(*cmd, **kwargs)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 228, in execute
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] cmd=sanitized_cmd)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] ProcessExecutionError: Unexpected error while running command.
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd2
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Exit code: 2
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Stdout: u''
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Stderr: u''
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b]

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: python-nova 1:2015.1~b2-0ubuntu5~cloud0 [modified: usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py] [origin: Canonical]
ProcVersionSignature: User Name 3.16.0-31.41~14.04.1-generic 3.16.7-ckt5
Uname: Linux 3.16.0-31-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Fri Feb 27 11:13:43 2015
NovaConf: Error: [Errno 13] Permission denied: '/etc/nova/nova.conf'
PackageArchitecture: all
SourcePackage: nova
UpgradeStatus: No upgrade log present (probably fresh install)

James Page (james-page) wrote :
James Page (james-page) wrote :

Confirming this is >= kilo-2 only.

summary: - blkid calls need to handle 0 or 2 return codes
+ VFS blkid calls need to handle 0 or 2 return codes
James Page (james-page) on 2015-02-27
Changed in nova (Ubuntu):
assignee: nobody → James Page (james-page)
Changed in nova:
assignee: nobody → James Page (james-page)
Changed in nova (Ubuntu):
status: New → Triaged
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.openstack.org/159797

Changed in nova:
status: New → In Progress
Changed in nova:
importance: Undecided → Low
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 1:2015.1~b2-0ubuntu6

---------------
nova (1:2015.1~b2-0ubuntu6) vivid; urgency=medium

  * d/p/fixup-blkid.patch: Cherry pick inflight fix for return code
    handling of blkid calls in libvirt kvm compute driver (LP: #1426324).
 -- James Page <email address hidden> Tue, 03 Mar 2015 14:42:44 +0000

Changed in nova (Ubuntu):
status: Triaged → Fix Released
James Page (james-page) on 2015-03-05
Changed in cloud-archive:
status: New → Fix Committed
James Page (james-page) on 2015-03-16
Changed in cloud-archive:
status: Fix Committed → Fix Released
Matt Riedemann (mriedem) on 2015-05-11
tags: added: kilo-backport-potential

Reviewed: https://review.openstack.org/159797
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=051fced477cdead4e307c23fc343da427beb6c81
Submitter: Jenkins
Branch: master

commit 051fced477cdead4e307c23fc343da427beb6c81
Author: James Page <email address hidden>
Date: Fri Feb 27 11:27:30 2015 +0000

    Handle return code 2 from blkid calls

    blkid returns code 2 if the requested TYPE key is not found
    for the specified device.

    Ensure that this situation is handled correctly; blkid will
    not return any data, so the return value will be empty.

    Change-Id: I38cd7b53cdfd694c0eba1b6054d8e4c33759b0b8
    Closes-Bug: #1426324

Changed in nova:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/182716
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1b941f1d61ed830cc43aa99b105a1c5629ceab42
Submitter: Jenkins
Branch: stable/kilo

commit 1b941f1d61ed830cc43aa99b105a1c5629ceab42
Author: James Page <email address hidden>
Date: Fri Feb 27 11:27:30 2015 +0000

    Handle return code 2 from blkid calls

    blkid returns code 2 if the requested TYPE key is not found
    for the specified device.

    Ensure that this situation is handled correctly; blkid will
    not return any data, so the return value will be empty.

    Change-Id: I38cd7b53cdfd694c0eba1b6054d8e4c33759b0b8
    Closes-Bug: #1426324
    (cherry picked from commit 051fced477cdead4e307c23fc343da427beb6c81)

tags: added: in-stable-kilo
Thierry Carrez (ttx) on 2015-06-24
Changed in nova:
milestone: none → liberty-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2015-10-15
Changed in nova:
milestone: liberty-1 → 12.0.0
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers