VFS blkid calls need to handle 0 or 2 return codes

Bug #1426324 reported by James Page
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
James Page
Kilo
Fix Released
Undecided
Unassigned
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
nova (Ubuntu)
Fix Released
High
James Page

Bug Description

kilo-2 introduce blkid calls for fs detection on all new instances; if the specified key is not found on the block device, blkid will return 2 instead of 0 - nova needs to deal with this:

2015-02-27 10:48:51.270 3062 INFO nova.virt.disk.vfs.api [-] Unable to import guestfs, falling back to VFSLocalFS
2015-02-27 10:48:51.476 3062 ERROR nova.compute.manager [-] [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Instance failed to spawn
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Traceback (most recent call last):
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2328, in _build_resources
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] yield resources
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2198, in _build_and_run_instance
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] flavor=flavor)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2329, in spawn
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] admin_pass=admin_password)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2728, in _create_image
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] project_id=instance['project_id'])
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 230, in cache
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] *args, **kwargs)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 507, in create_image
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] copy_qcow2_image(base, self.path, size)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] return f(*args, **kwargs)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 473, in copy_qcow2_image
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] disk.extend(target, size, use_cow=True)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 183, in extend
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] if not is_image_extendable(image, use_cow):
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 235, in is_image_extendable
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] if fs.get_image_fs() in SUPPORTED_FS_TO_EXTEND:
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py", line 167, in get_image_fs
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] run_as_root=True)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 190, in execute
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] return processutils.execute(*cmd, **kwargs)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 228, in execute
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] cmd=sanitized_cmd)
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] ProcessExecutionError: Unexpected error while running command.
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blkid -o value -s TYPE /dev/nbd2
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Exit code: 2
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Stdout: u''
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b] Stderr: u''
2015-02-27 10:48:51.476 3062 TRACE nova.compute.manager [instance: 1aa12a52-c91b-49b4-9636-63c39f7ba72b]

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: python-nova 1:2015.1~b2-0ubuntu5~cloud0 [modified: usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py] [origin: Canonical]
ProcVersionSignature: User Name 3.16.0-31.41~14.04.1-generic 3.16.7-ckt5
Uname: Linux 3.16.0-31-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Fri Feb 27 11:13:43 2015
NovaConf: Error: [Errno 13] Permission denied: '/etc/nova/nova.conf'
PackageArchitecture: all
SourcePackage: nova
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Confirming this is >= kilo-2 only.

summary: - blkid calls need to handle 0 or 2 return codes
+ VFS blkid calls need to handle 0 or 2 return codes
James Page (james-page)
Changed in nova (Ubuntu):
assignee: nobody → James Page (james-page)
Changed in nova:
assignee: nobody → James Page (james-page)
Changed in nova (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/159797

Changed in nova:
status: New → In Progress
Changed in nova:
importance: Undecided → Low
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 1:2015.1~b2-0ubuntu6

---------------
nova (1:2015.1~b2-0ubuntu6) vivid; urgency=medium

  * d/p/fixup-blkid.patch: Cherry pick inflight fix for return code
    handling of blkid calls in libvirt kvm compute driver (LP: #1426324).
 -- James Page <email address hidden> Tue, 03 Mar 2015 14:42:44 +0000

Changed in nova (Ubuntu):
status: Triaged → Fix Released
James Page (james-page)
Changed in cloud-archive:
status: New → Fix Committed
James Page (james-page)
Changed in cloud-archive:
status: Fix Committed → Fix Released
Matt Riedemann (mriedem)
tags: added: kilo-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/159797
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=051fced477cdead4e307c23fc343da427beb6c81
Submitter: Jenkins
Branch: master

commit 051fced477cdead4e307c23fc343da427beb6c81
Author: James Page <email address hidden>
Date: Fri Feb 27 11:27:30 2015 +0000

    Handle return code 2 from blkid calls

    blkid returns code 2 if the requested TYPE key is not found
    for the specified device.

    Ensure that this situation is handled correctly; blkid will
    not return any data, so the return value will be empty.

    Change-Id: I38cd7b53cdfd694c0eba1b6054d8e4c33759b0b8
    Closes-Bug: #1426324

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/182716

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/182716
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1b941f1d61ed830cc43aa99b105a1c5629ceab42
Submitter: Jenkins
Branch: stable/kilo

commit 1b941f1d61ed830cc43aa99b105a1c5629ceab42
Author: James Page <email address hidden>
Date: Fri Feb 27 11:27:30 2015 +0000

    Handle return code 2 from blkid calls

    blkid returns code 2 if the requested TYPE key is not found
    for the specified device.

    Ensure that this situation is handled correctly; blkid will
    not return any data, so the return value will be empty.

    Change-Id: I38cd7b53cdfd694c0eba1b6054d8e4c33759b0b8
    Closes-Bug: #1426324
    (cherry picked from commit 051fced477cdead4e307c23fc343da427beb6c81)

tags: added: in-stable-kilo
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-1 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.