GuestFS fails to mount image for data injection

[Salvatore Orlando]

A GuestFS error is causing injection to fail. This result in a warning for metadata injection but results in a spawn error for key injection.

This is logged with debug level:

Exception AttributeError: "GuestFS instance has no attribute '_o'" in <bound method GuestFS.__del__ of <guestfs.GuestFS instance at 0x3ea9f38>> ignored
febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup

And causes this error: http://paste.openstack.org/show/62293/

Full logs available here: http://logs.openstack.org/58/63558/8/check/check-tempest-dsvm-neutron-pg/108e4ca/logs

Interestingly, it seems guestfs was not actually used when the relevant patches went throught the gate checks:
https://review.openstack.org/#/c/70237/
https://review.openstack.org/#/c/70354/

This was expected for patch #70354 but sounds strange for patch #70237

Finally, The traceback seeems different from that of bug 1221985

[Anita Kuno (anteaya)]

Here is my first attempt at a logstash fingerprint for this bug: http://bit.ly/1gAUkaI

It affects the following jobs:
check-tempest-dsvm-neutron-pg 27.2%
check-tempest-dsvm-neutron-isolated 27.2%
check-tempest-dsvm-ironic-postgres-nv 14.4%
check-tempest-dsvm-ironic-nv 12.4%
check-tempest-dsvm-ironic 4.8%

[Anita Kuno (anteaya)]

message:"NovaException: Error mounting /opt/stack/data/nova/instances/" AND message:"/disk with libguestfs"

[Anita Kuno (anteaya)]

Better fingerprint: message:"NovaException: Error mounting /opt/stack/data/nova/instances/" AND message:"/disk with libguestfs" AND filename: "logs/screen-n-cpu.txt"

http://bit.ly/1eH63xA

[Matt Riedemann (mriedem)]

e-r query patch: https://review.openstack.org/#/c/70550/

[Richard Jones (rjones-redhat)]

The actual error:

    ext2: parent directory not found: /lib: File not found by ext2_lookup

In this case it means you need to run:

    sudo update-guestfs-appliance

(This is for Debian/Ubuntu only)

[Salvatore Orlando (salvatore-orlando)]

Thanks Richard.
I was looking for something like that yesterday, but could not find an answer by myself.

[Davanum Srinivas (DIMS) (dims-v)]

"This error message is usually printed when supermin runs out of space on /tmp or /var/tmp" according to http://askubuntu.com/questions/302289/error-mounting-disk-image-using-guestmount

[John Garbutt (johngarbutt)]

Could be a test setup issue, need more information.

[Sahid Orentino (sahid-ferdjaoui)]

We probably have to detect if guestfs is well configured even if we can import it

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120467
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6a0e6209cab2ced467b63ce9ce69a41daae669fe
Submitter: Jenkins
Branch: master

commit 6a0e6209cab2ced467b63ce9ce69a41daae669fe
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Wed Sep 10 16:05:22 2014 +0000

    disk/vfs: ensure guestfs capabilities

    Ensures that guestfs is available and well configured. Also
    when checking for capabilities this commit introduces an in-memory
    flag to avoid repeating the process if succeeded.
    If we are able to load guestfs but something wrong happens a fatal
    error is raised.

    DocImpact: We should add a not for ubuntu like systems that use
    libguestfs to execute the command 'update-guestfs-appliance' for
    configuring it.

    Closes-Bug: #1275267
    Closes-Bug: #1157922
    Change-Id: I97b3a23829ea1f3aadfe08ca6448b35010d2f312