Image file injection is broken on CentOS due to a missing dependency of libguestfs

[Roman Podoliaka]

When I try to boot an instance injecting a public SSH key, the boot fails.

Seems that one of libquestfs dependencies is missing:

  guestfsd: error while loading shared libraries: libhivex.so.0: cannot open shared object file: No such file or directory

libguestfs *must* depend on libhivex

[Roman Podoliaka (rpodolyaka)]

we should check this comment https://answers.launchpad.net/nova/+question/167348

[Roman Podoliaka (rpodolyaka)]

Ignore my last comment, this is a different issue with libguestds and cirros images.

[Roman Podoliaka (rpodolyaka)]

mos-linux team, could you please rebuild the libguestfs package and make it depend on libhivex on CentOS?

[Albert Syriy (asyriy)]

Roman, could you please add steps to reproduce the bug ?

[Roman Podoliaka (rpodolyaka)]

So the steps to reproduce would be:

1. Deploy an OpenStack cluster using the latest 5.1/5.0.1 MOS ISO. Choose CentOS 6.5 when asked for OS to be used.

2. Try to boot an instance injecting an arbitrary file:

   echo "Hello, World!" > data.txt
   nova boot --image TestVM --file /data.txt=data.txt --flavor m1.tiny demo

3. SSH to your instance.

4. Check if path /data.txt exists.

Expected result:

The booted VM has an injected file - /data.txt.

Actual result:

Instance boot fails.

[Roman Podoliaka (rpodolyaka)]

libguestfs specific issue here is a missing shared library:

  guestfsd: error while loading shared libraries: libhivex.so.0: cannot open shared object file: No such file or directory

You can trigger this error by running the libguestfs check on a compute node:

libguestfs-test-tool

Expected result:

libguestfs-test-tool exists with 0 error code.

Actual result:

libguestfs-test-tool fails with the following error message "libhivex.so.0: cannot open shared object file: No such file or directory"

[Roman Podoliaka (rpodolyaka)]

Installing of the corresponding RPM package solves the issue for me, but it would be really great, if we could add the missing dependency to libguestfs package.

[OSCI Robot (oscirobot)]

Package hivex has been built from changeset: http://gerrit.mirantis.com/28542
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-28542/centos

[Albert Syriy (asyriy)]

The package hivex was commited to repository https://gerrit.mirantis.com/#/c/28542/
The custom ISO has been built with the hivex packages, but the has not been included as dependency for libguestfs. The reason hivex package has not appeared on the OBS repository mirrors.
Link to the ISO is http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/290/

The task is finished as soon as the hivex package is reviewed and appears on the OBS mirros.

[Albert Syriy (asyriy)]

Met an issue with building the libguestfs packages. (It's a blocker for me.)
See details at the link
https://bugs.launchpad.net/mos/+bug/1372983

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/28796
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-28796/centos

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/28404
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-28404/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-28404/centos"

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/29372
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-29372/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-29372/centos"

[Albert Syriy (asyriy)]

Please review and merge if there is not comments

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/28404
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-28404/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-28404/centos"

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/28796
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-28796/centos

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/29429
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29429/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29429/centos"

[Albert Syriy (asyriy)]

Due to luck of network access the libguestfs build with the pre-compiled appliance.
The package was built successfully, but during the installation test faced the issue (see the output):
+ /usr/bin/libguestfs-test-tool
     ************************************************************
     * IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SELinux: Enforcing
library version: 1.20.11rhel=6,release=2.mira4
guestfs_get_append: (null)
guestfs_get_attach_method: appliance
guestfs_get_autosync: 1
guestfs_get_cachedir: /var/tmp
guestfs_get_direct: 0
guestfs_get_memsize: 500
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_qemu: /usr/bin/qemu-kvm
guestfs_get_recovery_proc: 1
guestfs_get_selinux: 0
guestfs_get_smp: 1
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: attach-method=appliance
libguestfs: launch: tmpdir=/tmp/libguestfs0adqtv
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: command: run: febootstrap-supermin-helper
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ -f checksum
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ x86_64
supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null)
supermin helper [00002ms] inputs[0] = /usr/lib64/guestfs/supermin.d
checking modpath /lib/modules/2.6.32-358.2.1.el6.x86_64 is a directory
picked vmlinuz-2.6.32-358.2.1.el6.x86_64 because modpath /lib/modules/2.6.32-358.2.1.el6.x86_64 exists
supermin helper [00004ms] finished creating kernel
supermin helper [00004ms] visiting /usr/lib64/guestfs/supermin.d
supermin helper [00004ms] visiting /usr/lib64/guestfs/supermin.d/daemon.img
supermin helper [00004ms] visiting /usr/lib64/guestfs/supermin.d/init.img
supermin helper [00004ms] visiting /usr/lib64/guestfs/supermin.d/udev-rules.img
supermin helper [00004ms] adding kernel modules
supermin helper [00234ms] finished creating appliance
libguestfs: checksum of existing appliance: 0e1c5945c4770888dc09aa2a8ec16676e74ea1bd1516ba0462dc3e0c79695937
libguestfs: [00241ms] begin building supermin appliance
libguestfs: [00241ms] run supermin-helper
libguestfs: command: run: febootstrap-supermin-helper
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ x86_64
libguestfs: command: run: \ /var/tmp/guestfs.W7RIue/kernel
libguestfs: command: run: \ /var/tmp/guestfs.W7RIue/initrd
libguestfs: command: run: \ /var/tmp/guestfs.W7RIue/root
supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = /var/tmp/guestfs.W7RIue/kernel, initrd = /var/tmp/guestfs.W7RIue/initrd, appliance = /var/tmp/guestfs.W7RIue/root
supermin helper [00000ms] inputs[0] ...

[Albert Syriy (asyriy)]

Workaround for building libguestfs without a network access doesn't work correctly on our building system.
http://libguestfs.org/download/binaries/appliance/README.txt

The build libguestfs package are luck of hostfiles (list of binaries and libraries installed on the host system) and base.img files in the rmp, what installed to the /usr/lib64/guestfs/supermin.d/ folder during installation.

As the result, the appliance could be restored by supermin correctly and libguestfs doesn't work.

To build correctly libguestfs, the build machine should contains the same list of rpm (binaries) and the structure of folders, what would be used on the host system, where the libguestfs is going to be installed.

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset:
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29429/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29429/centos"

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/29429
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29429/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-29429/centos"

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: http://gerrit.mirantis.com/28796
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-28796/centos

[Albert Syriy (asyriy)]

The fix is under review now.
branch 5.1.1
https://gerrit.mirantis.com/#/c/29429/
branch 6.0
https://gerrit.mirantis.com/#/c/28796/

Please merge if there are not comments.

[OSCI Robot (oscirobot)]

Package hivex has been built from changeset: http://gerrit.mirantis.com/29783
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-master-29783/centos

[OSCI Robot (oscirobot)]

Package hivex has been built from changeset: http://gerrit.mirantis.com/29783
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-master/centos

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: https://review.fuel-infra.org/354
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-354/centos

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: https://review.fuel-infra.org/354
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable/centos

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: https://review.fuel-infra.org/360
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-360/centos

[Albert Syriy (asyriy)]

The fix for the branch 5.1.1 has been merged at:
https://review.fuel-infra.org/#/c/354/

The fix for the branch 6.0 has been merged, please see:
https://review.fuel-infra.org/#/c/360/

[Roman Podoliaka (rpodolyaka)]

what about 5.0.3?

[Albert Syriy (asyriy)]

Unfortunately there is not the branch 5.0.3 for libguestfs now.
I have made reqest to add the branch.
https://bugs.launchpad.net/fuel/+bug/1386099

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: https://review.fuel-infra.org/439
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable-439/centos

[Albert Syriy (asyriy)]

The fix for the branch 5.0.3 has been merged:
https://review.fuel-infra.org/#/c/439/

Build could be find at the link
http://osci-jenkins.srt.mirantis.net:8080/job/f-i-build-rpm-deps-request/49/

[OSCI Robot (oscirobot)]

Package libguestfs has been built from changeset: https://review.fuel-infra.org/439
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable/centos

[Alexander Gubanov (ogubanov)]

I checked it on mos 5.1.1 and 6.0 - fixed.
Proof:

release: 5.1.1
build_id: 2014-11-23_21-01-00
build_number: '34'

[root@node-2 ~]# nova boot --image TestVM --file /data.txt=./data.txt --flavor m1.tiny --nic net-id=07434cbc-5a17-4da4-87e0-353236dcb1c6 test3

[root@node-2 ~]# ip netns exec qdhcp-07434cbc-5a17-4da4-87e0-353236dcb1c6 ssh cirros@192.168.111.6
cirros@192.168.111.6's password:
$ ls -l /data*
-rw-r--r-- 1 root root 13 Dec 12 08:12 /data.txt
$ cat /data*
Hello, World

----

release: '6.0'
build_id: 2014-12-09_19-59-53
build_number: '48'

[root@node-1 ~]# echo 'hello' > data.txt
[root@node-1 ~]# nova boot --image TestVM --file /data.txt=./data.txt --flavor m1.micro --nic net-id=5ed80282-2dbc-4d3b-88bf-0c552a923b8a test2
[root@node-1 ~]# ip netns exec qrouter-c1b03574-e458-4c1a-a3a5-71e3e80fc777 ssh cirros@192.168.111.5
The authenticity of host '192.168.111.5 (192.168.111.5)' can't be established.
cirros@192.168.111.5's password:
$ cat /data.txt
hello