[library] File injection during instance launching doesn't work on Ubuntu

[Alexander Rubtsov]

Release: Mirantis OpenStack 5.0
OS: Ubuntu
Networking: Nova-network (FlatDHCP)
Environment: KVM, 1 controller, 1 compute

Steps to reproduce:
from controller node:
echo "Hello World!" > ./test.txt
nova boot --image <image_ID> --flavor 1 --nic net-id=<network_ID> --file /root/test.txt=./test.txt instance-test

On compute node in /var/log/nova-all.log:
Jun 28 00:29:47 node-8 nova-nova.compute.manager ERROR: Instance failed to spawn
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1720, in _spawn
block_device_info)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in spawn
admin_pass=admin_password)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2697, in _create_image
instance, network_info, admin_pass, files, suffix)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2537, in _inject_data
instance=instance)
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2531, in _inject_data
mandatory=('files',))
File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 353, in inject_data
fs.setup()
File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 132, in setup
{'imgfile': self.imgfile, 'e': e})
NovaException: Error mounting /var/lib/nova/instances/dab50d4d-a884-4b1c-9db4-ca3c8e323f22/disk with libguestfs (external command failed, see earlier error messages)

Then we tried to create an instance without injection and it was launched successfully.
If we go on compute node and try to execute the following commands:
mkdir ./tmp
guestmount -r -a /var/lib/nova/instances/8180ff35-e371-4c69-a357-168e43e49970/disk -i /root/tmp/
We get the same error

Then we execute on the compute node:
update-guestfs-appliance
After that we can mount guest fs and we can launch an instance with injection

[Dmitry Borodaenko (angdraug)]

update-guestfs-appliance was supposed to be run from postinst script of libguestfs0 deb package:

case "$1" in
    configure|reconfigure)
        . /usr/share/debconf/confmodule
        db_get libguestfs/update-appliance
        if [ "$RET" = true ]; then
            update-guestfs-appliance >&2 || true
        fi
    ;;

The libguestfs/update-appliance is set to true by puppet here:
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/files/guestfs.seed
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/manifests/utilities.pp

[Dmitry Borodaenko (angdraug)]

Please provide a diagnostic snapshot. We need to figure out why update-guestfs-appliance wasn't run during deployment, or why it ran and failed.

[Bogdan Dobrelya (bogdando)]

could be related https://bugs.launchpad.net/fuel/+bug/1337314

[Dmitry Borodaenko (angdraug)]

This bug was Incomplete for over a month, lets add test coverage for this use case and close it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124325

[Andrey Sledzinskiy (asledzinskiy)]

Seems that issue is still reproducable
It was reproduced on cluster with CentOS, Flat nova-network, Simple, 1 controller, 1 compute
Instance creation with attached file failed
Logs are attached

[Andrey Sledzinskiy (asledzinskiy)]

Also I tried to reproduce it on
{

    "build_id": "2014-09-26_00-01-46",
    "ostf_sha": "32cf2541ab8c8aa82f1e7b1d83a74e570ca2b7c9",
    "build_number": "529",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "9c10fc7ffd4c59d940812c580e32f4304d6ae267",
    "production": "docker",
    "fuelmain_sha": "a7f7546e157d0fc80e9a5f75aadda125648e0e64",
    "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-09-26_00-01-46",
                "ostf_sha": "32cf2541ab8c8aa82f1e7b1d83a74e570ca2b7c9",
                "build_number": "529",
                "api": "1.0",
                "nailgun_sha": "9c10fc7ffd4c59d940812c580e32f4304d6ae267",
                "production": "docker",
                "fuelmain_sha": "a7f7546e157d0fc80e9a5f75aadda125648e0e64",
                "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "11c7260a3f91ac3c9e3327b9b21be32f6ade79e0"
            }
        }
    },
    "fuellib_sha": "11c7260a3f91ac3c9e3327b9b21be32f6ade79e0"

}

Cluster with Ubuntu, Flat nova-network, Simple, 1 controller, 1 compute
Instance was created but there was no /root/test.txt file

Logs are attached

[Andrey Sledzinskiy (asledzinskiy)]

Re-check it on latest master
Issue is reproduced on CentOS - instance isn't created with file injection (logs are attached)
On Ubuntu instance is created but there is no /root/test.txt file on instance
{

    "build_id": "2014-09-30_00-01-46",
    "ostf_sha": "727cd3cc93e255635fef13c80baee649ced075be",
    "build_number": "537",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "618d45b0cceef67b4da616167cab3161d55e9b52",
    "production": "docker",
    "fuelmain_sha": "9108f78577c7139353ace42f256e79d19564c98b",
    "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-09-30_00-01-46",
                "ostf_sha": "727cd3cc93e255635fef13c80baee649ced075be",
                "build_number": "537",
                "api": "1.0",
                "nailgun_sha": "618d45b0cceef67b4da616167cab3161d55e9b52",
                "production": "docker",
                "fuelmain_sha": "9108f78577c7139353ace42f256e79d19564c98b",
                "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "73f2e3ca9dca9809de31471e15b58c634a81254b"
            }
        }
    },
    "fuellib_sha": "73f2e3ca9dca9809de31471e15b58c634a81254b"

}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124325
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=53e71f7169f8c76d18ceb42477e80a39889b29b0
Submitter: Jenkins
Branch: master

commit 53e71f7169f8c76d18ceb42477e80a39889b29b0
Author: asledzinskiy <email address hidden>
Date: Fri Sep 26 11:05:08 2014 +0300

    Add test on creating instance with
    file injection

    Change-Id: I92dfe1da854dfc662f1b8b77e85e1e295b5790e6
    Related-Bug: #1335697

[Stanislav Makar (smakar)]

VMs are spawning without problem, but the feature file injection doesnt wok
I got such string from logs:
<182>Nov 24 12:23:15 node-4 nova-compute Unable to import guestfsfalling back to VFSLocalFS

[Stanislav Makar (smakar)]

Below some logs that are connected with it and are presented during vm creation in /var/log/syslog:

<3>Nov 24 11:27:08 node-4 kernel: [ 1370.718014] EXT4-fs (nbd5): VFS: Can't find ext4 filesystem
<3>Nov 24 11:27:08 node-4 kernel: [ 1370.721165] FAT-fs (nbd5): bogus number of reserved sectors
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.724521] ffff88005e503000: eb 48 90 00 00 00 00 00 00 00 00 00 00 00 00 00 .H..............
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.728925] ffff88005e503010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.736931] ffff88005e503020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.743751] ffff88005e503030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 02 ................
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.749295] XFS (nbd5): Internal error xfs_sb_read_verify at line 789 of file /build/buildd/linux-lts-saucy-3.11.0/fs/xfs/xfs_mount.c. Caller 0xffffffffa000b495
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.749295]
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.759078] XFS (nbd5): Corruption detected. Unmount and run xfs_repair
<3>Nov 24 11:27:09 node-4 kernel: [ 1370.802595] block nbd5: Unexpected reply (ffff88005e943c08)
<27>Nov 24 11:27:09 node-4 ntpd[18331]: bind(36) AF_INET6 fe80::d41a:1fff:feea:98be%13#123 flags 0x11 failed: Cannot assign requested address
<27>Nov 24 11:27:09 node-4 ntpd[18331]: unable to create socket on br100 (19) for fe80::d41a:1fff:feea:98be#123
<27>Nov 24 11:27:11 node-4 dnsmasq[21085]: failed to load names from /var/lib/nova/networks/nova-br100.hosts: No such file or directory
<3>Nov 24 12:23:15 node-4 kernel: [ 4737.665315] EXT4-fs (nbd11): VFS: Can't find ext4 filesystem
<3>Nov 24 12:23:15 node-4 kernel: [ 4737.669327] FAT-fs (nbd11): bogus number of reserved sectors
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.672965] ffff88005e840000: eb 48 90 00 00 00 00 00 00 00 00 00 00 00 00 00 .H..............
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.677691] ffff88005e840010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.684460] ffff88005e840020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.702385] ffff88005e840030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 02 ................
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.707152] XFS (nbd11): Internal error xfs_sb_read_verify at line 789 of file /build/buildd/linux-lts-saucy-3.11.0/fs/xfs/xfs_mount.c. Caller 0xffffffffa000b495
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.707152]
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.716063] XFS (nbd11): Corruption detected. Unmount and run xfs_repair
<3>Nov 24 12:23:16 node-4 kernel: [ 4737.770959] block nbd11: Unexpected reply (ffff88005e8e5c08)

and in dmesg as well.
Looks like nova uses NBD instead of guestfs.

[Stanislav Makar (smakar)]

- Packages are absent in ubuntu repo:
   python-libguestfs
   libguestfs-tools
- Packeges are not installed
   https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/manifests/utilities.pp
   https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/manifests/init.pp lines 284 and 382
   https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/compute.pp line 241

That's why NBD is using instead of libguestfs.

[Stanislav Makar (smakar)]

I have installed the version 1.14 from original Ubuntu 12.04 repo and
I have caught this
WARNING nova.virt.disk.vfs.guestfs [req-7c7ca124-e736-4edf-9d6c-c1fc015f32bc None] Unable to force
 TCG mode, libguestfs too old? GuestFS instance has no attribute 'set_backend_settings'

It would be great to have 1.26 or 1.28 of below packages:
python-libguestfs
libguestfs-tools
libguestfs0

[Stanislav Makar (smakar)]

Sorry
python-guestfs

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/137602

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/137606

[Stanislav Makar (smakar)]

depends on https://bugs.launchpad.net/fuel/+bug/1397014

[Stanislav Makar (smakar)]

upgraded to 1.18 - doesn't work , the same
propose to enable this feature and use different approaches for different OS:
     CentOS - use libguestfs
     Ubuntu - use NBD due to the problems with libguestfs

Note:
On CentOS python-guestfs is installed as dependency of package openstack-nova-compute, so it isn't necessary to install, just enable by the option inject_partition=-1.
On Ubuntu - there isn't so, then we don't install it manually and set inject_partition=1. If we dont install it, nova-compute uses nbd.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/138286

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137606
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5cc403a167ecd2d179a309d1003151e7a55567cb
Submitter: Jenkins
Branch: master

commit 5cc403a167ecd2d179a309d1003151e7a55567cb
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 27 11:15:51 2014 +0000

    Enable the server personality feature

    http://docs.openstack.org/api/openstack-compute/2/content/Server_Personality-d1e2543.html
    -Use different approaches for different OS:
     CentOS - use libguestfs
     Ubuntu - use NBD due to the problems with libguestfs

    Change-Id: Id2139b415af3f8835a6d4e9ee90e81121caf7c95
    Closes-bug: #1335697

[Anastasia Palkina (apalkina)]

Verified on ISO #49

"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}

[Anastasia Palkina (apalkina)]

Verified for CentOS and Ubuntu on ISO #49

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/138286
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9e50ab7a055d13362c00a10536e83d74a1565c6d
Submitter: Jenkins
Branch: stable/5.1

commit 9e50ab7a055d13362c00a10536e83d74a1565c6d
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 27 11:15:51 2014 +0000

    Enable the server personality feature

    http://docs.openstack.org/api/openstack-compute/2/content/Server_Personality-d1e2543.html
    -Use different approaches for different OS:
     CentOS - use libguestfs
     Ubuntu - use NBD due to the problems with libguestfs

    Change-Id: Id2139b415af3f8835a6d4e9ee90e81121caf7c95
    Closes-bug: #1335697

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stanislav Makar (<email address hidden>) on branch: master
Review: https://review.openstack.org/137602
Reason: not needed