OSTF Launch instance, create snapshot, launch instance from snapshot failed

[Oleksiy Butenko]

Test OSTF Launch instance, create snapshot, launch instance from snapshot failed

Steps:
1. Deploy env 9.2 SNAPSHOT#458
    - Controller, Telemetry - MongoDB (3)
    - Compute, Cinder (2)
    - Ironic (1)
    - Neutron with VLAN
2. Run OSTF

Expected result: all ostf tests - pass

Actual result:
Test OSTF Launch instance, create snapshot, launch instance from snapshot failed
_______________

[Oleksiy Butenko (obutenko)]

Reproduced on all tempest ci env, so it's blocker for tempest-ci

[Oleksiy Butenko (obutenko)]

if you need real env with bug feel free to contact me, I can provide access for you.

[Oleksiy Butenko (obutenko)]

https://goo.gl/b9JMjl - Diagnostic Snapshot

[Anna Babich (ababich)]

Traceback from nova-compute for failed test: http://paste.openstack.org/show/587562/

[Roman Podoliaka (rpodolyaka)]

I believe, the error above is caused by the fact that qemu / libvirt versions were updated, but the corresponding apparmor profiles have been not:

[ 4205.932731] type=1400 audit(1478015150.867:91): apparmor="DENIED" operation="mknod" profile="libvirt-29799294-e48d-46d4-b69e-2523bc334c66" name="/tmp/memfd-K1MPBa" pid=134065 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=109 ouid=109
[ 4238.772782] type=1400 audit(1478015183.707:96): apparmor="DENIED" operation="mknod" profile="libvirt-f510c840-2788-420d-b3e5-c587a52f2242" name="/tmp/memfd-i7BQBm" pid=136798 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=109 ouid=109

apparently, libvirt now *defaults* to vhost networking and this part was not properly added to the apparmor profiles we ship:

./hw/virtio/vhost.c- if (!(hdev->features & (0x1ULL << VHOST_F_LOG_ALL))) {
./hw/virtio/vhost.c- error_setg(&hdev->migration_blocker,
./hw/virtio/vhost.c- "Migration disabled: vhost lacks VHOST_F_LOG_ALL feature.");
./hw/virtio/vhost.c- } else if (!qemu_memfd_check()) {
./hw/virtio/vhost.c- error_setg(&hdev->migration_blocker,
./hw/virtio/vhost.c: "Migration disabled: failed to allocate shared memory");
./hw/virtio/vhost.c- }
./hw/virtio/vhost.c- }

[Roman Podoliaka (rpodolyaka)]

MOS Linux, please take a look at this.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/28086

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/28086
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: aebfeb916151ce13ea9af5bcbae261e471db0713
Author: Ivan Suzdal <email address hidden>
Date: Wed Nov 2 10:32:41 2016

Temporary change apparmor rules

Qemu should be able to write into /tmp
during live vm migration.

TODO: Remove this changes after closing LP#1626972.

Change-Id: I6086e22aee0d32b6c28dc89bc408e6f99fd96248
Closes-Bug: #1638269

[Anna Babich (ababich)]

It's still reproduced: https://bugs.launchpad.net/mos/+bug/1638920

[Ivan Suzdal (isuzdal)]

Anna, it seems like you use too old libvirt version. Fix was implemented in 1.3.1-1ubuntu6~u1404+mos6 version, in your snapshot I can see 1.3.1-1ubuntu6~u1404+mos4 version. Please, try with newer version.

[Sergii Turivnyi (sturivnyi)]

It reproduced on 9.2 snapshot #465

[Anna Babich (ababich)]

We can see that 1.3.1-1ubuntu6~u1404+mos6 version for libvirt was proposed starting from http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-02-134322, but it looks like snapshot #465 was deployed with older snapshot repos:
ababich@srv138-bud:~$ cat /home/jenkins/env_inject.properties
SNAPSHOT_ID=#465
UPDATE_MASTER='true'
EXTRA_RPM_REPOS='mos-os,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/os-2016-06-23-135731/x86_64|mos-proposed,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-11-02-092322/x86_64|mos-updates,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/updates-2016-10-12-105639/x86_64|mos-security,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/security-2016-06-23-140002/x86_64|mos-holdback,http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/holdback-2016-06-23-140047/x86_64'
UPDATE_FUEL_MIRROR='http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/os-2016-06-23-135731/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-11-02-092322/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/updates-2016-10-12-105639/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/security-2016-06-23-140002/x86_64 http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/holdback-2016-06-23-140047/x86_64'
EXTRA_DEB_REPOS='mos-proposed,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-02-104321 mos9.0-proposed main restricted|mos-updates,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-02-104321 mos9.0-updates main restricted|mos-security,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-02-104321 mos9.0-security main restricted|mos-holdback,deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-11-02-104321 mos9.0-holdback main restricted'
ababich@srv138-bud:~$

So, we need to fix the gap in EnvInject variables and check that the next run of Tempest job will use actual ones.

[Oleksiy Butenko (obutenko)]

we need wait for new snapshot and check it.
now job 9.x.snapshot broken https://product-ci.infra.mirantis.net/job/9.x.snapshot/ - it's root cause, why we don't have new snapshot on labs

[Yury Tregubov (ytregubov)]

The problem is not seen after 9.2 mos snapshot #498