Smoke test failed with ostf error:Create volume and attach it to instance (failure)

[Nastya Urlapova]

Deploy cluster in ha mode with 1 controller and Neutron VXLAN

        Scenario:
            1. Create cluster
            2. Add 1 node with controller role
            3. Add 2 nodes with compute role
            4. Run network verification
            5. Deploy the cluster
            6. Run network verification
            7. Run OSTF

Test failed on ostf step:
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Create volume and attach it to instance (failure) Time limit exceeded while waiting for volume becoming 'available' to finish. Please refer to OpenStack logs for more details.

!!! Build 254 after Centos 7 merge

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "254"
  build_id: "254"
  fuel-nailgun_sha: "0e09dce510927f2cc490b898e5fe3f813bd791be"
  python-fuelclient_sha: "f033192b84263f0e699458a4274289a5198ae7e4"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "48fd58676debcc85951db68df6d77c22daa55e52"
  fuel-library_sha: "ab7e51f345ffb7c256e0f61addcf86553d7c3867"
  fuel-ostf_sha: "23b7ae2a1a57de5a3e1861ffb7805394ca339cc2"
  fuel-mirror_sha: "8d945c9cd3a0831e7d830e22d3f7ae467af2b057"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "f38a1dc55b1bb0e2b833a62b9025df558e2d8896"

[Tatyanka (tatyana-leontovich)]

From this tests volume hangs in creating status. Also on this env volume from volume bootable tests hangs with deleteing status. Attampts to delete it manually do not help, even with force delete:
http://paste.openstack.org/show/480963/

[Tatyanka (tatyana-leontovich)]

<159>Dec 5 15:27:53 node-3 cinder-scheduler: publish_errors = False
<159>Dec 5 15:27:53 node-3 cinder-scheduler: oslo_versionedobjects.fatal_exception_format_errors = False
<158>Dec 5 15:49:55 node-3 cinder-api: Unable to delete volume: 70573e79-0f63-4d48-9b28-6589bd87d230, volume must be available or error, but is creating.
<158>Dec 5 15:50:15 node-3 cinder-api: Unable to delete volume: 3240bba0-0c2a-4668-af92-04e6a9f6ef83, volume must be available or error, but is deleting.
<158>Dec 5 17:23:28 node-3 cinder-api: Unable to delete volume: 70573e79-0f63-4d48-9b28-6589bd87d230, volume must be available or error, but is creating.
<158>Dec 5 17:24:06 node-3 cinder-api: Unable to delete volume: 3240bba0-0c2a-4668-af92-04e6a9f6ef83, volume must be available or error, but is deleting.

[Dmitry Teselkin (teselkin-d)]

We've investigated the issue and found that the test failed with timeout because 'lvremove' command executed by cinder-volume service hanged, it's process went to D state.

Rebooting VM hard (via sysrq-trigger) didn't helped - after every reboot cinder volume came back and an attempt to remove it exited in lvremove in D state.

We managed to fix it by downgrading the kernel to 'linux-image-3.13.0-71'. After a reboot (to use new kernel) we removed the volume without any issue.

I'm attaching log files of our attempts to remove the volume with both kernels. The command used:

    strace -f lvremove -f cinder 2>&1 | tee lvremove.log

Log files renamed to match kernel names used.

[Ivan Kolodyazhny (e0ne)]

Dmitry's comments sounds right.

I've found a strange behaviour during investigation this issue: https://paste.mirantis.net/show/1538/. I'm not sure yet: is it related or not?

We've got related issue, but it was reproduced only on bare-metal nodes with Rally: https://bugs.launchpad.net/mos/+bug/1369524

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Aleksandra Fedorova <email address hidden>
Review: https://review.fuel-infra.org/14435

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14435
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: 24ac0699752d91ce8bdf3efff28c0659bd885395
Author: Aleksandra Fedorova <email address hidden>
Date: Sun Dec 6 21:11:04 2015

Temporary pin UBUNTU_MIRROR.

Ubuntu kernel 3.13.0-72 which comes in recent trusty-proposed repo
causes failures of smoke_neutron test group. We temporary pin Ubuntu
mirror to snapshot with 3.13.0-71 kernel.

It is enough to pin mirror in test_all job, as swarm always uses the
same mirror id.

Change-Id: Idd140982bf112d70a5a77084b64695468291701d
Related-Bug: #1523092

[Nastya Urlapova (aurlapova)]

Ivan, Aleksandra's review was related to issue.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Panchenko <email address hidden>
Review: https://review.fuel-infra.org/15949

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/15949
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: 7a04ac48d156e279704c21095a598e83cb10fc0a
Author: Artem Panchenko <email address hidden>
Date: Sun Jan 3 10:55:23 2016

Revert "Temporary pin UBUNTU_MIRROR."

This reverts commit 24ac0699752d91ce8bdf3efff28c0659bd885395.
Closes-bug: #1530633
Related-bug: #1523092

Change-Id: I9eb9cf49031c8e1c084ff29d71f6731c9a58cd56

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Panchenko <email address hidden>
Review: https://review.fuel-infra.org/15960

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/15960
Submitter: Dmitry Kaigarodsev <email address hidden>
Branch: master

Commit: beb0011f754aba84185d9c0e0e96223e5d319792
Author: Artem Panchenko <email address hidden>
Date: Mon Jan 4 18:56:20 2016

Remove temporary pin of Ubuntu mirror

Change-Id: I7d40e06a336cee8eb623fc724335b12696288833
Closes-bug: #1530633
Related-bug: #1523092