Create volume and boot instance from it failed on step server deletion

[Tatyanka]

Destroy two controllers and check pacemaker status is correct

Scenario:
1. Destroy first controller
2. Check pacemaker status
3. Run OSTF
4. Revert environment
5. Destroy second controller
6. Check pacemaker status
7. Run OSTF

Actual Result:
OStf failed on step 7:
Create volume and boot instance from it (failure)
Instance do not become active, so deletion starts, and failed by timeout
fuel_health.test: DEBUG: Waiting for <Server: ost1_test-boot-volume-instance1099375625> to get to ACTIVE status. Currently in build status
fuel_health.test: DEBUG: Sleeping for 10 seconds
fuel_health.common.test_mixins: INFO: STEP:5, verify action: 'server deletion'
fuel_health.nmanager: DEBUG: Deleting server.
fuel_health.test: DEBUG: Sleeping for 10 seconds
fuel_health.test: DEBUG: Sleeping for 10 seconds
fuel_health.common.test_mixins: INFO: Timeout 30s exceeded for server deletion
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 223, in __exit__
    raise AssertionError(msg)
AssertionError: Time limit exceeded while waiting for server deletion to finish.

So looks like create instace after destructive actions take a liitle bit more time, so may we need to increase timeout for instance creation

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "408"
  build_id: "408"
  fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "7ef751bdc0e4601310e85b8bf713a62ed4aee305"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "8bb8c70efc61bcf633e02d6054dbf5ec8dcf6699"
  fuelmenu_sha: "2a0def56276f0fc30fd949605eeefc43e5d7cc6c"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "62573cb2a8aa54845de9303b4a30935a90e1db61"

[Egor Kotko (ykotko)]

Seems the root of issue not only in time-outs. We are waiting for dictionary as response, but got the error message:
ERROR: Gateway Time-out (HTTP 504)

Steps to reproduce:
1). Deploy cluster: 3Controllers, 2Computes, 1Cinder
2). Destroy (Force Shutoff) one controller
3). Create cinder volume

Expected:
Volume will be created in ~2min
Response will be like(in cli):
http://paste.openstack.org/show/483768/
in rest api - dictionary

Actual:
Response: error message:
http://paste.openstack.org/show/483769/
but the volume was created.
Approximate time of volume creation ~5 min

https://drive.google.com/file/d/0BzWDM1PONYEub0FpdXlMdzQ3S1U/view?usp=sharing

[Dmitry Pyzhov (dpyzhov)]

This is incorrect behaviour of cinder. Moving to mos-cinder team.

[Yuriy Nesenenko (ynesenenko)]

Looks like the used env is slow according to the appropriate time of volume creation (the above mentioned ~2-5min). So I don't think that it is incorrect behaviour of cinder.

[Yuriy Nesenenko (ynesenenko)]

I think that the used env is slow according to the appropriate time of volume creation (the above mentioned ~2-5min). Please check it out on a faster environment. The appropriate time of volume creation should be < 1 min.

[Tatyanka (tatyana-leontovich)]

@Ivan - please use snapshots and described steps to reproduce, also ping mos -qa for the help here

[Tatyanka (tatyana-leontovich)]

Note according to test rail, the same issue happens on baremetal env https://mirantis.testrail.com/index.php?/tests/view/2465860&group_by=tests:status_id&group_order=asc&group_id=8 - so do not thinks that problem is in slow bm environment that appears only after destructive actions

[Egor Kotko (ykotko)]

Reproduced again on the baremetal environment, sometimes it is possible reproduce the problem not from the first time.
Destroy controller start test if was not reproduced:
start destroyed controller wait it is online and destroy another one after start the test.

https://drive.google.com/file/d/0BzWDM1PONYEuM0pYOUNwTGt0aG8/view?usp=sharing

[Roman Podoliaka (rpodolyaka)]

Another failure here: http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom_system_test/1669/consoleFull

[Ivan Kolodyazhny (e0ne)]

Upstream bug: https://bugs.launchpad.net/cinder/+bug/1539057

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/16539

[Roman Podoliaka (rpodolyaka)]

Do we have a stable repro? Also, why RabbitMQ is down?

The last thing I want to do here is introduce a last minute fix, which can break other things. Maybe it makes to move this to -updates?

[Tatyanka (tatyana-leontovich)]

+ occurrence https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/123/testReport/%28root%29/ha_neutron_destroy_controllers/ha_neutron_destroy_controllers/

[Tatyanka (tatyana-leontovich)]

+ occurrence
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/123/testReport/%28root%29/ha_neutron_test_3_1_rabbit_failover/ha_neutron_test_3_1_rabbit_failover/

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/16539

[Roman Podoliaka (rpodolyaka)]

We haven't seen in this in the field yet, only on CI (rarely). The proposed fix is too risky to be merged at this stage of the release cycle. We believe it's safe to move this to a MU and continue investigation of this issue.

[Ivan Kolodyazhny (e0ne)]

We've got a hacky workaround. Proper fix [1] requires fix in oslo.messaging to support 'retry' and 'timeout' params per client. For now, these params are global.

[1] https://review.openstack.org/274148 - PoC

[Roman Podoliaka (rpodolyaka)]

Given the fact we haven't actually seen this in the field, I suggest we skip the release notes part.

[Dmitry Belyaninov (dbelyaninov)]

Also applicable for 7.0
https://mirantis.testrail.com/index.php?/tests/view/4044632

[Dmitry Mescheryakov (dmitrymex)]

The original issue is definitely the same as #1560097. Since we already tracking all the work there, I marking current bug as duplicate.

Re issue that Egor have found in comment #2 - it is al least a very different issue, but it might be that that one is a wrong test as well. For instance, it is not clear if Egor waited for RabbitMQ to recover after destructive action. Please file a separate issue if you experience it again and also please post snippets of the logs with timestamps indicating when the issue occurred, because it is completely unclear from the comment.