gate-devstack-dsvm-cells fails in volumes exercise with "Server ex-vol-inst not deleted"

[Matt Riedemann]

http://logs.openstack.org/02/153902/2/gate/gate-devstack-dsvm-cells/14ce82b/console.html#_2015-02-10_04_21_33_685

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZGV2c3RhY2svZXhlcmNpc2VzL3ZvbHVtZXMuc2hcIiBBTkQgbWVzc2FnZTpcIlNlcnZlciBleC12b2wtaW5zdCBub3QgZGVsZXRlZFwiIEFORCB0YWdzOlwiY29uc29sZVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDIzNTc5NDY1MDQ0fQ==

6 hits in 7 days, check and gate, all failures.

[Matt Riedemann (mriedem)]

It first tries to delete here:

http://logs.openstack.org/02/153902/2/gate/gate-devstack-dsvm-cells/14ce82b/console.html#_2015-02-10_04_18_50_557

This is the instance uuid: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb

[Matt Riedemann (mriedem)]

Looks like the exercise is first trying to delete the instance, which doesn't exist, then does a loop doing nova show on the instance until it's gone (times out at 90 seconds).

Then it boots the instance:

http://logs.openstack.org/02/153902/2/gate/gate-devstack-dsvm-cells/14ce82b/console.html#_2015-02-10_04_18_52_638

Then it loops for about a minute waiting for the instance to be ACTIVE.

Then the test creates a volume and then tries to attach the volume to the instance. The volume attach completes in 12 seconds.

Then detaches the volume (that takes 6 seconds). Deletes the volume in 4 seconds. Then it loops for a minute waiting for the server to delete and times out.

The delete starts at:

http://logs.openstack.org/02/153902/2/gate/gate-devstack-dsvm-cells/14ce82b/console.html#_2015-02-10_04_20_32_450

And times out at:

http://logs.openstack.org/02/153902/2/gate/gate-devstack-dsvm-cells/14ce82b/console.html#_2015-02-10_04_21_33_683

After the delete request, I see this error in the n-cpu logs:

http://logs.openstack.org/02/153902/2/gate/gate-devstack-dsvm-cells/14ce82b/logs/screen-n-cpu.txt.gz#_2015-02-10_04_20_36_663

2015-02-10 04:20:36.663 ERROR nova.compute.manager [req-d2130958-4f41-4c39-9794-55c249ced5b5 demo demo] [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] Setting instance vm_state to ERROR
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] Traceback (most recent call last):
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] File "/opt/stack/new/nova/nova/compute/manager.py", line 2585, in do_terminate_instance
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] self._delete_instance(context, instance, bdms, quotas)
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] File "/opt/stack/new/nova/nova/hooks.py", line 149, in inner
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] rv = f(*args, **kwargs)
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] File "/opt/stack/new/nova/nova/compute/manager.py", line 2554, in _delete_instance
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] quotas.rollback()
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] six.reraise(self.type_, self.value, self.tb)
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] File "/opt/stack/new/nova/nova/compute/manager.py", line 2530, in _delete_instance
2015-02-10 04:20:36.663 18842 TRACE nova.compute.manager [instance: d76c4dc3-4fbb-45f0-a62a-24f6d470d8fb] self._shutdown_instance(context, instance, bdms)
2015-02-10 04:20:36.663 18842 TRACE no...

[Matt Riedemann (mriedem)]

e-r query: https://review.openstack.org/#/c/154508/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/154566

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154567

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/154566
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0ff69b5660a22463915429911cc4cea53882f9db
Submitter: Jenkins
Branch: master

commit 0ff69b5660a22463915429911cc4cea53882f9db
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 10 07:23:39 2015 -0800

    Use dot notation on instance object fields in _delete_instance

    The _delete_instance method is getting object parameters now so update
    the docstring to specify the types on the formal parameters and also
    access the instance.vm_state field using dot notation.

    Related-Bug: #1420322

    Change-Id: Iecaabeba4181983d2b6dca9de6521408d5384c24

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/154567
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9dafd17c8208a507d97c5aa3ca9f22062b521f07
Submitter: Jenkins
Branch: master

commit 9dafd17c8208a507d97c5aa3ca9f22062b521f07
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 10 08:20:14 2015 -0800

    libvirt: remove _destroy_instance_files shim

    The instance parameter passed to _destroy_instance_files is a nova
    object now so we can remove the shim and todo and move that code up to
    the cleanup method. A note is added to clarify the usage of the
    'clean_attempts' system_metadata.

    The cleanup method is called by destroy and _create_domain_and_network
    which are also getting an instance object.

    Also changes one remaining place in _create_domain_and_network where the
    instance.uuid field was accessed like a dict (changed to use dot
    notation to re-enforce that it's an object).

    Partial-Bug: #1420322

    Change-Id: Ia4d204df233fd40a9008e6cc010446c62e18a3b9

[melanie witt (melwitt)]

https://review.openstack.org/#/c/155659/