test_cancel_update_server_with_port failing intermittently

Bug #1643268 reported by Rabi Mishra
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Incomplete
Undecided
Unassigned
OpenStack Heat
New
High
Unassigned

Bug Description

test_cancel_update_server_with_port seems to be failing[1] intermittently with non-convergence with

2016-11-20 04:36:45.142985 | 2016-11-20 04:36:45.141 | heat_integrationtests.common.exceptions.StackBuildErrorException: Stack CancelUpdateTest-297766966/b0208a66-631e-4716-8785-91819afaf1a5 is in ROLLBACK_FAILED status due to 'ClientException: resources.Server: Failed to attach network adapter device to 01ce1b55-167d-4b16-a265-35661f4c1b48 (HTTP 500) (Request-ID: req-c7044e7e-b245-4d78-9709-b92c544673b9)'

From the nova logs it looks like a libvirt error[2]. Not checked if this is a nova issue or not. Filing this bug to track it.

[1] http://logs.openstack.org/76/398476/1/gate/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/1e38d87/console.html#_2016-11-20_04_36_45_142985

[2] http://logs.openstack.org/76/398476/1/gate/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/1e38d87/logs/screen-n-cpu.txt.gz?level=ERROR#_2016-11-20_04_26_56_121

Thomas Herve (therve)
Changed in heat:
importance: Undecided → Critical
importance: Critical → High
milestone: none → ocata-3
milestone: ocata-3 → ocata-2
Revision history for this message
Thomas Herve (therve) wrote :

It seems it started happening with the move to xenial, so maybe a libvirt issue. https://bugs.launchpad.net/tacker/+bug/1515768 is a previous occurrence of it.

Adding nova has thre is a 500, so there ought to be a bug.

Revision history for this message
Rabi Mishra (rabi) wrote :

It seems there is already a similar nova bug[1] Looks like interface detach is slower in xenial libvirt.

[1] https://bugs.launchpad.net/nova/+bug/1607714

Revision history for this message
Rabi Mishra (rabi) wrote :

The above mentioned bug is not yet resolved in nova. IMHO, we've 2 options to reduce the number of failures at the gate.

a. Add some sleep after the port detach during rollback[1] which may reduce the number of failures.
b. Skip test_cancel_update_server_with_port till it's fixed in nova.

[1] https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server_network_mixin.py#L505

summary: - test_cancel_update_server_with_port failing with non-convergence
- intermittently
+ test_cancel_update_server_with_port failing intermittently
Revision history for this message
Rabi Mishra (rabi) wrote :

It seems nova tempest tests[1] are skipped currently. This also seems to be happening for both convergence and non-convergence.

[1] https://review.openstack.org/#/c/348955/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/400046

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/400046
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ea3eb60f9098f5e2119f5a0de2eedc1fcf979fff
Submitter: Jenkins
Branch: master

commit ea3eb60f9098f5e2119f5a0de2eedc1fcf979fff
Author: rabi <email address hidden>
Date: Mon Nov 21 10:44:54 2016 +0530

    Skip test_cancel_update_server_with_port

    Skip test_cancel_update_server_with_port till bug 1607714 is
    fixed in nova. Related tempest nova tests are skipped
    currently[1].

    [1] https://review.openstack.org/#/c/348955/

    Change-Id: Id08866d5b428a64778423892455495c0182335af
    Related-Bug: #1643268

Rabi Mishra (rabi)
Changed in heat:
milestone: ocata-2 → ocata-3
Revision history for this message
Sean Dague (sdague) wrote :

From a nova perspective this probably has to be investigated as to whether the port was built correctly and timely from neutron.

Changed in nova:
status: New → Incomplete
Rabi Mishra (rabi)
Changed in heat:
milestone: ocata-3 → next
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.