test_server_multi_create_auto_allocate failing with "Failed to allocate the network(s), not rescheduling"

[Armando Migliaccio]

http://logs.openstack.org/93/356593/3/check/gate-tempest-dsvm-neutron-dvr/5396726/logs/testr_results.html.gz

http://logs.openstack.org/78/355078/7/check/gate-tempest-dsvm-neutron-linuxbridge/913f31a/logs/testr_results.html.gz

Traceback (most recent call last):
  File "tempest/api/compute/admin/test_auto_allocate_network.py", line 177, in test_server_multi_create_auto_allocate
    min_count=3)
  File "tempest/common/compute.py", line 167, in create_test_server
    % server['id'])
  File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "tempest/common/compute.py", line 149, in create_test_server
    clients.servers_client, server['id'], wait_until)
  File "tempest/common/waiters.py", line 75, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server 600e0bed-d693-476f-9e72-5ef34d9e00d8 failed to build and is in ERROR status
Details: {u'created': u'2016-08-21T23:02:49Z', u'code': 500, u'message': u'Build of instance 600e0bed-d693-476f-9e72-5ef34d9e00d8 aborted: Failed to allocate the network(s), not rescheduling.'}

This test:

https://review.openstack.org/#/c/327191/

merged Aug 20.

[Armando Migliaccio (armando-migliaccio)]

We got 22 failures in the gate so far in the gate queue:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20test_server_multi_create_auto_allocate%5C%22

[Armando Migliaccio (armando-migliaccio)]

http://logs.openstack.org/36/335536/17/check/gate-tempest-dsvm-neutron-linuxbridge/5cd2f53/logs/screen-q-svc.txt.gz?level=TRACE#_2016-08-22_11_12_06_071

The cleanup logic [1] is unfortunately not resilient enough.

[1] https://github.com/openstack/neutron/blob/master/neutron/services/auto_allocate/db.py#L336

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/358753

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/358771

[Paul Vinciguerra (pvinci)]

Hit this bug here: http://logs.openstack.org/59/358359/5/check/gate-tempest-dsvm-neutron-src-oslo.messaging/a615346/console.html

2016-08-23 00:03:01.763015 | Captured traceback:
2016-08-23 00:03:01.763040 | ~~~~~~~~~~~~~~~~~~~
2016-08-23 00:03:01.763075 | Traceback (most recent call last):
2016-08-23 00:03:01.763153 | File "tempest/api/compute/admin/test_auto_allocate_network.py", line 177, in test_server_multi_create_auto_allocate
2016-08-23 00:03:01.763179 | min_count=3)
2016-08-23 00:03:01.763231 | File "tempest/common/compute.py", line 167, in create_test_server
2016-08-23 00:03:01.763258 | % server['id'])
2016-08-23 00:03:01.763340 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-23 00:03:01.763370 | self.force_reraise()
2016-08-23 00:03:01.763454 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-23 00:03:01.763517 | six.reraise(self.type_, self.value, self.tb)
2016-08-23 00:03:01.763604 | File "tempest/common/compute.py", line 149, in create_test_server
2016-08-23 00:03:01.763655 | clients.servers_client, server['id'], wait_until)
2016-08-23 00:03:01.763710 | File "tempest/common/waiters.py", line 75, in wait_for_server_status
2016-08-23 00:03:01.763758 | server_id=server_id)
2016-08-23 00:03:01.763938 | tempest.exceptions.BuildErrorException: Server 895257b2-65f3-4588-8e25-eaa8c57befe9 failed to build and is in ERROR status
2016-08-23 00:03:01.764095 | Details: {u'code': 500, u'created': u'2016-08-22T23:31:17Z', u'message': u'Build of instance 895257b2-65f3-4588-8e25-eaa8c57befe9 aborted: Failed to allocate the network(s), not rescheduling.'}

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/358771
Reason: superseded by https://review.openstack.org/#/c/357662/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/358753
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=11849b7279c5e7afe686fab6f4b21ce6f7366c14
Submitter: Jenkins
Branch: master

commit 11849b7279c5e7afe686fab6f4b21ce6f7366c14
Author: Kevin Benton <email address hidden>
Date: Fri Aug 19 21:04:15 2016 -0700

    Make auto allocate cleanup retry

    This adds a retry decorator to the auto allocate cleanup logic
    so it can handle retriable errors during cleanup. It also adds
    catches for notfound errors to make the function idempotent so
    it works on retries.

    Change-Id: I3d62af3018fb834ec85771d8bc8e7379cc80b72a
    Closes-Bug: #1615710

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/361639

[Matt Riedemann (mriedem)]

The tempest test skip is reverted here:

https://review.openstack.org/#/c/361671/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/361639

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.