ExternalGatewayForFloatingIPNotFound exception raised in gate-tempest-dsvm-neutron-full

Bug #1584920 reported by Armando Migliaccio
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Kevin Benton

Bug Description

http://logs.openstack.org/10/316610/3/gate/gate-tempest-dsvm-neutron-full/89ec2d5/logs/testr_results.html.gz

Traceback (most recent call last):
  File "tempest/test.py", line 113, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_network_advanced_server_ops.py", line 112, in test_server_connectivity_reboot
    server, keypair, floating_ip = self._setup_network_and_servers()
  File "tempest/scenario/test_network_advanced_server_ops.py", line 65, in _setup_network_and_servers
    floating_ip = self.create_floating_ip(server, public_network_id)
  File "tempest/scenario/manager.py", line 858, in create_floating_ip
    fixed_ip_address=ip4
  File "tempest/lib/services/network/floating_ips_client.py", line 21, in create_floatingip
    return self.create_resource(uri, post_data)
  File "tempest/lib/services/network/base.py", line 60, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "tempest/lib/common/rest_client.py", line 259, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "tempest/lib/common/rest_client.py", line 641, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 694, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'type': u'ExternalGatewayForFloatingIPNotFound', u'detail': u'', u'message': u'External network 2698f174-8255-4521-927a-4d70212c3e94 is not reachable from subnet a7516d4c-3c9a-48c0-97ee-83059e727294. Therefore, cannot associate Port f73aa386-f1b3-4f88-bcf4-c2e03a565d15 with a Floating IP.'}

http://logs.openstack.org/09/312009/11/gate/gate-tempest-dsvm-neutron-full/0b89afa/logs/testr_results.html.gz

Traceback (most recent call last):
  File "tempest/test.py", line 112, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "tempest/scenario/test_network_basic_ops.py", line 398, in test_network_basic_ops
    self._setup_network_and_servers()
  File "tempest/scenario/test_network_basic_ops.py", line 123, in _setup_network_and_servers
    floating_ip = self.create_floating_ip(server)
  File "tempest/scenario/manager.py", line 858, in create_floating_ip
    fixed_ip_address=ip4
  File "tempest/lib/services/network/floating_ips_client.py", line 21, in create_floatingip
    return self.create_resource(uri, post_data)
  File "tempest/lib/services/network/base.py", line 60, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "tempest/lib/common/rest_client.py", line 260, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "tempest/lib/common/rest_client.py", line 649, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 747, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'External network bbb07097-8453-41ed-b9fc-a017f5bf7c20 is not reachable from subnet e1dcd01e-5a58-498a-b2a7-58167fd67e76. Therefore, cannot associate Port 603843ca-f04a-4301-9582-4a359e5db66a with a Floating IP.', u'detail': u'', u'type': u'ExternalGatewayForFloatingIPNotFound'}

tags: added: gate-failure
Changed in neutron:
importance: Undecided → High
Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

This isn't very common but it seems to show up a couple times per day over the last 9 - 10 days [1]. Given than the logstash history is only about 10 days (I think) it is impossible to tell if it started on the 16th or if it has been happening for a while.

I looked through the neutron log around the time of the failure and didn't see any errors around the time this test ran. So I think we'll just have to take a look at the tests that are failing and try to imagine why this particular exception would be thrown. I can't think of a better way right now.

[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ExternalGatewayForFloatingIPNotFound%5C%22

Changed in neutron:
assignee: nobody → Carl Baldwin (carl-baldwin)
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

For what it could figure out it seems like the test fails because it is trying to allocate a floating IP and associate it to a port that exists on a subnet for which Neutron says it is not connected to the external network via a router. However from [1] and [2] (which is the span of time where the test is executed) it seems the setup is correct, i.e. the router is uplinked to the external network and connected to the subnet. Now if I look at [3], which is where the exception is raised, I wonder what change might have caused the regression...

[1] http://logs.openstack.org/10/316610/3/gate/gate-tempest-dsvm-neutron-full/89ec2d5/logs/tempest.txt.gz#_2016-05-17_09_15_18_221
[2] http://logs.openstack.org/10/316610/3/gate/gate-tempest-dsvm-neutron-full/89ec2d5/logs/tempest.txt.gz#_2016-05-17_09_15_29_945
[3] https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L908

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/322372

Changed in neutron:
assignee: Carl Baldwin (carl-baldwin) → Kevin Benton (kevinbenton)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/322372
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5109d16bc52abb20636c94c0208fb498d1c626eb
Submitter: Jenkins
Branch: master

commit 5109d16bc52abb20636c94c0208fb498d1c626eb
Author: Kevin Benton <email address hidden>
Date: Sat May 14 23:44:23 2016 -0700

    Make deepcopy of update body in API layer

    This adjusts the update handler in the API layer to pass a deepcopy
    of the original update body to ensure that the original is used on
    any generated retry requests.

    Without this update handlers that modified the body as they processed
    it (e.g. the external gateway update for routers) would be missing
    data on a retry request.

    This adds a test case for the bug that was caused by this.

    Closes-Bug: #1584920
    Change-Id: I88fb9d45eb82ef48e0d865decdcc7fad1c5fb361

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/322625

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/322626

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/mitaka)

Reviewed: https://review.openstack.org/322625
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7a2cf43bc23224ec9c3d238c324d467658f8b506
Submitter: Jenkins
Branch: stable/mitaka

commit 7a2cf43bc23224ec9c3d238c324d467658f8b506
Author: Kevin Benton <email address hidden>
Date: Sat May 14 23:44:23 2016 -0700

    Make deepcopy of update body in API layer

    This adjusts the update handler in the API layer to pass a deepcopy
    of the original update body to ensure that the original is used on
    any generated retry requests.

    Without this update handlers that modified the body as they processed
    it (e.g. the external gateway update for routers) would be missing
    data on a retry request.

    This adds a test case for the bug that was caused by this.

    Closes-Bug: #1584920
    Change-Id: I88fb9d45eb82ef48e0d865decdcc7fad1c5fb361
    (cherry picked from commit 5109d16bc52abb20636c94c0208fb498d1c626eb)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/322626
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=483ceec08395c4bfdfadd23cf628d2149b33964f
Submitter: Jenkins
Branch: stable/liberty

commit 483ceec08395c4bfdfadd23cf628d2149b33964f
Author: Kevin Benton <email address hidden>
Date: Sat May 14 23:44:23 2016 -0700

    Make deepcopy of update body in API layer

    This adjusts the update handler in the API layer to pass a deepcopy
    of the original update body to ensure that the original is used on
    any generated retry requests.

    Without this update handlers that modified the body as they processed
    it (e.g. the external gateway update for routers) would be missing
    data on a retry request.

    This adds a test case for the bug that was caused by this.

    Closes-Bug: #1584920
    Change-Id: I88fb9d45eb82ef48e0d865decdcc7fad1c5fb361
    (cherry picked from commit 5109d16bc52abb20636c94c0208fb498d1c626eb)

tags: added: in-stable-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 9.0.0.0b1

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/neutron 7.1.1

This issue was fixed in the openstack/neutron 7.1.1 release.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/neutron 8.1.2

This issue was fixed in the openstack/neutron 8.1.2 release.

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.