Unable to associate floating IP 172.24.5.4 to fixed IP 10.1.0.5 for instance ...

[Imre Farkas]

Gate seems to be failing because of:

2016-01-25 12:27:22.139 ERROR nova.api.openstack.compute.floating_ips [req-ebc3999e-b054-400f-a85e-bc3446a49f44 tempest-BaremetalBasicOps-1761846040 tempest-BaremetalBasicOps-1606508311] Unable to associate floating IP 172.24.5.4 to fixed IP 10.1.0.5 for instance 213b10ac-06eb-4ea0-b638-263f2b75a329. Error: Router a771a17f-0224-4293-b757-59fcc6d21410 could not be found
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last):
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/new/nova/nova/api/openstack/compute/floating_ips.py", line 238, in _add_floating_ip
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips fixed_address=fixed_address)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/new/nova/nova/network/base_api.py", line 77, in wrapper
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips res = f(self, context, *args, **kwargs)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 1203, in associate_floating_ip
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips client.update_floatingip(fip['id'], {'floatingip': param})
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 100, in with_params
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips ret = self.function(instance, *args, **kwargs)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 714, in update_floatingip
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips return self.put(self.floatingip_path % (floatingip), body=body)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in put
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips headers=headers, params=params)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 243, in retry_request
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips headers=headers, params=params)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 206, in do_request
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips self._handle_fault_response(status_code, replybody)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips exception_handler_v20(status_code, des_error_body)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 69, in exception_handler_v20
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips status_code=status_code)
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips NotFound: Router a771a17f-0224-4293-b757-59fcc6d21410 could not be found
2016-01-25 12:27:22.139 2247 ERROR nova.api.openstack.compute.floating_ips

http://logs.openstack.org/10/255310/10/check/gate-tempest-dsvm-ironic-pxe_ssh/67c7e80/logs/screen-n-api.txt.gz#_2016-01-25_12_27_22_139

[Vladyslav Drok (vdrok)]

In this run, I see the following in the q-svc log:

http://logs.openstack.org/10/255310/10/check/gate-tempest-dsvm-ironic-pxe_ssh/67c7e80/logs/screen-q-svc.txt.gz#_2016-01-25_12_27_22_134

and in the other run:

http://logs.openstack.org/18/267918/4/check/gate-tempest-dsvm-ironic-pxe_ssh/5c62aec/logs/screen-q-svc.txt.gz#_2016-01-22_21_45_08_868

[Vladyslav Drok (vdrok)]

It is also reproducible locally, but the router is in fact still there:

$ neutron router-list
+--------------------------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| aa710b2d-9f19-4ef6-8df6-59a9ea9ee28c | router1 | {"network_id": "d793c2c6-90c4-45c0-aa83-551c4dc6c5cd", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "9b283d07-68be-4978-9d7f-9b49d05dbfd4", "ip_address": "172.24.4.2"}, {"subnet_id": "d66f90ce-c040-46ec-9c37-d19b4b3c2d4e", "ip_address": "2001:db8::1"}]} | False | False |
+--------------------------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+

[aeva black (tenbrae)]

Setting Importance to High because this is seriously affecting our CI jobs. Kibana shows 390 failures between Jan 22 and 26. The errors begin appearing on Jan 22

I've proposed a query to E-R here: https://review.openstack.org/272541

[Ruby Loo (rloo)]

Looks like the issue might be due to a neutron change, jroll proposed this revert: https://review.openstack.org/#/c/272564/

[Ryan Tidwell (ryan-tidwell)]

I looked through the Neutron q-svc log, I'm not yet convinced reverting this change will help. I didn't see anything that would indicate to me that Neutron notifications are raising exceptions or interfering with CRUD on routers of floating ip's.

[Ryan Tidwell (ryan-tidwell)]

I have a fix in hand, but a quick workaround would be to ensure that in the job the tenant requesting the floating IP association is the same as the owner of the router. That may not be possible, but it would get you around the problem.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/272709

[Vladyslav Drok (vdrok)]

Hi Ryan, thanks for looking into this. Yep, owning the router by the tenant is not an option for this job, its whole point is to ensure that demo tenant can spawn ironic instances. The revert worked too, but I guess it breaks some other stuff you've implemented already :) And the timeouts in ironic pxe_ipa job may be because of the hardware - I see that it was run on core 2 duo t7700 :(

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/272709
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0e5c3fa44d72fef56346d1cdd2426aa6fe3a3475
Submitter: Jenkins
Branch: master

commit 0e5c3fa44d72fef56346d1cdd2426aa6fe3a3475
Author: Ryan Tidwell <email address hidden>
Date: Tue Jan 26 11:46:07 2016 -0800

    Elevate context for router lookups during floating IP association.

    During a floating IP association, the tenant making the request
    may not always be the owner of the router. To make the association,
    Neutron needs to query the router details internally but needs to
    use an elevated context to do so. Otherwise, the user sees a
    cryptic error stating that the router doesn't exist.

    Change-Id: If2bd6baa785ff879c61ce12e70b62e0ba25635f5
    Closes-Bug: #1537754

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jim Rollenhagen (<email address hidden>) on branch: master
Review: https://review.openstack.org/272564
Reason: Thanks Ryan/Brian :)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.