Neutron incorrectly selects subnet

[David O Neill]

Distro bionic
Openstack version: openstack testing cloud

When using the command

openstack server add floating ip --fixed-ip-address 10.66.0.18 juju-53a7bc-north-0 10.245.163.125

for the following machine

ubuntu@dmzoneill-bastion:~$ openstack server list -c ID -c Name -c Networks
+--------------------------------------+--------------------------+-----------------------------------------------------------------+
| ID | Name | Networks |
+--------------------------------------+--------------------------+-----------------------------------------------------------------+
| 063c2e8e-4d57-4267-a506-4c7b336e71b6 | juju-53a7bc-north-0 | north=10.66.0.18; south=10.55.0.9 |
| 670bd827-f570-439f-b17f-9aaaa1710849 | juju-6de968-south-0 | south=10.55.0.4 |
| 0cd3c498-2826-4868-9384-e12e0799f903 | juju-855490-default-0 | dmzoneill_admin_net=10.5.0.6 |
| 0b72ccd8-b694-45c7-86be-870511426140 | juju-370447-controller-0 | north=10.66.0.14; south=10.55.0.3; dmzoneill_admin_net=10.5.0.8 |
| 40c68cb2-4e20-4d15-a82c-4c4252b8a0da | dmzoneill-bastion | dmzoneill_admin_net=10.5.0.7, 10.245.162.200 |
+--------------------------------------+--------------------------+-----------------------------------------------------------------+

Neutron returns the error

GET call to network for http://10.245.161.159:9696/v2.0/ports?device_id=063c2e8e-4d57-4267-a506-4c7b336e71b6 used request id req-d915168a-32c6-4c74-9a83-1ef090b376d8
Manager serverstack ran task network.GET.ports in 0.122786998749s
Manager serverstack running task network.PUT.floatingips
REQ: curl -g -i -X PUT http://10.245.161.159:9696/v2.0/floatingips/0c771099-ca95-4447-8a60-5f64a590d943 -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.18.4 CPython/2.7.15+" -H "Content-Type: application/json" -H "X-Auth-Token: {SHA1}9cf2688baf3b5c3e5dea7e2f7faa6554ee1b6bfb" -d '{"floatingip": {"fixed_ip_address": "10.66.0.18", "port_id": "6f8626ba-ae8c-492a-93ad-3f349c600a3b"}}'
http://10.245.161.159:9696 "PUT /v2.0/floatingips/0c771099-ca95-4447-8a60-5f64a590d943 HTTP/1.1" 400 169
RESP: [400] Content-Type: application/json Content-Length: 169 X-Openstack-Request-Id: req-1a8be2ec-456e-483d-a4e0-5ab204c55c2d Date: Fri, 07 Feb 2020 14:45:13 GMT Connection: keep-alive
RESP BODY: {"NeutronError": {"message": "Bad floatingip request: Port 6f8626ba-ae8c-492a-93ad-3f349c600a3b does not have fixed ip 10.66.0.18.", "type": "BadRequest", "detail": ""}}

Neutron seems to look at the list of networks associated with server and pops the last network (south) from the list.

Enutron selects the port south 6f8626ba-ae8c-492a-93ad-3f349c600a3b which is not in that subnet and errors

ubuntu@dmzoneill-bastion:~$ openstack port list -c ID -c "Fixed IP Addresses"
+--------------------------------------+---------------------------------------------------------------------------+
| ID | Fixed IP Addresses |
+--------------------------------------+---------------------------------------------------------------------------+
| 1ad11c89-8574-49fc-9efe-b4a0b73b01eb | ip_address='10.55.0.2', subnet_id='eec9530d-df77-41eb-8e85-9ef6e45931a7' |
| 36b16756-01fe-49c7-9e14-41abaf0059f9 | ip_address='10.5.0.8', subnet_id='7edee502-ab23-46af-b446-17c233d11a94' |
| 3e20a885-6421-4acd-b936-56b3bcb4930f | ip_address='10.5.0.6', subnet_id='7edee502-ab23-46af-b446-17c233d11a94' |
| 410beab6-c2ed-4d31-bc3d-4457a3c28b5f | ip_address='10.55.0.4', subnet_id='eec9530d-df77-41eb-8e85-9ef6e45931a7' |
| 4e1defd4-1185-4f54-bb93-768dbf8d6436 | ip_address='10.66.0.14', subnet_id='fe280e59-e8e0-4cdc-923a-1b15e45b95ce' |
| 56d6a38d-0698-491b-9996-b239a7d95d5b | ip_address='10.55.0.3', subnet_id='eec9530d-df77-41eb-8e85-9ef6e45931a7' |
| 67558707-ed0b-47b6-be0e-0da0bf2871b5 | ip_address='10.66.0.2', subnet_id='fe280e59-e8e0-4cdc-923a-1b15e45b95ce' |
| 6d3c6101-75c3-47d0-a496-83cc62092f2d | ip_address='10.5.0.7', subnet_id='7edee502-ab23-46af-b446-17c233d11a94' |
| 6f8626ba-ae8c-492a-93ad-3f349c600a3b | ip_address='10.55.0.9', subnet_id='eec9530d-df77-41eb-8e85-9ef6e45931a7' |
| 778899dd-7c15-47a9-8968-261088aa14bf | ip_address='10.66.0.18', subnet_id='fe280e59-e8e0-4cdc-923a-1b15e45b95ce' |
| 7a07c64b-6d7b-4a57-b553-96459337f4cc | ip_address='10.66.0.1', subnet_id='fe280e59-e8e0-4cdc-923a-1b15e45b95ce' |
| 8ea03785-c046-4840-be82-0cd4bad5b4e8 | ip_address='10.5.0.2', subnet_id='7edee502-ab23-46af-b446-17c233d11a94' |
| cf0f0aea-e962-4ddf-ac7a-22c43ad483b0 | ip_address='10.5.0.1', subnet_id='7edee502-ab23-46af-b446-17c233d11a94' |
+--------------------------------------+---------------------------------------------------------------------------+

The work around is to use the floating ip interface

openstack floating ip set --port 778899dd-7c15-47a9-8968-261088aa14bf --fixed-ip-address 10.66.0.18 10.245.163.125

ubuntu@dmzoneill-bastion:~$ openstack server list -c ID -c Name -c Networks
+--------------------------------------+--------------------------+-----------------------------------------------------------------+
| ID | Name | Networks |
+--------------------------------------+--------------------------+-----------------------------------------------------------------+
| 063c2e8e-4d57-4267-a506-4c7b336e71b6 | juju-53a7bc-north-0 | north=10.66.0.18, 10.245.163.125; south=10.55.0.9 |
| 670bd827-f570-439f-b17f-9aaaa1710849 | juju-6de968-south-0 | south=10.55.0.4 |
| 0cd3c498-2826-4868-9384-e12e0799f903 | juju-855490-default-0 | dmzoneill_admin_net=10.5.0.6 |
| 0b72ccd8-b694-45c7-86be-870511426140 | juju-370447-controller-0 | north=10.66.0.14; south=10.55.0.3; dmzoneill_admin_net=10.5.0.8 |
| 40c68cb2-4e20-4d15-a82c-4c4252b8a0da | dmzoneill-bastion | dmzoneill_admin_net=10.5.0.7, 10.245.162.200 |
+--------------------------------------+--------------------------+-----------------------------------------------------------------+

[Brian Haley (brian-haley)]

I'm not sure this is a bug in neutron, but possibly in either the client or openstacksdk.

The PUT has the incorrect port id:

{"floatingip": {"fixed_ip_address": "10.66.0.18", "port_id": "6f8626ba-ae8c-492a-93ad-3f349c600a3b"}}'

It should have 778899dd-7c15-47a9-8968-261088aa14bf

Can you run the command with an increased verbosity using -vv/-vvv ? I'm curious what the GET returns for port information, for example this command:

GET call to network for http://10.245.161.159:9696/v2.0/ports?device_id=063c2e8e-4d57-4267-a506-4c7b336e71b6

Something I also noticed is that the client code in openstackclient/compute/v2/server.py is getting all the ports and then filtering if a fixed_ip is given, it could have passed the fixed_ip to "port list" to have it filter for it, at least I think that should work and might also fix the issue.

[Akihiro Motoki (amotoki)]

I agree with Brian that it is not a bug in neutron and possibly it is a bug in python-openstackcliet or openstacksdk.
Which version of python-openstackclient and openstacksdk do you use?

Looking at openstackclient/compute/v2/server.py and its history, we had an improvement to handle multiple ports of server in stein release. If you use rocky or older releases of OSC, you may hit this kind of problem.

[1] https://github.com/openstack/python-openstackclient/blob/e07324e30fbb24e89fd63d1c5a5fe485f693a45c/openstackclient/compute/v2/server.py#L269

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]