Router port: no dataplane update on change

[Volodymyr Litovka]

Hi colleagues,

I'm using Heat to provision Neutron and faced the bug which is rather Neutron's than Heat's, since Neutron receives all information, provisions control plane ('openstack port/router show' gives updated config), but do nothing in dataplane.

My config, which I'm sending to Heat in order to change existing stack - https://pastebin.com/7J6emmSt
Changes are: qnet2-gwport's ip_address changed from 1.1.1.1 to 1.1.1.2
Information about environment is below.

After I apply changes, I see everything is updated to 1.1.1.2 in 'openstack show' outputs:

$ openstack port show cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d
[ ... ]
| fixed_ips | ip_address='1.1.1.2', subnet_id='95f6bce5-cb46-43f6-86b5-16a71306e54e'
| id | cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d
| device_id | 771bbc70-a4e9-4f42-9c51-71ba3e6b525f

$ openstack router show 771bbc70-a4e9-4f42-9c51-71ba3e6b525f
[ ... ]
| id | 771bbc70-a4e9-4f42-9c51-71ba3e6b525f
| interfaces_info | [{"subnet_id": "95f6bce5-cb46-43f6-86b5-16a71306e54e", "ip_address": "1.1.1.2", "port_id": "cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d"}]

but actual ip address in netns is old:

# ip netns exec qrouter-771bbc70-a4e9-4f42-9c51-71ba3e6b525f ip a
[ ... ]
265: qr-cbb4bb4b-7c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d0:1c:a0:97:50:fb brd ff:ff:ff:ff:ff:ff
    inet 1.1.1.1/24 brd 1.1.1.255 scope global qr-cbb4bb4b-7c

Logs with debug are available here - https://pastebin.com/Aj8WrGUt . There you will find that neutron-openvswitch-agent see changes to port:

===
Resource Port cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d updated (revision_number 26->29).

Old fields: {'fixed_ips': [IPAllocation(ip_address=1.1.1.1,network_id=d608a42b-984c-46c8-9537-8249aee4ce29,port_id=cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d,subnet_id=95f6bce5-cb46-43f6-86b5-16a71306e54e)], 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute1',level=0,port_id=cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d,segment=NetworkSegment(a72cc202-b6ec-49ed-8e42-6f7875a9131b))]}

New fields: {'fixed_ips': [IPAllocation(ip_address=1.1.1.2,network_id=d608a42b-984c-46c8-9537-8249aee4ce29,port_id=cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d,subnet_id=95f6bce5-cb46-43f6-86b5-16a71306e54e)], 'binding_levels': [PortBindingLevel(driver='openvswitch',host='compute1',level=0,port_id=cbb4bb4b-7ce4-4e58-b81e-fc6262d8c53d,segment=NetworkSegment(a72cc202-b6ec-49ed-8e42-6f7875a9131b))]}
===

but later reports "Transaction caused no change do_commit" for all commands ("Running txn command") that openvswitch-agent runs.

ENVIRONMENT:

OS: Ubuntu 16.04.5 LTS (Xenial Xerus)
Kernel: 4.15.0-24-generic #26~16.04.1-Ubuntu SMP
Neutron: 2:12.0.3-0ubuntu1~cloud0 (Queens)
Openvswitch: 2.9.0-0ubuntu1~cloud0
Patches for bug 1776468 (https://bugs.launchpad.net/neutron/+bug/1776468/) are applied.

Any help on this issue is highly appreciated. Any additional information is available upon request.

Thank you!

[Brian Haley (brian-haley)]

Interesting.

I tried the following:

$ openstack port set --no-fixed-ip --fixed-ip ip-address=10.0.0.22 342c052d-2a51-42e2-90ff-5e56a3d9197c

$ openstack port list
| 342c052d-2a51-42e2-90ff-5e56a3d9197c | | fa:16:3e:74:b7:bb | ip_address='10.0.0.22', subnet_id='b15fee4f-b8f1-4945-98be-6c44ba96bf0e'

$ openstack router show router1
| interfaces_info | [{"subnet_id": "b15fee4f-b8f1-4945-98be-6c44ba96bf0e", "ip_address": "10.0.0.22", "port_id": "342c052d-2a51-42e2-90ff-5e56a3d9197c"}, {"subnet_id": "57ebbdbb-e7e7-40cc-a06d-fdf26c489214", "ip_address": "fdba:10:ba82::1", "port_id": "d9b77c8d-3449-4a94-8707-a301a2be18e5"}] |

But I never saw the l3-agent get notified watching it's log, so the qr- interface was never changed.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

We should not be allowing options to update the router port after it has been configured. This would create lot more issues. I think there is a patch out there to prevent the router ports from getting updated.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I suspect that this is a duplicate of https://bugs.launchpad.net/neutron/+bug/1796824

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Since we are going to block any IP address update to some of the device owners as stated in the bug above #1796824.
We can mark this as duplicate.