linuxbridge-agent not reassigning all IP address when creating new bridge

[Ralf Haferkamp]

When creating a new bridge the linuxbridge-agent (ensure_bridge()) is calling update_interface_ip_details(bridge_name, interface, ips, gateway) to reassign all IP addresses and routes from the underlying physical interface to the bridge.

However when the underlying interface has multiple addresses assigned that share the same subnet it will only reassign the first IP address to the bridge. E.g. consider this underlying interface

7: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:77:77:01:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.2/24 brd 192.168.122.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet 192.168.122.4/24 brd 192.168.122.255 scope global secondary bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:77ff:fe77:101/64 scope link
       valid_lft forever preferred_lft forever

After the agent took over and create a bridge on it will only have 192.168.122.2/24 assigned.

I believe the bug was introduced with: https://review.openstack.org/#/c/474170/

Relevant log excerpt (this is a newton based install, but the problem exists in master as well):

2017-10-27 14:18:26.464 21233 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Done starting bridge brqcd296387-00 for subinterface bond0 ensure_bridge /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:383
2017-10-27 14:18:26.464 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command: ['ip', 'addr', 'show', 'brqcd296387-00', 'to', '192.168.122.2/24'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89
2017-10-27 14:18:26.480 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.480 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command (rootwrap daemon): ['ip', '-4', 'addr', 'add', '192.168.122.2/24', 'scope', 'global', 'dev', 'brqcd296387-00', 'brd', '192.168.122.255'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-10-27 14:18:26.483 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.483 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command: ['ip', 'addr', 'show', 'brqcd296387-00', 'to', '192.168.122.4/24'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89
2017-10-27 14:18:26.488 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.489 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command (rootwrap daemon): ['ip', '-4', 'route', 'replace', 'default', 'via', '192.168.122.1', 'metric', '100', 'dev', 'brqcd296387-00'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-10-27 14:18:26.493 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.493 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command (rootwrap daemon): ['ip', '-4', 'route', 'del', 'default', 'via', '192.168.122.1', 'dev', 'bond0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-10-27 14:18:26.495 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.496 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command (rootwrap daemon): ['ip', '-4', 'addr', 'del', '192.168.122.2/24', 'dev', 'bond0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-10-27 14:18:26.498 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.498 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command (rootwrap daemon): ['ip', '-4', 'addr', 'del', '192.168.122.4/24', 'dev', 'bond0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-10-27 14:18:26.501 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-10-27 14:18:26.501 21233 DEBUG neutron.agent.linux.utils [req-63495c60-52fd-4d0c-acf3-2ccb80aead39 - - - - -] Running command (rootwrap daemon): ['brctl', 'addif', 'brqcd296387-00', 'bond0'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105

[Brian Haley (brian-haley)]

Yes, I think I see the problem. ip['cidr'] has a prefix length on the end, so it's not matching correctly. For example:

--> ip a s lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

--> ip a s lo to 127.0.0.2/8
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever

--> ip a s lo to 127.0.0.2

I think we have to change get_devices_with_ip() in the ip_lib code, let me send out a test patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/515801

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/515801
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7b8289253c355f39ac84bc6ea36d265a8be0e381
Submitter: Zuul
Branch: master

commit 7b8289253c355f39ac84bc6ea36d265a8be0e381
Author: Brian Haley <email address hidden>
Date: Fri Oct 27 14:01:52 2017 -0400

    Fix callers of get_devices_with_ip() to pass addresses

    If callers of get_devices_with_ip(), or
    device.addr.list(to=address) pass an ip_cidr, it
    could match any ip_cidr in that range on the interface.
    Callers need to pass the IP without the prefix portion in
    order to match it exactly. Added a helper utility to
    strip the cidr part from a ip_cidr.

    Determined the unit test for this can't actually check
    this case since we are mocking the return value from
    /sbin/ip, so modified it to just make sure the dict
    is correct.

    Added a functional test that adds two IP addresses in
    the same IP range to verify that we actually filter
    correctly when a 'to=IP' is specified.

    Change-Id: I3a95b3bb72a43f322ad23892d8959398aac22a1c
    Closes-bug: #1728080

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/531701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531701
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=15d843a2cafb6ef44974fac7628baf6a38cdd950
Submitter: Zuul
Branch: stable/pike

commit 15d843a2cafb6ef44974fac7628baf6a38cdd950
Author: Brian Haley <email address hidden>
Date: Fri Oct 27 14:01:52 2017 -0400

    Fix callers of get_devices_with_ip() to pass addresses

    If callers of get_devices_with_ip(), or
    device.addr.list(to=address) pass an ip_cidr, it
    could match any ip_cidr in that range on the interface.
    Callers need to pass the IP without the prefix portion in
    order to match it exactly. Added a helper utility to
    strip the cidr part from a ip_cidr.

    Determined the unit test for this can't actually check
    this case since we are mocking the return value from
    /sbin/ip, so modified it to just make sure the dict
    is correct.

    Added a functional test that adds two IP addresses in
    the same IP range to verify that we actually filter
    correctly when a 'to=IP' is specified.

    Change-Id: I3a95b3bb72a43f322ad23892d8959398aac22a1c
    Closes-bug: #1728080
    (cherry picked from commit 7b8289253c355f39ac84bc6ea36d265a8be0e381)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/534720

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/534720
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2b220321469a4ac2e7c1788e07bd9a72b07f93ea
Submitter: Zuul
Branch: stable/ocata

commit 2b220321469a4ac2e7c1788e07bd9a72b07f93ea
Author: Brian Haley <email address hidden>
Date: Fri Oct 27 14:01:52 2017 -0400

    Fix callers of get_devices_with_ip() to pass addresses

    If callers of get_devices_with_ip(), or
    device.addr.list(to=address) pass an ip_cidr, it
    could match any ip_cidr in that range on the interface.
    Callers need to pass the IP without the prefix portion in
    order to match it exactly. Added a helper utility to
    strip the cidr part from a ip_cidr.

    Determined the unit test for this can't actually check
    this case since we are mocking the return value from
    /sbin/ip, so modified it to just make sure the dict
    is correct.

    Added a functional test that adds two IP addresses in
    the same IP range to verify that we actually filter
    correctly when a 'to=IP' is specified.

    Change-Id: I3a95b3bb72a43f322ad23892d8959398aac22a1c
    Closes-bug: #1728080
    (cherry picked from commit 7b8289253c355f39ac84bc6ea36d265a8be0e381)
    (cherry picked from commit 15d843a2cafb6ef44974fac7628baf6a38cdd950)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.3 release.