vlan interface routes lost when parent bond interface is changed

[Harald Jensås]

[root@os-net-config os-net-config]# ip r
default via 192.168.122.1 dev eth0 proto static metric 100
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.98 metric 100

[root@os-net-config os-net-config]# cat config.json
{
    "network_config": [
        {
            "name": "bond0",
            "type": "linux_bond",
            "use_dhcp": false,
            "bonding_options": "mode=1 miimon=111",
            "members": [
                {
                    "type": "interface",
                    "name": "nic2",
                    "primary": true
                },
                {
                    "type": "interface",
                    "name": "nic3"
                }
            ],
        },
        {
            "device": "bond0",
            "type": "vlan",
            "vlan_id": 610,
            "addresses": [{"ip_netmask": "172.20.1.10/26"}],
            "routes": [{"ip_netmask": "172.20.2.0/26", "next_hop": "172.20.1.62"}],
        },
        {
            "device": "bond0",
            "vlan_id": 611,
            "type": "vlan",
            "addresses": [{"ip_netmask": "172.20.1.70/26"}],
            "routes": [{"ip_netmask": "172.20.2.64/26", "next_hop": "172.20.1.126"}],
        },
        {
            "device": "bond0",
            "type": "vlan",
            "vlan_id": 612,
            "dns_servers": [
                "192.168.122.1"
            ],
            "addresses": [{"ip_netmask": "172.20.1.130/26"}],
            "routes": [{"ip_netmask": "172.20.2.128/26", "next_hop": "172.20.1.190"}],
        }
    ]
}

[root@os-net-config os-net-config]# os-net-config -c /etc/os-net-config/config.json -v
[2018/11/30 01:14:46 PM] [INFO] Using config file at: /etc/os-net-config/config.json
[2018/11/30 01:14:46 PM] [INFO] Ifcfg net config provider created.
[2018/11/30 01:14:46 PM] [INFO] Not using any mapping file.
[2018/11/30 01:14:47 PM] [INFO] Finding active nics
[2018/11/30 01:14:47 PM] [INFO] vlan612 is not an active nic
[2018/11/30 01:14:47 PM] [INFO] vlan611 is not an active nic
[2018/11/30 01:14:47 PM] [INFO] vlan610 is not an active nic
[2018/11/30 01:14:47 PM] [INFO] bond0 is not an active nic
[2018/11/30 01:14:47 PM] [INFO] bonding_masters is not an active nic
[2018/11/30 01:14:47 PM] [INFO] eth1 is an embedded active nic
[2018/11/30 01:14:47 PM] [INFO] eth0 is an embedded active nic
[2018/11/30 01:14:47 PM] [INFO] eth2 is an embedded active nic
[2018/11/30 01:14:47 PM] [INFO] lo is not an active nic
[2018/11/30 01:14:47 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)
[2018/11/30 01:14:47 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']
[2018/11/30 01:14:47 PM] [INFO] nic3 mapped to: eth2
[2018/11/30 01:14:47 PM] [INFO] nic2 mapped to: eth1
[2018/11/30 01:14:47 PM] [INFO] nic1 mapped to: eth0
[2018/11/30 01:14:47 PM] [INFO] adding linux bond: bond0
[2018/11/30 01:14:47 PM] [INFO] adding interface: eth1
[2018/11/30 01:14:47 PM] [INFO] adding interface: eth2
[2018/11/30 01:14:47 PM] [INFO] adding vlan: vlan610
[2018/11/30 01:14:47 PM] [INFO] adding custom route for interface: vlan610
[2018/11/30 01:14:47 PM] [INFO] adding vlan: vlan611
[2018/11/30 01:14:47 PM] [INFO] adding custom route for interface: vlan611
[2018/11/30 01:14:47 PM] [INFO] adding vlan: vlan612
[2018/11/30 01:14:47 PM] [INFO] adding custom route for interface: vlan612
[2018/11/30 01:14:47 PM] [INFO] applying network configs...
[2018/11/30 01:14:47 PM] [INFO] running ifdown on interface: vlan612
[2018/11/30 01:14:47 PM] [INFO] running ifdown on interface: vlan611
[2018/11/30 01:14:47 PM] [INFO] running ifdown on interface: vlan610
[2018/11/30 01:14:47 PM] [INFO] running ifdown on interface: eth2
[2018/11/30 01:14:47 PM] [INFO] running ifdown on interface: eth1
[2018/11/30 01:14:47 PM] [INFO] running ifdown on interface: bond0
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan612
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan610
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-vlan611
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-bond0
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan610
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan611
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-vlan612
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eth2
[2018/11/30 01:14:47 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-eth1
[2018/11/30 01:14:47 PM] [INFO] running ifup on interface: eth2
[2018/11/30 01:14:47 PM] [INFO] running ifup on interface: eth1
[2018/11/30 01:14:47 PM] [INFO] running ifup on interface: bond0
[2018/11/30 01:14:47 PM] [INFO] running ifup on interface: vlan612
[2018/11/30 01:14:51 PM] [INFO] running ifup on interface: vlan611
[2018/11/30 01:14:55 PM] [INFO] running ifup on interface: vlan610

[root@os-net-config os-net-config]# ip r
default via 192.168.122.1 dev eth0 proto static metric 100
169.254.0.0/16 dev bond0 scope link metric 1005
169.254.0.0/16 dev vlan612 scope link metric 1006
169.254.0.0/16 dev vlan611 scope link metric 1007
169.254.0.0/16 dev vlan610 scope link metric 1008
172.20.1.0/26 dev vlan610 proto kernel scope link src 172.20.1.10
172.20.1.64/26 dev vlan611 proto kernel scope link src 172.20.1.70
172.20.1.128/26 dev vlan612 proto kernel scope link src 172.20.1.130
172.20.2.0/26 via 172.20.1.62 dev vlan610
172.20.2.64/26 via 172.20.1.126 dev vlan611
172.20.2.128/26 via 172.20.1.190 dev vlan612
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.98 metric 100

[root@os-net-config os-net-config]# sed -i s/miimon=111/miimon=100/g /etc/os-net-config/config.json

[root@os-net-config os-net-config]# os-net-config -c /etc/os-net-config/config.json -v
[2018/11/30 01:15:53 PM] [INFO] Using config file at: /etc/os-net-config/config.json
[2018/11/30 01:15:53 PM] [INFO] Ifcfg net config provider created.
[2018/11/30 01:15:53 PM] [INFO] Not using any mapping file.
[2018/11/30 01:15:53 PM] [INFO] Finding active nics
[2018/11/30 01:15:53 PM] [INFO] vlan612 is not an active nic
[2018/11/30 01:15:53 PM] [INFO] vlan611 is not an active nic
[2018/11/30 01:15:53 PM] [INFO] vlan610 is not an active nic
[2018/11/30 01:15:53 PM] [INFO] bond0 is not an active nic
[2018/11/30 01:15:53 PM] [INFO] bonding_masters is not an active nic
[2018/11/30 01:15:53 PM] [INFO] eth1 is an embedded active nic
[2018/11/30 01:15:53 PM] [INFO] eth0 is an embedded active nic
[2018/11/30 01:15:53 PM] [INFO] eth2 is an embedded active nic
[2018/11/30 01:15:53 PM] [INFO] lo is not an active nic
[2018/11/30 01:15:53 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)
[2018/11/30 01:15:53 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']
[2018/11/30 01:15:53 PM] [INFO] nic3 mapped to: eth2
[2018/11/30 01:15:53 PM] [INFO] nic2 mapped to: eth1
[2018/11/30 01:15:53 PM] [INFO] nic1 mapped to: eth0
[2018/11/30 01:15:53 PM] [INFO] adding linux bond: bond0
[2018/11/30 01:15:53 PM] [INFO] adding interface: eth1
[2018/11/30 01:15:53 PM] [INFO] adding interface: eth2
[2018/11/30 01:15:53 PM] [INFO] adding vlan: vlan610
[2018/11/30 01:15:53 PM] [INFO] adding custom route for interface: vlan610
[2018/11/30 01:15:53 PM] [INFO] adding vlan: vlan611
[2018/11/30 01:15:53 PM] [INFO] adding custom route for interface: vlan611
[2018/11/30 01:15:53 PM] [INFO] adding vlan: vlan612
[2018/11/30 01:15:53 PM] [INFO] adding custom route for interface: vlan612
[2018/11/30 01:15:53 PM] [INFO] applying network configs...
[2018/11/30 01:15:53 PM] [INFO] No changes required for interface: eth2
[2018/11/30 01:15:53 PM] [INFO] No changes required for interface: eth1
[2018/11/30 01:15:53 PM] [INFO] No changes required for vlan interface: vlan612
[2018/11/30 01:15:53 PM] [INFO] No changes required for vlan interface: vlan611
[2018/11/30 01:15:53 PM] [INFO] No changes required for vlan interface: vlan610
[2018/11/30 01:15:53 PM] [INFO] running ifdown on interface: eth2
[2018/11/30 01:15:54 PM] [INFO] running ifdown on interface: eth1
[2018/11/30 01:15:54 PM] [INFO] running ifdown on interface: bond0
[2018/11/30 01:15:54 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-bond0
[2018/11/30 01:15:54 PM] [INFO] running ifup on interface: eth2
[2018/11/30 01:15:54 PM] [INFO] running ifup on interface: eth1
[2018/11/30 01:15:54 PM] [INFO] running ifup on interface: bond0

[root@os-net-config os-net-config]# ip r
default via 192.168.122.1 dev eth0 proto static metric 100
169.254.0.0/16 dev bond0 scope link metric 1005
172.20.1.0/26 dev vlan610 proto kernel scope link src 172.20.1.10
172.20.1.64/26 dev vlan611 proto kernel scope link src 172.20.1.70
172.20.1.128/26 dev vlan612 proto kernel scope link src 172.20.1.130
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.98 metric 100

The following routes are no longer present:
172.20.2.0/26 via 172.20.1.62 dev vlan610
172.20.2.64/26 via 172.20.1.126 dev vlan611
172.20.2.128/26 via 172.20.1.190 dev vlan612

The routes are in the configuration files:
[root@os-net-config os-net-config]# cat /etc/sysconfig/network-scripts/route-vlan61*
172.20.2.0/26 via 172.20.1.62 dev vlan610
172.20.2.64/26 via 172.20.1.126 dev vlan611
172.20.2.128/26 via 172.20.1.190 dev vlan612

Bringing the vlan interfaces down and up again adds the routes:
172.20.2.0/26 via 172.20.1.62 dev vlan610
172.20.2.64/26 via 172.20.1.126 dev vlan611
172.20.2.128/26 via 172.20.1.190 dev vlan612

[Harald Jensås (harald-jensas)]

Proposed fix: https://review.openstack.org/621298

Test result in reproducer:
--------------------------

[root@os-net-config ~]# cat /etc/os-net-config/config.json
{
    "network_config": [
        {
            "name": "bond0",
            "type": "linux_bond",
            "use_dhcp": false,
            "bonding_options": "mode=1 miimon=111",
            "members": [
                {
                    "type": "interface",
                    "name": "nic2",
                    "primary": true
                },
                {
                    "type": "interface",
                    "name": "nic3"
                }
            ],
        },
        {
            "device": "bond0",
            "type": "vlan",
            "vlan_id": 610,
            "addresses": [{"ip_netmask": "172.20.1.10/26"}],
            "routes": [{"ip_netmask": "172.20.2.0/26", "next_hop": "172.20.1.62"}],
        },
        {
            "device": "bond0",
            "vlan_id": 611,
            "type": "vlan",
            "addresses": [{"ip_netmask": "172.20.1.70/26"}],
            "routes": [{"ip_netmask": "172.20.2.64/26", "next_hop": "172.20.1.126"}],
        },
        {
            "device": "bond0",
            "type": "vlan",
            "vlan_id": 612,
            "dns_servers": [
                "192.168.122.1"
            ],
            "addresses": [{"ip_netmask": "172.20.1.130/26"}],
            "routes": [{"ip_netmask": "172.20.2.128/26", "next_hop": "172.20.1.190"}],
        }
    ]
}

Initial configuration apply:
----------------------------

[root@os-net-config ~]# os-net-config -c os-net-config -c /etc/os-net-config/config.json -v
[2018/12/01 07:56:26 PM] [INFO] Using config file at: /etc/os-net-config/config.json
[2018/12/01 07:56:26 PM] [INFO] Ifcfg net config provider created.
[2018/12/01 07:56:26 PM] [INFO] Not using any mapping file.
[2018/12/01 07:56:26 PM] [INFO] Finding active nics
[2018/12/01 07:56:26 PM] [INFO] lo is not an active nic
[2018/12/01 07:56:26 PM] [INFO] eth2 is an embedded active nic
[2018/12/01 07:56:26 PM] [INFO] eth0 is an embedded active nic
[2018/12/01 07:56:26 PM] [INFO] eth1 is an embedded active nic
[2018/12/01 07:56:26 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)
[2018/12/01 07:56:26 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']
[2018/12/01 07:56:26 PM] [INFO] nic3 mapped to: eth2
[2018/12/01 07:56:26 PM] [INFO] nic2 mapped to: eth1
[2018/12/01 07:56:26 PM] [INFO] nic1 mapped to: eth0
[2018/12/01 07:56:26 PM] [INFO] adding linux bond: bond0
[2018/12/01 07:56:26 PM] [INFO] adding interface: eth1
[2018/12/01 07:56:26 PM] [INFO] adding interface: eth2
[2018/12/01 07:56:26 PM] [INFO] adding vlan: vlan610
[2018/12/01 07:56:26 PM] [INFO] adding custom route for interface: vlan610
[2018/12/01 07:56:26 PM] [INFO] adding vlan: vlan611
[2018/12/01 07:56:26 PM] [INFO] adding custom route for interface: vlan611
[2018/12/01 07:56:26 PM] [INFO] adding vlan: vlan612
[2018/12/01 07:56:26 PM] [INFO] adding custom route for interface: vlan612
[2018/12/01 07:56:26 PM] [INFO] applying network configs...
[2018/12/01 07:56:26 PM] [INFO] running ifdown on interface: vlan612
[2018/...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/621990

[Harald Jensås (harald-jensas)]

(The bug number on the master patch was incorrect. :( )

Fix merged to: master
https://review.openstack.org/621298

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/621977

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/621978

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/621989

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621978
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=93f7f9dc4b1ecab3968bfd2fb5ae3d5b6ffcf400
Submitter: Zuul
Branch: stable/queens

commit 93f7f9dc4b1ecab3968bfd2fb5ae3d5b6ffcf400
Author: Harald Jensås <email address hidden>
Date: Fri Nov 30 10:54:06 2018 +0100

    Fix - restart VLAN interface on underlying device change

    When a change is needed on bond, os-net-config write the
    configuration and restart nic and bond. But it does not
    restart vlan on top of the bond. The results is that all
    routes for vlan interfaces are lost.

    This moves the processing of VLAN's so that it run's
    after any interfaces, bridges, bonds etc. Then
    concatenates the restart lists from other interfaces and
    do a check if 'PHYSDEV' of a VLAN interface is being
    restarted.

    Closes-Bug: #1806027
    Related: RHBZ#1654987
    Change-Id: I246da54cf02d06466f52d210f89d82ca9e2a6ef2
    (cherry picked from commit 19067b4e77ec618ffef67acc771869ecbc33ad34)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621989
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=2f6f6cb6ab190cf8c7bbe5c113d8279863cfff03
Submitter: Zuul
Branch: stable/pike

commit 2f6f6cb6ab190cf8c7bbe5c113d8279863cfff03
Author: Harald Jensås <email address hidden>
Date: Fri Nov 30 10:54:06 2018 +0100

    Fix - restart VLAN interface on underlying device change

    When a change is needed on bond, os-net-config write the
    configuration and restart nic and bond. But it does not
    restart vlan on top of the bond. The results is that all
    routes for vlan interfaces are lost.

    This moves the processing of VLAN's so that it run's
    after any interfaces, bridges, bonds etc. Then
    concatenates the restart lists from other interfaces and
    do a check if 'PHYSDEV' of a VLAN interface is being
    restarted.

    Closes-Bug: #1806027
    Related: RHBZ#1654987
    Change-Id: I246da54cf02d06466f52d210f89d82ca9e2a6ef2
    (cherry picked from commit 19067b4e77ec618ffef67acc771869ecbc33ad34)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621977
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=f7007b689e78154f26c2704da4804641e8beae50
Submitter: Zuul
Branch: stable/rocky

commit f7007b689e78154f26c2704da4804641e8beae50
Author: Harald Jensås <email address hidden>
Date: Fri Nov 30 10:54:06 2018 +0100

    Fix - restart VLAN interface on underlying device change

    When a change is needed on bond, os-net-config write the
    configuration and restart nic and bond. But it does not
    restart vlan on top of the bond. The results is that all
    routes for vlan interfaces are lost.

    This moves the processing of VLAN's so that it run's
    after any interfaces, bridges, bonds etc. Then
    concatenates the restart lists from other interfaces and
    do a check if 'PHYSDEV' of a VLAN interface is being
    restarted.

    Closes-Bug: #1806027
    Related: RHBZ#1654987
    Change-Id: I246da54cf02d06466f52d210f89d82ca9e2a6ef2
    (cherry picked from commit 19067b4e77ec618ffef67acc771869ecbc33ad34)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621990
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=0743c2c8036c441f4dbfb95750984831de1ec139
Submitter: Zuul
Branch: stable/ocata

commit 0743c2c8036c441f4dbfb95750984831de1ec139
Author: Harald Jensås <email address hidden>
Date: Fri Nov 30 10:54:06 2018 +0100

    Fix - restart VLAN interface on underlying device change

    When a change is needed on bond, os-net-config write the
    configuration and restart nic and bond. But it does not
    restart vlan on top of the bond. The results is that all
    routes for vlan interfaces are lost.

    This moves the processing of VLAN's so that it run's
    after any interfaces, bridges, bonds etc. Then
    concatenates the restart lists from other interfaces and
    do a check if 'PHYSDEV' of a VLAN interface is being
    restarted.

    Closes-Bug: #1806027
    Related: RHBZ#1654987
    Change-Id: I246da54cf02d06466f52d210f89d82ca9e2a6ef2
    (cherry picked from commit 19067b4e77ec618ffef67acc771869ecbc33ad34)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config 9.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config 8.4.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config 7.3.9 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config ocata-eol release.