l23network::l3::route does not survive reboot

[Aleksandr Didenko]

Custom routes created with l23network::l3::route resource are not saved and can't survive system reboot.

Steps to reproduce:

1) On any deployed node write the following manifest:

$gws = hiera('default_gateway')
include l23network
l23network::l3::route { 'testroute':
 destination => '1.2.3.0/24',
 gateway => $gws[0],
 metric => 100
}

2) Apply it with 'puppet apply' command

3) Check that custom route is in place:
# ip ro | grep 1.2.3.0
1.2.3.0/24 via 10.146.0.1 dev br-fw-admin metric 100

4) Reboot the node

5) Check that custom route is in place

Expected result:
route is in place in step 5 (just like in step 3)

Actual result:
route is missing

Version info:
fuel-9.0 iso build #191

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/318619

[Sergey Vasilenko (xenolog)]

For saving routes into interfaces file you should use l23network::l2::port (bond, bridge) or l23network::l3::ifconfig for interface, which used for passing traffic to the gateway. This resource should be present into the same task, where l23network::l3::route used.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318619
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=eac904e05ca8e9b0c3c5f8fc34d3aff1bcd98b37
Submitter: Jenkins
Branch: master

commit eac904e05ca8e9b0c3c5f8fc34d3aff1bcd98b37
Author: Sergey Vasilenko <email address hidden>
Date: Thu May 19 15:36:56 2016 +0300

    Fix route-generation process for l23_stored_config resource

    also add some tests for cover it

    Change-Id: I2f448d8ed5a2c81ab6a03e49b25583a2e450c613
    Closes-bug: #1580581

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/367226

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367226
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9e8a616fd7be8a4f7762fda77e92b545ef0d25aa
Submitter: Jenkins
Branch: stable/mitaka

commit 9e8a616fd7be8a4f7762fda77e92b545ef0d25aa
Author: Sergey Vasilenko <email address hidden>
Date: Thu May 19 15:36:56 2016 +0300

    Fix route-generation process for l23_stored_config resource

    also add some tests for cover it

    Change-Id: I2f448d8ed5a2c81ab6a03e49b25583a2e450c613
    Closes-bug: #1580581
    (cherry picked from commit eac904e05ca8e9b0c3c5f8fc34d3aff1bcd98b37)

[Timur Nurlygayanov (tnurlygayanov)]

root@node-3:~# puppet apply bug.pp
Error: Could not find data item default_gateway in any Hiera data file and no default supplied at /root/bug.pp:1 on node node-3.test.domain.local
Error: Could not find data item default_gateway in any Hiera data file and no default supplied at /root/bug.pp:1 on node node-3.test.domain.local

[Timur Nurlygayanov (tnurlygayanov)]

root@node-3:~# cat bug.pp
$gws = ['10.109.18.1']
include l23network
l23network::l3::route { 'testroute':
 destination => '1.2.3.0/24',
 gateway => $gws[0],
 metric => 100
}

root@node-3:~# puppet apply bug.pp
Notice: Compiled catalog for node-3.test.domain.local in environment production in 2.33 seconds
Notice: /Stage[main]/Main/L23network::L3::Route[testroute]/L3_route[1.2.3.0/24,metric:100]/ensure: created
Notice: /Stage[main]/L23network/Enable_hotplug[global]/ensure: created
Notice: Finished catalog run in 3.12 seconds
root@node-3:~# ip ro | grep 1.2.3.0
1.2.3.0/24 via 10.109.18.1 dev br-ex metric 100

After the reboot:

root@node-3:~# ip ro | grep 1.2.3.0
root@node-3:~#

[Timur Nurlygayanov (tnurlygayanov)]

This was tested on MOS 9.1:

root@node-3:~# cat /etc/apt/sources.list.d/mos-proposed.list
# This file is managed by Puppet. DO NOT EDIT.
# mos-proposed
deb http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2016-09-09-040322 mos9.0-proposed main restricted

[Sergey Vasilenko (xenolog)]

Timur, Your example is wrong. You code should looks like:

    $gws = ['10.109.18.1']
    class { 'l23network':
      use_ovs => false
    }

    l23network::l3::ifconfig { 'eth1':
      ipaddr => ['10.109.18.5/24'],
      gateway => $gws[0],
    } ->
    l23network::l3::route { 'testroute':
     destination => '1.2.3.0/24',
     gateway => $gws[0],
     metric => 100
    }

[Timur Nurlygayanov (tnurlygayanov)]

Steps To Verify:
1. Create puppet manifest (see detailed steps) on controller node and run it.
2. Reboot node
3. Make sure that route wasn't removed after the reboot.

root@node-3:~# cat bug.pp
$gws = ['10.109.18.1']
class { 'l23network':
  use_ovs => false
}

l23network::l3::ifconfig { 'enp0s4':
  ipaddr => ['10.109.18.5/24'],
  gateway => $gws[0],
} ->
l23network::l3::route { 'testroute':
 destination => '1.2.3.0/24',
 gateway => $gws[0],
 metric => 100
}
root@node-3:~#

root@node-3:~# puppet apply bug.pp
Notice: Compiled catalog for node-3.test.domain.local in environment production in 1.07 seconds
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/method: method changed 'manual' to 'static'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/bridge: defined 'bridge' as 'absent'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/ipaddr: defined 'ipaddr' as '10.109.18.5/24'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/gateway: defined 'gateway' as '10.109.18.1'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/bond_slaves: defined 'bond_slaves' as 'absent'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/routes: defined 'routes' as '
---
  "1.2.3.0/24,metric:100":
    gateway: "10.109.18.1"
    destination: "1.2.3.0/24"
    metric: "100"
'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L3_ifconfig[enp0s4]/ensure: created
!!! Iproute can not setup new gateway.
!!! May be default gateway with same metric already exists:
default via 10.109.18.1 dev br-ex
10.109.15.0/24 dev br-fw-admin proto kernel scope link src 10.109.15.4
10.109.16.0/24 dev br-mgmt proto kernel scope link src 10.109.16.6
10.109.17.0/24 dev br-storage proto kernel scope link src 10.109.17.4
10.109.18.0/24 dev br-ex proto kernel scope link src 10.109.18.6
10.109.18.0/24 dev enp0s4 proto kernel scope link src 10.109.18.5
240.0.0.0/30 dev hapr-host proto kernel scope link src 240.0.0.1
240.0.0.4/30 dev vr-host-base proto kernel scope link src 240.0.0.5

Notice: /Stage[main]/Main/L23network::L3::Route[testroute]/L3_route[1.2.3.0/24,metric:100]/ensure: created
Notice: /Stage[main]/L23network/Enable_hotplug[global]/ensure: created
Notice: Finished catalog run in 8.59 seconds

After the reboot:
root@node-3:~# ip ro | grep 1.2.3.0
root@node-3:~#

[Timur Nurlygayanov (tnurlygayanov)]

Verified on MOS 9.1 snapshot 9.0-2016-09-11-182323

root@node-1:~# puppet apply bug.pp
Notice: Compiled catalog for node-1.test.domain.local in environment production in 1.21 seconds
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/method: method changed 'manual' to 'static'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/bridge: defined 'bridge' as 'absent'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/ipaddr: defined 'ipaddr' as '10.109.8.10/24'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/bond_slaves: defined 'bond_slaves' as 'absent'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L23_stored_config[enp0s4]/routes: defined 'routes' as '
---
  "1.2.3.0/24,metric:100":
    gateway: "10.109.8.1"
    destination: "1.2.3.0/24"
    metric: "100"
'
Notice: /Stage[main]/Main/L23network::L3::Ifconfig[enp0s4]/L3_ifconfig[enp0s4]/ensure: created
Notice: /Stage[main]/Main/L23network::L3::Route[testroute]/L3_route[1.2.3.0/24,metric:100]/ensure: created
Notice: /Stage[main]/L23network/Enable_hotplug[global]/ensure: created
Notice: Finished catalog run in 12.11 seconds
root@node-1:~# cat bug.pp

$gws = ['10.109.8.1']

class { 'l23network':
  use_ovs => false
}

l23network::l3::ifconfig { 'enp0s6':
  ipaddr => ['10.109.8.10/24'],
} ->
l23network::l3::route { 'testroute':
 destination => '1.2.3.0/24',
 gateway => $gws[0],
 metric => 100
}
root@node-1:~# ip route | grep 1.2.3
1.2.3.0/24 via 10.109.8.1 dev br-ex metric 100
root@node-1:~#

After the reboot:
root@node-1:~# ip route | grep 1.2.3
1.2.3.0/24 via 10.109.8.1 dev br-ex metric 100
root@node-1:~#