[fuelmenu] Gateway unset by l23network randomly

[Oleg S. Gelbukh]

Steps to reproduce:

1. Install Fuel with default settings and kernel parameters (1 net interface)
2. Enter fuel-menu by pressing the key after reboot
3. Switch to Network Settings tab and enter new values in IP Address and Default Gateway fields:
    orig values:
IP Address: 10.20.0.2
Default Gateway: 10.20.0.1
    new values:
IP Address: 10.109.0.2
Default Gateway: 10.109.0.1
4. Select Apply at the bottom and press Enter

Expected result:
Gateway setting on the top of the tab and the input field Default Gateway contain the same value entered earlier

Actual result:
The input field is empty.
'Gateway: None' displayed in the top part of the tab, in interface settings.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "490"
  build_id: "490"
  fuel-nailgun_sha: "9c78c70c46a3463f9cfcfac78974fe27d32a5f05"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "e869072139670bb8bbfde00ef04dec3d189f5927"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "f7a008e6801ba0072b08302a740174aec506078a"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "88c9f2978f9bf38810941b596c2d435f89ad6943"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "3145e00d1fb780bd0d59e899f12a76143b256ea1"

[Matthew Mosesohn (raytrac3r)]

Oleg, I can't reproduce this in my local environment, but I can tell you here's how the gateway feature works:
1 - If you modify a network interface where the gateway is currently set, it gets unset (so we can apply a new one)
or if it's a DHCP interface, we also unset gateway manually.
However, if you have eth1 with a gateway, then configure eth0 gateway, eth1's gateway stays. You must unset eth1's gateway before eth0's gateway becomes the default. The highest interface # wins in this case.
2 - we run l23network in Puppet to apply config changes. This ultimately sets the new gateway.
A new default gateway is left as None if the new config can't ping that gateway. That leads me to think that it's your network.

I need you to provide logs and if possible, access to your env to see this in action. There are logs in /var/log/puppet and /var/log/fuelmenu.log to inspect.

[Matthew Mosesohn (raytrac3r)]

Vladimir Khlyunev managed to reproduce this. What happens is sometimes puppet l23network module appears to set the gateway, but it ends up getting unset. It's quite intermittent and only happens about 10% of the time during extensive testing. Logs of my tests are here:
http://paste.openstack.org/show/S9EnvoaxwrTgJlQwpYBE/

[Oleg S. Gelbukh (gelbuhos)]

Network Manager seems to be responsible for unsetting the GW. In my case, changing BOOTPROTO from 'none' to 'static' in /etc/sysconfig/networking-scripts/ifcfg-eth0 and manual setting the gw helped me, so it might be an issue with that parameter in l23network.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/274338

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274338
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3f732b24260c1239f3d168556a8cc3075ea8af95
Submitter: Jenkins
Branch: master

commit 3f732b24260c1239f3d168556a8cc3075ea8af95
Author: Stanislav Makar <email address hidden>
Date: Sat Jan 30 09:24:58 2016 +0000

    Disable NetworkManager by default

    Add the possibility to use NetworkManager or not, but disabled by default.
    It is not recommended because it leads to unpredictable behaviour.

    Change-Id: I8e080fd5e4ceac9e2e26dafdf50e7f11fb6c8f04
    Closes-bug: #1538757

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275026

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275026
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0b2294dc5849945b2f31ca72f93ab681a429b5f7
Submitter: Jenkins
Branch: stable/8.0

commit 0b2294dc5849945b2f31ca72f93ab681a429b5f7
Author: Stanislav Makar <email address hidden>
Date: Sat Jan 30 09:24:58 2016 +0000

    Disable NetworkManager by default

    Add the possibility to use NetworkManager or not, but disabled by default.
    It is not recommended because it leads to unpredictable behaviour.

    Change-Id: I8e080fd5e4ceac9e2e26dafdf50e7f11fb6c8f04
    Closes-bug: #1538757
    (cherry picked from commit 3f732b24260c1239f3d168556a8cc3075ea8af95)

[Matthew Mosesohn (raytrac3r)]

fuelmenu is still not fixed because the changes were made to l23network base class, not l23network::l3:ifconfig.

We need to call class l23network, but it's not possible to do this without 'modprobe bonding' being called. Can this be worked around?

[Sergey Vasilenko (xenolog)]

L23network REQUIRED should be initialized before using ANY calls.
By design.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275575

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275271
Committed: https://git.openstack.org/cgit/openstack/fuel-menu/commit/?id=fba0825d97e3d192dea4209c25baab0a28587369
Submitter: Jenkins
Branch: master

commit fba0825d97e3d192dea4209c25baab0a28587369
Author: Matthew Mosesohn <email address hidden>
Date: Tue Feb 2 18:32:25 2016 +0300

    Call l23network init for interface config

    l23network base class now has fixes for networkmanager
    which was creating some issues with interface gateways.

    Includes a workaround for disabling bonding kernel module
    because it gets enabled automatically in l23network class
    initialization. (details in LP#1541028)

    Change-Id: Ib50c2d3672f7e10cff5cc3c29d7b3dbf5d428a86
    Closes-Bug: #1538757

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275635

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275635
Committed: https://git.openstack.org/cgit/openstack/fuel-menu/commit/?id=e071216cb214e34b4d861478033425ee6a54a3be
Submitter: Jenkins
Branch: stable/8.0

commit e071216cb214e34b4d861478033425ee6a54a3be
Author: Matthew Mosesohn <email address hidden>
Date: Tue Feb 2 18:32:25 2016 +0300

    Call l23network init for interface config

    l23network base class now has fixes for networkmanager
    which was creating some issues with interface gateways.

    Includes a workaround for disabling bonding kernel module
    because it gets enabled automatically in l23network class
    initialization. (details in LP#1541028)

    Change-Id: Ib50c2d3672f7e10cff5cc3c29d7b3dbf5d428a86
    Closes-Bug: #1538757
    (cherry picked from commit fba0825d97e3d192dea4209c25baab0a28587369)

[Matthew Mosesohn (raytrac3r)]

The fix for this landed after RC1 iso. This cannot be verified on RC1 iso, but any ISO created after Feb 3.

[Artem Hrechanychenko (agrechanichenko)]

verified on 571 iso - Build #571 (Feb 15, 2016 7:42:00 PM)

if change ip and gateway - all works fine
if change only ip - https://bugs.launchpad.net/fuel/+bug/1546103

[Andrey Lavrentyev (alavrentyev)]

verified on #402 iso 9.0-mos

no empty input fields observed