Fuel agent genereate wrong rules for 70-persistent-net.rules

[Nikita Koshikov]

Steps to reproduce:

MOS7 with MU3
1)Create env with hardware nodes(in this case it's DL180 G9s)
2)Assign networks to interfaces
3)Start deployment

Actual results:
Deployment stuck, because hardware servers never come back after deployment. This is because eth NIC for admin network renamed by udev in wrong way.
Here is file generated by fuel-agent:

root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", KERNEL="eth*", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", KERNEL="eth*", NAME="eth3"
# Generated by fuel-agent during provisioning: END

Problem here is that when udev first time rename interface to something like eth1->rename2. The next rename is not works, because it doesn't match KERNLER='eth*' rule.

It can be resolved really easy, we just need to delete KERNEL="eth*", from rule definition. Below is fixed file, that did the trick
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", NAME="eth3"
# Generated by fuel-agent during provisioning: END

The same system was able to correctly rename interfaces:
[ 56.442750] systemd-udevd[6750]: renamed network interface eth1 to rename3
[ 56.510752] systemd-udevd[6799]: renamed network interface eth2 to eth1
[ 56.554635] systemd-udevd[6750]: renamed network interface rename3 to eth2

The buggy code is here:
Master - https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L920-L923
MOS7 - https://github.com/openstack/fuel-agent/blob/stable/7.0/fuel_agent/manager.py#L442-L452

Expected result:
Deployment should finish successfully.

lspic output:
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
0c:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
0c:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

Steps to Verify:
1) Ensure that initramfs image contains 70-persistent-net.rules. That image is located under the following wildcard /boot/initrd*. The rules should be found under the path lib/udev/rules.d/70-persistent-net.rules
2) Ensure that renaming is happening very early. Search dmesg.log for 'renamed' entries. They should have happened very early and right after the NICs become recognized by udev. Not after dozens of seconds of uptime.

[Dmitry Klenov (dklenov)]

Starting from 8.0 default interface naming schema is biosdevnaming. So not assigning bug to 8.0 and 9.0.

Although there is a possibility to switch back to persistent naming ethX. Although it is not default. So will add 10.0 with medium priority.

[Alexander Gordeev (a-gordeev)]

it's solely related with udev.

I'd rather ask for help mos-linux team. Maybe they know how to fix it better and all possible side effects of the proposed change.

[Dmitry Teselkin (teselkin-d)]

The solution looks good, I don't think there is a way 'better'.

[Alexander Gordeev (a-gordeev)]

I've heard that inside of kernel NICs are still named under ethX names, even with persistent naming scheme.

[Nikita Koshikov (nkoshikov)]

Here is complete workaround, that we applied on customer deployment:

--- /tmp/i/usr/lib/python2.6/site-packages/fuel_agent/manager.orig 2016-05-23 17:45:52.424936068 +0000
+++ /tmp/i/usr/lib/python2.6/site-packages/fuel_agent/manager.py 2016-05-20 21:48:08.180124582 +0000
@@ -446,8 +446,8 @@
             for mapping in self.driver.configdrive_scheme.\
                     common.udevrules.split(','):
                 mac_addr, nic_name = mapping.split('_')
- f.write('SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", '
- 'ATTR{address}=="%s", ATTR{type}=="1", KERNEL=="eth*",'
+ f.write('SUBSYSTEM=="net", DRIVERS=="?*", '
+ 'ATTR{address}=="%s", ATTR{type}=="1",'
                         ' NAME="%s"\n' % (mac_addr, nic_name))
             f.write('# Generated by fuel-agent during provisioning: END\n')
         # FIXME(agordeev): Disable net-generator that will add new etries to

But that was not enough - for some unknown race, 1 reboot always provide wrong interface names, even with correct udev rules generated.

Here is second hack, that actually fix problem with first boot:
--- /tmp/i/usr/share/fuel-agent/cloud-init-templates/boothook_ubuntu.jinja2.orig 2016-05-23 17:50:49.651719024 +0000
+++ /tmp/i/usr/share/fuel-agent/cloud-init-templates/boothook_ubuntu.jinja2 2016-05-21 04:43:33.326960788 +0000
@@ -16,6 +16,10 @@

 # udev persistent net
 cloud-init-per instance udev_persistent_net1 /etc/init.d/networking stop
+cloud-init-per instance unload_module_igb /bin/sh -c "rmmod igb || true"
+cloud-init-per instance unload_module_bnx2x /bin/sh -c "rmmod bnx2x || true"
+cloud-init-per instance load_module_igb /bin/sh -c "modprobe igb || true"
+cloud-init-per instance load_module_bnx2x /bin/sh -c "modprobe bnx2x || true"

 ADMIN_MAC={{ common.admin_mac }}
 ADMIN_IF=$(echo {{ common.udevrules }} | sed 's/[,=]/\n/g' | grep "$ADMIN_MAC" | cut -d_ -f2 | head -1)
And of course it customer specific, due to Intel and Broadcom NICs.

[Alexander Gordeev (a-gordeev)]

>+cloud-init-per instance load_module_igb /bin/sh -c "modprobe igb || true"
>+cloud-init-per instance load_module_bnx2x /bin/sh -c "modprobe bnx2x || true"

looks like you changed the order of loading of kernel modules. So, NICs will be processed by udev in the non-alphanumeric order. (Perhaps, kernel loads bnx2x earlier than igb, due to alpha numeric ordering of modules)

[Alexander Gordeev (a-gordeev)]

I have a silly explanation, why udev rules didn't work from the first boot. I might be wrong, but

systemd-udevd resides in initrd image.
So, then when system boots up, udev gets started and it doesn't have access to full copy of /etc

First time NIC will be processed without the rule /etc/udev/rules.d/70-persistent-net.rules (as initrd doesn't contain it)

Cloud-init operates in the middle of system boot a bit later than systemd/initrd.

In order to enforce NIC remaning, you're removing kernel modules for these NIC and loading them back. So, NIC gets enumarated accoring to the generated udev rule.

Perhaps, fuel-agent should just rebuild initrd after generating new udev rules. So, newly generated udev rules gets applied at the first boot during initrd processing.

[Alexander Gordeev (a-gordeev)]

This is serious.

It seems that we didn't encountered this issue earlier because of lack mixed NIC hardware configuration.

[Alexander Gordeev (a-gordeev)]

So much doubts if the issue is really related with KERNLER='eth*' part of udev rules.

[Aleksey Zvyagintsev (azvyagintsev)]

can we simply remove this filter option(kernel i mean) ?
SUBSYSTEM=="net" and mac attribute should be enough to determinate correct interface.

[Nikita Koshikov (nkoshikov)]

Alexander, if I leave KERNEL='eth*', the next reboot I will have wrong order - this is observed and confirmed.

[Albert Syriy (asyriy)]

The best way to fix the issue (IMHO) is migrate to predictable interface naming schema.

Udev discover interfaces in the arbitrary order and as a result assign ethX names in arbitrary order too.

The boot process has two phases. At the beginning the initrd/iniramfs image (with udev rules and corresponding kernel modules) are downloaded into memory and Udev initialize hardware (and assign the names to it) temporary file system in memory are using at that moment. Later (the second phase, when disk has been mounted and Ubuntu has switched to the persistent file system) Udev triggers second initialization and runs all udev rules again, pretend like the devices just have been discovered and assign names. At the second phase you could easily get "device busy" if attempt to rename interfaces being in active state (for example according the udev rule you should rename eth0 to eth1 and vise verse). This renaming fails.

Please check if you have the same set of udev network rules on bootstrap, packed in initrd image and in the deployed Ubuntu image. Please put attention to the presence of the 75-net-description.rules on initrd

[Alexander Rubtsov (arubtsov)]

sla2 for MOS 7.0-updates

[Alexander Rubtsov (arubtsov)]

Please disregard my latest comment
The correct tag is:
sla1 for MOS 7.0-updates

[Anton Matveev (amatveev)]

priority for 7.0 has been changed to high, since it affects one of the customers

[Alexander Gordeev (a-gordeev)]

preliminary action actions, so far:
* adjust udev rules template string [remove kernel=eth thing]
* move admin NIC config generation out of cloud-init boothooks to fuel-agent utils. that means, admin network config and udev rules will be applied on the first boot. no need to restart networking in cloud-init templates anymore.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351824

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/354272

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/354704

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/354705

[Alexander Gordeev (a-gordeev)]

added steps to verify

[Alexander Gordeev (a-gordeev)]

Regarding the steps to reproduce.

NIC which is connected to admin (pxe) network should be renamed in the middle of provisioning right before the node will boot into the target os.

This is relatively easy to achieve under VMs by swapping NIC order at PCI bus.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354704
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=3aa4734050c2f258c10a4240f0fd1b75b87cde31
Submitter: Jenkins
Branch: stable/7.0

commit 3aa4734050c2f258c10a4240f0fd1b75b87cde31
Author: Alexander Gordeev <email address hidden>
Date: Fri Aug 5 19:02:36 2016 +0300

    Configure admin NIC prior the first system boot

    Having admin NIC confugured in the middle of system booting
    is very fragile and error-prone approach.
    It's better to configure it ahead of first booting time.

    Also, there's no need for freaky networking restarting inside of
    cloud-init boothooks anymore.

    Change-Id: I4a820bc6cea3b32bc5a5961321426b6c712bc78a
    Backported-from: I4c278341e8b40eb8d9b100bed1d9a650f27b5c5b
    Related-Bug: #1583815

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354705
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=5c35bbd811b0c4697b161b5e2a327f8be96ca8ae
Submitter: Jenkins
Branch: stable/7.0

commit 5c35bbd811b0c4697b161b5e2a327f8be96ca8ae
Author: Alexander Gordeev <email address hidden>
Date: Thu Aug 11 20:22:25 2016 +0300

    Include udev rules for NIC mappings into initramfs

    Sometimes, udev can't resolve NIC renamings according to
    the its rules. Especially when there're a bunch of NICs
    and one of them has been used for PXE booting.
    Udev stuggles with late renamings when a system is actually up
    and network services are already running too. In the result,
    admin NIC gets its temporary name like renamedX.

    Moving udev rules into initramfs, thorouhgly allows udev to rename
    NIC without any flaws. Since nobody uses NIC during early initramfs
    running time that practically means nothing will prevent the
    completion of NIC renaming.

    Change-Id: Idfecf974f0fbb825e6378f33b0ec8a3c8fa7dd40
    Backported-From: I9f87833e7d4c864913b7e6113d416614493b23f7
    Closes-Bug: #1583815

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 7.0 + MU5 updates.

Bug is not reproduced on virtual environment.

Steps to verify:
1) Create env with virtual nodes, assign networks to interfaces, start deployment
2) During provisioning when node reboots, perform the following actions:
- virsh edit <slave-name>
- swap 'slot' values at interfaces (for example, for interfaces with source network 'fuel_7_admin' and 'fuel_7_public') > save changes
- virsh destroy <slave-name>
- virsh start <slave-name>
3) Wait till deployment is finished.

Actual results:
Interfaces are renamed correctly:
root@node-6:~# cat /var/log/dmesg | grep renamed
[ 4.043525] systemd-udevd[109]: renamed network interface eth0 to rename2
[ 4.044531] systemd-udevd[107]: renamed network interface eth1 to eth0
[ 4.094133] systemd-udevd[109]: renamed network interface rename2 to eth1
and then deployment is finished successfully.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351824
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=da7a5784733d5d0706c517bafb2215e3ea9f10f8
Submitter: Jenkins
Branch: master

commit da7a5784733d5d0706c517bafb2215e3ea9f10f8
Author: Alexander Gordeev <email address hidden>
Date: Fri Aug 5 19:02:36 2016 +0300

    Configure admin NIC prior the first system boot

    Having admin NIC confugured in the middle of system booting
    is very fragile and error-prone approach.
    It's better to configure it ahead of first booting time.

    Also, there's no need for freaky networking restarting inside of
    cloud-init boothooks anymore.

    Change-Id: I4c278341e8b40eb8d9b100bed1d9a650f27b5c5b
    Related-Bug: #1583815

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/354272
Reason: This repo is retired now, no further work will get merged.