PXE interface flapping

[Dmitry Klenov]

Steps to reproduce:

1) Boot node into bootstrap
2) eth4 - admin if
3)deploy env
4)reset env
5) node again into bootstrap
6) eth0 - admin if

It is expected that admin interface name does not change

[Dmitry Klenov (dklenov)]

User impact:

Issue is seen in ~25% cases on the H/W which is planned to be used as production one in customer environment. When issue happens, it breaks deployment.

[Igor Marnat (imarnat)]

The fix for the issue: https://review.openstack.org/#/c/220728/

[Alexander Gordeev (a-gordeev)]

Folks, this report is definitely way too incomplete.

at least, we need to know:
1) the exact version of fuel on which the customer have run into this issue.
2) log files. It's not clear whether eth4 and eth0 have the different MACs or eth4 was just renamed to eth0 by applying udev's mapping from nailgun.

could the original issue be related with https://bugs.launchpad.net/fuel/+bug/1455610 or https://bugs.launchpad.net/fuel/+bug/1466148 ?

[Alexei Sheplyakov (asheplyakov)]

@Aleksandr,

I think the exact version of Fuel is not very useful. Instead we need the hardware data (lspci -vvv, dmidecode) and boot logs (dmesg)

[Igor Marnat (imarnat)]

Correction to comment #2: it's possible that the fix suggested is partial, not help fully. Still investigating

[Alexei Sheplyakov (asheplyakov)]

The bug has nothing to do with Ceph, I don't quite understand why it has been assigned to me.

[Alexei Sheplyakov (asheplyakov)]

Perhaps nailgun should maintain the state (in particular the network interface naming) for the bootstrap nodes too.
Alternatively it can use the stateless interfaces' naming, for instance http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames

[Alexei Sheplyakov (asheplyakov)]

> The fix for the issue: https://review.openstack.org/#/c/220728

Nope, that patch is supposed to fix a different problem

[Aleksey Zvyagintsev (azvyagintsev)]

 1) the exact version of fuel on which the customer have run into this issue.
A: reproduces at least on fuel7.0-286

2)It's not clear whether eth4 and eth0 have the different MACs or eth4 was just renamed to eth0 by applying udev's mapping from nailgun.
A:eth4=>renamed to eth0 , mac address stays same (same cobbler dhcp address, same node id. )

3) lspci -vvv, dmidecode and logs
A:will be provided after tests.

[Alexander Gordeev (a-gordeev)]

Ok, Thanks for the answers.

@Aleksey Zvyagintsev (azvyagintsev), Could you additionaly provide the contents of /etc/udev/rules.d ?

[Aleksey Kasatkin (alekseyk-ru)]

Any logs, please!

[slava valyavskiy (slava-val-al)]

In our current case this bug leads to misconfiguration of ALL bond interfaces, not only for admin interface.

[slava valyavskiy (slava-val-al)]

I have attached nailgun log files from master node.

[Ksenia Svechnikova (kdemina)]

Reproduced 1/20 times:

ISO 288
Run ps aux on this node, when eth4 should be PXE:

root 2423 0.0 0.0 9172 612 ? Ss 11:50 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0
root 2781 0.0 0.0 249344 1560 ? Sl 11:54 0:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root 2845 0.0 0.0 66268 1128 ? Ss 11:54 0:00 /usr/sbin/sshd
root 14824 0.1 0.0 68828 3512 ? Ss 12:31 0:00 \_ sshd: root@pts/0
root 14826 0.1 0.0 108352 1792 pts/0 Ss 12:31 0:00 \_ -bash
root 15096 0.0 0.0 110796 1656 pts/0 R+ 12:31 0:00 \_ ps auxf

[Bartosz Kupidura (zynzel)]

Probably main reason of that bug is 2 kind of NICs installed. We use there igb AND ixgbe driver.
So depending which module will be loaded first, admin interface can be found on eth0 or eth4.

70-persistent-net.rules before reboot:
# PCI device 0x8086:0x1521 (igb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="ec:f4:bb:cd:45:54", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

70-persistent-net.rules after reboot:
# PCI device 0x8086:0x1521 (igb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="ec:f4:bb:cd:45:54", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

[Eugene Bogdanov (ebogdanov)]

Raising priority to Critical as agreed with Dmitry Klenov.

[Eugene Bogdanov (ebogdanov)]

Moving back to 7.0 scope - this is a blocker for Telco team.

[Igor Marnat (imarnat)]

JFYI, Albert works on the fix which _partially_ relates to the problem with hardcoded interface naming: https://review.openstack.org/#/c/223939

[Dmitry Klenov (dklenov)]

The bug should be fixed by a commit made for https://bugs.launchpad.net/fuel/+bug/1466148. Commit: https://review.openstack.org/#/c/221235/.

So moving issue to fix released.

[Dmitry Klenov (dklenov)]

Sorry, to Fix Committed.

[Ksenia Svechnikova (kdemina)]

Issue was not reproduced with ISO#295

[Vitaly Sedelnik (vsedelnik)]

Assigned back to Alexei Sheplyakov for 8.0 - I assigned to Alexey Shtokolov by mistake, sorry for confusion.

[Mike Scherbakov (mihgen)]

Folks,
how can it be fixed for 7.0 but not for 8.0? Are we following our process of backporting fixes?

[Aleksey Kasatkin (alekseyk-ru)]

But https://review.openstack.org/#/c/221235/ is the backport from 8.0 master ( of https://review.openstack.org/#/c/220741/ ), so it should be fixed in 8.0 as well.