| 2016-05-19 22:01:11 |
Nikita Koshikov |
bug |
|
|
added bug |
| 2016-05-19 23:00:00 |
Nikita Koshikov |
description |
Steps to reproduce:
MOS7 with MU3
1)Create env with hardware nodes(in this case it's DL180 G9s)
2)Assign networks to interfaces
3)Start deployment
Actual results:
Deployment stuck, because hardware servers never come back after deployment. This is because eth NIC for admin network renamed by udev in wrong way.
Here is file generated by fuel-agent:
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", KERNEL="eth*", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", KERNEL="eth*", NAME="eth3"
# Generated by fuel-agent during provisioning: END
Problem here is that when udev first time rename interface to something like eth1->rename2. The next rename is not works, because it doesn't match KERNLER='eth*' rule.
It can be resolved really easy, we just need to delete KERNEL="eth*", from rule definition. Below is fixed file, that did the trick
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", NAME="eth3"
# Generated by fuel-agent during provisioning: END
The same system was able to correctly rename interfaces:
[ 56.442750] systemd-udevd[6750]: renamed network interface eth1 to rename3
[ 56.510752] systemd-udevd[6799]: renamed network interface eth2 to eth1
[ 56.554635] systemd-udevd[6750]: renamed network interface rename3 to eth2
The buggy code is here: https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L920-L923
Expected result:
Deployment should finish successfully.
lspic output:
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
0c:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
0c:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) |
Steps to reproduce:
MOS7 with MU3
1)Create env with hardware nodes(in this case it's DL180 G9s)
2)Assign networks to interfaces
3)Start deployment
Actual results:
Deployment stuck, because hardware servers never come back after deployment. This is because eth NIC for admin network renamed by udev in wrong way.
Here is file generated by fuel-agent:
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", KERNEL="eth*", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", KERNEL="eth*", NAME="eth3"
# Generated by fuel-agent during provisioning: END
Problem here is that when udev first time rename interface to something like eth1->rename2. The next rename is not works, because it doesn't match KERNLER='eth*' rule.
It can be resolved really easy, we just need to delete KERNEL="eth*", from rule definition. Below is fixed file, that did the trick
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", NAME="eth3"
# Generated by fuel-agent during provisioning: END
The same system was able to correctly rename interfaces:
[ 56.442750] systemd-udevd[6750]: renamed network interface eth1 to rename3
[ 56.510752] systemd-udevd[6799]: renamed network interface eth2 to eth1
[ 56.554635] systemd-udevd[6750]: renamed network interface rename3 to eth2
The buggy code is here:
Master - https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L920-L923
MOS7 - https://github.com/openstack/fuel-agent/blob/stable/7.0/fuel_agent/manager.py#L442-L452
Expected result:
Deployment should finish successfully.
lspic output:
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
0c:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
0c:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
nominated for series |
|
fuel/newton |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
bug task added |
|
fuel/newton |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
nominated for series |
|
fuel/7.0.x |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
bug task added |
|
fuel/7.0.x |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
nominated for series |
|
fuel/8.0.x |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
bug task added |
|
fuel/8.0.x |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
nominated for series |
|
fuel/mitaka |
|
| 2016-05-20 07:29:50 |
Dmitry Klenov |
bug task added |
|
fuel/mitaka |
|
| 2016-05-20 07:32:09 |
Dmitry Klenov |
bug task deleted |
fuel/8.0.x |
|
|
| 2016-05-20 07:32:14 |
Dmitry Klenov |
bug task deleted |
fuel/mitaka |
|
|
| 2016-05-20 07:34:13 |
Dmitry Klenov |
fuel/newton: milestone |
|
10.0 |
|
| 2016-05-20 07:34:20 |
Dmitry Klenov |
fuel/newton: assignee |
|
Fuel Toolbox (fuel-toolbox) |
|
| 2016-05-20 07:34:26 |
Dmitry Klenov |
fuel/7.0.x: assignee |
|
Fuel Toolbox (fuel-toolbox) |
|
| 2016-05-20 07:34:29 |
Dmitry Klenov |
fuel/7.0.x: milestone |
|
7.0-updates |
|
| 2016-05-20 07:34:31 |
Dmitry Klenov |
fuel/newton: importance |
Undecided |
Medium |
|
| 2016-05-20 07:34:33 |
Dmitry Klenov |
fuel/7.0.x: importance |
Undecided |
Medium |
|
| 2016-05-20 07:34:35 |
Dmitry Klenov |
fuel/newton: status |
New |
Confirmed |
|
| 2016-05-20 07:34:39 |
Dmitry Klenov |
fuel/7.0.x: status |
New |
Confirmed |
|
| 2016-05-20 07:35:05 |
Dmitry Klenov |
tags |
customer-found |
area-library customer-found |
|
| 2016-07-06 21:12:28 |
Maksim Malchuk |
bug task deleted |
fuel/newton |
|
|
| 2016-08-02 15:48:03 |
Alexander Rubtsov |
tags |
area-library customer-found |
area-library customer-found sla2 |
|
| 2016-08-02 15:54:31 |
Alexander Rubtsov |
tags |
area-library customer-found sla2 |
area-library customer-found sla1 |
|
| 2016-08-02 16:02:17 |
Anton Matveev |
fuel/7.0.x: importance |
Medium |
High |
|
| 2016-08-04 10:19:25 |
Anton Matveev |
fuel/7.0.x: milestone |
7.0-updates |
7.0-mu-5 |
|
| 2016-08-04 12:39:02 |
Alexander Gordeev |
fuel: assignee |
Fuel Toolbox (fuel-toolbox) |
Alexander Gordeev (a-gordeev) |
|
| 2016-08-05 16:04:46 |
OpenStack Infra |
fuel: status |
Confirmed |
In Progress |
|
| 2016-08-17 13:15:14 |
Alexander Gordeev |
description |
Steps to reproduce:
MOS7 with MU3
1)Create env with hardware nodes(in this case it's DL180 G9s)
2)Assign networks to interfaces
3)Start deployment
Actual results:
Deployment stuck, because hardware servers never come back after deployment. This is because eth NIC for admin network renamed by udev in wrong way.
Here is file generated by fuel-agent:
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", KERNEL="eth*", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", KERNEL="eth*", NAME="eth3"
# Generated by fuel-agent during provisioning: END
Problem here is that when udev first time rename interface to something like eth1->rename2. The next rename is not works, because it doesn't match KERNLER='eth*' rule.
It can be resolved really easy, we just need to delete KERNEL="eth*", from rule definition. Below is fixed file, that did the trick
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", NAME="eth3"
# Generated by fuel-agent during provisioning: END
The same system was able to correctly rename interfaces:
[ 56.442750] systemd-udevd[6750]: renamed network interface eth1 to rename3
[ 56.510752] systemd-udevd[6799]: renamed network interface eth2 to eth1
[ 56.554635] systemd-udevd[6750]: renamed network interface rename3 to eth2
The buggy code is here:
Master - https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L920-L923
MOS7 - https://github.com/openstack/fuel-agent/blob/stable/7.0/fuel_agent/manager.py#L442-L452
Expected result:
Deployment should finish successfully.
lspic output:
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
0c:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
0c:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10) |
Steps to reproduce:
MOS7 with MU3
1)Create env with hardware nodes(in this case it's DL180 G9s)
2)Assign networks to interfaces
3)Start deployment
Actual results:
Deployment stuck, because hardware servers never come back after deployment. This is because eth NIC for admin network renamed by udev in wrong way.
Here is file generated by fuel-agent:
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", KERNEL="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", KERNEL="eth*", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", KERNEL="eth*", NAME="eth3"
# Generated by fuel-agent during provisioning: END
Problem here is that when udev first time rename interface to something like eth1->rename2. The next rename is not works, because it doesn't match KERNLER='eth*' rule.
It can be resolved really easy, we just need to delete KERNEL="eth*", from rule definition. Below is fixed file, that did the trick
root@node-47:~# cat /etc/udev/rules.d/70-persistent-net.rules
# Generated by fuel-agent during provisioning: BEGIN
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:78", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:78", ATTR{type}=="1", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:02:17:7c", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="50:65:f3:6c:45:79", ATTR{type}=="1", NAME="eth3"
# Generated by fuel-agent during provisioning: END
The same system was able to correctly rename interfaces:
[ 56.442750] systemd-udevd[6750]: renamed network interface eth1 to rename3
[ 56.510752] systemd-udevd[6799]: renamed network interface eth2 to eth1
[ 56.554635] systemd-udevd[6750]: renamed network interface rename3 to eth2
The buggy code is here:
Master - https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L920-L923
MOS7 - https://github.com/openstack/fuel-agent/blob/stable/7.0/fuel_agent/manager.py#L442-L452
Expected result:
Deployment should finish successfully.
lspic output:
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
0c:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
0c:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
Steps to Verify:
1) Ensure that initramfs image contains 70-persistent-net.rules. That image is located under the following wildcard /boot/initrd*. The rules should be found under the path lib/udev/rules.d/70-persistent-net.rules
2) Ensure that renaming is happening very early. Search dmesg.log for 'renamed' entries. They should have happened very early and right after the NICs become recognized by udev. Not after dozens of seconds of uptime. |
|
| 2016-08-19 13:46:16 |
Denis Meltsaykin |
fuel/7.0.x: status |
Confirmed |
Fix Committed |
|
| 2016-08-23 06:34:34 |
TatyanaGladysheva |
tags |
area-library customer-found sla1 |
area-library customer-found on-verification sla1 |
|
| 2016-08-24 09:15:48 |
TatyanaGladysheva |
tags |
area-library customer-found on-verification sla1 |
area-library customer-found sla1 |
|
| 2016-08-24 09:15:54 |
TatyanaGladysheva |
fuel/7.0.x: status |
Fix Committed |
Fix Released |
|
| 2016-12-22 18:27:05 |
Dmitry Pyzhov |
fuel: milestone |
10.0 |
10.1 |
|
