Bug #1657750 “Random bond down on nodes with 6 interfaces during...” : Series mitaka : Bugs : Fuel for OpenStack

Revision history for this message

Sergey Galkin (sgalkin) wrote on 2017-01-19:

#1

6 interfaces configuration Edit (70.4 KiB, image/png)

Oleksiy Molchanov (omolchanov) on 2017-01-20

Changed in fuel:
milestone:	none → 9.3
assignee:	nobody → Fuel Sustaining (fuel-sustaining-team)
importance:	Undecided → High

Revision history for this message

Sergey Galkin (sgalkin) wrote on 2017-01-20:

#2

Workaround in action

root@node-2312:~# ping 10.21.0.2
PING 10.21.0.2 (10.21.0.2) 56(84) bytes of data
From 10.21.1.78 icmp_seq=1 Destination Host Unreachable
From 10.21.1.78 icmp_seq=2 Destination Host Unreachable
From 10.21.1.78 icmp_seq=3 Destination Host Unreachable
^C
--- 10.21.0.2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3007ms

root@node-2312:~# for i in eno1 eno2 enp3s0f0 enp3s0f1; do ifdown ${i}; done
ifdown: interface eno2 not configured
ifdown: interface enp3s0f0 not configured
ifdown: interface enp3s0f1 not configured

root@node-2312:~# for i in eno1 eno2 enp3s0f0 enp3s0f1; do ifup ${i}; done
ntp stop/waiting
ntp start/running, process 149331
ntp stop/waiting
ntp start/running, process 149463
ntp stop/waiting
ntp start/running, process 149612
ntp stop/waiting
ntp start/running, process 149873

root@node-2312:~# ping 10.21.0.2
PING 10.21.0.2 (10.21.0.2) 56(84) bytes of data.
64 bytes from 10.21.0.2: icmp_seq=1 ttl=64 time=0.395 ms

Revision history for this message

Sergey Galkin (sgalkin) wrote on 2017-01-23:

#3

Workaround is not workaround.
After
for i in eno1 eno2 enp3s0f0 enp3s0f1; do ifdown ${i}; for i in eno1 eno2 enp3s0f0 enp3s0f1; do ifup ${i}; done
and restart deployment the node with workaround gone offline again

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2017-01-23:

#4

Please also provide the following information for the problem node:

- /etc/astute.yaml
- output of this command:
for i in eno1 eno2 enp3s0f0 enp3s0f1; do ethtool ${i}; done
- output of this command:
cat /proc/net/bonding/bond0
- output of this command:
lshw

Revision history for this message

Leontii Istomin (listomin) wrote on 2017-01-23:

#5

Unfortunately we removed the env with repro, therefore can't provide you with /etc/astute.yaml

Bond parameters were the following:
- action: add-bond
   bond_properties:
   lacp_rate: fast
   mode: 802.3ad
   xmit_hash_policy: layer3+4
   bridge: br-fw-admin
   interface_properties:
   vendor_specific:
   disable_offloading: true
   interfaces:
   - eno1
   - eno2
   - enp3s0f0
   - enp3s0f1
   name: bond0

root@bootstrap:~# ethtool -i eno1
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001863 16.5.20
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# ethtool -i eno2
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001863 16.5.20
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# ethtool -i enp3s0f0
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001866 16.5.20
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# ethtool -i enp3s0f1
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001866 16.5.20
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# lspci | grep Ether
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Unfortunately we removed the env with repro, therefore can't provide you with /etc/astute.yaml

Bond parameters were the following:
 - action: add-bond
   bond_properties:
   lacp_rate: fast
   mode: 802.3ad
   xmit_hash_policy: layer3+4
   bridge: br-fw-admin
   interface_properties:
   vendor_specific:
   disable_offloading: true
   interfaces:
   - eno1
   - eno2
   - enp3s0f0
   - enp3s0f1
   name: bond0

root@bootstrap:~# ethtool -i eno1
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001863 16.5.20
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# ethtool -i eno2
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001863 16.5.20
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# ethtool -i enp3s0f0
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001866 16.5.20
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# ethtool -i enp3s0f1
driver: i40e
version: 1.4.25-k
firmware-version: 4.41 0x80001866 16.5.20
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@bootstrap:~# lspci | grep Ether
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Revision history for this message

Leontii Istomin (listomin) wrote on 2017-01-23:

#6

Have found /etc/interfaces.d/ifcfg-bond0 conig:
auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2 enp3s0f0 enp3s0f1
bond-mode 802.3ad
bond-miimon 1000
bond-use-carrier 1
bond-lacp-rate fast
bond-updelay 30000
bond-downdelay 10000
bond-ad-select bandwidth
bond-xmit-hash-policy layer3+4

delays parameters multiplied by 10 because of we tried the following patch:http://paste.openstack.org/show/596089/

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2017-01-23:

#7

Unfortunately when bond fails rsyslog is not able to send logs to Fuel node anymore, so we can't find anything related to bond0 failure in the archive attached above. Logs from the failed node and the information I asked previously are needed to research this issue further.

Revision history for this message

Sergey Galkin (sgalkin) wrote on 2017-01-23:

#8

logs from failed node (Untitled (30:f0) - node-1990 - osscr01r12c31) available on http://mos-scale-share.mirantis.com/var-log-node-bond-error.tar.gz

Revision history for this message

Sergey Galkin (sgalkin) wrote on 2017-01-26:

#9

Download full text (13.2 KiB)

dmesg from failed node

[ 608.776981] bonding: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
[ 608.787064] Bridge firewalling registered
[ 608.824705] 8021q: 802.1Q VLAN Support v1.8
[ 608.824720] 8021q: adding VLAN 0 to HW filter on device eno1
[ 608.824725] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0
[ 627.044017] device ovs-system entered promiscuous mode
[ 627.044227] netlink: Unknown key attribute (type=20, max=19).
[ 627.095276] device br-floating entered promiscuous mode
[ 627.183961] device p_ff798dba-0 entered promiscuous mode
[ 627.190043] br-ex: port 1(p_ff798dba-0) entered forwarding state
[ 627.190050] br-ex: port 1(p_ff798dba-0) entered forwarding state
[ 633.413407] i40e 0000:01:00.0 eno1: removing f8:bc:12:06:c9:70 vid=0
[ 633.413736] bonding: bond0: Adding slave eno1.
[ 633.413813] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70
[ 633.419025] 8021q: adding VLAN 0 to HW filter on device eno1
[ 633.419040] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0
[ 633.419242] bonding: bond0: enslaving eno1 as an active interface with an up link.
[ 633.443242] bonding: bond0: Adding slave eno2.
[ 633.443247] i40e 0000:01:00.1 eno2: set new mac address f8:bc:12:06:c9:70
[ 633.461615] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[ 633.461788] 8021q: adding VLAN 0 to HW filter on device eno2
[ 633.461790] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0
[ 633.461947] bonding: bond0: enslaving eno2 as an active interface with an up link.
[ 633.487139] bonding: bond0: Adding slave enp3s0f0.
[ 633.487145] i40e 0000:03:00.0 enp3s0f0: set new mac address f8:bc:12:06:c9:70
[ 633.488363] i40e 0000:01:00.1 eno2: NIC Link is Down
[ 633.507526] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[ 633.507639] 8021q: adding VLAN 0 to HW filter on device enp3s0f0
[ 633.507641] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0
[ 633.507799] bonding: bond0: enslaving enp3s0f0 as an active interface with an up link.
[ 633.532945] bonding: bond0: Adding slave enp3s0f1.
[ 633.532950] i40e 0000:03:00.1 enp3s0f1: set new mac address f8:bc:12:06:c9:70
[ 633.534239] i40e 0000:03:00.0 enp3s0f0: NIC Link is Down
[ 633.562735] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[ 633.562886] 8021q: adding VLAN 0 to HW filter on device enp3s0f1
[ 633.562888] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0
[ 633.563047] bonding: bond0: enslaving enp3s0f1 as an active interface with an up link.
[ 633.576316] bonding: bond0: Removing slave eno1.
[ 633.576568] bonding: bond0: releasing active interface eno1
[ 633.576572] bonding: bond0: Warning: the permanent HWaddr of eno1 - f8:bc:12:06:c9:70 - is still in use by bond0. Set the HWaddr of eno1 to a different address to avoid conflicts.
[ 633.588549] i40e 0000:03:00.1 enp3s0f1: NIC Link is Down
[ 633.590506] i40e 0000:01:00.0 eno1: removing f8:bc:12:06:c9:70 vid=0
[ 633.590518] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70
[ 633.590675] bonding: bond0: Removing slave eno2.
[ 633.590901...

dmesg from failed node

[  608.776981] bonding: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
[  608.787064] Bridge firewalling registered
[  608.824705] 8021q: 802.1Q VLAN Support v1.8
[  608.824720] 8021q: adding VLAN 0 to HW filter on device eno1
[  608.824725] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0
[  627.044017] device ovs-system entered promiscuous mode
[  627.044227] netlink: Unknown key attribute (type=20, max=19).
[  627.095276] device br-floating entered promiscuous mode
[  627.183961] device p_ff798dba-0 entered promiscuous mode
[  627.190043] br-ex: port 1(p_ff798dba-0) entered forwarding state
[  627.190050] br-ex: port 1(p_ff798dba-0) entered forwarding state
[  633.413407] i40e 0000:01:00.0 eno1: removing f8:bc:12:06:c9:70 vid=0
[  633.413736] bonding: bond0: Adding slave eno1.
[  633.413813] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70
[  633.419025] 8021q: adding VLAN 0 to HW filter on device eno1
[  633.419040] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0
[  633.419242] bonding: bond0: enslaving eno1 as an active interface with an up link.
[  633.443242] bonding: bond0: Adding slave eno2.
[  633.443247] i40e 0000:01:00.1 eno2: set new mac address f8:bc:12:06:c9:70
[  633.461615] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  633.461788] 8021q: adding VLAN 0 to HW filter on device eno2
[  633.461790] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0
[  633.461947] bonding: bond0: enslaving eno2 as an active interface with an up link.
[  633.487139] bonding: bond0: Adding slave enp3s0f0.
[  633.487145] i40e 0000:03:00.0 enp3s0f0: set new mac address f8:bc:12:06:c9:70
[  633.488363] i40e 0000:01:00.1 eno2: NIC Link is Down
[  633.507526] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  633.507639] 8021q: adding VLAN 0 to HW filter on device enp3s0f0
[  633.507641] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0
[  633.507799] bonding: bond0: enslaving enp3s0f0 as an active interface with an up link.
[  633.532945] bonding: bond0: Adding slave enp3s0f1.
[  633.532950] i40e 0000:03:00.1 enp3s0f1: set new mac address f8:bc:12:06:c9:70
[  633.534239] i40e 0000:03:00.0 enp3s0f0: NIC Link is Down
[  633.562735] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  633.562886] 8021q: adding VLAN 0 to HW filter on device enp3s0f1
[  633.562888] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0
[  633.563047] bonding: bond0: enslaving enp3s0f1 as an active interface with an up link.
[  633.576316] bonding: bond0: Removing slave eno1.
[  633.576568] bonding: bond0: releasing active interface eno1
[  633.576572] bonding: bond0: Warning: the permanent HWaddr of eno1 - f8:bc:12:06:c9:70 - is still in use by bond0. Set the HWaddr of eno1 to a different address to avoid conflicts.
[  633.588549] i40e 0000:03:00.1 enp3s0f1: NIC Link is Down
[  633.590506] i40e 0000:01:00.0 eno1: removing f8:bc:12:06:c9:70 vid=0
[  633.590518] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70
[  633.590675] bonding: bond0: Removing slave eno2.
[  633.590901] bonding: bond0: releasing active interface eno2
[  633.602292] i40e 0000:01:00.1 eno2: removing f8:bc:12:06:c9:70 vid=0
[  633.602304] i40e 0000:01:00.1 eno2: returning to hw mac address f8:bc:12:06:c9:72
[  633.618626] bonding: bond0: Removing slave enp3s0f0.
[  633.618855] bonding: bond0: releasing active interface enp3s0f0
[  633.638077] i40e 0000:03:00.0 enp3s0f0: removing f8:bc:12:06:c9:70 vid=0
[  633.638094] i40e 0000:03:00.0 enp3s0f0: returning to hw mac address 3c:fd:fe:02:2d:60
[  633.642373] bonding: bond0: Removing slave enp3s0f1.
[  633.642683] bonding: bond0: releasing active interface enp3s0f1
[  633.662351] i40e 0000:03:00.1 enp3s0f1: removing f8:bc:12:06:c9:70 vid=0
[  633.662363] i40e 0000:03:00.1 enp3s0f1: returning to hw mac address 3c:fd:fe:02:2d:62
[  633.666528] bonding: bond0: setting mode to 802.3ad (4).
[  634.667631] bonding: bond0: setting xmit hash policy to layer3+4 (1).
[  635.668797] bonding: bond0: Setting LACP rate to fast (1).
[  635.668905] bonding: bond0: Setting ad_select to bandwidth (1).
[  635.668993] bonding: bond0: Setting MII monitoring interval to 100.
[  635.669148] bonding: bond0: Setting up delay to 3000.
[  635.669234] bonding: bond0: Setting down delay to 1000.
[  635.669266] bonding: bond0: Adding slave eno1.
[  635.669337] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70
[  635.675477] 8021q: adding VLAN 0 to HW filter on device eno1
[  635.675480] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0
[  635.675660] bonding: bond0: enslaving eno1 as a backup interface with an up link.
[  635.676409] bonding: bond0: Adding slave eno2.
[  635.676413] i40e 0000:01:00.1 eno2: set new mac address f8:bc:12:06:c9:70
[  635.696349] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  635.696425] 8021q: adding VLAN 0 to HW filter on device eno2
[  635.696427] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0
[  635.696653] bonding: bond0: enslaving eno2 as a backup interface with an up link.
[  635.696738] bonding: bond0: Adding slave enp3s0f0.
[  635.696742] i40e 0000:03:00.0 enp3s0f0: set new mac address f8:bc:12:06:c9:70
[  635.716348] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  635.716427] 8021q: adding VLAN 0 to HW filter on device enp3s0f0
[  635.716429] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0
[  635.716647] bonding: bond0: enslaving enp3s0f0 as a backup interface with an up link.
[  635.716736] bonding: bond0: Adding slave enp3s0f1.
[  635.716752] i40e 0000:03:00.1 enp3s0f1: set new mac address f8:bc:12:06:c9:70
[  635.732771] i40e 0000:01:00.1 eno2: NIC Link is Down
[  635.734497] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  635.734577] 8021q: adding VLAN 0 to HW filter on device enp3s0f1
[  635.734579] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0
[  635.734759] bonding: bond0: enslaving enp3s0f1 as a backup interface with an up link.
[  635.738019] IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready
[  635.738022] 8021q: adding VLAN 0 to HW filter on device bond0
[  635.738064] bonding: bond0: link status down again after 0 ms for interface eno2.
[  635.738067] bonding: bond0: link status definitely up for interface eno1, 10000 Mbps full duplex.
[  635.738086] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[  635.754640] device bond0 entered promiscuous mode
[  635.754642] device eno1 entered promiscuous mode
[  635.754707] device eno2 entered promiscuous mode
[  635.754766] device enp3s0f0 entered promiscuous mode
[  635.754822] device enp3s0f1 entered promiscuous mode
[  635.755134] br-fw-admin: port 1(bond0) entered forwarding state
[  635.755138] br-fw-admin: port 1(bond0) entered forwarding state
[  635.762593] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=132
[  635.762599] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=132
[  635.762604] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=132
[  635.762608] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=132
[  635.800738] device bond0.132 entered promiscuous mode
[  635.801587] i40e 0000:03:00.0 enp3s0f0: NIC Link is Down
[  635.804015] br-mgmt: port 1(bond0.132) entered forwarding state
[  635.804020] br-mgmt: port 1(bond0.132) entered forwarding state
[  635.814834] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=134
[  635.814842] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=134
[  635.814850] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=134
[  635.814858] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=134
[  635.834379] bonding: bond0: link status down again after 100 ms for interface enp3s0f0.
[  635.834381] bonding: bond0: link status down again after 100 ms for interface enp3s0f1.
[  636.059121] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0
[  636.059127] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=132
[  636.059139] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=134
[  636.071886] device bond0.134 entered promiscuous mode
[  636.075210] br-storage: port 1(bond0.134) entered forwarding state
[  636.075214] br-storage: port 1(bond0.134) entered forwarding state
[  636.085437] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=138
[  636.085446] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=138
[  636.085456] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=138
[  636.085464] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=138
[  636.102846] i40e 0000:03:00.1 enp3s0f1: NIC Link is Down
[  636.121106] device bond0.138 entered promiscuous mode
[  636.134689] bonding: bond0: link status down for interface eno1, disabling it in 1000 ms.
[  636.324540] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0
[  636.324542] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=132
[  636.324547] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=134
[  636.324552] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=138
[  636.329875] br-mesh: port 1(bond0.138) entered forwarding state
[  636.329880] br-mesh: port 1(bond0.138) entered forwarding state
[  636.334870] bonding: bond0: link status up again after 200 ms for interface eno1.
[  636.577370] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0
[  636.577376] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=132
[  636.577391] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=134
[  636.577405] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=138
[  636.582521] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=130
[  636.582535] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=130
[  636.582547] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=130
[  636.582562] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=130
[  636.859873] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0
[  636.859880] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=130
[  636.859905] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=132
[  636.859927] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=134
[  636.859934] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=138
[  636.864828] device bond0.130 entered promiscuous mode
[  636.868743] br-ex: port 2(bond0.130) entered forwarding state
[  636.868747] br-ex: port 2(bond0.130) entered forwarding state
[  636.938158] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  636.939471] bonding: bond0: link status up for interface enp3s0f1, enabling it in 3000 ms.
[  636.988358] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  637.039577] bonding: bond0: link status up for interface eno2, enabling it in 3000 ms.
[  637.149576] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[  637.239728] bonding: bond0: link status up for interface enp3s0f0, enabling it in 3000 ms.
[  639.942376] bonding: bond0: link status definitely up for interface enp3s0f1, 10000 Mbps full duplex.
[  639.999486] bonding: bond0: Setting use_carrier to 1.
[  640.000145] bonding: bond0: setting xmit hash policy to layer3+4 (1).
[  640.000168] bonding: bond0: Setting MII monitoring interval to 100.
[  640.000169] bonding: bond0: Note: Updating updelay (to 3000) since it is a multiple of the miimon value.
[  640.000170] bonding: bond0: Note: Updating downdelay (to 1000) since it is a multiple of the miimon value.
[  640.000182] bonding: bond0: Setting down delay to 1000.
[  640.000192] bonding: bond0: Setting up delay to 3000.
[  640.000813] bonding: bond0: Unable to update ad_select because interface is up.
[  640.010202] bonding: bond0: unable to update mode because interface is up.
[  640.018893] bonding: bond0: Unable to update LACP rate because interface is up.
[  640.033583] type=1400 audit(1485467998.744:39): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=32972 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  640.033589] type=1400 audit(1485467998.744:40): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=32972 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  640.042617] bonding: bond0: link status definitely up for interface eno2, 10000 Mbps full duplex.
[  640.133351] type=1400 audit(1485467998.844:41): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=33305 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  640.133357] type=1400 audit(1485467998.844:42): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=33305 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  640.242817] bonding: bond0: link status definitely up for interface enp3s0f0, 10000 Mbps full duplex.
[  642.232579] br-ex: port 1(p_ff798dba-0) entered forwarding state
[  650.784903] br-fw-admin: port 1(bond0) entered forwarding state
[  650.848965] br-mgmt: port 1(bond0.132) entered forwarding state
[  651.105214] br-storage: port 1(bond0.134) entered forwarding state
[  651.361474] br-mesh: port 1(bond0.138) entered forwarding state
[  651.938026] br-ex: port 2(bond0.130) entered forwarding state

Oleksiy Molchanov (omolchanov) on 2017-01-30

Changed in fuel:
milestone:	9.x-updates → 11.0
status:	New → Confirmed
tags:	added: area-library

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-31: Related fix proposed to fuel-library (master)

#10

Related fix proposed to branch: master
Review: https://review.openstack.org/427123

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-06: Related fix merged to fuel-library (master)

#11

Reviewed: https://review.openstack.org/427123
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6fde41c44af5ce785053fdf40eddad65a1a1bff4
Submitter: Jenkins
Branch: master

commit 6fde41c44af5ce785053fdf40eddad65a1a1bff4
Author: Sergey Vasilenko <email address hidden>
Date: Mon Jan 30 17:53:13 2017 +0300

Prevent a bond for unrequired re-assembles

    * Assemble bond members under bond in UP state
    * FIX lost bridge parameter
    * FIX lost 'use_carrier' bond property
    * waiting for bond UP after slaves added
    * additional flush IP addresses from slave while bond assemble

    Change-Id: I568d852e65dc5d5c246e11deb0740f4a608f5ecc
    Closes-bug: #1658981
    Related-bug: #1657750

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-07: Related fix proposed to fuel-library (stable/newton)

#12

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/430173

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-07: Related fix proposed to fuel-library (stable/mitaka)

#13

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/430174

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-11: Related fix merged to fuel-library (stable/mitaka)

#14

Reviewed: https://review.openstack.org/430174
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a848384706cdf69e7241ef4f51c9e2534805f586
Submitter: Jenkins
Branch: stable/mitaka

commit a848384706cdf69e7241ef4f51c9e2534805f586
Author: Sergey Vasilenko <email address hidden>
Date: Mon Jan 30 17:53:13 2017 +0300

Prevent a bond for unrequired re-assembles

    * Assemble bond members under bond in UP state
    * FIX lost bridge parameter
    * FIX lost 'use_carrier' bond property
    * waiting for bond UP after slaves added
    * additional flush IP addresses from slave while bond assemble

    Change-Id: I568d852e65dc5d5c246e11deb0740f4a608f5ecc
    Closes-bug: #1658981
    Related-bug: #1657750
    (cherry picked from commit 6fde41c44af5ce785053fdf40eddad65a1a1bff4)

tags:

added: in-stable-mitaka

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-11: Related fix merged to fuel-library (stable/newton)

#15

Reviewed: https://review.openstack.org/430173
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=dedeaa789c78f8f4a9282e87210468a90eb09b37
Submitter: Jenkins
Branch: stable/newton

commit dedeaa789c78f8f4a9282e87210468a90eb09b37
Author: Sergey Vasilenko <email address hidden>
Date: Mon Jan 30 17:53:13 2017 +0300

Prevent a bond for unrequired re-assembles

    * Assemble bond members under bond in UP state
    * FIX lost bridge parameter
    * FIX lost 'use_carrier' bond property
    * waiting for bond UP after slaves added
    * additional flush IP addresses from slave while bond assemble

    Change-Id: I568d852e65dc5d5c246e11deb0740f4a608f5ecc
    Closes-bug: #1658981
    Related-bug: #1657750
    (cherry picked from commit 6fde41c44af5ce785053fdf40eddad65a1a1bff4)

tags:

added: in-stable-newton

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Confirmed	High	Fuel Sustaining	Fuel for OpenStack 11.0
Nominated for Ocata by Oleksiy Molchanov
	Mitaka	Confirmed	High	Fuel Sustaining	Fuel for OpenStack 9.x-updates
	Newton	Confirmed	High	Fuel Sustaining	Fuel for OpenStack 10.1

Fuel for OpenStack

Random bond down on nodes with 6 interfaces during deployment

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches