dmesg from failed node [ 608.776981] bonding: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) [ 608.787064] Bridge firewalling registered [ 608.824705] 8021q: 802.1Q VLAN Support v1.8 [ 608.824720] 8021q: adding VLAN 0 to HW filter on device eno1 [ 608.824725] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0 [ 627.044017] device ovs-system entered promiscuous mode [ 627.044227] netlink: Unknown key attribute (type=20, max=19). [ 627.095276] device br-floating entered promiscuous mode [ 627.183961] device p_ff798dba-0 entered promiscuous mode [ 627.190043] br-ex: port 1(p_ff798dba-0) entered forwarding state [ 627.190050] br-ex: port 1(p_ff798dba-0) entered forwarding state [ 633.413407] i40e 0000:01:00.0 eno1: removing f8:bc:12:06:c9:70 vid=0 [ 633.413736] bonding: bond0: Adding slave eno1. [ 633.413813] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70 [ 633.419025] 8021q: adding VLAN 0 to HW filter on device eno1 [ 633.419040] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0 [ 633.419242] bonding: bond0: enslaving eno1 as an active interface with an up link. [ 633.443242] bonding: bond0: Adding slave eno2. [ 633.443247] i40e 0000:01:00.1 eno2: set new mac address f8:bc:12:06:c9:70 [ 633.461615] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 633.461788] 8021q: adding VLAN 0 to HW filter on device eno2 [ 633.461790] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0 [ 633.461947] bonding: bond0: enslaving eno2 as an active interface with an up link. [ 633.487139] bonding: bond0: Adding slave enp3s0f0. [ 633.487145] i40e 0000:03:00.0 enp3s0f0: set new mac address f8:bc:12:06:c9:70 [ 633.488363] i40e 0000:01:00.1 eno2: NIC Link is Down [ 633.507526] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 633.507639] 8021q: adding VLAN 0 to HW filter on device enp3s0f0 [ 633.507641] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0 [ 633.507799] bonding: bond0: enslaving enp3s0f0 as an active interface with an up link. [ 633.532945] bonding: bond0: Adding slave enp3s0f1. [ 633.532950] i40e 0000:03:00.1 enp3s0f1: set new mac address f8:bc:12:06:c9:70 [ 633.534239] i40e 0000:03:00.0 enp3s0f0: NIC Link is Down [ 633.562735] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 633.562886] 8021q: adding VLAN 0 to HW filter on device enp3s0f1 [ 633.562888] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0 [ 633.563047] bonding: bond0: enslaving enp3s0f1 as an active interface with an up link. [ 633.576316] bonding: bond0: Removing slave eno1. [ 633.576568] bonding: bond0: releasing active interface eno1 [ 633.576572] bonding: bond0: Warning: the permanent HWaddr of eno1 - f8:bc:12:06:c9:70 - is still in use by bond0. Set the HWaddr of eno1 to a different address to avoid conflicts. [ 633.588549] i40e 0000:03:00.1 enp3s0f1: NIC Link is Down [ 633.590506] i40e 0000:01:00.0 eno1: removing f8:bc:12:06:c9:70 vid=0 [ 633.590518] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70 [ 633.590675] bonding: bond0: Removing slave eno2. [ 633.590901] bonding: bond0: releasing active interface eno2 [ 633.602292] i40e 0000:01:00.1 eno2: removing f8:bc:12:06:c9:70 vid=0 [ 633.602304] i40e 0000:01:00.1 eno2: returning to hw mac address f8:bc:12:06:c9:72 [ 633.618626] bonding: bond0: Removing slave enp3s0f0. [ 633.618855] bonding: bond0: releasing active interface enp3s0f0 [ 633.638077] i40e 0000:03:00.0 enp3s0f0: removing f8:bc:12:06:c9:70 vid=0 [ 633.638094] i40e 0000:03:00.0 enp3s0f0: returning to hw mac address 3c:fd:fe:02:2d:60 [ 633.642373] bonding: bond0: Removing slave enp3s0f1. [ 633.642683] bonding: bond0: releasing active interface enp3s0f1 [ 633.662351] i40e 0000:03:00.1 enp3s0f1: removing f8:bc:12:06:c9:70 vid=0 [ 633.662363] i40e 0000:03:00.1 enp3s0f1: returning to hw mac address 3c:fd:fe:02:2d:62 [ 633.666528] bonding: bond0: setting mode to 802.3ad (4). [ 634.667631] bonding: bond0: setting xmit hash policy to layer3+4 (1). [ 635.668797] bonding: bond0: Setting LACP rate to fast (1). [ 635.668905] bonding: bond0: Setting ad_select to bandwidth (1). [ 635.668993] bonding: bond0: Setting MII monitoring interval to 100. [ 635.669148] bonding: bond0: Setting up delay to 3000. [ 635.669234] bonding: bond0: Setting down delay to 1000. [ 635.669266] bonding: bond0: Adding slave eno1. [ 635.669337] i40e 0000:01:00.0 eno1: already using mac address f8:bc:12:06:c9:70 [ 635.675477] 8021q: adding VLAN 0 to HW filter on device eno1 [ 635.675480] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0 [ 635.675660] bonding: bond0: enslaving eno1 as a backup interface with an up link. [ 635.676409] bonding: bond0: Adding slave eno2. [ 635.676413] i40e 0000:01:00.1 eno2: set new mac address f8:bc:12:06:c9:70 [ 635.696349] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 635.696425] 8021q: adding VLAN 0 to HW filter on device eno2 [ 635.696427] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0 [ 635.696653] bonding: bond0: enslaving eno2 as a backup interface with an up link. [ 635.696738] bonding: bond0: Adding slave enp3s0f0. [ 635.696742] i40e 0000:03:00.0 enp3s0f0: set new mac address f8:bc:12:06:c9:70 [ 635.716348] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 635.716427] 8021q: adding VLAN 0 to HW filter on device enp3s0f0 [ 635.716429] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0 [ 635.716647] bonding: bond0: enslaving enp3s0f0 as a backup interface with an up link. [ 635.716736] bonding: bond0: Adding slave enp3s0f1. [ 635.716752] i40e 0000:03:00.1 enp3s0f1: set new mac address f8:bc:12:06:c9:70 [ 635.732771] i40e 0000:01:00.1 eno2: NIC Link is Down [ 635.734497] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 635.734577] 8021q: adding VLAN 0 to HW filter on device enp3s0f1 [ 635.734579] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0 [ 635.734759] bonding: bond0: enslaving enp3s0f1 as a backup interface with an up link. [ 635.738019] IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready [ 635.738022] 8021q: adding VLAN 0 to HW filter on device bond0 [ 635.738064] bonding: bond0: link status down again after 0 ms for interface eno2. [ 635.738067] bonding: bond0: link status definitely up for interface eno1, 10000 Mbps full duplex. [ 635.738086] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready [ 635.754640] device bond0 entered promiscuous mode [ 635.754642] device eno1 entered promiscuous mode [ 635.754707] device eno2 entered promiscuous mode [ 635.754766] device enp3s0f0 entered promiscuous mode [ 635.754822] device enp3s0f1 entered promiscuous mode [ 635.755134] br-fw-admin: port 1(bond0) entered forwarding state [ 635.755138] br-fw-admin: port 1(bond0) entered forwarding state [ 635.762593] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=132 [ 635.762599] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=132 [ 635.762604] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=132 [ 635.762608] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=132 [ 635.800738] device bond0.132 entered promiscuous mode [ 635.801587] i40e 0000:03:00.0 enp3s0f0: NIC Link is Down [ 635.804015] br-mgmt: port 1(bond0.132) entered forwarding state [ 635.804020] br-mgmt: port 1(bond0.132) entered forwarding state [ 635.814834] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=134 [ 635.814842] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=134 [ 635.814850] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=134 [ 635.814858] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=134 [ 635.834379] bonding: bond0: link status down again after 100 ms for interface enp3s0f0. [ 635.834381] bonding: bond0: link status down again after 100 ms for interface enp3s0f1. [ 636.059121] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=0 [ 636.059127] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=132 [ 636.059139] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=134 [ 636.071886] device bond0.134 entered promiscuous mode [ 636.075210] br-storage: port 1(bond0.134) entered forwarding state [ 636.075214] br-storage: port 1(bond0.134) entered forwarding state [ 636.085437] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=138 [ 636.085446] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=138 [ 636.085456] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=138 [ 636.085464] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=138 [ 636.102846] i40e 0000:03:00.1 enp3s0f1: NIC Link is Down [ 636.121106] device bond0.138 entered promiscuous mode [ 636.134689] bonding: bond0: link status down for interface eno1, disabling it in 1000 ms. [ 636.324540] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=0 [ 636.324542] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=132 [ 636.324547] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=134 [ 636.324552] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=138 [ 636.329875] br-mesh: port 1(bond0.138) entered forwarding state [ 636.329880] br-mesh: port 1(bond0.138) entered forwarding state [ 636.334870] bonding: bond0: link status up again after 200 ms for interface eno1. [ 636.577370] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=0 [ 636.577376] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=132 [ 636.577391] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=134 [ 636.577405] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=138 [ 636.582521] i40e 0000:01:00.0 eno1: adding f8:bc:12:06:c9:70 vid=130 [ 636.582535] i40e 0000:01:00.1 eno2: adding f8:bc:12:06:c9:70 vid=130 [ 636.582547] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=130 [ 636.582562] i40e 0000:03:00.1 enp3s0f1: adding f8:bc:12:06:c9:70 vid=130 [ 636.859873] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=0 [ 636.859880] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=130 [ 636.859905] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=132 [ 636.859927] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=134 [ 636.859934] i40e 0000:03:00.0 enp3s0f0: adding f8:bc:12:06:c9:70 vid=138 [ 636.864828] device bond0.130 entered promiscuous mode [ 636.868743] br-ex: port 2(bond0.130) entered forwarding state [ 636.868747] br-ex: port 2(bond0.130) entered forwarding state [ 636.938158] i40e 0000:03:00.1 enp3s0f1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 636.939471] bonding: bond0: link status up for interface enp3s0f1, enabling it in 3000 ms. [ 636.988358] i40e 0000:01:00.1 eno2: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 637.039577] bonding: bond0: link status up for interface eno2, enabling it in 3000 ms. [ 637.149576] i40e 0000:03:00.0 enp3s0f0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None [ 637.239728] bonding: bond0: link status up for interface enp3s0f0, enabling it in 3000 ms. [ 639.942376] bonding: bond0: link status definitely up for interface enp3s0f1, 10000 Mbps full duplex. [ 639.999486] bonding: bond0: Setting use_carrier to 1. [ 640.000145] bonding: bond0: setting xmit hash policy to layer3+4 (1). [ 640.000168] bonding: bond0: Setting MII monitoring interval to 100. [ 640.000169] bonding: bond0: Note: Updating updelay (to 3000) since it is a multiple of the miimon value. [ 640.000170] bonding: bond0: Note: Updating downdelay (to 1000) since it is a multiple of the miimon value. [ 640.000182] bonding: bond0: Setting down delay to 1000. [ 640.000192] bonding: bond0: Setting up delay to 3000. [ 640.000813] bonding: bond0: Unable to update ad_select because interface is up. [ 640.010202] bonding: bond0: unable to update mode because interface is up. [ 640.018893] bonding: bond0: Unable to update LACP rate because interface is up. [ 640.033583] type=1400 audit(1485467998.744:39): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=32972 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [ 640.033589] type=1400 audit(1485467998.744:40): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=32972 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [ 640.042617] bonding: bond0: link status definitely up for interface eno2, 10000 Mbps full duplex. [ 640.133351] type=1400 audit(1485467998.844:41): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=33305 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [ 640.133357] type=1400 audit(1485467998.844:42): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=33305 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 [ 640.242817] bonding: bond0: link status definitely up for interface enp3s0f0, 10000 Mbps full duplex. [ 642.232579] br-ex: port 1(p_ff798dba-0) entered forwarding state [ 650.784903] br-fw-admin: port 1(bond0) entered forwarding state [ 650.848965] br-mgmt: port 1(bond0.132) entered forwarding state [ 651.105214] br-storage: port 1(bond0.134) entered forwarding state [ 651.361474] br-mesh: port 1(bond0.138) entered forwarding state [ 651.938026] br-ex: port 2(bond0.130) entered forwarding state