Activity log for bug #1676329

Date Who What changed Old value New value Message
2017-03-27 09:30:56 Sergii bug added bug
2017-03-27 10:21:11 Sergii description Detailed bug description: After rebooting compute node one of dpdk interface belonging to balance-tcp bonding mode is lost. ovs-vsctl show: Bridge br-prv Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "bond0" Interface "dpdk1" type: dpdk error: "could not open network device dpdk1 (No such device)" Interface "dpdk0" type: dpdk Port br-prv Interface br-prv type: internal Port phy-br-prv Interface phy-br-prv type: patch options: {peer=int-br-prv} ovs_version: "2.6.1" Steps to reproduce: 1. Deploy cluster with dpdk and bond mode balance-tcp. (In my case I used 2 compute nodes) 2. Configure LACP in ovs and on the hardware switch ovs-vsctl set port bond0 lacp=active ovs-vsctl set port bond0 other_config:lacp-time=slow 3. Check that LACP in negotiated root@node-3:~# ovs-appctl bond/show bond0 ---- bond0 ---- bond_mode: balance-tcp bond may use recirculation: yes, Recirc-ID : 1 bond-hash-basis: 0 updelay: 3000 ms downdelay: 1000 ms next rebalance: 7774 ms lacp_status: `negotiated` active slave mac: 00:25:90:0a:4b:dc(dpdk0) 2. Reboot any compute node 3. Look at port status in ovs ovs-vsctl show Expected results: LACP works correctly. Actual result: Could not open network device dpdk0 or 1. Only one interface is available Reproducibility: Checked on builds 1499, 1507,1513. Each time after rebooting compute node. Description of the environment: Fuel 10 build 1513 1 controller, 2 computes, 1 BASE-OS node. Bonding 2x10G on both compute nodes with DPDK and balance-tcp mode. Additional information: root@node-1:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:0a:00.0 'I350 Gigabit Network Connection' if=enp10s0f0 drv=igb unused=igb_uio 0000:0a:00.1 'I350 Gigabit Network Connection' if=enp10s0f1 drv=igb unused=igb_uio 0000:81:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f0,ens11f0d1 drv=i40e unused=igb_uio 0000:81:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f1,ens11f1d1 drv=i40e unused=igb_uio Fuel diagnostic snapshot: Detailed bug description:  After rebooting compute node one of dpdk interface belonging to balance-tcp bonding mode is lost.  ovs-vsctl show:     Bridge br-prv         Controller "tcp:127.0.0.1:6633"             is_connected: true         fail_mode: secure         Port "bond0"             Interface "dpdk1"                 type: dpdk                 error: "could not open network device dpdk1 (No such device)"             Interface "dpdk0"                 type: dpdk         Port br-prv             Interface br-prv                 type: internal         Port phy-br-prv             Interface phy-br-prv                 type: patch                 options: {peer=int-br-prv}     ovs_version: "2.6.1" Steps to reproduce:  1. Deploy cluster with dpdk and bond mode balance-tcp. (In my case I used 2 compute nodes)  2. Configure LACP in ovs and on the hardware switch         ovs-vsctl set port bond0 lacp=active         ovs-vsctl set port bond0 other_config:lacp-time=slow  3. Check that LACP in negotiated         root@node-3:~# ovs-appctl bond/show bond0         ---- bond0 ----         bond_mode: balance-tcp         bond may use recirculation: yes, Recirc-ID : 1         bond-hash-basis: 0         updelay: 3000 ms         downdelay: 1000 ms         next rebalance: 7774 ms         lacp_status: `negotiated`         active slave mac: 00:25:90:0a:4b:dc(dpdk0)  2. Reboot any compute node  3. Look at port status in ovs         `ovs-vsctl show` Expected results:  LACP works correctly. Actual result:  Could not open network device dpdk0 or 1. Only one interface is available Reproducibility:  Checked on builds 1499, 1507,1513. Each time after rebooting compute node. Description of the environment:  Fuel 10 build 1513  1 controller, 2 computes, 1 BASE-OS node. Bonding 2x10G on both compute nodes with DPDK and balance-tcp mode. Additional information:  root@node-1:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:0a:00.0 'I350 Gigabit Network Connection' if=enp10s0f0 drv=igb unused=igb_uio 0000:0a:00.1 'I350 Gigabit Network Connection' if=enp10s0f1 drv=igb unused=igb_uio 0000:81:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f0,ens11f0d1 drv=i40e unused=igb_uio 0000:81:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f1,ens11f1d1 drv=i40e unused=igb_uio Fuel diagnostic snapshot:
2017-03-27 10:21:39 Sergii description Detailed bug description:  After rebooting compute node one of dpdk interface belonging to balance-tcp bonding mode is lost.  ovs-vsctl show:     Bridge br-prv         Controller "tcp:127.0.0.1:6633"             is_connected: true         fail_mode: secure         Port "bond0"             Interface "dpdk1"                 type: dpdk                 error: "could not open network device dpdk1 (No such device)"             Interface "dpdk0"                 type: dpdk         Port br-prv             Interface br-prv                 type: internal         Port phy-br-prv             Interface phy-br-prv                 type: patch                 options: {peer=int-br-prv}     ovs_version: "2.6.1" Steps to reproduce:  1. Deploy cluster with dpdk and bond mode balance-tcp. (In my case I used 2 compute nodes)  2. Configure LACP in ovs and on the hardware switch         ovs-vsctl set port bond0 lacp=active         ovs-vsctl set port bond0 other_config:lacp-time=slow  3. Check that LACP in negotiated         root@node-3:~# ovs-appctl bond/show bond0         ---- bond0 ----         bond_mode: balance-tcp         bond may use recirculation: yes, Recirc-ID : 1         bond-hash-basis: 0         updelay: 3000 ms         downdelay: 1000 ms         next rebalance: 7774 ms         lacp_status: `negotiated`         active slave mac: 00:25:90:0a:4b:dc(dpdk0)  2. Reboot any compute node  3. Look at port status in ovs         `ovs-vsctl show` Expected results:  LACP works correctly. Actual result:  Could not open network device dpdk0 or 1. Only one interface is available Reproducibility:  Checked on builds 1499, 1507,1513. Each time after rebooting compute node. Description of the environment:  Fuel 10 build 1513  1 controller, 2 computes, 1 BASE-OS node. Bonding 2x10G on both compute nodes with DPDK and balance-tcp mode. Additional information:  root@node-1:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:0a:00.0 'I350 Gigabit Network Connection' if=enp10s0f0 drv=igb unused=igb_uio 0000:0a:00.1 'I350 Gigabit Network Connection' if=enp10s0f1 drv=igb unused=igb_uio 0000:81:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f0,ens11f0d1 drv=i40e unused=igb_uio 0000:81:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f1,ens11f1d1 drv=i40e unused=igb_uio Fuel diagnostic snapshot: Detailed bug description:  After rebooting compute node one of dpdk interface belonging to balance-tcp bonding mode is lost.  ovs-vsctl show:     Bridge br-prv         Controller "tcp:127.0.0.1:6633"             is_connected: true         fail_mode: secure         Port "bond0"             Interface "dpdk1"                 type: dpdk                 error: "could not open network device dpdk1 (No such device)"             Interface "dpdk0"                 type: dpdk         Port br-prv             Interface br-prv                 type: internal         Port phy-br-prv             Interface phy-br-prv                 type: patch                 options: {peer=int-br-prv}     ovs_version: "2.6.1" Steps to reproduce:  1. Deploy cluster with dpdk and bond mode balance-tcp. (In my case I used 2 compute nodes)  2. Configure LACP in ovs and on the hardware switch         ovs-vsctl set port bond0 lacp=active         ovs-vsctl set port bond0 other_config:lacp-time=slow  3. Check that LACP in negotiated         root@node-3:~# ovs-appctl bond/show bond0         ---- bond0 ----         bond_mode: balance-tcp         bond may use recirculation: yes, Recirc-ID : 1         bond-hash-basis: 0         updelay: 3000 ms         downdelay: 1000 ms         next rebalance: 7774 ms         lacp_status: negotiated         active slave mac: 00:25:90:0a:4b:dc(dpdk0)  2. Reboot any compute node  3. Look at port status in ovs         ovs-vsctl show Expected results:  LACP works correctly. Actual result:  Could not open network device dpdk0 or 1. Only one interface is available Reproducibility:  Checked on builds 1499, 1507,1513. Each time after rebooting compute node. Description of the environment:  Fuel 10 build 1513  1 controller, 2 computes, 1 BASE-OS node. Bonding 2x10G on both compute nodes with DPDK and balance-tcp mode. Additional information:  root@node-1:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:0a:00.0 'I350 Gigabit Network Connection' if=enp10s0f0 drv=igb unused=igb_uio 0000:0a:00.1 'I350 Gigabit Network Connection' if=enp10s0f1 drv=igb unused=igb_uio 0000:81:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f0,ens11f0d1 drv=i40e unused=igb_uio 0000:81:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f1,ens11f1d1 drv=i40e unused=igb_uio Fuel diagnostic snapshot:
2017-03-27 11:31:30 Oleksiy Molchanov fuel: status New Incomplete
2017-03-27 11:31:40 Oleksiy Molchanov fuel: assignee Sergii (sgudz)
2017-03-27 12:42:12 Sergii description Detailed bug description:  After rebooting compute node one of dpdk interface belonging to balance-tcp bonding mode is lost.  ovs-vsctl show:     Bridge br-prv         Controller "tcp:127.0.0.1:6633"             is_connected: true         fail_mode: secure         Port "bond0"             Interface "dpdk1"                 type: dpdk                 error: "could not open network device dpdk1 (No such device)"             Interface "dpdk0"                 type: dpdk         Port br-prv             Interface br-prv                 type: internal         Port phy-br-prv             Interface phy-br-prv                 type: patch                 options: {peer=int-br-prv}     ovs_version: "2.6.1" Steps to reproduce:  1. Deploy cluster with dpdk and bond mode balance-tcp. (In my case I used 2 compute nodes)  2. Configure LACP in ovs and on the hardware switch         ovs-vsctl set port bond0 lacp=active         ovs-vsctl set port bond0 other_config:lacp-time=slow  3. Check that LACP in negotiated         root@node-3:~# ovs-appctl bond/show bond0         ---- bond0 ----         bond_mode: balance-tcp         bond may use recirculation: yes, Recirc-ID : 1         bond-hash-basis: 0         updelay: 3000 ms         downdelay: 1000 ms         next rebalance: 7774 ms         lacp_status: negotiated         active slave mac: 00:25:90:0a:4b:dc(dpdk0)  2. Reboot any compute node  3. Look at port status in ovs         ovs-vsctl show Expected results:  LACP works correctly. Actual result:  Could not open network device dpdk0 or 1. Only one interface is available Reproducibility:  Checked on builds 1499, 1507,1513. Each time after rebooting compute node. Description of the environment:  Fuel 10 build 1513  1 controller, 2 computes, 1 BASE-OS node. Bonding 2x10G on both compute nodes with DPDK and balance-tcp mode. Additional information:  root@node-1:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:0a:00.0 'I350 Gigabit Network Connection' if=enp10s0f0 drv=igb unused=igb_uio 0000:0a:00.1 'I350 Gigabit Network Connection' if=enp10s0f1 drv=igb unused=igb_uio 0000:81:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f0,ens11f0d1 drv=i40e unused=igb_uio 0000:81:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f1,ens11f1d1 drv=i40e unused=igb_uio Fuel diagnostic snapshot: Detailed bug description:  After rebooting compute node one of dpdk interface belonging to balance-tcp bonding mode is lost.  ovs-vsctl show:     Bridge br-prv         Controller "tcp:127.0.0.1:6633"             is_connected: true         fail_mode: secure         Port "bond0"             Interface "dpdk1"                 type: dpdk                 error: "could not open network device dpdk1 (No such device)"             Interface "dpdk0"                 type: dpdk         Port br-prv             Interface br-prv                 type: internal         Port phy-br-prv             Interface phy-br-prv                 type: patch                 options: {peer=int-br-prv}     ovs_version: "2.6.1" Steps to reproduce:  1. Deploy cluster with dpdk and bond mode balance-tcp. (In my case I used 2 compute nodes)  2. Configure LACP in ovs and on the hardware switch         ovs-vsctl set port bond0 lacp=active         ovs-vsctl set port bond0 other_config:lacp-time=slow  3. Check that LACP in negotiated         root@node-3:~# ovs-appctl bond/show bond0         ---- bond0 ----         bond_mode: balance-tcp         bond may use recirculation: yes, Recirc-ID : 1         bond-hash-basis: 0         updelay: 3000 ms         downdelay: 1000 ms         next rebalance: 7774 ms         lacp_status: negotiated         active slave mac: 00:25:90:0a:4b:dc(dpdk0)  2. Reboot any compute node  3. Look at port status in ovs         ovs-vsctl show Expected results:  LACP works correctly. Actual result:  Could not open network device dpdk0 or 1. Only one interface is available Reproducibility:  Checked on builds 1499, 1507,1513. Each time after rebooting compute node. Description of the environment:  Fuel 10 build 1513  1 controller, 2 computes, 1 BASE-OS node. Bonding 2x10G on both compute nodes with DPDK and balance-tcp mode. Additional information:  root@node-1:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:0a:00.0 'I350 Gigabit Network Connection' if=enp10s0f0 drv=igb unused=igb_uio 0000:0a:00.1 'I350 Gigabit Network Connection' if=enp10s0f1 drv=igb unused=igb_uio 0000:81:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f0,ens11f0d1 drv=i40e unused=igb_uio 0000:81:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=ens11f1,ens11f1d1 drv=i40e unused=igb_uio Fuel diagnostic snapshot: http://mos-scale-share.mirantis.com/sgudz/fuel-snapshot-2017-03-27_08-56-43.tar
2017-03-27 12:59:10 Oleksiy Molchanov fuel: status Incomplete Confirmed
2017-03-27 12:59:16 Oleksiy Molchanov fuel: assignee Sergii (sgudz) Fuel Sustaining (fuel-sustaining-team)
2017-03-27 12:59:19 Oleksiy Molchanov fuel: importance Undecided High
2017-03-29 16:19:33 Pavel tags blocker-for-qa
2017-03-29 17:08:04 Sergey Philippov bug added subscriber Sergey Philippov
2017-03-30 10:57:17 Pavel tags blocker-for-qa
2017-03-30 17:25:04 Atsuko Ito fuel: assignee Fuel Sustaining (fuel-sustaining-team) Ivan Suzdal (isuzdal)
2017-03-30 17:29:31 Fuel Devops McRobotson fuel: status Confirmed In Progress
2017-03-31 07:19:12 Michael Polenchuk bug added subscriber Michael Polenchuk
2017-04-28 09:55:34 Fuel Devops McRobotson fuel: status In Progress Fix Committed