DPDK: Vrouter with Bond on VF is not coming up

[Vinod Nair]

On a Compute with bond on VF( 1 VF from each port is assigned to bond). Provision went through fine
also before the compute was provisioned bond was up was able to ping the next-hop

DPDK: Vrouter with Bond on VF is not coming up

root@cs-scale-2:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/2 Socket: unix
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3 MTU:1514 Ref:2
            RX port packets:437 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:437 bytes:37582 errors:437
            TX packets:0 bytes:0 errors:0

EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:81:00.1 on NUMA socket 1
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
2016-04-21 14:02:10,516 VROUTER: Found 0 eth device(s)
2016-04-21 14:02:10,516 VROUTER: Using 4 forwarding lcore(s)
2016-04-21 14:02:10,516 VROUTER: Using 0 IO lcore(s)
2016-04-21 14:02:10,516 VROUTER: Using 5 service lcores
2016-04-21 14:02:10,516 VROUTER: set fd limit to 4096 (prev 1024, max 4096)
2016-04-21 14:02:10,532 VROUTER: Starting NetLink...
2016-04-21 14:02:10,532 USOCK: usock_alloc[7f94fb34b700]: new socket FD 41
2016-04-21 14:02:10,532 USOCK: usock_alloc[7f94fb34b700]: setting socket FD 41 send buff size.
Buffer size set to 18320000 (requested 9216000)
2016-04-21 14:02:10,532 VROUTER: NetLink TCP socket FD is 41
2016-04-21 14:02:10,532 VROUTER: uvhost Unix socket FD is 42
2016-04-21 14:02:10,532 VROUTER: Lcore 10: distributing MPLSoGRE packets to [11,12,13]
2016-04-21 14:02:10,532 VROUTER: Lcore 11: distributing MPLSoGRE packets to [10,12,13]
2016-04-21 14:02:10,532 VROUTER: Lcore 12: distributing MPLSoGRE packets to [10,11,13]
2016-04-21 14:02:10,532 VROUTER: Lcore 13: distributing MPLSoGRE packets to [10,11,12]
2016-04-21 14:02:10,532 UVHOST: Starting uvhost server...
2016-04-21 14:02:10,532 UVHOST: server event FD is 43
2016-04-21 14:02:10,532 UVHOST: server socket FD is 44
2016-04-21 14:02:11,532 VROUTER: Retrying connection for socket 42...
2016-04-21 14:02:11,532 UVHOST: Handling connection FD 44...
2016-04-21 14:02:11,532 UVHOST: FD 44 accepted new NetLink connection FD 47
2016-04-21 14:02:11,541 DPCORE: vrouter soft reset start
2016-04-21 14:02:11,560 DPCORE: vrouter soft reset done (0)
2016-04-21 14:02:34,852 VROUTER: Error adding vif 0 eth device 0: invalid PMD 0 (must be less than 0)
2016-04-21 14:02:34,852 VROUTER: Deleting vif 0 eth device
2016-04-21 14:02:34,852 VROUTER: error deleting eth dev : already removed
2016-04-21 14:04:18,858 VROUTER: Error adding vif 0 eth device bond0: no port ID found for PCI 0000:00:00.0
2016-04-21 14:04:18,858 VROUTER: Deleting vif 0 eth device
2016-04-21 14:04:18,858 VROUTER: error deleting eth dev : already removed
2016-04-21 14:04:18,858 VROUTER: Adding vif 2 (gen. 3) packet device unix
2016-04-21 14:04:18,858 USOCK: usock_alloc[7f94fb34b700]: new socket FD 48
2016-04-21 14:04:18,858 USOCK: usock_alloc[7f94fb34b700]: setting socket FD 48 send buff size.
Buffer size set to 18320000 (requested 9216000)

TESTBED.py snippet

env.dpdk = {
    host2: { 'huge_pages' : '50', 'coremask' : '0xf'},
}

env.sriov = {
     host2 :[ {'interface' : 'p1p1', 'VF' : 7, 'physnets' : ['physnet1']},{'interface' : 'p1p2', 'VF' : 7, 'physnets' : ['physnet2']} ]
}

bond= {
         host1 : { 'name': 'bond0', 'member': ['p1p1','p1p2'], 'mode': '802.3ad', 'xmit_hash_policy': 'layer3+4' },
         host2 : { 'name': 'bond0', 'member': ['p1p1_4','p1p2_4'], 'mode': 'active-backup'},
}

#OPTIONAL SEPARATION OF MANAGEMENT AND CONTROL + DATA
#====================================================
#Control Interface
control_data = {
    host1 : { 'ip': '13.1.0.1/24', 'gw' : '13.1.0.254', 'device':'bond0' },
    host2 : { 'ip': '13.1.0.2/24', 'gw' : '13.1.0.254', 'device':'bond0'}

}

Version: 3.0.2.0-31~kilo

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20397
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20400
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20401
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20402
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20397
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/69e568b3fcadb1389d8439d55589e6c177fe1ca1
Submitter: Zuul
Branch: R3.0

commit 69e568b3fcadb1389d8439d55589e6c177fe1ca1
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:08:42 2016 -0700

Bring up all interfaces after creating VFs (so that bond interface can
come up after VFs are added).

Change-Id: I4b299a8578483907f3cd2eef706f4d09ecc8fdf4
Partial-Bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20400
Committed: http://github.org/Juniper/contrail-packaging/commit/9d4058960dcd260f0477ee128920f8b42a438521
Submitter: Zuul
Branch: R3.0

commit 9d4058960dcd260f0477ee128920f8b42a438521
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:15:35 2016 -0700

Wait for slave interfaces to come up before vrouter-dpdk starts up,
especially if the vrouter physical interface is a vlan interface.

Change-Id: Idb313d6a6a081c74cf3764121922139d913eb997
Partial-bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20401
Committed: http://github.org/Juniper/contrail-provisioning/commit/ac54dc34bebdfb3f7da1aa7b5a99cbdcd91fa57c
Submitter: Zuul
Branch: R3.0

commit ac54dc34bebdfb3f7da1aa7b5a99cbdcd91fa57c
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:17:50 2016 -0700

- When provisioning a compute node with VFs, use the same command
as setup_interface.

- Avoid bringing down the PF when the VF comes up. Otherwise, addition
of the VF to a bond fails, probably due a bug in the ixgbe driver.

- Modify /etc/network/interfaces to add the right commands for VFs

- Enslave VFs explicitly in /etc/network/interfaces to work around flaky
behavior in ixgbe diver.

Change-Id: Ia2d85d0fb19f2d4b30d1f4eac8602913c7f2ea0a
Partial-bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20402
Committed: http://github.org/Juniper/contrail-vrouter/commit/fbf75259310e214979718ca5990010bb437d12c9
Submitter: Zuul
Branch: R3.0

commit fbf75259310e214979718ca5990010bb437d12c9
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:23:47 2016 -0700

- Set the VLAN filter for VFs if vrouter is provisioned on a VLAN.

Change-Id: I33e5f26b50c0532c3a802acba8025f3a268be4fb
Closes-bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20439
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20440
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20439
Committed: http://github.org/Juniper/contrail-provisioning/commit/4158fcef4ed2ea54d48064d0ca680c8141eb1c1d
Submitter: Zuul
Branch: R3.0

commit 4158fcef4ed2ea54d48064d0ca680c8141eb1c1d
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 13:51:11 2016 -0700

Prevent prompt for user/password when running fab setup_all

Change-Id: I7d8faa246f7752716f876fc330cd9a278823bbba
Closes-bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20440
Committed: http://github.org/Juniper/contrail-vrouter/commit/a03c7c5ee14b4928802c280cc24b82e4d35985d3
Submitter: Zuul
Branch: R3.0

commit a03c7c5ee14b4928802c280cc24b82e4d35985d3
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 13:52:58 2016 -0700

Set vlan filter only when vrouter physical interface is a VF.

Change-Id: Iad9a60ba9fa737167a650cd80997ff149cc8c9cf
Closes-bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20553
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20553
Committed: http://github.org/Juniper/contrail-provisioning/commit/9d08ef1adfadff79e12502fda41f47a789d10f11
Submitter: Zuul
Branch: R3.0

commit 9d08ef1adfadff79e12502fda41f47a789d10f11
Author: Raja Sivaramakrishnan <email address hidden>
Date: Mon May 23 18:21:44 2016 -0700

vlan interface creation fails if the pre-up does "ifconfig up" of the vlan
interface that is being brought up. This is not an issue for physical interfaces,
but fails if the VLAN is created on top of SR-IOV VFs. post-up should be
used instead of pre-up as the interface would have been created by then.

Change-Id: If7988947cf758776e36cbe5df87e9f55e40c44bb
Closes-bug: 1573255

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20598
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20598
Committed: http://github.org/Juniper/contrail-vrouter/commit/306b142c09770887a4d94ce4c1d0b30a58275319
Submitter: Zuul
Branch: R3.0

commit 306b142c09770887a4d94ce4c1d0b30a58275319
Author: Raja Sivaramakrishnan <email address hidden>
Date: Tue May 24 23:31:38 2016 -0700

Set MAC address of slave interface explicitly if it is a SR-IOV VF.

Change-Id: I4d62d23fff5d9e34477f3d706197322c1f2c94a7
Closes-bug: 1573255

[Vinod Nair (vinodnair)]

Active-standby mode we don't support with dpdk rite

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21389
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21390
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21392
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21393
Submitter: Raja Sivaramakrishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21389
Committed: http://github.org/Juniper/contrail-provisioning/commit/89549526db727d59c8f8a5e837217c62c2e16cc0
Submitter: Zuul
Branch: master

commit 89549526db727d59c8f8a5e837217c62c2e16cc0
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu Jun 23 14:56:22 2016 -0700

Merge below 3.0 changes into master.

- When provisioning a compute node with VFs, use the same command
as setup_interface.

- Avoid bringing down the PF when the VF comes up. Otherwise, addition
of the VF to a bond fails, probably due a bug in the ixgbe driver.

- Modify /etc/network/interfaces to add the right commands for VFs

- Enslave VFs explicitly in /etc/network/interfaces to work around flaky
behavior in ixgbe diver.

- Prevent prompt for user/password when running fab setup_all

- vlan interface creation fails if the pre-up does "ifconfig up" of the vlan
interface that is being brought up. This is not an issue for physical interfaces,
but fails if the VLAN is created on top of SR-IOV VFs. post-up should be
used instead of pre-up as the interface would have been created by then.

- Bringing up interfaces one at a time intstead of all.

Change-Id: I55a583e8e89a31c3b313ad223bae2976e3f344f3
Closes-Bug: 1573255 1585876

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21390
Committed: http://github.org/Juniper/contrail-packaging/commit/2ffade6cf26f80cec6e8f3d952721872a28afc78
Submitter: Zuul
Branch: master

commit 2ffade6cf26f80cec6e8f3d952721872a28afc78
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu Jun 23 15:16:56 2016 -0700

Merge following change from 3.0

commit 792717186faf520a4d09f022846cccfecf290652
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:15:35 2016 -0700

Wait for slave interfaces to come up before vrouter-dpdk starts up,
especially if the vrouter physical interface is a vlan interface.

Change-Id: Idb313d6a6a081c74cf3764121922139d913eb997
Partial-bug: 1573255

Change-Id: Icfaf3a33d79c5ed67b306ec7fe86cfb4d59d3ea2

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21392
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/32f77a777dcd5958352af486fd1b215f14598eec
Submitter: Zuul
Branch: master

commit 32f77a777dcd5958352af486fd1b215f14598eec
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu Jun 23 15:23:34 2016 -0700

Merge following 3.0 commit into master.

commit 4c0421eac744db57f69ac06a9e8ad51fe39519cf
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:08:42 2016 -0700

Bring up all interfaces after creating VFs (so that bond interface can
come up after VFs are added).

Change-Id: I4b299a8578483907f3cd2eef706f4d09ecc8fdf4
Partial-Bug: 1573255

Change-Id: I3877c8f9b97e02cf765cd0cbcf3bde9ef859395d

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21393
Committed: http://github.org/Juniper/contrail-vrouter/commit/c71dea5c6364f26b4f4dd9fa63b5656ca8daf9a5
Submitter: Zuul
Branch: master

commit c71dea5c6364f26b4f4dd9fa63b5656ca8daf9a5
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu Jun 23 15:26:01 2016 -0700

Merge following 3.0 commits into master.

commit 5a9838a91c5061f5e7a22b882f1f3dd8504baf8d
Author: Raja Sivaramakrishnan <email address hidden>
Date: Wed Jun 1 16:56:17 2016 -0700

Fix a race condition which resulted in the pkt0 thread polling too soon
(before the eventfd could be added to the poll).

Change-Id: I51c1484d7aa8b67fcb2a03c42ef73894158129da
Closes-Bug: 1587275

commit cfa2f6e92ae11885724b868daea667d556921e64
Author: Raja Sivaramakrishnan <email address hidden>
Date: Tue May 24 23:31:38 2016 -0700

Set MAC address of slave interface explicitly if it is a SR-IOV VF.

Change-Id: I4d62d23fff5d9e34477f3d706197322c1f2c94a7
Closes-bug: 1573255

commit b060bc7dca5bb2368a6ccca7420ab8a3bf1eb74c
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 13:52:58 2016 -0700

Set vlan filter only when vrouter physical interface is a VF.

Change-Id: Iad9a60ba9fa737167a650cd80997ff149cc8c9cf
Closes-bug: 1573255

commit 1979d7e728e6d293ddedcfb06c74eaaaa0d5e363
Author: Raja Sivaramakrishnan <email address hidden>
Date: Thu May 19 00:23:47 2016 -0700

- Set the VLAN filter for VFs if vrouter is provisioned on a VLAN.

Change-Id: I33e5f26b50c0532c3a802acba8025f3a268be4fb
Closes-bug: 1573255

Change-Id: I725ee4558c0ddc9131039f9fa7c7f361395ce608