CC13: DPDK vrouter drops packets >= 1500

Bug #1795839 reported by Bernhard Koessler
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Committed
Critical
Kiran
Trunk
Fix Committed
Critical
Kiran

Bug Description

In a CC13/Contrail 5.0.1 setup with Intel X710 Fortville NICs, most DPDK vrouters drop with default 1500 MTU and above. It can be seen that the interface drop counter increases on the trnasmitting side.

Possibly related to
CC13: DPDK vrouter fragments packets >= 1500 from non-DPDK workloads
https://bugs.launchpad.net/juniperopenstack/+bug/1795834

Tests:
- Test setup: 2 DPDK packet gen images on 2 different DPDK compute nodes (overcloud63m-compdpdk-22 to overcloud63m-compdpdk-25)
- DPDK packetgen traffic with packet < 1400 works in both directions without issues
- DPDK packetgen traffic with packets 1511 works only in one direction (from overcloud63m-compdpdk-22 to overcloud63m-compdpdk-25) but not the other. If overcloud63m-compdpdk-25 tries to send large packets, nothing goes out on vif 0 and I see the interface drop counter go up
- Non-DPDK traffic (exiting packetgen and using ping) for small packets works in both direction
- Non-DPDK traffic for large packets gets fragmented (despite 9000 configured on vhost0) and only works in the same direction as the DPDK traffic (from overcloud63m-compdpdk-22 to overcloud63m-compdpdk-25) --> this is covered in https://bugs.launchpad.net/juniperopenstack/+bug/1795834
- Note: kernel mode vrouters work fine with all packet sizes, also from working DPDK node 22 to kernel mode all works.

test logs attached.

Tags: dpdk vrouter cc13
Revision history for this message
Bernhard Koessler (bkoessler) wrote :
Revision history for this message
Bernhard Koessler (bkoessler) wrote :
Revision history for this message
Vinod Nair (vinodnair) wrote :

Kiran is working on the fix.

Revision history for this message
Sivakumar Ganapathy (hotlava51) wrote :

This bug is seen in a corner case only when vrouter mtu < vm mtu. Not critical for 5.0.2 and hence removing it from the release.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/46957
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/46958
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/46958
Committed: http://github.com/Juniper/contrail-vrouter/commit/514d42fd16f7cf4a7dc4cc5bf110cdbbe4aa3436
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 514d42fd16f7cf4a7dc4cc5bf110cdbbe4aa3436
Author: Kiran Kasim <email address hidden>
Date: Fri Oct 12 09:54:05 2018 -0700

DPDK: Fix for vhost0 MTU not getting set

After an agent soft reset, vif 0 gets deleted and added back. But due to
a timing issue, before it gets added back, the vhost0 MTU notification
used to come resulting in MTU not getting set (Since vif is not added
yet). Fix is to query the MTU from PMD during vif 0 addition.

Change-Id: I2b102b82e21fcc137e62db748d8982bdbbdf87e2
Closes-Bug: 1795839

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/46957
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/48248
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/48248
Committed: http://github.com/Juniper/contrail-vrouter/commit/7504900d9e3e3c696306428e1bcd96dd7dab0ea5
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 7504900d9e3e3c696306428e1bcd96dd7dab0ea5
Author: Kiran Kasim <email address hidden>
Date: Mon Dec 17 23:00:54 2018 -0800

DPDK: Fix for vhost0 MTU not getting set

After an agent soft reset, vif 0 gets deleted and added back. But due to
a timing issue, before it gets added back, the vhost0 MTU notification
used to come resulting in MTU not getting set (Since vif is not added
yet). Fix is to query the MTU from PMD during vif 0 addition.

Change-Id: I262f2e62cbd390b3c8bae2bdc777fc1ac4249886
Closes-Bug: 1795839

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/49229
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/49230
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2.3.x

Review in progress for https://review.opencontrail.org/49231
Submitter: Kiran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/49229
Committed: http://github.com/Juniper/contrail-vrouter/commit/1d38cf70d5e2b4361d5265c37f4b662236e2c088
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 1d38cf70d5e2b4361d5265c37f4b662236e2c088
Author: Kiran Kasim <email address hidden>
Date: Fri Oct 12 09:54:05 2018 -0700

DPDK: Fix for vhost0 MTU not getting set

After an agent soft reset, vif 0 gets deleted and added back. But due to
a timing issue, before it gets added back, the vhost0 MTU notification
used to come resulting in MTU not getting set (Since vif is not added
yet). Fix is to query the MTU from PMD during vif 0 addition.

Change-Id: I2b102b82e21fcc137e62db748d8982bdbbdf87e2
Closes-Bug: 1795839
(cherry picked from commit 514d42fd16f7cf4a7dc4cc5bf110cdbbe4aa3436)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/49230
Committed: http://github.com/Juniper/contrail-vrouter/commit/928a6084116c3834e123cf075aee6d598be1b9d2
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 928a6084116c3834e123cf075aee6d598be1b9d2
Author: Kiran Kasim <email address hidden>
Date: Fri Oct 12 09:54:05 2018 -0700

DPDK: Fix for vhost0 MTU not getting set

After an agent soft reset, vif 0 gets deleted and added back. But due to
a timing issue, before it gets added back, the vhost0 MTU notification
used to come resulting in MTU not getting set (Since vif is not added
yet). Fix is to query the MTU from PMD during vif 0 addition.

Change-Id: I2b102b82e21fcc137e62db748d8982bdbbdf87e2
Closes-Bug: 1795839
(cherry picked from commit 514d42fd16f7cf4a7dc4cc5bf110cdbbe4aa3436)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.