Log dropped packets in vrouter for troubleshooting

[richard roberts]

Today troubleshooting drops in vrouter is difficult and actualy challenging in production-like environment (there a many drops - hard to sort out which matters and which do not)

Agregated Drops are indeed reported in analytics, which actually fears end customers ("oh look contrail is dropping !"), although drop are in most cases a very legimitate action.

Current drop hunt is based on dropstats / vif --get-drop-stats / tcpdump or vifdump" this is time consuming.

This LP intends to greatly simplify the drop hunting and saves a lot of time by simply logging dropped packets.

The rationale is that, in many cases the simple fact to identify the dropped packet (info such as IP SRC/DST/ports/etc...) gives a great troubleshooting information on where to look at. The faster we get it the faster the resolution is.

Minimal requirements:
- log dropped packets in the vrouter (it can be a memory buffer for performances)
- cli to see the drops buffer log
- we must be able to disable/enable the logging of drops packets
  * this is either due to possible performance impact or for legal reason (certification telecom authorities / data privacy / etc may not allow to log by default)
- what has to be logged:
  * input interface (and make it easy to filter with grep so we can get packets from a given interface easily)
  * drop code reason (correlation with drop stats)
  * packet dump with at least the important header/first xxx bytes (say 128 ?)
  * packet length

Optional:
- export drop packets to analytics with compute_id, drop code, input interface, and packet header or dump.
- default logging mode vs non logging mode: if we see perfomances impact, we should implement the logging mode only in troubleshooting situation
(+ this can raises some certifcation issues due to confidentiality/)
- per interface loggging ?
- can we have the mac header too () ?
- improve analytics drop view (we should be able to see per vroutre drops counts, record/click on it to see the dropped packets).

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47334
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47336
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47334
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47336
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47334
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47336
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47334
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47336
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47334
Submitter: haji mohamed ashraf ali (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/47334
Committed: http://github.com/Juniper/contrail-vrouter/commit/f77a5b28036fd66a70598e19c870c27eb2c647ec
Submitter: Vinay Vithal Mahuli (<email address hidden>)
Branch: master

commit f77a5b28036fd66a70598e19c870c27eb2c647ec
Author: hajim <email address hidden>
Date: Wed Dec 5 14:11:18 2018 +0530

vrouter pkt drop log buffer infra. This feature
is used to log additional information when a packet is dropped in
vrouter module.
To run pkt drop log : dropstats -log <core number>
If core number specified as 0, it will log for all cores.
Currently below features are enabled.
- Log pkt drops info for all cores
- provided support to enable/disable pkt drop log during load time
by providing "options vrouter vr_pkt_droplog_buf_en=1" in
/etc/modprobe.d/vrouter.conf file.
- Provided support to configure MAX.number of records during load
time providing "options vrouter vr_pkt_droplog_bufsz=<value> in
/etc/modprobe.d/vrouter.conf file.
- Pkt Drop log support can be enabled/disabled during runtime using
sysctl by doing "echo 1
>/proc/sys/net/vrouter/pkt_drop_log_enable"
- We can also enable Minimum logging(drop reason,timestamp, src/dst
IP and file location) by doing "echo 1 >
/proc/sys/net/vrouter/pkt_drop_log_min_enable"
Change-Id:I9b19c22b86a6f604e1bebf5d2e59e173ee5d88cf
Partial-Bug: 1728010

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/47336
Committed: http://github.com/Juniper/contrail-controller/commit/b98d6b8487dab6ba95e6599586077ee44b3350f3
Submitter: Vinay Vithal Mahuli (<email address hidden>)
Branch: master

commit b98d6b8487dab6ba95e6599586077ee44b3350f3
Author: hajim <email address hidden>
Date: Mon Oct 29 15:21:47 2018 +0530

Vrouter drop stats log buffer Infra

Change-Id: I146b650cdec968f15744ddd58fbdc4099a0752d3
Depends-On: I9b19c22b86a6f604e1bebf5d2e59e173ee5d88cf
Partial-Bug: 1728010

[Jeba Paulaiyan (jebap)]

Agenda:

       Let us sync up and conclude on this bug. This is gating 5.0.3 release.

Dev is saying it is fixed, PS is saying it is fixed, QA is saying it is not fixed.

Bill, Sivakumar, I made you optional to keep you updated.

.........................................................................................................................................
Join online meeting <https://meet.juniper.net/jebap/SL2OM964>
Trouble Joining? Try Skype Web App <https://meet.juniper.net/jebap/SL2OM964?sl=1>

Join by Phone
[phone],, 15777450#
ILYNC (45962),, 15777450#
[phone],, 15777450#
[phone],, 15777450#
[phone],, 15777450#
Find a local number <https://dialin.juniper.net>

Conference ID: 15777450

Forgot your dial-in PIN? <https://dialin.juniper.net> | First online meeting? <https://support.office.microsoft.com/en-us/article/join-a-skype-for-business-meeting-3862be6d-758a-4064-a016-67c0febf3cd5?ui=en-US&rs=en-US&ad=US#OS_Type=Mac>

Please consider with whom you are communicating, including non-Juniper personnel of companies federated on Skype for Business, before sharing any confidential information. Non-disclosure agreements may apply.
.........................................................................................................................................

[Kiran (kiran-kn80)]

Hi,
We discussed about this in the call. Vinod needed to set MF bit in the header after which the issue was resolved.

Command used: hping3 -0 172.16.1.3 -H 47 -d -x

Thanks

From: Jeba Paulaiyan <email address hidden>
Date: Saturday, February 16, 2019 at 12:55 AM
To: Veyis Ceylan <email address hidden>, haji mohamed ashraf ali <email address hidden>, Kiran KN <email address hidden>, Vinod Nair <email address hidden>
Cc: Bill Nicholson <email address hidden>, Sivakumar Ganapathy <email address hidden>
Subject: Conclude on JCB-218856 for 5.0.3 release

Agenda:

       Let us sync up and conclude on this bug. This is gating 5.0.3 release.

Dev is saying it is fixed, PS is saying it is fixed, QA is saying it is not fixed.

Bill, Sivakumar, I made you optional to keep you updated.

.........................................................................................................................................
Join online meeting <https://meet.juniper.net/jebap/SL2OM964>
Trouble Joining? Try Skype Web App <https://meet.juniper.net/jebap/SL2OM964?sl=1>

Join by Phone
[phone],, 15777450#
ILYNC (45962),, 15777450#
[phone],, 15777450#
[phone],, 15777450#
[phone],, 15777450#
Find a local number <https://dialin.juniper.net>

Conference ID: 15777450

Forgot your dial-in PIN? <https://dialin.juniper.net> | First online meeting? <https://support.office.microsoft.com/en-us/article/join-a-skype-for-business-meeting-3862be6d-758a-4064-a016-67c0febf3cd5?ui=en-US&rs=en-US&ad=US#OS_Type=Mac>

Please consider with whom you are communicating, including non-Juniper personnel of companies federated on Skype for Business, before sharing any confidential information. Non-disclosure agreements may apply.
.........................................................................................................................................