sendpkts scapy command is timing out

Bug #1718142 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Fix Committed
High
Vedamurthy Joshi
R4.0
Fix Committed
High
Vedamurthy Joshi
R4.1
Fix Committed
High
Vedamurthy Joshi
Trunk
Fix Committed
High
Vedamurthy Joshi

Bug Description

At times, sendpkts command is timing out and causes the script to fail as seen in
http://10.204.216.50/Docs/logs/4.1.0.0-54_jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-173_1505778324.96/logs//testecmpsanity.log (test_ecmp_svc_in_network_with_static_route_no_policy)

tcutils.fabutils.log:

2017-09-19 03:00:28,822 - DEBUG - sendpkts --name sendudp_761223_3 --stop
2017-09-19 03:01:09,656 - DEBUG - Running remote_cmd, Cmd : recvpkts --name recvudp_761223_3 --stop, host_string: ubuntu@169.254.0.6, password: Nonegateway: root@10.204.216.99, gateway password: c0ntrail123
2017-09-19 03:01:09,802 - DEBUG - recvpkts --name recvudp_761223_3 --stop
2017-09-19 03:02:15,931 - ERROR - Unable to run command sendpkts --name sendudp_689772_1 --stop: Command failed to finish in 120 seconds
Traceback (most recent call last):
  File "tcutils/fabutils.py", line 111, in remote_cmd
    output = _run(cmd, timeout=timeout, pty=not as_daemon, shell=shell)
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 677, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 1146, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 931, in _run_command
    capture_buffer_size=capture_buffer_size)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 813, in _execute
    worker.raise_if_needed()
  File "/usr/local/lib/python2.7/dist-packages/fabric/thread_handling.py", line 12, in wrapper
    callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/io.py", line 32, in output_loop
    OutputLooper(*args, **kwargs).loop()
  File "/usr/local/lib/python2.7/dist-packages/fabric/io.py", line 87, in loop
    raise CommandTimeout(timeout=self.timeout)
CommandTimeout: Command failed to finish in 120 seconds

Jeba Paulaiyan (jebap)
tags: added: sanity
Revision history for this message
Ganesha HV (ganeshahv) wrote :

Need setup in that state.

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

Ubuntu-traffic has v6 family also enabled always. When scapy’s sendpkts is run, it continuously turns the intf into promisc mode off and on very quickly
This also seems to cause v6 intf to do dhcpv6 always.
While this is happening, issuing a stop on sendpkts hangs the cmd for few mins(resulting in script failure) and returns later.
Disabling v6 on the intf fixed the issue
Options:

1)Per Mallams observations, having a timeout of 240sec(4 mins) did seem to avoid the timeout failure(instead of the current 120s)
2) Disable v6 on the intf before running scapy : seems clumsy.

3) Turn promisc off only when scapy does send() (i.e not send/receive ) here… https://github.com/Juniper/contrail-test-ci/blob/master/tcutils/pkgs/Traffic/traffic/core/generator.py#L176
For ICMP, it would still have promisc ON

Option 3 seems a better option

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/35948
Submitter: Vedamurthy Joshi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/35996
Submitter: Vedamurthy Joshi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/35997
Submitter: Vedamurthy Joshi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/35998
Submitter: Vedamurthy Joshi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/35948
Committed: http://github.com/Juniper/contrail-test-ci/commit/aa17a714b26d84fce184ba20757fc240b0557367
Submitter: Zuul (<email address hidden>)
Branch: master

commit aa17a714b26d84fce184ba20757fc240b0557367
Author: Vedu Joshi <email address hidden>
Date: Tue Sep 26 16:24:04 2017 +0530

Avoid continuous promisc mode flaps

Avoid continuous promisc mode flaps when doing sendpkts
ubuntu-traffic image has also been updated with this change

Change-Id: I326f52cdabe74a748e9b75d58d45da5661c87830
Closes-Bug: 1718142

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/35998
Committed: http://github.com/Juniper/contrail-test-ci/commit/f33ccd8cb9d21882bd47cb6a986aef5ea8bcc219
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit f33ccd8cb9d21882bd47cb6a986aef5ea8bcc219
Author: Vedu Joshi <email address hidden>
Date: Tue Sep 26 16:24:04 2017 +0530

Avoid continuous promisc mode flaps

Avoid continuous promisc mode flaps when doing sendpkts
ubuntu-traffic image has also been updated with this change

Change-Id: I326f52cdabe74a748e9b75d58d45da5661c87830
Closes-Bug: 1718142

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/35996
Committed: http://github.com/Juniper/contrail-test-ci/commit/d484161716504c381307a64ae828b46342da9773
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit d484161716504c381307a64ae828b46342da9773
Author: Vedu Joshi <email address hidden>
Date: Tue Sep 26 16:24:04 2017 +0530

Avoid continuous promisc mode flaps

Avoid continuous promisc mode flaps when doing sendpkts
ubuntu-traffic image has also been updated with this change

Change-Id: I326f52cdabe74a748e9b75d58d45da5661c87830
Closes-Bug: 1718142

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/35997
Committed: http://github.com/Juniper/contrail-test-ci/commit/e40859fb61029590024a13d8bc76c96d5d1977bd
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit e40859fb61029590024a13d8bc76c96d5d1977bd
Author: Vedu Joshi <email address hidden>
Date: Tue Sep 26 16:24:04 2017 +0530

Avoid continuous promisc mode flaps

Avoid continuous promisc mode flaps when doing sendpkts
ubuntu-traffic image has also been updated with this change

Change-Id: I326f52cdabe74a748e9b75d58d45da5661c87830
Closes-Bug: 1718142

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/40095
Submitter: Ankit Jain (<email address hidden>)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.