[14.04 3058] kernel crash in agent while running port mirroring testcase

[Suresh]

Kernel crash seen in compute node while running test_intf_mirror_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3 testcase in test_intf_mirror script.

  Created 2 VM in nodea35 and mirroring the packets in nodea34. kernel crash is seen in nodea35 when pinging between VM's.

  Copied the dump and dmesg logs of the crash to stack@10.204.216.49 (stack/stack@123) at /home/stack/suresh/kernel_crash/201704211406

Logs:
====
> /contrail-test/common/servicechain/mirror/verify.py(307)verify_port_mirroring()
-> session = ssh(host['host_ip'], host['username'], host['password'])
(Pdb) l
302 svm = mirr_vm.vm_obj
303 if svm.status == 'ACTIVE':
304 svm_name = svm.name
305 host = self.get_svm_compute(svm_name)
306 tapintf = self.get_svm_tapintf(svm_name)
307 -> session = ssh(host['host_ip'], host['username'], host['password'])
308 pcap = self.start_tcpdump(session, tapintf)
309 assert src_vm.ping_with_certainty(dst_vm.vm_ip, count=5, size='1400')
310 self.logger.info('Ping from %s to %s executed with c=5, expected mirrored packets 5 Ingress,5 Egress count = 10'
311 % (src_vm.vm_ip, dst_vm.vm_ip))
312 exp_count = 10
(Pdb) n
> /contrail-test/common/servicechain/mirror/verify.py(308)verify_port_mirroring()
-> pcap = self.start_tcpdump(session, tapintf)
(Pdb) n
2017-04-21 13:55:24,959 - INFO - Staring tcpdump to capture the mirrored packets.
> /contrail-test/common/servicechain/mirror/verify.py(309)verify_port_mirroring()
-> assert src_vm.ping_with_certainty(dst_vm.vm_ip, count=5, size='1400')
(Pdb) n
2017-04-21 13:57:33,050 - WARNING - Unable to run command ping -s 1400 -c 5 -W 1 204.200.183.3: Command failed to finish in 120 seconds
2017-04-21 13:57:38,053 - WARNING - Got exception in ping_to_ip:argument of type 'NoneType' is not iterable
2017-04-21 13:57:39,060 - WARNING - Unable to run command ping -s 1400 -c 5 -W 1 204.200.183.3: Low level socket error connecting to host 10.204.216.31 on port 22: Unable to connect to port 22 on 10.204.216.31 (tried 1 time)
2017-04-21 13:57:44,065 - WARNING - Got exception in ping_to_ip:argument of type 'NoneType' is not iterable
2017-04-21 13:57:45,071 - WARNING - Unable to run command ping -s 1400 -c 5 -W 1 204.200.183.3: Low level socket error connecting to host 10.204.216.31 on port 22: Unable to connect to port 22 on 10.204.216.31 (tried 1 time)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30790
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30790
Committed: http://github.com/Juniper/contrail-vrouter/commit/e18e6557a2e1053eabab946c292d452cdb8df325
Submitter: Zuul (<email address hidden>)
Branch: master

commit e18e6557a2e1053eabab946c292d452cdb8df325
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31041
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31041
Committed: http://github.com/Juniper/contrail-vrouter/commit/3fbe0e1ff87a2c4751cec45a442e03ee16f2eb55
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 3fbe0e1ff87a2c4751cec45a442e03ee16f2eb55
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31152
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31153
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31154
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/31155
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31154
Committed: http://github.com/Juniper/contrail-vrouter/commit/d7ea2cce0b162e064f5b0201cc692fb260939ae0
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit d7ea2cce0b162e064f5b0201cc692fb260939ae0
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31152
Committed: http://github.com/Juniper/contrail-vrouter/commit/808c4646efc7dcae45630542c0e123ec02139600
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit 808c4646efc7dcae45630542c0e123ec02139600
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31155
Committed: http://github.com/Juniper/contrail-vrouter/commit/a6eb88fcf43a827da615768e34eef1697192183b
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit a6eb88fcf43a827da615768e34eef1697192183b
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/31153
Committed: http://github.com/Juniper/contrail-vrouter/commit/e431f08a615b3956c29f962efe820c18a42aac60
Submitter: Zuul (<email address hidden>)
Branch: R2.22.x

commit e431f08a615b3956c29f962efe820c18a42aac60
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32593
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32593
Committed: http://github.com/Juniper/contrail-vrouter/commit/4fb89e127de1a70aba07b0835ce86bf876bbf66b
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 4fb89e127de1a70aba07b0835ce86bf876bbf66b
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181