It¹s kind of tricky to figure out if there was a memory leak after the
reboot. We are trying to plot it for the existing hypervisors to see if
they are running into memory leaks. Do you want us to check any specific
stuff?
Btw, we have seen more hypervisors with kernel 3.13 crashing in last 2
days.
Thanks,
Varun
On 4/28/15, 2:21 AM, "Anand H. Krishnan" <email address hidden> wrote:
Hey Anand,
It¹s kind of tricky to figure out if there was a memory leak after the
reboot. We are trying to plot it for the existing hypervisors to see if
they are running into memory leaks. Do you want us to check any specific
stuff?
Btw, we have seen more hypervisors with kernel 3.13 crashing in last 2
days.
Thanks,
Varun
On 4/28/15, 2:21 AM, "Anand H. Krishnan" <email address hidden> wrote:
>Hi /bugs.launchpad .net/bugs/ 1449162 25T12:28: 05.944331+ 00:00 b0c010ash2018 kernel: [535885.190684] 25T12:28: 05.944348+ 00:00 b0c010ash2018 kernel: [535885.215308] 657>] kmem_cache_ alloc+0x77/ 0x1f0 25T12:28: 05.944349+ 00:00 b0c010ash2018 kernel: [535885.230779] 25T12:28: 05.944350+ 00:00 b0c010ash2018 kernel: [535885.247356] 25T12:28: 05.944351+ 00:00 b0c010ash2018 kernel: [535885.264374] iscsi bonding temp_thermal intel_powerclamp coretemp kvm_intel kvm dm_multipath 25T12:28: 05.944354+ 00:00 b0c010ash2018 kernel: [535885.575624] 25T12:28: 05.944355+ 00:00 b0c010ash2018 kernel: [535885.654294] 25T12:28: 05.944356+ 00:00 b0c010ash2018 kernel: [535885.737038] 25T12:28: 05.944356+ 00:00 b0c010ash2018 kernel: [535885.826571] ffffffff811b065 7>] [<ffffffff811b0 657>] alloc+0x77/ 0x1f0 25T12:28: 05.944357+ 00:00 b0c010ash2018 kernel: [535885.923753] 0ffd90 EFLAGS: 00010282 25T12:28: 05.944358+ 00:00 b0c010ash2018 kernel: [535885.974387] 25T12:28: 05.944359+ 00:00 b0c010ash2018 kernel: [535886.078298] 25T12:28: 05.944360+ 00:00 b0c010ash2018 kernel: [535886.188858] 25T12:28: 05.944363+ 00:00 b0c010ash2018 kernel: [535886.302853] 25T12:28: 05.944365+ 00:00 b0c010ash2018 kernel: [535886.418492] 25T12:28: 05.944387+ 00:00 b0c010ash2018 kernel: [535886.536915] 0(0000) GS:ffff88181fbc 0000(0000) 000000000000000 0 25T12:28: 05.944388+ 00:00 b0c010ash2018 kernel: [535886.656231] 25T12:28: 05.944393+ 00:00 b0c010ash2018 kernel: [535886.716245] 25T12:28: 05.944394+ 00:00 b0c010ash2018 kernel: [535886.833304] 25T12:28: 05.944395+ 00:00 b0c010ash2018 kernel: [535886.889899] 25T12:28: 05.944395+ 00:00 b0c010ash2018 kernel: [535887.004037] 25T12:28: 05.944396+ 00:00 b0c010ash2018 kernel: [535887.117912] 25T12:28: 05.944397+ 00:00 b0c010ash2018 kernel: [535887.232613] 25T12:28: 05.944397+ 00:00 b0c010ash2018 kernel: [535887.288242] be94>] ? alloc_pid+ 0x24/0x2e0 25T12:28: 05.944400+ 00:00 b0c010ash2018 kernel: [535887.344143] be94>] alloc_pid+ 0x24/0x2e0 25T12:28: 05.944401+ 00:00 b0c010ash2018 kernel: [535887.398737] 998a>] copy_process. part.27+ 0x8ca/0xf50 25T12:28: 05.944401+ 00:00 b0c010ash2018 kernel: [535887.452662] 331b>] ? audit_filter_ rules.isra. 7+0x55b/ 0xad0 25T12:28: 05.944402+ 00:00 b0c010ash2018 kernel: [535887.506223] a090>] copy_process+ 0x80/0x90 25T12:28: 05.944402+ 00:00 b0c010ash2018 kernel: [535887.558678] a1d2>] do_fork+0x62/0x280 25T12:28: 05.944403+ 00:00 b0c010ash2018 kernel: [535887.610125] 3924>] ? audit_filter_ syscall+ 0x94/0xe0 25T12:28: 05.944412+ 00:00 b0c010ash2018 kernel: [535887.661481] a476>] SyS_clone+0x16/0x20 25T12:28: 05.944413+ 00:00 b0c010ash2018 kernel: [535887.711599] ead9>] stub_clone+ 0x69/0x90 25T12:28: 05.944414+ 00:00 b0c010ash2018 kernel: [535887.760582] e77d>] ? system_ call_fastpath+ 0x1a/0x1f 25T12:28: 05.944414+ 00:00 b0c010ash2018 kernel: [535887.808776] 25T12:28: 05.944414+ 00:00 b0c010ash2018 kernel: [535887.808776] 25T12:28: 05.944415+ 00:00 b0c010ash2018 kernel: [535887.954161] 657>] kmem_cache_ alloc+0x77/ 0x1f0 25T12:28: 05.944416+ 00:00 b0c010ash2018 kernel: [535888.001499] 25T12:28: 05.944416+ 00:00 b0c010ash2018 kernel: [535888.047379] 25T12:28: 05.944419+ 00:00 b0c010ash2018 kernel: [535888.157161] 25T12:28: 06.040139+ 00:00 b0c010ash2018 kernel: [535888.207652] /bugs.launchpad .net/opencontra il/+bug/ 1449162/ +subscriptions
>
>Can you please check whether there is a steady memory leak before this
>crash happened? It is difficult to see what has gone wrong without a
>dump.
>
>Thanks,
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https:/
>
>Title:
> Seeing kernel backtrace and vrouter process stuck in INIT state with
> contrail 2.01 and kernel 3.13
>
>Status in OpenContrail:
> New
>
>Bug description:
> We are seeing these frequent kernel backtraces with contrail running
> on 2.01-41 and kernel version 3.13. The only way to recover after this
> is a hypervisor reboot. Need to dig into the root-cause of this as
> this is seriously affecting our Uptime.
>
> Following is the backtrace:
> 2015-04-
>BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
> 2015-04-
>IP: [<ffffffff811b0
> 2015-04-
>PGD 12df28d067 PUD 12d69d0067 PMD 0
> 2015-04-
>Oops: 0000 [#1] SMP
> 2015-04-
>Modules linked in: veth vhost_net macvtap macvlan vhost xt_conntrack
>iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
>nf_nat nf_conntrack 8021q mrp garp bridge stp llc ip6table_filter
>ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables nbd
>vrouter(OX) ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr
>iscsi_tcp libiscsi_tcp libiscsi scsi_transport_
>x86_pkg_
>crct10dif_pclmul crc32_pclmul dcdbas ghash_clmulni_intel scsi_dh xfs
>gpio_ich joydev ipmi_devintf aesni_intel msr ablk_helper lp mac_hid
>cryptd sb_edac edac_core lrw gf128mul parport mei_me wmi ioatdma lpc_ich
>mei ipmi_si glue_helper libcrc32c shpchp acpi_power_meter aes_x86_64
>hid_generic usbhid hid ixgbe igb megaraid_sas dca i2c_algo_bit ptp
>pps_core mdio
> 2015-04-
>CPU: 28 PID: 45796 Comm: openstack-statu Tainted: G OX
>3.13.0-49-generic #81~precise1-Ubuntu
> 2015-04-
>Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.4.3 07/09/2014
> 2015-04-
>task: ffff8812f2ce0000 ti: ffff8812c70fe000 task.ti: ffff8812c70fe000
> 2015-04-
>RIP: 0010:[<
>kmem_cache_
> 2015-04-
>RSP: 0018:ffff8812c7
> 2015-04-
>RAX: 0000000000000000 RBX: 0000000001200011 RCX: 000000000003e8bc
> 2015-04-
>RDX: 000000000003e8bb RSI: 00000000000000d0 RDI: 00000000000162a0
> 2015-04-
>RBP: ffff8812c70ffde0 R08: ffff88181fbd62a0 R09: ffffffff8108be94
> 2015-04-
>R10: ffff88187fffbf00 R11: 00007fff8841b000 R12: 0000000000000001
> 2015-04-
>R13: ffff88181f403900 R14: ffff88181f403900 R15: 00000000000000d0
> 2015-04-
>FS: 00007fbc1850670
>knlGS:
> 2015-04-
>CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2015-04-
>CR2: 0000000000000001 CR3: 00000012d6b18000 CR4: 00000000001427e0
> 2015-04-
>Stack:
> 2015-04-
>ffffffff8108be94 ffff8817ef6a0aa8 ffff8817f7189880 0000000000000000
> 2015-04-
>ffff8812c70ffde0 0000000001200011 0000000000000000 ffffffff81c452a0
> 2015-04-
>0000000000000000 0000000000000000 ffff8812c70ffe10 ffffffff8108be94
> 2015-04-
>Call Trace:
> 2015-04-
>[<ffffffff8108
> 2015-04-
>[<ffffffff8108
> 2015-04-
>[<ffffffff8106
> 2015-04-
>[<ffffffff8110
> 2015-04-
>[<ffffffff8106
> 2015-04-
>[<ffffffff8106
> 2015-04-
>[<ffffffff8110
> 2015-04-
>[<ffffffff8106
> 2015-04-
>[<ffffffff8176
> 2015-04-
>[<ffffffff8176
> 2015-04-
>Code: 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 5e 01 00 00
>48 85 c0 0f 84 55 01 00 00 49 63 45 20 49 8b 7d 00 48 8d 4a 01 <49> 8b 1c
>04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 b7 49 63
> 2015-04-
>Code: 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 5e 01 00 00
>48 85 c0 0f 84 55 01 00 00 49 63 45 20 49 8b 7d 00 48 8d 4a 01 <49> 8b 1c
>04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 b7 49 63
> 2015-04-
>RIP [<ffffffff811b0
> 2015-04-
>RSP <ffff8812c70ffd90>
> 2015-04-
>CR2: 0000000000000001
> 2015-04-
>---[ end trace 8e74a782f5824da3 ]---
> 2015-04-
>[sched_delayed] sched: RT throttling activated
>
>To manage notifications about this bug go to:
>https:/