During source NAT development, I got some vrouter kernel trace. To confirm, I ran a simple test : $ cd /opt/stack/contrail/controller/src/config/utils $ while true; do echo -n "creating... "; python contrail_veth_port.py --subnet 10.0.0.0/24 test_vm private; echo ok; echo -n "deleting... "; python contrail_veth_port.py --delete --subnet 10.0.0.0/24 test_vm private; echo ok; done And 12 hours later I got that kernel trace: 44321 Jul 26 23:17:32 localhost kernel: [118557.448900] RTNL: assertion failed at /build/buildd/linux-3.2.0/net/core/dev.c (3218) 44322 Jul 26 23:17:32 localhost kernel: [118557.450256] Pid: 13339, comm: contrail-vroute Tainted: G O 3.2.0-64-virtual #97-Ubuntu 44323 Jul 26 23:17:32 localhost kernel: [118557.450259] Call Trace: 44324 Jul 26 23:17:32 localhost kernel: [118557.450272] [] netdev_rx_handler_unregister+0x59/0x60 44325 Jul 26 23:17:32 localhost kernel: [118557.450313] [] linux_if_del_tap+0x90/0xa0 [vrouter] 44326 Jul 26 23:17:32 localhost kernel: [118557.450320] [] eth_drv_del+0x1b/0x60 [vrouter] 44327 Jul 26 23:17:32 localhost kernel: [118557.450331] [] vif_delete+0x2e/0x40 [vrouter] 44328 Jul 26 23:17:32 localhost kernel: [118557.450340] [] vr_interface_req_process+0x13c/0x250 [vrouter] 44329 Jul 26 23:17:32 localhost kernel: [118557.450345] [] sandesh_decode_one+0x10a/0x1d0 [vrouter] 44330 Jul 26 23:17:32 localhost kernel: [118557.450352] [] ? thrift_binary_protocol_write_field_begin+0x80/0x80 [vrouter] 44331 Jul 26 23:17:32 localhost kernel: [118557.450358] [] ? thrift_binary_protocol_write_message_end+0x10/0x10 [vrouter] 44332 Jul 26 23:17:32 localhost kernel: [118557.450364] [] ? thrift_binary_protocol_write_sandesh_begin+0x20/0x20 [vrouter] 44333 Jul 26 23:17:32 localhost kernel: [118557.450369] [] ? thrift_protocol_skip+0x240/0x240 [vrouter] 44334 Jul 26 23:17:32 localhost kernel: [118557.450374] [] ? thrift_binary_protocol_write_sandesh_end+0x10/0x10 [vrouter] 44335 Jul 26 23:17:32 localhost kernel: [118557.450379] [] ? thrift_binary_protocol_write_struct_begin+0x10/0x10 [vrouter] 44336 Jul 26 23:17:32 localhost kernel: [118557.450384] [] ? thrift_binary_protocol_write_set_begin+0x10/0x10 [vrouter] 44337 Jul 26 23:17:32 localhost kernel: [118557.450389] [] ? thrift_binary_protocol_write_struct_end+0x10/0x10 [vrouter] 44338 Jul 26 23:17:32 localhost kernel: [118557.450395] [] ? thrift_binary_protocol_write_map_begin+0xa0/0xa0 [vrouter] 44339 Jul 26 23:17:32 localhost kernel: [118557.450400] [] ? thrift_binary_protocol_write_list_begin+0x80/0x80 [vrouter] 44340 Jul 26 23:17:32 localhost kernel: [118557.450405] [] ? thrift_binary_protocol_write_field_end+0x10/0x10 [vrouter] 44341 Jul 26 23:17:32 localhost kernel: [118557.450410] [] ? thrift_binary_protocol_write_bool+0x20/0x20 [vrouter] 44342 Jul 26 23:17:32 localhost kernel: [118557.450415] [] ? thrift_binary_protocol_write_map_end+0x10/0x10 [vrouter] 44343 Jul 26 23:17:32 localhost kernel: [118557.450420] [] ? thrift_binary_protocol_write_field_stop+0x20/0x20 [vrouter] 44344 Jul 26 23:17:32 localhost kernel: [118557.450425] [] ? thrift_binary_protocol_write_list_end+0x10/0x10 [vrouter] 44345 Jul 26 23:17:32 localhost kernel: [118557.450430] [] ? thrift_binary_protocol_write_string+0x50/0x50 [vrouter] 44346 Jul 26 23:17:32 localhost kernel: [118557.450435] [] ? thrift_binary_protocol_write_i16+0x40/0x40 [vrouter] 44347 Jul 26 23:17:32 localhost kernel: [118557.450440] [] ? thrift_binary_protocol_write_i32+0x30/0x30 [vrouter] 44348 Jul 26 23:17:32 localhost kernel: [118557.450445] [] ? thrift_binary_protocol_write_i64+0x60/0x60 [vrouter] 44349 Jul 26 23:17:32 localhost kernel: [118557.450450] [] ? thrift_binary_protocol_write_u16+0x40/0x40 [vrouter] 44350 Jul 26 23:17:32 localhost kernel: [118557.450455] [] ? thrift_binary_protocol_write_ipv4+0x10/0x10 [vrouter] 44351 Jul 26 23:17:32 localhost kernel: [118557.450460] [] ? thrift_binary_protocol_write_u64+0x60/0x60 [vrouter] 44352 Jul 26 23:17:32 localhost kernel: [118557.450465] [] ? thrift_binary_protocol_write_binary+0x90/0x90 [vrouter] 44353 Jul 26 23:17:32 localhost kernel: [118557.450469] [] ? thrift_binary_protocol_write_u32+0x30/0x30 [vrouter] 44354 Jul 26 23:17:32 localhost kernel: [118557.450474] [] ? thrift_binary_protocol_write_set_end+0x10/0x10 [vrouter] 44355 Jul 26 23:17:32 localhost kernel: [118557.450479] [] ? thrift_binary_protocol_write_byte+0x30/0x30 [vrouter] 44356 Jul 26 23:17:32 localhost kernel: [118557.450484] [] ? thrift_binary_protocol_read_message_begin+0x100/0x100 [vrouter] 44357 Jul 26 23:17:32 localhost kernel: [118557.450489] [] ? thrift_binary_protocol_write_byte+0x30/0x30 [vrouter] 44358 Jul 26 23:17:32 localhost kernel: [118557.450494] [] ? thrift_binary_protocol_read_field_begin+0xa0/0xa0 [vrouter] 44359 Jul 26 23:17:32 localhost kernel: [118557.450500] [] ? thrift_binary_protocol_read_message_end+0x10/0x10 [vrouter] 44360 Jul 26 23:17:32 localhost kernel: [118557.450504] [] ? thrift_binary_protocol_read_sandesh_begin+0x20/0x20 [vrouter] 44361 Jul 26 23:17:32 localhost kernel: [118557.450510] [] ? thrift_binary_protocol_write_double+0x10/0x10 [vrouter] 44362 Jul 26 23:17:32 localhost kernel: [118557.450515] [] ? thrift_binary_protocol_read_sandesh_end+0x10/0x10 [vrouter] 44363 Jul 26 23:17:32 localhost kernel: [118557.450520] [] ? thrift_binary_protocol_read_struct_begin+0x20/0x20 [vrouter] 44364 Jul 26 23:17:32 localhost kernel: [118557.450525] [] ? thrift_binary_protocol_read_map_begin+0xd0/0xd0 [vrouter] 44365 Jul 26 23:17:32 localhost kernel: [118557.450530] [] ? thrift_binary_protocol_read_struct_end+0x10/0x10 [vrouter] 44366 Jul 26 23:17:32 localhost kernel: [118557.450535] [] ? thrift_binary_protocol_read_list_begin+0xb0/0xb0 [vrouter] 44367 Jul 26 23:17:32 localhost kernel: [118557.450540] [] ? thrift_binary_protocol_read_field_end+0x10/0x10 [vrouter] 44368 Jul 26 23:17:32 localhost kernel: [118557.450545] [] ? thrift_binary_protocol_read_set_begin+0x10/0x10 [vrouter] 44369 Jul 26 23:17:32 localhost kernel: [118557.450550] [] ? thrift_binary_protocol_read_map_end+0x10/0x10 [vrouter] 44370 Jul 26 23:17:32 localhost kernel: [118557.450554] [] ? thrift_binary_protocol_read_string+0xd0/0xd0 [vrouter] 44371 Jul 26 23:17:32 localhost kernel: [118557.450559] [] ? thrift_binary_protocol_read_list_end+0x10/0x10 [vrouter] 44372 Jul 26 23:17:32 localhost kernel: [118557.450564] [] ? thrift_binary_protocol_read_byte+0x50/0x50 [vrouter] 44373 Jul 26 23:17:32 localhost kernel: [118557.450569] [] ? thrift_binary_protocol_read_i64+0x80/0x80 [vrouter] 44374 Jul 26 23:17:32 localhost kernel: [118557.450574] [] ? thrift_binary_protocol_read_i32+0x50/0x50 [vrouter] 44375 Jul 26 23:17:32 localhost kernel: [118557.450579] [] ? thrift_binary_protocol_read_u16+0x50/0x50 [vrouter] 44376 Jul 26 23:17:32 localhost kernel: [118557.450584] [] ? thrift_binary_protocol_read_u64+0x80/0x80 [vrouter] 44377 Jul 26 23:17:32 localhost kernel: [118557.450589] [] ? thrift_binary_protocol_read_ipv4+0x10/0x10 [vrouter] 44378 Jul 26 23:17:32 localhost kernel: [118557.450593] [] ? thrift_binary_protocol_read_bool+0x50/0x50 [vrouter] 44379 Jul 26 23:17:32 localhost kernel: [118557.450598] [] ? thrift_binary_protocol_read_double+0x10/0x10 [vrouter] 44380 Jul 26 23:17:32 localhost kernel: [118557.450603] [] ? thrift_binary_protocol_read_u32+0x50/0x50 [vrouter] 44381 Jul 26 23:17:32 localhost kernel: [118557.450608] [] ? thrift_binary_protocol_read_set_end+0x10/0x10 [vrouter] 44382 Jul 26 23:17:32 localhost kernel: [118557.450613] [] ? thrift_binary_protocol_read_binary+0xe0/0xe0 [vrouter] 44383 Jul 26 23:17:32 localhost kernel: [118557.450618] [] ? thrift_binary_protocol_read_i16+0x50/0x50 [vrouter] 44384 Jul 26 23:17:32 localhost kernel: [118557.450622] [] ? thrift_binary_protocol_read_binary+0xe0/0xe0 [vrouter] 44385 Jul 26 23:17:32 localhost kernel: [118557.450628] [] ? thrift_transport_flush+0x20/0x20 [vrouter] 44386 Jul 26 23:17:32 localhost kernel: [118557.450634] [] ? thrift_memory_buffer_is_open+0x10/0x10 [vrouter] 44387 Jul 26 23:17:32 localhost kernel: [118557.450639] [] ? thrift_memory_buffer_open+0x10/0x10 [vrouter] 44388 Jul 26 23:17:32 localhost kernel: [118557.450644] [] ? thrift_memory_buffer_flush+0x10/0x10 [vrouter] 44389 Jul 26 23:17:32 localhost kernel: [118557.450649] [] ? thrift_memory_buffer_close+0x10/0x10 [vrouter] 44390 Jul 26 23:17:32 localhost kernel: [118557.450654] [] ? thrift_memory_buffer_read+0x90/0x90 [vrouter] 44391 Jul 26 23:17:32 localhost kernel: [118557.450659] [] ? thrift_memory_buffer_read_end+0x10/0x10 [vrouter] 44392 Jul 26 23:17:32 localhost kernel: [118557.450664] [] ? thrift_memory_buffer_write_end+0x10/0x10 [vrouter] 44393 Jul 26 23:17:32 localhost kernel: [118557.450669] [] ? sandesh_hdr_free+0x10/0x10 [vrouter] 44394 Jul 26 23:17:32 localhost kernel: [118557.450674] [] sandesh_decode+0x46/0x80 [vrouter] 44395 Jul 26 23:17:32 localhost kernel: [118557.450681] [] sandesh_proto_decode+0x24/0x30 [vrouter] 44396 Jul 26 23:17:32 localhost kernel: [118557.450688] [] vr_message_request+0x40/0x50 [vrouter] 44397 Jul 26 23:17:32 localhost kernel: [118557.450694] [] netlink_trans_request+0x5c/0x210 [vrouter] 44398 Jul 26 23:17:32 localhost kernel: [118557.450699] [] ? genl_family_find_byid+0x2f/0x60 44399 Jul 26 23:17:32 localhost kernel: [118557.450702] [] genl_rcv_msg+0x1d5/0x250 44400 Jul 26 23:17:32 localhost kernel: [118557.450705] [] ? netlink_unicast+0x2be/0x300 44401 Jul 26 23:17:32 localhost kernel: [118557.450708] [] ? genl_rcv+0x40/0x40 44402 Jul 26 23:17:32 localhost kernel: [118557.450710] [] netlink_rcv_skb+0xa9/0xd0 44403 Jul 26 23:17:32 localhost kernel: [118557.450713] [] genl_rcv+0x25/0x40 44404 Jul 26 23:17:32 localhost kernel: [118557.450716] [] netlink_unicast+0x2b0/0x300 44405 Jul 26 23:17:32 localhost kernel: [118557.450719] [] ? __alloc_skb+0x8c/0x240 44406 Jul 26 23:17:32 localhost kernel: [118557.450722] [] netlink_sendmsg+0x2de/0x390 44407 Jul 26 23:17:32 localhost kernel: [118557.450726] [] sock_sendmsg+0x10e/0x130 44408 Jul 26 23:17:32 localhost kernel: [118557.450730] [] ? set_next_entity+0xad/0xd0 44409 Jul 26 23:17:32 localhost kernel: [118557.450733] [] ? finish_task_switch+0x4a/0xf0 44410 Jul 26 23:17:32 localhost kernel: [118557.450737] [] ? move_addr_to_kernel+0x64/0x70 44411 Jul 26 23:17:32 localhost kernel: [118557.450740] [] ? verify_iovec+0x52/0xd0 44412 Jul 26 23:17:32 localhost kernel: [118557.450743] [] ___sys_sendmsg+0x3bf/0x3e0 44413 Jul 26 23:17:32 localhost kernel: [118557.450752] [] ? try_to_wake_up+0x18e/0x200 44414 Jul 26 23:17:32 localhost kernel: [118557.450755] [] ? wake_up_state+0x10/0x20 44415 Jul 26 23:17:32 localhost kernel: [118557.450764] [] ? wake_futex+0x76/0xa0 44416 Jul 26 23:17:32 localhost kernel: [118557.450767] [] ? futex_wake+0x113/0x130 44417 Jul 26 23:17:32 localhost kernel: [118557.450771] [] __sys_sendmsg+0x49/0x90 44418 Jul 26 23:17:32 localhost kernel: [118557.450774] [] sys_sendmsg+0x19/0x20 44419 Jul 26 23:17:32 localhost kernel: [118557.450780] [] system_call_fastpath+0x16/0x1b I got that trace 12 hours later again, but the test continue to work 2 days more without any errors. I remark another problem. 3 days after I started the test, the loop test take 30 seconds more to run (ie. 5-6 seconds at the beginning and 35 seconds 3 days after).