While deleting VN/VMI/LI repeatedly, Control-node suddenly rebooted.

[Daisuke Nakajima]

root@contrail-03:~# date
Tue Jan 12 10:03:48 JST 2016
root@contrail-03:~# contrail-status -d
== Contrail Control ==
supervisor-control: active
contrail-control active pid 78609, uptime 0:05:01
contrail-control-nodemgr active pid 56679, uptime 1 day, 16:40:19
contrail-dns active pid 56681, uptime 1 day, 16:40:19
contrail-named active pid 56682, uptime 1 day, 16:40:19

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api active pid 20144, uptime 1 day, 17:09:23
contrail-analytics-nodemgr active pid 20139, uptime 1 day, 17:09:23
contrail-collector active pid 20142, uptime 1 day, 17:09:23
contrail-query-engine timeout pid 20143, uptime 1 day, 17:09:23
contrail-snmp-collector active pid 20140, uptime 1 day, 17:09:23
contrail-topology active pid 20141, uptime 1 day, 17:09:23

2016-01-12 Tue 09:55:17:500.736 JST contrail-03 [Thread 140658342655744, Pid 56680]: IFMap [SYS_WARN]: IFMapIdentifierNotFound: Cant find identifier default-domain:logical2000:test-vn722:test-vn722 controller/src/ifmap/ifmap_server_table.cc 241
2016-01-12 Tue 09:55:17:502.790 JST contrail-03 [Thread 140658342655744, Pid 56680]: IFMap [SYS_WARN]: IFMapIdentifierNotFound: Cant find identifier target:64512:8001815 controller/src/ifmap/ifmap_server_table.cc 241
2016-01-12 Tue 09:55:17:503.536 JST contrail-03 [Thread 140658342655744, Pid 56680]: IFMap [SYS_WARN]: IFMapIdentifierNotFound: Cant find identifier default-domain:logical2000:test-vn1540:test-vn1540 controller/src/ifmap/ifmap_server_table.cc 241
2016-01-12 Tue 09:55:17:504.024 JST contrail-03 [Thread 140658342655744, Pid 56680]: BGP [SYS_WARN]: XmppPeerInstanceLog: XMPP Peer contrail-03:10.84.50.5 in instance default-domain:logical2000:test-vn413:test-vn413 : Received route without pending subscribe controller/src/bgp/bgp_xmpp_channel.cc 846
2016-01-12 Tue 09:55:17:504.177 JST contrail-03 [Thread 140658342655744, Pid 56680]: BGP [SYS_WARN]: XmppPeerInstanceLog: XMPP Peer contrail-03:10.84.50.5 in instance default-domain:logical2000:test-vn413:test-vn413 : Received route without pending subscribe controller/src/bgp/bgp_xmpp_channel.cc 846
2016-01-12 Tue 09:55:17:504.356 JST contrail-03 [Thread 140658342655744, Pid 56680]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.84.50.2:5269::10.84.50.5:35831 > Write failed due to error: 125 category: system message: Operation canceled controller/src/io/tcp_session.cc 362
2016-01-12 Tue 09:58:49:274.759 JST contrail-03 [Thread 140580678604736, Pid 78609]: SANDESH: Logging: DISABLED -> ENABLED
2016-01-12 Tue 09:58:49:274.909 JST contrail-03 [Thread 140580678604736, Pid 78609]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_NOTICE ] log4level: [ NOTSET ] -> [ WARN ]
2016-01-12 Tue 09:58:49:275.634 JST contrail-03 [Thread 140580678604736, Pid 78609]: Starting Bgp Server at port 179
2016-01-12 Tue 09:58:49:278.061 JST contrail-03 [Thread 140580678604736, Pid 78609]: SANDESH: No Client: 1452560329277993 SandeshModuleClientTrace: data= [ name = contrail-03:Control:contrail-control:0 client_info= [ status = Idle successful_connections = 0 pid = 78609 http_port = 8083 start_time = 1452560329277901 collector_name = primary = 0.0.0.0:0 secondary = 0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] ]
2016-01-12 Tue 09:58:49:278.691 JST contrail-03 [Thread 140580519061248, Pid 78609]: SANDESH: Send FAILED: 1452560329278593 NodeStatusUVE: data= [ name = contrail-03 process_status= [ [ [ module_id = contrail-control instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], [ type = Discovery name = xmpp-server server_addrs= [ [ (*_iter6) = 10.84.50.100:5998, ] ] status = Initializing description = Publish ], ] ] description = Number of connections:2, Expected:5 ], ] ] ]
2016-01-12 Tue 09:58:49:278.766 JST contrail-03 [Thread 140580519061248, Pid 78609]: SANDESH: Send FAILED: 1452560329278658 SandeshModuleClientTrace: data= [ name = contrail-03:Control:contrail-control:0 client_info= [ status = Disconnect successful_connections = 0 pid = 78609 http_port = 8083 start_time = 1452560329277901 collector_name = primary = 0.0.0.0:0 secondary = 0.0.0.0:0 rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] ]

root@contrail-03:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21.1-22 22
contrail-config 2.21.1-22 22
contrail-config-openstack 2.21.1-22 22
contrail-control 2.21.1-22 22
contrail-dns 2.21.1-22 22
contrail-f5 2.21.1-22 22
contrail-fabric-utils 2.21.1-22 22
contrail-install-packages 2.21.1-22~juno 22
contrail-lib 2.21.1-22 22
contrail-nodemgr 2.21.1-22 22
contrail-openstack-analytics 2.21.1-22 22
contrail-openstack-config 2.21.1-22 22
contrail-openstack-control 2.21.1-22 22
contrail-openstack-database 2.21.1-22 22
contrail-openstack-webui 2.21.1-22 22
contrail-setup 2.21.1-22 22
contrail-utils 2.21.1-22 22
contrail-web-controller 2.21.1-22 22
contrail-web-core 2.21.1-22 22
ifmap-python-client 0.1-2 22
ifmap-server 0.3.2-1contrail1 22
neutron-plugin-contrail 2.21.1-22 22
python-contrail 2.21.1-22 22
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail22

[Daisuke Nakajima (dnakajima)]

This issue was seen on R2.21-build30, too.
When control-node restarted, crash file was generated at /var/crash instead of core file.

drwxrwxrwt 2 root root 4096 Mar 31 11:28 ./
drwxr-xr-x 13 root root 4096 Jul 17 2015 ../
-rw-rw---- 1 root root 0 Mar 31 11:19 .lock
-rw-r----- 1 contrail contrail 114538164 Mar 31 11:31 _usr_bin_contrail-control.108.crash

[Nischal Sheth (nsheth)]

Marking Incomplete - please open a new bug if this is seen again.