VM UVE showing empty data during sanity tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Juniper Openstack |
Fix Committed
|
High
|
Vedamurthy Joshi |
Bug Description
2.0 Build 2427 Centos Havana multi-node multi-interface testbed
This has happened during multiple tests in the sanity run :
http://
Logs and cassandra logs are in /cs-shared/
In the testcase test_svc_mirroring, the case because it didnt find the VM uve for vn2_vm2
2014-10-31 22:43:06,877 - WARNING - Failed to get VM vn2_vm2, ID 55b6e5b0-
2014-10-31 22:43:08,879 - INFO - Verifying the vm in opserver^M
2014-10-31 22:43:08,879 - INFO - Verifying in collector 10.204.217.6 ...^M
Key Error^M
2014-10-31 22:43:08,886 - WARNING - Failed to get VM vn2_vm2, ID 55b6e5b0-
2014-10-31 22:43:10,888 - INFO - Verifying the vm in opserver^M
2014-10-31 22:43:10,889 - INFO - Verifying in collector 10.204.217.6 ...^M
Key Error^M
Changed in juniperopenstack: | |
assignee: | Megh Bhatt (meghb) → Vedamurthy Joshi (vedujoshi) |
status: | New → Fix Committed |
The issue was caused due to the Sandesh send queue getting stuck with ready_to_send_ being set to false due to a race condition. This issue was introduced as part of recent checkin https:/ /github. com/Juniper/ contrail- sandesh/ commit/ fc865df0e682e5b 63c852fdf7b4710 035000e518 which removed mutex from SandeshWriter and replaced it by using tbb::atomic to update ready_to_send_. However the mutex is needed since TcpSession::Send and TcpSession: :WriteReady callback can happen in parallel and hence the mutex is needed to serialize the execution of both and updating of the ready_to_send_.
Following is the gdb analysis of the gcore of contrail- vrouter- agent:
[root@nodec60 ~]# gdb /var/tmp/ contrail- vrouter- agent /var/tmp/core.6504 gnu.org/ licenses/ gpl.html> redhat- linux-gnu" . www.gnu. org/software/ gdb/bugs/>... contrail- vrouter- agent.. .done.
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-
For bug reporting instructions, please see:
<http://
Reading symbols from /var/tmp/
warning: core file may not match specified executable file. '*-debug* ' install /usr/lib/ debug/. build-id/ b5/d86fbcf0ccb0 3331e6c7c73897b 96845e0a4eb libxml2. so.2... (no debugging symbols found)...done. libxml2. so.2 libthriftasio. so.0... done. libthriftasio. so.0 libthrift- 0.8.0.so. ..done. libthrift- 0.8.0.so libboost_ filesystem. so.1.48. 0...(no debugging symbols found)...done. libboost_ filesystem. so.1.48. 0 libboost_ program_ options. so.1.48. 0...(no debugging symbols found)...done. libboost_ program_ options. so.1.48. 0 libboost_ regex.so. 1.48.0. ..(no debugging symbols found)...done. libboost_ regex.so. 1.48.0 librt.so. 1...(no debugging symbols found)...done. libboost_ system. so.1.48. 0...(no debugging symbols found)...done. libboost_ system. so.1.48. 0 liblog4cplus- 1.1.so. 7...done. liblog4cplus- 1.1.so. 7 libpthread. so.0... (no debugging symbols found)...done. libpthread. so.0 libtbb_ debug.so. 2...done. libtbb_ debug.so. 2 libstdc+ +.so.6. ..(no debugging symbols found)...done. libstdc+ +.so...
[New Thread 7013]
[New Thread 7014]
[New Thread 7015]
[New Thread 7016]
[New Thread 11208]
[New Thread 6504]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo=
Reading symbols from /usr/lib64/
Loaded symbols for /usr/lib64/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib64/
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /lib64/
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/
Reading symbols from /usr/lib/
Loaded symbols for /usr/lib/
Reading symbols from /usr/lib64/
Loaded symbols for /usr/lib64/