Vcenter-as-compute: contrail-collector core seen #54 build

[Sarath]

Topology : vcenter-compute 3 node HA
Version : 5.0 #54-CB

Zhiqiang triaged the setup and knows the problem

root@5a10s31:~# docker exec -it analytics bash
contrail-status
contrail-status
root@5a10s31(analytics):/# contrail-status
exit
== Contrail Analytics ==
contrail-collector: active
contrail-analytics-api: active
contrail-query-engine: active
contrail-alarm-gen: active
contrail-snmp-collector: active
contrail-topology: active
contrail-analytics-nodemgr: active
========Run time service failures=============
/var/crashes/core.contrail-collec.3340.5a10s31.1512691169
root@5a10s31(analytics):/# exit
exit
root@5a10s31:~# ^H
-bash: :s^H: substitution failed
root@5a10s31:~# docker exec -it analyticsdb bash
contrail-status
contrail-status
root@5a10s31(analyticsdb):/# contrail-status
exit
== Contrail Database ==
contrail-database: active

kafka: active
contrail-database-nodemgr: active
========Run time service failures=============
/var/crashes/core.contrail-collec.3340.5a10s31.1512691169
root@5a10s31(analyticsdb):/# exit
exit
root@5a10s31:~# docker exec -it controller bash
contrail-status
contrail-status
root@5a10s31(controller):/# contrail-status
exit
== Contrail Control ==
contrail-control: active
contrail-named: active
contrail-dns: active
contrail-control-nodemgr: active
== Contrail Config ==
contrail-api: active
contrail-schema: backup
contrail-svc-monitor: backup
contrail-device-manager: backup
contrail-config-nodemgr: active
== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui: active
contrail-webui-middleware: active
== Contrail Support Services ==
zookeeper: active
rabbitmq-server: inactive (disabled on boot)
========Run time service failures=============
/var/crashes/core.contrail-collec.3340.5a10s31.1512691169
root@5a10s31(controller):/# exit
exit
root@5a10s31:~# ssh root@10.87.36.19
root@10.87.36.19's password:
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-62-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Fri Dec 8 12:48:10 PST 2017

  System load: 0.52 Processes: 120
  Usage of /: 4.4% of 54.72GB Users logged in: 1
  Memory usage: 6% IP address for vhost0: 10.87.36.19
  Swap usage: 0%

  Graph this data and manage this system at:
    https://landscape.canonical.com/

128 packages can be updated.
56 updates are security updates.

Last login: Fri Dec 8 12:39:27 2017 from 10.87.36.10
root@contrailvm-5a10s27:~# contrail-status
== Contrail vRouter ==
contrail-vrouter-agent: active
contrail-vrouter-nodemgr: initializing (NTP state unsynchronized.)
root@contrailvm-5a10s27:~#
root@contrailvm-5a10s27:~#

root@5a10s31:/var/crashes# gdb vizd core.contrail-collec.3340.5a10s31.1512691169
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from vizd...done.

warning: core file may not match specified executable file.
[New LWP 3359]
[New LWP 3364]
[New LWP 3349]
[New LWP 3845]
[New LWP 3366]
[New LWP 3354]
[New LWP 3347]
[New LWP 3348]
[New LWP 3350]
[New LWP 3351]
[New LWP 3352]
[New LWP 3353]
[New LWP 3355]
[New LWP 3357]
[New LWP 3362]
[New LWP 3846]
[New LWP 3363]
[New LWP 3356]
[New LWP 3365]
[New LWP 3847]
[New LWP 3340]
[New LWP 3358]

warning: Could not load shared library symbols for 21 libraries, e.g. /usr/lib/x86_64-linux-gnu/libcassandra.so.2.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f63b4f4d428 in sigandset (dest=0xd0c, left=0xd1f, right=0x6) at sigandset.c:33
33 sigandset.c: No such file or directory.
[Current thread is 1 (Thread 0x7f63a5648700 (LWP 3359))]
(gdb) bt
#0 0x00007f63b4f4d428 in sigandset (dest=0xd0c, left=0xd1f, right=0x6) at sigandset.c:33
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb)

[Sarath (nsarath)]

nsarath@ubuntu-build02:/auto/cores/1737308$ ls -l
total 261172
-rwxrwxrwx 1 nsarath test 8069120 Dec 9 01:14 analytics-logs.tar
-rwxrwxrwx 1 nsarath test 231849984 Dec 9 01:11 core.contrail-collec.3340.5a10s31.1512691169
-rwxrwxrwx 1 nsarath test 26460160 Dec 9 01:12 vrouter-log.tar
nsarath@ubuntu-build02:/auto/cores/1737308$

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38283
Submitter: mkheni (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38284
Submitter: mkheni (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38283
Committed: http://github.com/Juniper/contrail-controller/commit/491b37bcdb015f1e7008e94a1c09a24006f81823
Submitter: Zuul (<email address hidden>)
Branch: master

commit 491b37bcdb015f1e7008e94a1c09a24006f81823
Author: mkheni <email address hidden>
Date: Tue Dec 12 18:26:23 2017 -0800

cql_if asserts if it cannot find table metadata.

This could happen if the schema created on one node is not
yet propogated to the node serving the request. Removed the
assert and instead in that case just fail the write.

Change-Id: If24d4c28f074cfc43d50e97aea72629b3ba01ee3
Closes-bug: #1737861

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38284
Committed: http://github.com/Juniper/contrail-controller/commit/5958821885b38bb4022bb04097c07ef181e7a94b
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 5958821885b38bb4022bb04097c07ef181e7a94b
Author: mkheni <email address hidden>
Date: Tue Dec 12 18:26:23 2017 -0800

cql_if asserts if it cannot find table metadata.

This could happen if the schema created on one node is not
yet propogated to the node serving the request. Removed the
assert and instead in that case just fail the write.

Change-Id: If24d4c28f074cfc43d50e97aea72629b3ba01ee3
Closes-bug: #1737861

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38772
Submitter: mkheni (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/38772
Committed: http://github.com/Juniper/contrail-controller/commit/ae17f2ea783b183ce3db8c78ac0ceacc39b95158
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit ae17f2ea783b183ce3db8c78ac0ceacc39b95158
Author: mkheni <email address hidden>
Date: Tue Dec 12 18:26:23 2017 -0800

cql_if asserts if it cannot find table metadata.

This could happen if the schema created on one node is not
yet propogated to the node serving the request. Removed the
assert and instead in that case just fail the write.

Change-Id: If24d4c28f074cfc43d50e97aea72629b3ba01ee3
Closes-bug: #1737861
(cherry picked from commit 491b37bcdb015f1e7008e94a1c09a24006f81823)