[R4.0 build 9] collector core in 0x00002aca55269c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

[Suresh]

R4.0 Build 9 mitaka, collector core seen during sanity run

host1 = 'root@10.204.216.31'
host2 = 'root@10.204.216.30'
host3 = 'root@10.204.217.93'
host4 = 'root@10.204.217.94'
host5 = 'root@10.204.217.95'
host6 = 'root@10.204.217.96'

contrail-compute: [host1, host2],
openstack: [host3],
contrail-controller: [host4],
contrail-analytics: [host5, host6],
contrail-analyticsdb: [host4, host5, host6],
build: [host_build],

root@nodec55(analytics):/# contrail-status
== Contrail Analytics ==
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

========Run time service failures=============
/var/crashes/core.contrail-collec.2882.nodec55.1495423152
root@nodec55(analytics):/#

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00002aca55269c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00002aca55269c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00002aca5526d028 in __GI_abort () at abort.c:89
#2 0x00002aca55262bf6 in __assert_fail_base (fmt=0x2aca553b33b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x8ad119 "dmutex.Release()",
    file=file@entry=0x8ad1c8 "controller/src/analytics/db_handler.cc", line=line@entry=516, function=function@entry=0x8b2f60 "bool DbHandler::InitializeInternalLocked()") at assert.c:92
#3 0x00002aca55262ca2 in __GI___assert_fail (assertion=0x8ad119 "dmutex.Release()", file=0x8ad1c8 "controller/src/analytics/db_handler.cc", line=516,
    function=0x8b2f60 "bool DbHandler::InitializeInternalLocked()") at assert.c:101
#4 0x0000000000681f2f in ?? ()
#5 0x000000000068223c in ?? ()
#6 0x0000000000432380 in ?? ()
#7 0x00002aca55254f45 in __libc_start_main (main=0x430c00, argc=1, argv=0x7ffebb95f128, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffebb95f118)
    at libc-start.c:287
#8 0x000000000044123f in ?? ()

[Suresh (suresha)]

core copied to /cs-shared/bugs/1692453

[Raj Reddy (rajreddy)]

likely bug in

int ZookeeperClientImpl::DeleteNodeSync(const char *path, int *err) {
    int rc;
 retry:
    do {
        rc = zki_->ZooDelete(zk_handle_, path, -1);
    } while (IsZooErrorUnrecoverable(rc));
    if (IsZooErrorUnrecoverable(rc)) {
        // Reconnect
        Reconnect();
        goto retry;
    }
    if (rc != ZOK) {
        *err = errno;
    }
    return rc;
}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32073
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32074
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32073
Committed: http://github.com/Juniper/contrail-controller/commit/c130e974941bebce3146c38e16891f1cdf9efe91
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit c130e974941bebce3146c38e16891f1cdf9efe91
Author: Megh Bhatt <email address hidden>
Date: Wed May 24 00:34:03 2017 -0700

Fix ZooDelete in zookeeper client

Fix ZooDelete in zookeeper client to retry recoverable errors

Closes-Bug #1692453

Change-Id: If9a9d10d8675acd181aeb2f8436c1aa944cbb13f

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32074
Committed: http://github.com/Juniper/contrail-controller/commit/97b9d19cbe5abb90cb5c143147de49fd36622f37
Submitter: Zuul (<email address hidden>)
Branch: master

commit 97b9d19cbe5abb90cb5c143147de49fd36622f37
Author: Megh Bhatt <email address hidden>
Date: Wed May 24 00:34:03 2017 -0700

Fix ZooDelete in zookeeper client

Fix ZooDelete in zookeeper client to retry recoverable errors

Closes-Bug #1692453

Change-Id: If9a9d10d8675acd181aeb2f8436c1aa944cbb13f

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/33240
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/33241
Submitter: Sundaresan Rajangam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/33241
Committed: http://github.com/Juniper/contrail-controller/commit/079d4ea44cad5948bda1cf23505f06380910efa7
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 079d4ea44cad5948bda1cf23505f06380910efa7
Author: Megh Bhatt <email address hidden>
Date: Wed May 24 00:34:03 2017 -0700

Fix ZooDelete in zookeeper client

Fix ZooDelete in zookeeper client to retry recoverable errors

Closes-Bug #1692453

Change-Id: If9a9d10d8675acd181aeb2f8436c1aa944cbb13f
(cherry picked from commit 97b9d19cbe5abb90cb5c143147de49fd36622f37)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/33240
Committed: http://github.com/Juniper/contrail-controller/commit/1d26974477ae01ee38d88b216ccaf10c438f22d4
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 1d26974477ae01ee38d88b216ccaf10c438f22d4
Author: Megh Bhatt <email address hidden>
Date: Wed May 24 00:34:03 2017 -0700

Fix ZooDelete in zookeeper client

Fix ZooDelete in zookeeper client to retry recoverable errors

Closes-Bug #1692453

Change-Id: If9a9d10d8675acd181aeb2f8436c1aa944cbb13f
(cherry picked from commit 97b9d19cbe5abb90cb5c143147de49fd36622f37)