[ks8-R5.0]:Agent crash and Pod creation failed on R5.0 build 16

[Pulkit Tandon]

Configuration:
K8s 1.9.2
coat-5.0-15
Centos-7.4

Setup:
5 node setup.
1 Kube master. 3 Controller.
2 Agent+ K8s slaves

Description:
Created a service with tag "app=http-test"
Created 2 endpoints.
As soon as creation is done, router agent crashes.

Trace:

[root@nodec61 /]# gdb /usr/bin/contrail-vrouter-agent /var/crashes/core.contrail-vroute.13355.nodec61.1523964738
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/contrail-vrouter-agent...Reading symbols from /usr/bin/contrail-vrouter-agent...(no debugging symbols found)...done.
(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 13549]
[New LWP 13355]
[New LWP 13544]
[New LWP 13542]
[New LWP 13548]
[New LWP 13545]
[New LWP 13546]
[New LWP 13547]
[New LWP 13543]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0 0x00007fb9817761f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install contrail-vrouter-agent-5.0-16.el7.centos.x86_64
(gdb) bt full
#0 0x00007fb9817761f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007fb9817778e8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007fb98176f266 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007fb98176f312 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4 0x0000000000c15440 in AgentOperDBTable::ConfigEventHandler(IFMapNode*, DBEntry*) ()
No symbol table info available.
#5 0x0000000000c41714 in IFMapDependencyManager::ProcessChangeList() ()
No symbol table info available.
#6 0x0000000000ea4a57 in TaskTrigger::WorkerTask::Run() ()
No symbol table info available.
#7 0x0000000000e9e64f in TaskImpl::execute() ()
No symbol table info available.
#8 0x00007fb9823458ca in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) () from /lib64/libtbb.so.2
No symbol table info available.
#9 0x00007fb9823415b6 in tbb::internal::arena::process(tbb::internal::generic_scheduler&) () from /lib64/libtbb.so.2
No symbol table info available.
#10 0x00007fb982340c8b in tbb::internal::market::process(rml::job&) () from /lib64/libtbb.so.2
No symbol table info available.
#11 0x00007fb98233e67f in tbb::internal::rml::private_worker::run() () from /lib64/libtbb.so.2
No symbol table info available.
#12 0x00007fb98233e879 in tbb::internal::rml::private_worker::thread_routine(void*) () from /lib64/libtbb.so.2
No symbol table info available.
#13 0x00007fb982560e25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#14 0x00007fb98183934d in clone () from /lib64/libc.so.6
No symbol table info available.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/42034
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/42035
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/42034
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/42034
Committed: http://github.com/Juniper/contrail-controller/commit/981f2e4e41f729077fdd42944e2d55046de628ec
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 981f2e4e41f729077fdd42944e2d55046de628ec
Author: Hari Prasad Killi <email address hidden>
Date: Tue Apr 17 23:46:47 2018 +0530

Filter out tag nodes without uuid

Change-Id: I9b14bb38ff54016e04c523160a482f0b9ae6e88a
closes-bug: #1764821

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/42035
Committed: http://github.com/Juniper/contrail-controller/commit/28a014f1f97160d1f9ca3f04331566ec0a558893
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 28a014f1f97160d1f9ca3f04331566ec0a558893
Author: Hari Prasad Killi <email address hidden>
Date: Tue Apr 17 23:46:47 2018 +0530

Filter out tag nodes without uuid

Change-Id: I9b14bb38ff54016e04c523160a482f0b9ae6e88a
closes-bug: #1764821

[Pulkit Tandon (pulkitt)]

Old issue and started working long back.
Recent sanity run does not have this issue. Thus, tagging recent build IDs to close it.
ocata-master-89
ocata-R5.0-50