5.0.1: BGPaaS local-as. Agent crashes on bgpaas vm route scale

[Shashikiran H]

ocata-master-153

bgpass vm --- local as --- contrail --- ibgp --- mx

mx has 2 routing instance (left and right).
contrail has to 2 vns corresponding to each mx vrf(left and right).

bgpaas vm has 2 legs correspong to left and right VN and 2 groups with different local-as values to contrail corresponding to each VN leg.
I use bgpaas vm to move routes from left VN local-as to right VN local-as. Routes then go to contrail right vrf, and then back to mx.

I see the below agent crash:
[root@nodec31 ~]# gdb contrail-vrouter-agent core.contrail-vroute.961.nodec31.1530921939
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/contrail-vrouter-agent...done.
BFD: Warning: /root/core.contrail-vroute.961.nodec31.1530921939 is truncated: expected core file size >= 1045245952, found: 115347456.

warning: core file may not match specified executable file.
[New LWP 961]
[New LWP 970]
[New LWP 971]
[New LWP 972]
[New LWP 974]
[New LWP 975]
[New LWP 976]
[New LWP 968]
[New LWP 969]
Failed to read a valid object file image from memory.
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0 0x00007f98239c5277 in ?? ()
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fff4db97268:

[Manish Singh (manishs)]

Harsh, Do you have the core? Cant see any core files....

ls: cannot access /cs-shared/bugs/1781172: No such file or directory

[Kumar Harsh (hkumar)]

core is here:
[hkumar@nodem3 ~/io_tcp_agent_crash]

[Kumar Harsh (hkumar)]

Core file is truncated , please share new core file:

gdb contrail-vrouter-agent.debug core.contrail-vroute.961.nodec31.1530921939
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/contrail-vrouter-agent.debug...done.
BFD: Warning: /root/core.contrail-vroute.961.nodec31.1530921939 is truncated: expected core file size >= 1045245952, found: 115347456.

warning: core file may not match specified executable file.
[New LWP 961]
[New LWP 970]
[New LWP 971]
[New LWP 972]
[New LWP 974]
[New LWP 975]
[New LWP 976]
[New LWP 968]
[New LWP 969]
Cannot access memory at address 0x7f982737b128
Cannot access memory at address 0x7f982737b120
Failed to read a valid object file image from memory.
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0 0x00007f98239c5277 in ?? ()
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fff4db97268:
(gdb) q
[root@nodeg23 ~]#
[root@nodeg23 ~]# gdb c31_agent core.contrail-vroute.961.nodec31.1530921939
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/c31_agent...done.
BFD: Warning: /root/core.contrail-vroute.961.nodec31.1530921939 is truncated: expected core file size >= 1045245952, found: 115347456.

warning: core file may not match specified executable file.
[New LWP 961]
[New LWP 970]
[New LWP 971]
[New LWP 972]
[New LWP 974]
[New LWP 975]
[New LWP 976]
[New LWP 968]
[New LWP 969]
Failed to read a valid object file image from memory.
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0 0x00007f98239c5277 in ?? ()
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fff4db97268:

[root@nodeg23 ~]# strings core.contrail-vroute.961.nodec31.1530921939|grep build-info
{"build-info":[{"build-time":"2018-06-20 18:03:25.640239","build-hostname":"centos-7-4-builder-juniper-contrail-ci-c-0000057528.novalocal","build-user":"zuul","build-version":"5.1.0","build-id":"5.1.0-153.el7","build-number":"@contrail"}]}
{"build-info": [{"build-time": "2018-06-20 18:03:25.640239", "build-hostname": "centos-7-4-builder-juniper-contrail-ci-c-0000057528.novalocal", "build-user": "zuul", "build-version": "5.1.0"}]}
{"build-info": [{"build-time"...

[Shashikiran H (skiranh)]

Unable to repro this. The old steps don't seem to be creating a core.