Gateway_less_Fwd: contrail-control and contrail-dns crashes while applying policy on IP-Fabric virtual network

[Chandra Sekhar Reddy Mallam]

contrail-control and contrail-dns crashes while applying policy on IP-Fabric virtual network.

Build
------
R4.1.0.0 Build 23 Ubuntu 14.04 Mitaka

Topology
—————
Control/config/analytics node :nodei15
Compute nodes : nodek11, nodec23 and nodeb3

root@nodei15(controller):/# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-config 4.1.0.0-24 24
contrail-config-openstack 4.1.0.0-24 24
contrail-control 4.1.0.0-24 24
contrail-database-common 4.1.0.0-24 24
contrail-dns 4.1.0.0-24 24
contrail-f5 4.1.0.0-24 24
contrail-lib 4.1.0.0-24 24
contrail-nodemgr 4.1.0.0-24 24
contrail-openstack-control 4.1.0.0-24 24
contrail-openstack-webui 4.1.0.0-24 24
contrail-setup 4.1.0.0-24 24
contrail-utils 4.1.0.0-24 24
contrail-web-controller 4.1.0.0-24 24
contrail-web-core 4.1.0.0-24 24
contrail-web-storage 4.1.0.0-24 24
python-contrail 4.1.0.0-24 24
python-neutronclient 1:4.1.1-2~cloud0.2contrail 24
root@nodei15(controller):/# ls -ltr /var/crashes
total 725328
-rw------- 1 contrail contrail 203034624 Aug 28 11:23 core.contrail-dns.7229.nodei15.1503899629
-rw------- 1 contrail contrail 216072192 Aug 28 11:23 core.contrail-contro.14962.nodei15.1503899629
-rw------- 1 contrail contrail 203141120 Aug 28 11:23 core.contrail-contro.7576.nodei15.1503899630
-rw------- 1 contrail contrail 210595840 Aug 28 11:23 core.contrail-dns.7574.nodei15.1503899632
-rw------- 1 contrail contrail 202588160 Aug 28 11:23 core.contrail-contro.7865.nodei15.1503899633
-rw------- 1 contrail contrail 202121216 Aug 28 11:23 core.contrail-dns.8070.nodei15.1503899636
-rw------- 1 contrail contrail 202592256 Aug 28 11:23 core.contrail-contro.8262.nodei15.1503899636
-rw------- 1 contrail contrail 202223616 Aug 28 11:24 core.contrail-dns.8657.nodei15.1503899640
-rw------- 1 contrail contrail 210698240 Aug 28 11:24 core.contrail-contro.8716.nodei15.1503899640
-rw------- 1 contrail contrail 202006528 Aug 28 11:24 core.contrail-dns.8893.nodei15.1503899645
root@nodei15(controller):/#

root@nodei15(controller):/# contrail-status
== Contrail Control ==
contrail-control failed
contrail-control-nodemgr active
contrail-dns failed
contrail-named active

== Contrail Config ==
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-schema active
contrail-svc-monitor active

== Contrail Web UI ==
contrail-webui active
contrail-webui-middleware active

========Run time service failures=============
/var/crashes/core.contrail-contro.8716.nodei15.1503899640
/var/crashes/core.contrail-dns.8657.nodei15.1503899640
/var/crashes/core.contrail-contro.7865.nodei15.1503899633
/var/crashes/core.contrail-dns.8893.nodei15.1503899645
/var/crashes/core.contrail-contro.8262.nodei15.1503899636
/var/crashes/core.contrail-dns.7574.nodei15.1503899632
/var/crashes/core.contrail-contro.14962.nodei15.1503899629
/var/crashes/core.contrail-dns.8070.nodei15.1503899636
/var/crashes/core.contrail-dns.7229.nodei15.1503899629
/var/crashes/core.contrail-contro.7576.nodei15.1503899630
root@nodei15(controller):/#

Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00002b5204815c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00002b5204815c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00002b5204819028 in __GI_abort () at abort.c:89
#2 0x00002b520480ebf6 in __assert_fail_base (fmt=0x2b5204963018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xf2d5f2 "IsObject()", file=file@entry=0xf2d858 "build/include/rapidjson/document.h",
    line=line@entry=1075,
    function=function@entry=0xfb5960 "contrail_rapidjson::GenericValue<Encoding, Allocator>::ConstMemberIterator contrail_rapidjson::GenericValue<Encoding, Allocator>::MemberBegin() const [with Encoding = contrail_rapidjson::UTF8<>; Alloc"...) at assert.c:92
#3 0x00002b520480eca2 in __GI___assert_fail (assertion=0xf2d5f2 "IsObject()", file=0xf2d858 "build/include/rapidjson/document.h", line=1075,
    function=0xfb5960 "contrail_rapidjson::GenericValue<Encoding, Allocator>::ConstMemberIterator contrail_rapidjson::GenericValue<Encoding, Allocator>::MemberBegin() const [with Encoding = contrail_rapidjson::UTF8<>; Alloc"...)
    at assert.c:101
#4 0x000000000041faa2 in ?? ()
#5 0x0000000000b515c2 in ?? ()
#6 0x0000000000c19b4b in ?? ()
#7 0x0000000000c1a8cb in ?? ()
#8 0x0000000000c1ab46 in ?? ()
#9 0x0000000000c1b88e in ?? ()
#10 0x0000000000bfc019 in ?? ()
#11 0x0000000000c04ae2 in ?? ()
#12 0x0000000000c05956 in ?? ()
#13 0x0000000000bfddbc in ?? ()
#14 0x0000000000bfe02d in ?? ()
#15 0x0000000000719a47 in ?? ()
#16 0x0000000000711227 in ?? ()
#17 0x00002b5203ba3b3a in ?? () from /usr/lib/libtbb.so.2
#18 0x00002b5203b9f816 in ?? () from /usr/lib/libtbb.so.2
#19 0x00002b5203b9ef4b in ?? () from /usr/lib/libtbb.so.2
#20 0x00002b5203b9b0ff in ?? () from /usr/lib/libtbb.so.2
#21 0x00002b5203b9b2f9 in ?? () from /usr/lib/libtbb.so.2
#22 0x00002b520396d184 in start_thread (arg=0x2b520e450700) at pthread_create.c:312
#23 0x00002b52048dcffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

Following config on IP-Fabric virtual network
----------------------------------------------
Virtual Network

 {
virtual-network: {
virtual_network_network_id: 2,
virtual_machine_interface_back_refs: [
 {
to: [
"default-global-system-config",
"nodeb3",
"vhost0"
],
href: "http://10.204.217.127:8082/virtual-machine-interface/922be274-6067-4446-a553-9c16044914aa",
attr: null,
uuid: "922be274-6067-4446-a553-9c16044914aa"
},
 {
to: [
"default-global-system-config",
"nodek11",
"vhost0"
],
href: "http://10.204.217.127:8082/virtual-machine-interface/868552f9-507b-4547-bb28-a7d374dbd0a1",
attr: null,
uuid: "868552f9-507b-4547-bb28-a7d374dbd0a1"
},
 {
to: [
"default-global-system-config",
"nodec23",
"vhost0"
],
href: "http://10.204.217.127:8082/virtual-machine-interface/f6329d00-5341-49d4-9208-6116d3b5039c",
attr: null,
uuid: "f6329d00-5341-49d4-9208-6116d3b5039c"
}
],
parent_uuid: "85e75330-3840-4098-9e27-2cbe57b4613f",
network_policy_refs: [
 {
to: [
"default-domain",
"ctest-TestGWLessFWD-88028226",
"p1"
],
href: "http://10.204.217.127:8082/network-policy/16032bf2-be0d-409b-9ead-348c47cd5a1b",
attr: null,
uuid: "16032bf2-be0d-409b-9ead-348c47cd5a1b"
}
],
parent_href: "http://10.204.217.127:8082/project/85e75330-3840-4098-9e27-2cbe57b4613f",
parent_type: "project",
href: "http://10.204.217.127:8082/virtual-network/e4914540-25c0-4814-a25b-8fcf18be9738",
mac_learning_enabled: false,
id_perms: {
enable: true,
uuid: {
uuid_mslong: 16470021454084196000,
uuid_lslong: 11699102576661142000
},
creator: null,
created: "2017-08-18T10:02:48.369326",
user_visible: true,
last_modified: "2017-08-28T06:13:22.144539",
permissions: {
owner: "cloud-admin",
owner_access: 7,
other_access: 7,
group: "cloud-admin-group",
group_access: 7
},
description: null
},
port_security_enabled: true,
routing_instances: [
 {
to: [
"default-domain",
"default-project",
"ip-fabric",
"ip-fabric"
],
href: "http://10.204.217.127:8082/routing-instance/fa9e60c3-b402-474c-8dea-7c3394d06b54",
uuid: "fa9e60c3-b402-474c-8dea-7c3394d06b54"
},
 {
to: [
"default-domain",
"default-project",
"ip-fabric",
"__default__"
],
href: "http://10.204.217.127:8082/routing-instance/33dbfc98-45c4-4f4a-8fce-82fe17788dd0",
uuid: "33dbfc98-45c4-4f4a-8fce-82fe17788dd0"
}
],
pbb_etree_enable: false,
name: "ip-fabric",
fq_name: [
"default-domain",
"default-project",
"ip-fabric"
],
uuid: "e4914540-25c0-4814-a25b-8fcf18be9738",
mac_aging_time: 300,
virtual_network_refs: [
 {
to: [
"default-domain",
"ctest-TestGWLessFWD-92734979",
"vn1"
],
href: "http://10.204.217.127:8082/virtual-network/8a2f8d3b-2931-45fe-b32e-79d38a5b4173",
attr: null,
uuid: "8a2f8d3b-2931-45fe-b32e-79d38a5b4173"
},
 {
to: [
"default-domain",
"ctest-TestGWLessFWD-88028226",
"vn1"
],
href: "http://10.204.217.127:8082/virtual-network/5b5e3a99-2355-4da3-9b4b-d043045ff6ad",
attr: null,
uuid: "5b5e3a99-2355-4da3-9b4b-d043045ff6ad"
}
],
pbb_evpn_enable: false,
perms2: {
owner: "cloud-admin",
owner_access: 7,
global_access: 0,
share: [
]
},
flood_unknown_unicast: false,
layer2_control_word: false
}
}

[Chandra Sekhar Reddy Mallam (cmallam)]

Cores are copied to:
bhushana@10.204.216.50:/home/bhushana/Documents/technical/bugs/<bug-ID>

[Chandra Sekhar Reddy Mallam (cmallam)]

This crash is observed while attaching the policy to IP Fabric VN when VirtualNetworkPolicyType is None.

> /contrail-test/common/gw_less_fwd/base.py(382)setup_policy()
-> ip_fabric_vn_fq_name_str = "default-domain:default-project:ip-fabric"
(Pdb) n
> /contrail-test/common/gw_less_fwd/base.py(383)setup_policy()
-> ip_fabric_vn_obj = self.vnc_h.virtual_network_read(fq_name_str=ip_fabric_vn_fq_name_str)
(Pdb)
> /contrail-test/common/gw_less_fwd/base.py(384)setup_policy()
-> policy_obj = self.vnc_h.network_policy_read(fq_name=policy_fixture.policy_fq_name)
(Pdb)
> /contrail-test/common/gw_less_fwd/base.py(385)setup_policy()
-> ip_fabric_vn_obj.add_network_policy(policy_obj, None) <===============
(Pdb)
> /contrail-test/common/gw_less_fwd/base.py(386)setup_policy()
-> self.vnc_h.virtual_network_update(ip_fabric_vn_obj)

Crash is observed after above step.

[Pramodh D'Souza (psdsouza)]

can't access the core at bhushana@10.204.216.50:/home/bhushana/Documents/technical/bugs/<bug-ID>

Please provide a path that is accessible.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35856
Submitter: Pramodh D'Souza (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35857
Submitter: Pramodh D'Souza (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35857
Committed: http://github.com/Juniper/contrail-controller/commit/f8d7bb7f005929cd8b179c1af81de8de5291bd3b
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit f8d7bb7f005929cd8b179c1af81de8de5291bd3b
Author: Pramodh D'Souza <email address hidden>
Date: Fri Sep 22 18:14:54 2017 -0700

Fixes CN crash when ref attribute is null

Added check for attr "null" and handle it gracefully by forcing it to
be an empty set {}
Modified existing test data to test this case.
This is a double commit from mainline
Closes-bug: 1713407

Change-Id: Ia4eb01d9ef68831d2992ccd2a8dc3de137ca55fc

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35856
Committed: http://github.com/Juniper/contrail-controller/commit/8b25b80a9ea8e5acfe53d67afd1dca0818594bc6
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8b25b80a9ea8e5acfe53d67afd1dca0818594bc6
Author: Pramodh D'Souza <email address hidden>
Date: Fri Sep 22 16:05:54 2017 -0700

Fixes CN crash when ref attribute is null

Added check for attr "null" and handle it gracefully by forcing it to
be an empty set {}
Modified existing test data to test this case.
Closes-bug: 1713407

Change-Id: I035729e2dc0e8d403c55ffc032d6c49001a68e5f