Juniper Openstack

[Build CentOS release 6.5 2.10-37~icehouse.el6] Multi-Interface Sanity - vrouter crashed at NhDecode(NextHop const, PktInfo const, PktFlowInfo, PktControlInfo, PktControlInfo*, bool)

Bug #1425856 reported by Sandip Dey on 2015-02-26

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Fix Committed	High	Naveen N
R2.0	Fix Committed	High	Naveen N	Juniper Openstack r2.01
R2.1	Fix Committed	High	Naveen N	Juniper Openstack r2.1-fcs

Bug Description

Logs saved at :http://10.204.216.50/Docs/bugs/<bug-id>

BT
===
#0 0x000000000123fc8c in NextHop::id() const ()
#0 0x000000000123fc8c in NextHop::id() const ()
#1 0x00000000014aaedf in NhDecode(NextHop const*, PktInfo const*, PktFlowInfo*, PktControlInfo*, PktControlInfo*, bool) ()
#2 0x00000000014ab24a in RouteToOutInfo(AgentRoute const*, PktInfo const*, PktFlowInfo*, PktControlInfo*, PktControlInfo*) ()
#3 0x00000000014ad37f in PktFlowInfo::IngressProcess(PktInfo const*, PktControlInfo*, PktControlInfo*) ()
#4 0x00000000014adeb8 in PktFlowInfo::Process(PktInfo const*, PktControlInfo*, PktControlInfo*) ()
#5 0x00000000014b91e7 in FlowHandler::Run() ()
#6 0x00000000014b511f in Proto::ProcessProto(boost::shared_ptr<PktInfo>) ()
#7 0x00000000014b7543 in boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >::operator()(Proto*, boost::shared_ptr<PktInfo>) const ()
#8 0x00000000014b6f04 in bool boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> >::operator()<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list1<boost::shared_ptr<PktInfo>&> >(boost::_bi::type<bool>, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >&, boost::_bi::list1<boost::shared_ptr<PktInfo>&>&, long) ()
#9 0x00000000014b67ba in bool boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >::operator()<boost::shared_ptr<PktInfo> >(boost::shared_ptr<PktInfo>&) ()
#10 0x00000000014b62a9 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >, bool, boost::shared_ptr<PktInfo> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<PktInfo>) ()
#11 0x00000000014a9c2f in boost::function1<bool, boost::shared_ptr<PktInfo> >::operator()(boost::shared_ptr<PktInfo>) const ()
#12 0x00000000014b7928 in QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::RunQueue() ()
#13 0x00000000014b7760 in QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::Run() ()
#14 0x0000000001a84676 in TaskImpl::execute() ()
#15 0x00002b4ae65b018a in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x2b4b40000900, parent=...,
    child=0x41e26c0)
    at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/custom_scheduler.h:449
#16 0x00002b4ae65a7033 in tbb::internal::arena::process (this=0x41a8980, s=...)
    at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/arena.cpp:99
#17 0x00002b4ae65a5906 in tbb::internal::market::process (this=0x41a6500,
    j=...)
    at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/market.cpp:393
#18 0x00002b4ae65a04bc in tbb::internal::rml::private_worker::run (
    this=0x41a7200)
    at /ecbuilds/PipeLine/sb/third_party/tbb40_20111130oss/src/tbb/private_server.cpp:263
#19 0x00002b4ae65a0362 in tbb::internal::rml::private_worker::thread_routine (

Tags:

Revision history for this message

Naveen N (naveenn) wrote on 2015-02-26:

Though there was only one vgw in setup exporting 0/0 route, agent got a ecmp route for 0/0 route,
thought both next hop and label are same,there encap types differ.

<message from=\"<email address hidden>\" to=\"nodei5/bgp-peer\">
    <event xmlns=\"http://jabber.org/protocol/pubsub\">
        <items node=\"1/1/default-domain:admin:vgwvn:vgwvn\">
            <item id=\"0.0.0.0/0\">
                <entry>
                    <nlri>
                        <af>1</af>
                        <safi>1</safi>
                        <address>0.0.0.0/0</address>
                    </nlri>
                    <next-hops>
                        <next-hop>
                            <af>1</af>
                            <address>192.168.192.5</address>
                            <label>17</label>
                            <tunnel-encapsulation-list>
                                <tunnel-encapsulation>gre</tunnel-encapsulation>
                            </tunnel-encapsulation-list>
                        </next-hop>
                        <next-hop>
                            <af>1</af>
                            <address>192.168.192.5</address>
                            <label>17</label>
                            <tunnel-encapsulation-list>
                                <tunnel-encapsulation>gre</tunnel-encapsulation>
                                <tunnel-encapsulation>udp</tunnel-encapsulation>
                            </tunnel-encapsulation-list>
                        </next-hop>
                    </next-hops>
                    <version>1</version>
                    <virtual-network>default-domain:admin:vgwvn</virtual-network>
                    <sequence-number>0</sequence-number>
                    <security-group-list />
                    <local-preference>100</local-preference>
                </entry>
            </item>
        </items>

If there is a ecmp of vgw and traffic originates first from outside to contrail-domain VM,
agent was trying to get policy enabled next hop for reverse flow setup which would not
be created for vgw.
Fix for to take policy enabled next hop for reverse flow only destination is vm interface.

Though there was only one vgw in setup exporting 0/0 route, agent got a ecmp route for 0/0 route, 
thought both next hop and label are same,there encap types differ.

<message from=\"network-control@contrailsystems.com\" to=\"nodei5/bgp-peer\">
    <event xmlns=\"http://jabber.org/protocol/pubsub\">
        <items node=\"1/1/default-domain:admin:vgwvn:vgwvn\">
            <item id=\"0.0.0.0/0\">
                <entry>
                    <nlri>
                        <af>1</af>
                        <safi>1</safi>
                        <address>0.0.0.0/0</address>
                    </nlri>
                    <next-hops>
                        <next-hop>
                            <af>1</af>
                            <address>192.168.192.5</address>
                            <label>17</label>
                            <tunnel-encapsulation-list>
                                <tunnel-encapsulation>gre</tunnel-encapsulation>
                            </tunnel-encapsulation-list>
                        </next-hop>
                        <next-hop>
                            <af>1</af>
                            <address>192.168.192.5</address>
                            <label>17</label>
                            <tunnel-encapsulation-list>
                                <tunnel-encapsulation>gre</tunnel-encapsulation>
                                <tunnel-encapsulation>udp</tunnel-encapsulation>
                            </tunnel-encapsulation-list>
                        </next-hop>
                    </next-hops>
                    <version>1</version>
                    <virtual-network>default-domain:admin:vgwvn</virtual-network>
                    <sequence-number>0</sequence-number>
                    <security-group-list />
                    <local-preference>100</local-preference>
                </entry>
            </item>
        </items>

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-02-27: A change has been merged

Reviewed: https://review.opencontrail.org/7875
Committed: http://github.org/Juniper/contrail-controller/commit/f000a7fcd9c99fff809c87d3f30ed7e162ace1a8
Submitter: Zuul
Branch: R2.1

commit f000a7fcd9c99fff809c87d3f30ed7e162ace1a8
Author: Naveen N <email address hidden>
Date: Thu Feb 26 05:57:20 2015 -0800

* In a scenario where there are 2 vgw exporting same subnet route,
and traffic session gets initiated from external domain to
VM, then for setting reverse flow agent was trying to get policy
enabled interface nexthop for vgw, and for vgw we would only
create policy disabled nh. Fixing the same to pick
policy disabled nexthop for vgw destination
Closes-bug:#1425856

Change-Id: Ic0c61be47997a2c9ebe3196394cef21468fce8ca

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-02-28:

Reviewed: https://review.opencontrail.org/7905
Committed: http://github.org/Juniper/contrail-controller/commit/d0414d2e6eeae350ecf4189f894c475e0ef00d21
Submitter: Zuul
Branch: R2.0

commit d0414d2e6eeae350ecf4189f894c475e0ef00d21
Author: Naveen N <email address hidden>
Date: Fri Feb 27 01:58:02 2015 -0800

Change-Id: I23462aec80cc7fc351c376705f2fb2dc062acf16

information type:

Proprietary → Public

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-03-05: master

Review in progress for https://review.opencontrail.org/8032

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-03-05: A change has been merged

Download full text (5.5 KiB)

Reviewed: https://review.opencontrail.org/8032
Committed: http://github.org/Juniper/contrail-controller/commit/9d44b8f521cf02f70ecb7cd628754c64b8de3b9a
Submitter: Zuul
Branch: master

commit 9d44b8f521cf02f70ecb7cd628754c64b8de3b9a
Author: Hari <email address hidden>
Date: Tue Mar 3 21:29:48 2015 -0800

Merge the following commits from R2.1 branch (final).

Perf improvements for Logical Interface delete handling

Follow up on commit b4af4d67cef56ac85da5bb0d50bb55f882c70538 for delete
handling

When a link delete is notified to config listener, the IFMapNode in link
can be NULL. We used to ignore links with left/right node deleted.

Modified the link handler to pass the node-type when left/right node is
deleted. This way, the IFLinkToReq handler can take action based on the
type of node being deleted
(cherry picked from commit f1d7816dbdc886859caec5152d2b436ad4fb2a96)

Conflicts:
src/vnsw/agent/test-xml/test_xml_oper.cc

Holding the UUID from ID_PERMS in config DBState in DB Add and using it
for Config Delete
closes-bug: #1420209
(cherry picked from commit 6a9b4be6a0dcfb3980bd072a57f92761a548eb9c)

* Send ARP request packet on VMI interface instead of parent interface
such that vlan gets added, based on transmit vlan id
* If vlan-id is 0, then its treated as untagged interface, in agent
while framing the packet we were setting the tag in packet 0
avoiding the same same setting vlan id as invalid and programming
it as vlan-id 0 only in vrouter
Closes-bug:#1423122
(cherry picked from commit 38f1405b27e05e5d31a2ae6297ca42d915b368bd)

* If arp route is deleted twice, then we arp nexthop was not
getting deleted, since upon each arp route delete we enqueue
a request to modify nexthop to invalid and this might
result in arp entry creation, hence use operation RESYNC
instead of add/change
* Check for vrf while parsing on subnet address change, since
we might modify next vrf subnet routes if prefixx matches
Test case for same
Closes-bug:#1423502
(cherry picked from commit a65d557c27d15de5d7d0bf30fdc5a20fa93acf42)

Conflicts:
src/vnsw/agent/test/test_route.cc

reverting the unwated checkin of modifying the notify_count_ to bool

(cherry picked from commit 975f507d81ce89106814f4fced1d14b454dcdd5a)

* Whenever ICMP error packet is trapped for flow setup,
parse the inner payload and frame the flow key
Closes-bug:#1423848
(cherry picked from commit 81ee64728201b1d8234d790772ab3035874c1f5d)

* ARP route behind gateway should have proxy flag set
such that arp request on VM would be proxied by vrouter
Closes-bug:#1423122
(cherry picked from commit e72443e1a73d3372944981f15941e97a5a9f439f)

Remove unused UVEs

Remove UVE generated on add/delete of a link
Remove UVE generated on change of physical-device-vn entry.
Added a UT for physical-device-vn to validate add/delete of VMI to VN link

(cherry picked from commit 452b9341668f0a89455cc5e1847c0c7d70f0d3b2)

Holding the metadata of link while storing it in defer list. While
adding the link to Graph from defer list, the metadata also added back
to graph which makes the dependency tracker to work correctly

(cherry picked from commit 43d7a71e1c486a3c326d16e16134c25312b4953f)

Optimize config handling for SG and...

Reviewed:  https://review.opencontrail.org/8032
Committed: http://github.org/Juniper/contrail-controller/commit/9d44b8f521cf02f70ecb7cd628754c64b8de3b9a
Submitter: Zuul
Branch:    master

commit 9d44b8f521cf02f70ecb7cd628754c64b8de3b9a
Author: Hari <haripk@juniper.net>
Date:   Tue Mar 3 21:29:48 2015 -0800

Merge the following commits from R2.1 branch (final).

Perf improvements for Logical Interface delete handling

Follow up on commit b4af4d67cef56ac85da5bb0d50bb55f882c70538 for delete
handling

When a link delete is notified to config listener, the IFMapNode in link
can be NULL. We used to ignore links with left/right node deleted.

Conflicts:
src/vnsw/agent/test-xml/test_xml_oper.cc

Holding the UUID from ID_PERMS in config DBState in DB Add and using it
for Config Delete
closes-bug: #1420209
(cherry picked from commit 6a9b4be6a0dcfb3980bd072a57f92761a548eb9c)

Conflicts:
src/vnsw/agent/test/test_route.cc

reverting the unwated checkin of modifying the notify_count_ to bool

(cherry picked from commit 975f507d81ce89106814f4fced1d14b454dcdd5a)

* Whenever ICMP error packet is trapped for flow setup,
parse the inner payload and frame the flow key
Closes-bug:#1423848
(cherry picked from commit 81ee64728201b1d8234d790772ab3035874c1f5d)

* ARP route behind gateway should have proxy flag set
such that arp request on VM would be proxied by vrouter
Closes-bug:#1423122
(cherry picked from commit e72443e1a73d3372944981f15941e97a5a9f439f)

Remove unused UVEs

Remove UVE generated on add/delete of a link
Remove UVE generated on change of physical-device-vn entry.
Added a UT for physical-device-vn to validate add/delete of VMI to VN link

(cherry picked from commit 452b9341668f0a89455cc5e1847c0c7d70f0d3b2)

Holding the metadata of link while storing it in defer list. While
adding the link to Graph from defer list, the metadata also added back
to graph which makes the dependency tracker to work correctly

(cherry picked from commit 43d7a71e1c486a3c326d16e16134c25312b4953f)

Optimize config handling for SG and ACL

Follow up to commit
https://github.com/Juniper/contrail-controller/commit/b4af4d67cef56ac85da5bb0d50bb55f882c70538

On any change or link add/delete to a SG, we are invoking IFNodeToReq
for all connected VMI. Fix this by registering link handler for SG and
invoke IFNodeToReq only for peer VMI

On any change or link add/delete to a ACL, we are invoking IFNodeToReq
for all connected SG and VN. Fix this by registering link handler for ACL
and invoke IFNodeToReq only for peer SG / VN
(cherry picked from commit 0d33ec13e576165f68ff172a62b6dfb8cc73774f)

(cherry picked from commit f000a7fcd9c99fff809c87d3f30ed7e162ace1a8)

Fix Tor-Agent assert for KSyncEntry GetReference

Issue:
------
On connection close delete is triggered for all the KSync
Objects, creation of entry after this deletion in not
allowed.
Here Logical Switch Object is deleted and Unicast Remote
Entry Table is yet to receive a delete, in the meanwhile
a route table entry get notified and tries to take create
a temp entry for reference in Logical Switch Entry which
is already deleted

Fix:
----
Since delete trigger for Unicast Remote Entry Table happens
in context of VRF table walk, which will have a yeild in
between. during this time if a notification for route table
happens, it needs to be ignored.

Also fixing a memory leak in idl deletion path

Closes-Bug: 1424531
(cherry picked from commit d4ba9892ae816dc0b4d59d6dd7e3350d3b3a1745)

DHCP enable/disable bug fixes

1) Fix bug where dhcp enable flag toggling was not reflected in route -
Toggling of DHCP enable in interface results in route rebake. Also flag for
handle_dhcp is now moved from path to route.
2) Remove mac VM binding, instead add a route with DHCP peer path which can be
used to get binding. This path is having discard NH.
Closes-bug: #1417131

Conflicts:

src/vnsw/agent/oper/operdb_init.h
	src/vnsw/agent/test-xml/test_xml_oper.cc
	src/vnsw/agent/test/test_route.cc

Rename Dhcp path/peer to MacVmBinding.

Change-Id: I0643612ca907b85c548d4132502e00ca1498fdc1

Changed in juniperopenstack:
status:	New → Fix Committed

Nagabhushana R (bhushana) on 2015-04-21

Changed in juniperopenstack:
importance:	Undecided → High

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Juniper Openstack

[Build CentOS release 6.5 2.10-37~icehouse.el6] Multi-Interface Sanity - vrouter crashed at NhDecode(NextHop const*, PktInfo const*, PktFlowInfo*, PktControlInfo*, PktControlInfo*, bool)

Bug Description

Other bug subscribers

Remote bug watches

[Build CentOS release 6.5 2.10-37~icehouse.el6] Multi-Interface Sanity - vrouter crashed at NhDecode(NextHop const, PktInfo const, PktFlowInfo, PktControlInfo, PktControlInfo*, bool)