contrail-control crashes upon the deletion of VMs in OpenStack

[Martin Gerhard Loschwitz]

When deleting a number of VMs in OpenStack, contrail-control crashes reproducibly with the following exception:

contrail-control: controller/src/db/db_table_partition.cc:97: virtual void DBTablePartition::Remove(DBEntryBase*): Assertion `tree_.erase(*entry)' failed.

Pedro has debugged this and proposed the attached patch. With the patch attached, on our alpha platform, we're encountering regular crashes, too:

contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1u, 0>]: Assertion `!hook.is_linked()' failed.

We're also still seeing

contrail-control: controller/src/db/db_table_partition.cc:97: virtual void DBTablePartition::Remove(DBEntryBase*): Assertion `tree_.erase(*entry)' failed.

albeit less regularly.

Platform is Ubuntu 14.04.

[Martin Gerhard Loschwitz (martin-loschwitz)]

We're now also seeing a lot of these:

contrail-control: controller/src/db/db_table_partition.cc:97: virtual void DBTablePartition::Remove(DBEntryBase*): Assertion `tree_.erase(*entry)' failed.
contrail-control: controller/src/db/db_table_partition.cc:97: virtual void DBTablePartition::Remove(DBEntryBase*): Assertion `tree_.erase(*entry)' failed.
contrail-control: controller/src/bgp/bgp_ribout_updates.cc:251: virtual bool RibOutUpdates::PeerDequeue(int, IPeerUpdate*, const RibPeerSet&, RibPeerSet*): Assertion `start_marker->members.intersects(mready)' failed.
contrail-control: controller/src/db/db_table_partition.cc:97: virtual void DBTablePartition::Remove(DBEntryBase*): Assertion `tree_.erase(*entry)' failed.
contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1u, 0>]: Assertion `!hook.is_linked()' failed.
contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1u, 0>]: Assertion `!hook.is_linked()' failed.
contrail-control: controller/src/db/db_table_partition.cc:97: virtual void DBTablePartition::Remove(DBEntryBase*): Assertion `tree_.erase(*entry)' failed.
contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1u, 0>]: Assertion `!hook.is_linked()' failed.
contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1u, 0>]: Assertion `!hook.is_linked()' failed.
contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost::intrusive::detail::generic_hook<boost::intrusive::get_set_node_algo<void*, false>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1u, 0>]: Assertion `!hook.is_linked()' failed.
contrail-control: /usr/include/boost/intrusive/detail/utilities.hpp:322: void boost::intrusive::detail::destructor_impl(Hook&, boost::intrusive::detail::link_dispatch<(boost::intrusive::link_mode_type)1u>) [with Hook = boost...

[Martin Gerhard Loschwitz (martin-loschwitz)]

The patch originally suggested by Pedro doesn't work either:

diff -ruN contrail-1.20.old/controller/src/db/db_table.cc contrail-1.20/controller/src/db/db_table.cc
--- contrail-1.20.old/controller/src/db/db_table.cc 2014-11-03 06:37:36.000000000 +0000
+++ contrail-1.20/controller/src/db/db_table.cc 2014-12-05 07:23:38.962813814 +0000
@@ -187,7 +187,7 @@
 }

 DBTablePartBase *DBTable::GetTablePartition(const DBRequestKey *key) {
- int id = HashToPartition(Hash(key));
+ size_t id = HashToPartition(Hash(key));
     return GetTablePartition(id);
 }

@@ -205,7 +205,7 @@
 }

 DBEntry *DBTable::Find(const DBRequestKey *key) {
- int id = HashToPartition(Hash(key));
+ size_t id = HashToPartition(Hash(key));
     DBTablePartition *tbl_partition =
     static_cast<DBTablePartition *>(GetTablePartition(id));
     return tbl_partition->Find(key);

[Martin Gerhard Loschwitz (martin-loschwitz)]

 I can confirm that the following patch fixes the problem (courtesy, of course, of Pedro):

diff -ruN contrail-1.20.old/controller/src/bgp/evpn/evpn_route.h contrail-1.20/controller/src/bgp/evpn/evpn_route.h
--- contrail-1.20.old/controller/src/bgp/evpn/evpn_route.h 2014-11-03 06:37:18.000000000 +0000
+++ contrail-1.20/controller/src/bgp/evpn/evpn_route.h 2014-12-05 20:18:54.081169460 +0000
@@ -34,7 +34,7 @@

     RouteDistinguisher route_distinguisher() const { return rd_; }
     uint32_t tag() const { return tag_; }
- MacAddress mac_addr() const { return mac_addr_; }
+ const MacAddress & mac_addr() const { return mac_addr_; }
     Ip4Prefix ip_prefix() const { return ip_prefix_; }
     static size_t label_offset(const BgpProtoPrefix &prefix);

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5332
Committed: http://github.org/Juniper/contrail-controller/commit/ef1771d54b959ca7d8d01241aa5dfb887e096163
Submitter: Zuul
Branch: R1.10

commit ef1771d54b959ca7d8d01241aa5dfb887e096163
Author: Pedro Marques <email address hidden>
Date: Fri Dec 5 13:41:48 2014 -0800

Automatic copy constructor for MacAddress doesn't work as expected.
Add DISALLOW_COPY_AND_ASSIGN to prevent the copy constructor from being
used.

Change-Id: Id8ad9456f64a574c2bade9ed584614260cb9b8f6
Partial-Bug: #1399603

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5627
Committed: http://github.org/Juniper/contrail-controller/commit/f5f3e9834fb598dcaa3ffe18a2610fbf40fa1c6c
Submitter: Zuul
Branch: R2.0

commit f5f3e9834fb598dcaa3ffe18a2610fbf40fa1c6c
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Dec 14 16:11:36 2014 -0800

Backport Pedro's fix for bug 1399603

Automatic copy constructor for MacAddress doesn't work as expected.
Add DISALLOW_COPY_AND_ASSIGN to prevent the copy constructor from being
used.

Change-Id: Id8ad9456f64a574c2bade9ed584614260cb9b8f6
Partial-Bug: #1399603

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5626
Committed: http://github.org/Juniper/contrail-controller/commit/4e70c22001b17c10f99529278d48e19f1df697e3
Submitter: Zuul
Branch: master

commit 4e70c22001b17c10f99529278d48e19f1df697e3
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Dec 14 16:04:39 2014 -0800

Backport Pedro's fix for bug 1399603

Automatic copy constructor for MacAddress doesn't work as expected.
Add DISALLOW_COPY_AND_ASSIGN to prevent the copy constructor from being
used.

Change-Id: Id8ad9456f64a574c2bade9ed584614260cb9b8f6
Partial-Bug: #1399603

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5666
Committed: http://github.org/Juniper/contrail-controller/commit/6da724d897cd1fa78801a0fae4b3d31858fc0a29
Submitter: Zuul
Branch: master

commit 6da724d897cd1fa78801a0fae4b3d31858fc0a29
Author: Nischal Sheth <email address hidden>
Date: Sun Dec 7 00:25:09 2014 -0800

Return const refs to RouteDistinguisher and EthernetSegmentId

Automatic copy constructor and assignment operator do not work for
RouteDistinguisher and EthernetSegmentId.
Return const refs to these classes instead of returning by value.
The only exception is the static ::FromString method.

Change-Id: I1493d1ccf5dfeb8ef97b8d4c429c492cc61eddf5
Partial-Bug: 1399603

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5667
Committed: http://github.org/Juniper/contrail-controller/commit/41f87eb19b8374c6c913422a6bbfc1203e378fd5
Submitter: Zuul
Branch: R2.0

commit 41f87eb19b8374c6c913422a6bbfc1203e378fd5
Author: Nischal Sheth <email address hidden>
Date: Sun Dec 7 00:25:09 2014 -0800

Return const refs to RouteDistinguisher and EthernetSegmentId

Automatic copy constructor and assignment operator do not work for
RouteDistinguisher and EthernetSegmentId.
Return const refs to these classes instead of returning by value.
The only exception is the static ::FromString method.

Change-Id: I1493d1ccf5dfeb8ef97b8d4c429c492cc61eddf5
Partial-Bug: 1399603

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/5697
Committed: http://github.org/Juniper/contrail-controller/commit/fad6f2e02b6876c42f1f9155150b36a3d31d6206
Submitter: Zuul
Branch: R1.10

commit fad6f2e02b6876c42f1f9155150b36a3d31d6206
Author: Nischal Sheth <email address hidden>
Date: Tue Dec 16 10:28:39 2014 -0800

Return const ref to RouteDistinguisher

Automatic copy constructor and assignment operator do not work for
RouteDistinguisher.
Return const refs to this class instead of returning by value.
The only exception is the static ::FromString method.

Change-Id: I68d4d8694966f1ad9d485c1cf34bc5a19229815c
Partial-Bug: 1399603