Class 'networking_bgpvpn.neutron.db.bgpvpn_db.BGPVPNPortAssociationRoute' is not mapped

Bug #1815564 reported by Oleg Bondarev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-bgpvpn
New
Undecided
Unassigned

Bug Description

Neutron server fails to start with "Class 'networking_bgpvpn.neutron.db.bgpvpn_db.BGPVPNPortAssociationRoute' is not mapped" exception after pike -> queens upgrade with bgpvpn migrations applied. On some restarts it's "Class 'networking_bgpvpn.neutron.db.bgpvpn_db.BGPVPN' is not mapped".

It appears that if delete 'bgpvpn' relationship from 'class BGPVPNPortAssociationRoute' the exception is gone.

Revision history for this message
Oleg Bondarev (obondarev) wrote :

Not sure I understand relationships between classes BGPVPN, BGPVPNPortAssociation and BGPVPNPortAssociationRoute:

- BGPVPNPortAssociation has non-nullable foreign key bgpvpn_id to BGPVPN
- BGPVPNPortAssociationRoute has non-nullable foreign key port_association_id to BGPVPNPortAssociation
- BGPVPNPortAssociationRoute has nullable foreign key bgpvpn_id to BGPVPN

why BGPVPNPortAssociationRoute needs bgpvpn_id? Can it be different bgpvpn_id from corresponding BGPVPNPortAssociation object's bgpvpn_id?

Revision history for this message
Thomas Morin (tmmorin-orange) wrote :

Hi Oleg,

> why BGPVPNPortAssociationRoute needs bgpvpn_id?

A BGPVPN Port Association can include 'routes' of a type 'bgpvpn', which will result in redistribution routes from a BGPVPN into another.

(this is explained briefly in https://developer.openstack.org/api-ref/network/v2/#port-routes )

e.g:

/bgpvpn/bgpvpns/5d6820bf-a528-4e0c-85df-c2e771368d55/port_association/a02d408d-7493-4291-b7bc-56894d063693
{
    "port_association": {
        "port_id": "b58a6241-6e49-4b11-87c6-8e0606dde796",
        "routes": [
            {
                "type": "bgpvpn",
                "bgpvpn_id": "46a1a80b-7c42-4c45-88fd-b531e636969f"
            }
        ]
    }
}

... will result in routes advertized for import into BGPVPN 46a1a80b-7c42-4c45-88fd-b531e636969f being re-exported into BGPVPN 5d6820bf-a528-4e0c-85df-c2e771368d55

> Can it be different bgpvpn_id from corresponding BGPVPNPortAssociation object's bgpvpn_id?

Yes, and they will typically be different (not sure there's even a relevant case where they would be equal).

Revision history for this message
Thomas Morin (tmmorin-orange) wrote :

Your bug is surprising, because with alembic based migration, the DB schema is supposed to be the same whether we install from scratch a queens release, or if we apply migration script to upgrade from pike to queens.

Could you provide a dump of your DB schema after the migration to queens (for the bgpvpn tables) ?

Revision history for this message
Thomas Morin (tmmorin-orange) wrote :

Given that the error is not always the same, it looks a bit like an inconsistent behavior of sqlalchemy perhaps. I would first check that your version of sqlalchemy (and alembic) is close to the one use in Openstack CI for queens (see openstack/requirements -- or we can create a dummy gerrit change to trigger a devstack tempest run on stable/queens branch, if you want).

Revision history for this message
Thomas Morin (tmmorin-orange) wrote :

( Another idea, possibly totally irrelevant... I'm wondering if changing lazy='joined' into lazy='select', for the BGPVPNPortAssociationRoute relation, might have some effect -- at line https://github.com/openstack/networking-bgpvpn/blob/master/networking_bgpvpn/neutron/db/bgpvpn_db.py#L156 ... )

Revision history for this message
Oleg Bondarev (obondarev) wrote :

Hi Thomas, here's a dump for bgpvpn tables after migration: https://pastebin.com/t1WzbpNz

Revision history for this message
Oleg Bondarev (obondarev) wrote :
Download full text (6.5 KiB)

Copying here for history:
--
-- Table structure for table `alembic_version_bgpvpn`
--

DROP TABLE IF EXISTS `alembic_version_bgpvpn`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `alembic_version_bgpvpn` (
  `version_num` varchar(32) NOT NULL,
  PRIMARY KEY (`version_num`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `alembic_version_bgpvpn`
--

LOCK TABLES `alembic_version_bgpvpn` WRITE;
/*!40000 ALTER TABLE `alembic_version_bgpvpn` DISABLE KEYS */;
INSERT INTO `alembic_version_bgpvpn` VALUES ('23ce05e0a19f'),('666c706fea3b');
/*!40000 ALTER TABLE `alembic_version_bgpvpn` ENABLE KEYS */;
UNLOCK TABLES;

--
-- Table structure for table `bgpvpn_network_associations`
--

DROP TABLE IF EXISTS `bgpvpn_network_associations`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `bgpvpn_network_associations` (
  `id` varchar(36) NOT NULL,
  `project_id` varchar(255) NOT NULL,
  `bgpvpn_id` varchar(36) NOT NULL,
  `network_id` varchar(36) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `bgpvpn_id` (`bgpvpn_id`,`network_id`),
  KEY `network_id` (`network_id`),
  KEY `ix_bgpvpn_network_associations_project_id` (`project_id`),
  CONSTRAINT `bgpvpn_network_associations_ibfk_1` FOREIGN KEY (`network_id`) REFERENCES `networks` (`id`) ON DELETE CASCADE,
  CONSTRAINT `bgpvpn_network_associations_ibfk_2` FOREIGN KEY (`bgpvpn_id`) REFERENCES `bgpvpns` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `bgpvpn_network_associations`
--

LOCK TABLES `bgpvpn_network_associations` WRITE;
/*!40000 ALTER TABLE `bgpvpn_network_associations` DISABLE KEYS */;
/*!40000 ALTER TABLE `bgpvpn_network_associations` ENABLE KEYS */;
UNLOCK TABLES;

--
-- Table structure for table `bgpvpn_port_association_routes`
--

DROP TABLE IF EXISTS `bgpvpn_port_association_routes`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `bgpvpn_port_association_routes` (
  `id` varchar(36) NOT NULL,
  `port_association_id` varchar(36) NOT NULL,
  `type` enum('prefix','bgpvpn') NOT NULL,
  `local_pref` bigint(20) DEFAULT NULL,
  `prefix` varchar(49) DEFAULT NULL,
  `bgpvpn_id` varchar(36) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `port_association_id` (`port_association_id`),
  KEY `bgpvpn_id` (`bgpvpn_id`),
  CONSTRAINT `bgpvpn_port_association_routes_ibfk_1` FOREIGN KEY (`port_association_id`) REFERENCES `bgpvpn_port_associations` (`id`) ON DELETE CASCADE,
  CONSTRAINT `bgpvpn_port_association_routes_ibfk_2` FOREIGN KEY (`bgpvpn_id`) REFERENCES `bgpvpns` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `bgpvpn_port_association_routes`
--

LOCK TABLES `bgpvpn_port_association_routes` WRITE;
/*!40000 ALTER TABLE `bgpvpn_port_association_routes` DISABLE KEYS */;
/*!40000 ALTER TABLE `bgpvpn_port_association_routes`...

Read more...

Revision history for this message
Oleg Bondarev (obondarev) wrote :

Changing lazy='joined' into lazy='select', for the BGPVPNPortAssociationRoute port_association relation does not help, only after deletion of 'bgpvpn' relationship from 'class BGPVPNPortAssociationRoute' the exception is gone.

Revision history for this message
Thomas Morin (tmmorin-orange) wrote :

Thanks for trying the lazy=select ; that was a totally blind attempt...

I couldn't identify anything problematic is the DB dump, comparing with what we have on master (on which the neutron startup you have is not seen). This, in my understanding, tends to rule out an issue with the DB structure.

Could you share details about the version of sqlalchemy your are using ?

Revision history for this message
Oleg Bondarev (obondarev) wrote :

Sure:
root@ctl01:~# dpkg -l | grep sqlalchemy
ii python-sqlalchemy 1.0.13+ds1-1.1~u16.04+mcp2 all SQL toolkit and Object Relational Mapper for Python
root@ctl01:~# dpkg -l | grep alembic
ii alembic 0.8.10-1.1~u16.04+mcp2 all lightweight database migration tool for SQLAlchemy
ii python-alembic 0.8.10-1.1~u16.04+mcp2 all lightweight database migration tool for SQLAlchemy - Python 2.x

Revision history for this message
Thomas Morin (tmmorin-orange) wrote :

SQLAlchemy 1.0.13 is pretty old, isn't it ?

I see that stable/queens CI is using sqlalchemy 1.2.1 .

Even stable/pike CI is using a more recent sqlalchemy (1.1.12).

Would you have the possibility of trying a more recent version of SQLAlchemy, closer to what upstream validates for Queens ?

Revision history for this message
Oleg Bondarev (obondarev) wrote :
Download full text (20.5 KiB)

Ok, it looks like this is a race condition issue: if agent state report comes before all (BGPVPN exactly) db objects got initialized - we get errors.
If I stop all neutron agents, start neutron server, then start neutron agents some time after server - everything is good.
Error log:

2019-04-09 13:42:27,194.194 15604 INFO sqlalchemy.orm.mapper.Mapper [req-0392737c-e6c8-472f-8685-91190f882862 - - - - -] (BGPVPNPortAssociation|bgpvpn_port_associations) initialize prop routes
2019-04-09 13:42:27,197.197 15604 INFO sqlalchemy.orm.mapper.Mapper [req-917b59e9-2b38-4214-a808-6bf2872d708f - - - - -] (BGPVPNPortAssociationRoute|bgpvpn_port_association_routes) _configure_property(port_association, RelationshipProperty)
2019-04-09 13:42:27,197.197 15604 INFO sqlalchemy.orm.mapper.Mapper [req-917b59e9-2b38-4214-a808-6bf2872d708f - - - - -] (BGPVPNPortAssociationRoute|bgpvpn_port_association_routes) _configure_property(bgpvpn, RelationshipProperty)
2019-04-09 13:42:27,198.198 15604 INFO sqlalchemy.orm.mapper.Mapper [req-917b59e9-2b38-4214-a808-6bf2872d708f - - - - -] (BGPVPNPortAssociationRoute|bgpvpn_port_association_routes) _configure_property(id, Column)
2019-04-09 13:42:27,199.199 15604 INFO sqlalchemy.orm.mapper.Mapper [req-917b59e9-2b38-4214-a808-6bf2872d708f - - - - -] (BGPVPNPortAssociationRoute|bgpvpn_port_association_routes) _configure_property(port_association_id, Column)
2019-04-09 13:42:27,200.200 15604 INFO sqlalchemy.orm.mapper.Mapper [req-917b59e9-2b38-4214-a808-6bf2872d708f - - - - -] (BGPVPNPortAssociationRoute|bgpvpn_port_association_routes) _configure_property(type, Column)
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server [req-0392737c-e6c8-472f-8685-91190f882862 - - - - -] Exception during message handling: UnmappedClassError: Class 'networking_bgpvpn.neutron.db.bgpvpn_db.BGPVPNPortAssociationRoute' is not mapped
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
2019-04-09 13:42:27,201.201 15604 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wr...

Revision history for this message
Oleg Bondarev (obondarev) wrote :

Not sure what's special regarding BGPVPN objects, maybe it's just a coincidence that they are initialized last.
This environment has quite a lot of neutron extensions enabled so amount of DB objects might be a bit higher than in average deployment.
Anyway it seems some wait condition needed in neutron server to postpone state report handling until all DB objects are "constructed".

Revision history for this message
Oleg Bondarev (obondarev) wrote :

I filed new bug with proper description for neutron project: https://bugs.launchpad.net/neutron/+bug/1824299.
Marking this one as duplicate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-bgpvpn (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/659493

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-bgpvpn (master)

Reviewed: https://review.opendev.org/659493
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=b128f5e0dbb366a8b542b070e2cf81436d58911d
Submitter: Zuul
Branch: master

commit b128f5e0dbb366a8b542b070e2cf81436d58911d
Author: Oleg Bondarev <email address hidden>
Date: Thu May 16 12:30:11 2019 +0400

    Move db class definitions before orm relationships to those classes

    This is to fix race conditions on neutron server init.
    Please see bug for details.
    Also removes redundant relationship in BGPVPNPortAssociation
    as it is defined in BGPVPNPortAssociationRoute as well.

    Change-Id: Ibe51e51799c59fae1a8b412fd3d3535bbc96aa20
    Partial-Bug: #1824299
    Related-Bug: #1815564

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.