Sync router fails with db exception

[Maru Newby]

While investigating https://bugs.launchpad.net/neutron/+bug/1210664, salvatore-orlando discovered that a db exception was being raised during router syncing:

Traceback (most recent call last):
  File "/opt/stack/neutron/neutron/openstack/common/rpc/amqp.py", line 424, in _process_data
    **args)
  File "/opt/stack/neutron/neutron/common/rpc.py", line 44, in dispatch
    neutron_ctxt, version, method, namespace, **kwargs)
  File "/opt/stack/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)
  File "/opt/stack/neutron/neutron/db/l3_rpc_base.py", line 47, in sync_routers
    plugin.auto_schedule_routers(context, host, router_ids)
  File "/opt/stack/neutron/neutron/db/agentschedulers_db.py", line 303, in auto_schedule_routers
    self, context, host, router_ids)
  File "/opt/stack/neutron/neutron/scheduler/l3_agent_scheduler.py", line 113, in auto_schedule_routers
    context.session.add(binding)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__
    self.commit()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit
    self._prepare_impl()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl
    self.session.flush()
  File "/opt/stack/neutron/neutron/openstack/common/db/sqlalchemy/session.py", line 542, in _wrap
    raise exception.DBError(e)
DBError: (IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`routerl3agentbindings`, CONSTRAINT `routerl3agentbindings_ibfk_1` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') 'INSERT INTO routerl3agentbindings (id, router_id, l3_agent_id) VALUES (%s, %s, %s)' ('2df68c3d-f3c9-43d2-bf45-e2e57e84b054', 'c4502f1f-a093-4c7c-b161-929b6342509b', '85d6b60f-f3ff-4437-8f5a-af165087f3ea')

This can be reproduced by running the quantum smoke test in tempest (nosetests tempest/scenario/test_network_basic_ops.py). The smoke test passes - the exception only occurs at test cleanup. It may be that the router syncing code is working with stale state after router deletion.

[Salvatore Orlando (salvatore-orlando)]

I've raised this to High because if we disable auto-schema generation in Havana, the l3 agent scheduling - and possibly router synchronization - would be pretty much broken.

[Salvatore Orlando (salvatore-orlando)]

I haven't seen this in a while, scaling down to Incomplete to perform further checks.

[Xiang Hui (xianghui)]

This can be replicated by commenting "#router_id= "and setting "use_namespace=False", then restart neutron-l3-agent.
The router_id will be assigned the default value '', which mismatch DB format.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/47218

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47218
Committed: http://github.com/openstack/neutron/commit/3e1116eb0f1d94530707cd6ef4b37f17e9a13918
Submitter: Jenkins
Branch: master

commit 3e1116eb0f1d94530707cd6ef4b37f17e9a13918
Author: ZhiQiang Fan <email address hidden>
Date: Thu Sep 19 01:53:44 2013 +0800

    Ensure router exists when auto_schedule_routers

    Currently, the auto_schedule_routers() accepts parameter router_ids,
    which may contain invalid router ids, since we've already filtered
    them via plugin.get_routers(), we can directly use that safe object.

    Closes-Bug: #1217998
    Closes-Bug: #1210877

    Change-Id: I6196f16cca65fee4e848173d0a0a10fde967195d