Issue observed on current master (should be reproducible on mitaka too).
Steps to reproduce:
1) enable pluggable ipam be setting default value for 'ipam_driver' as 'internal' in patch set, upload into gerrit
2) validate test results for gate-tempest-dsvm-neutron-linuxbridge
3) observe that some router update related tests will fail
4) analyse screen-q-svc.txt.gz request processing to validate that after raising RetryRequest 'external_gateway_info' field is not present.
During retrying update router request 'external_gateway_info' field got missed, so update does not work correctly on retry in this case.
Before retry:
{u'router': {u'external_gateway_info': {u'network_id': u'e0e7e739-d64f-4eb8-b4ec-bb3ef9623ab9'}, u'name': u'tempest-router--1366126092', u'admin_state_up': False}}
After retry:
{u'router': {u'name': u'tempest-router--1366126092', u'admin_state_up': False}}
This issue is observed only on scale testing where concurent ip allocation happens for the same subnet.
Due to concurent data modifications one of the transactions is restarted by RetryRequest exception (compare-and-swap synchronization).
And for update router case restaring transaction leads to missing 'external_gateway_info' from original input.
Related log output:
2016-04-22 12:26:35.545 18234 DEBUG neutron.api.v2.base [req-efd63a02-c382-4268-8dc9-a7708a0fc205 tempest-RoutersIpV6Test-2037938894 -] Request body: {u'router': {u'external_gateway_info': {u'network_id': u'e0e7e739-d64f-4eb8-b4ec-bb3ef9623ab9'}, u'name': u'tempest-router--1366126092', u'admin_state_up': False}} prepare_request_body /opt/stack/new/neutron/neutron/api/v2/base.py:656
...
2016-04-22 12:26:36.873 18234 DEBUG oslo_db.api [req-efd63a02-c382-4268-8dc9-a7708a0fc205 tempest-RoutersIpV6Test-2037938894 -] Performing DB retry for function neutron.api.v2.base._update wrapper /usr/local/lib/python2.7/dist-packages/oslo_db/api.py:150
2016-04-22 12:26:36.874 18234 DEBUG neutron.api.v2.base [req-efd63a02-c382-4268-8dc9-a7708a0fc205 tempest-RoutersIpV6Test-2037938894 -] Request body: {u'router': {u'name': u'tempest-router--1366126092', u'admin_state_up': False}} prepare_request_body /opt/stack/new/neutron/neutron/api/v2/base.py:656
Full log available at [1]. Trace req-efd63a02-c382-4268-8dc9-a7708a0fc205
request processing. And [2] is failed test related to that issue.
[1] http://logs.openstack.org/23/181023/71/check/gate-tempest-dsvm-neutron-linuxbridge/a475a39/logs/screen-q-svc.txt.gz#_2016-04-22_12_26_35_545
[2] http://logs.openstack.org/23/181023/71/check/gate-tempest-dsvm-neutron-linuxbridge/a475a39/console.html#_2016-04-22_12_57_48_092
I want to mark this confirmed, but it needs some more info. What version of neutron? Repro steps?