Bug #1377241 “Lock wait timeout on delete port for DVR” : Bugs : neutron

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-03:

#1

Lockwait-timeout.txt Edit (36.6 KiB, text/plain)

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2014-10-03:

#2

Carl has something:

https://review.openstack.org/#/c/122880/

Can you verify that this might be related/fixes what you're seeing?

Changed in neutron:
status:	New → Confirmed
importance:	Undecided → High
assignee:	nobody → Carl Baldwin (carl-baldwin)

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-03:

#3

Will check the patch and run the script.

Armando Migliaccio (armando-migliaccio) on 2014-10-03

summary:

- OperationalError: Lock wait timeout exceeded seen occasionally when we
- run a script to clean all networks, ports and routers. This is seen with
- DVR
+ Lock wait timeout on delete port for DVR

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-06:

#4

Download full text (10.7 KiB)

I tested the above patch with the original script that caused the "lock wait".
The problem seems to be still there.

014-10-06 09:51:25.031 DEBUG neutron.openstack.common.lockutils [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Got semaphore "db-access" from (pid=17796) lock /opt/stack/neutron/neutron/openstack/common/lockutils.py:168
2014-10-06 09:51:25.092 DEBUG neutron.plugins.ml2.plugin [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Calling delete_port for 7bbcedcc-77ec-496f-9a54-d0ee1c2ec94e owned by network:router_interface_distributed from (pid=17796) delete_port /opt/stack/neutron/neutron/plugins/ml2/plugin.py:1045
2014-10-06 09:51:25.128 DEBUG neutron.db.l3_dvr_db [req-6a74e01a-0036-4ebd-9ae7-4e1d7c6aeb05 admin bfc91f91c80e4f64908a035506df281e] Subnet matches: ebfc8c6d-a4ed-4307-a80a-f4ec138255fd from (pid=17796) delete_csnat_router_interface_ports /opt/stack/neutron/neutron/db/l3_dvr_db.py:544
2014-10-06 09:51:25.128 DEBUG neutron.plugins.ml2.plugin [req-6a74e01a-0036-4ebd-9ae7-4e1d7c6aeb05 admin bfc91f91c80e4f64908a035506df281e] Deleting port e1ce4490-025a-4bee-9983-66bee9a5e7f4 from (pid=17796) delete_port /opt/stack/neutron/neutron/plugins/ml2/plugin.py:999
2014-10-06 09:51:25.129 DEBUG neutron.plugins.ml2.plugin [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] update_port_arp for port 7bbcedcc-77ec-496f-9a54-d0ee1c2ec94e, action del from (pid=17796) update_port_arp /opt/stack/neutron/neutron/plugins/ml2/plugin.py:825
2014-10-06 09:51:25.129 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Fanout notify l2population agents at q-agent-notifier the message remove_fdb_entries with {u'c6b32d25-8ae2-49f9-bb3b-85a7441a13ee': {'segment_id': 1002L, 'ports': {u'192.168.15.144': []}, 'network_type': u'vxlan'}} from (pid=17796) _notification_fanout /opt/stack/neutron/neutron/plugins/ml2/drivers/l2pop/rpc.py:40
2014-10-06 09:51:25.129 DEBUG neutron.common.rpc [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] neutron.plugins.ml2.drivers.l2pop.rpc.L2populationAgentNotifyAPI method fanout_cast called with arguments (<neutron.context.ContextBase object at 0x7f66c87a00d0>, {'args': {'fdb_entries': {u'c6b32d25-8ae2-49f9-bb3b-85a7441a13ee': {'ports': {u'192.168.15.144': []}, 'network_type': u'vxlan', 'segment_id': 1002L}}}, 'namespace': None, 'method': 'remove_fdb_entries'}) {'topic': 'q-agent-notifier-l2population-update'} from (pid=17796) wrapper /opt/stack/neutron/neutron/common/log.py:33
2014-10-06 09:51:25.131 DEBUG neutron.openstack.common.lockutils [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] Got semaphore "db-access" from (pid=17796) lock /opt/stack/neutron/neutron/openstack/common/lockutils.py:168
2014-10-06 09:51:25.210 DEBUG neutron.plugins.ml2.plugin [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] Calling delete_port for e1ce4490-025a-4bee-9983-66bee9a5e7f4 owned by network:router_centralized_snat from (pid=17796) delete_port /opt/stack/neutron/neutron/plugi...

I tested the above patch with the original script that caused the "lock wait".
The problem seems to be still there.

014-10-06 09:51:25.031 DEBUG neutron.openstack.common.lockutils [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Got semaphore "db-access" from (pid=17796) lock /opt/stack/neutron/neutron/openstack/common/lockutils.py:168
2014-10-06 09:51:25.092 DEBUG neutron.plugins.ml2.plugin [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Calling delete_port for 7bbcedcc-77ec-496f-9a54-d0ee1c2ec94e owned by network:router_interface_distributed from (pid=17796) delete_port /opt/stack/neutron/neutron/plugins/ml2/plugin.py:1045
2014-10-06 09:51:25.128 DEBUG neutron.db.l3_dvr_db [req-6a74e01a-0036-4ebd-9ae7-4e1d7c6aeb05 admin bfc91f91c80e4f64908a035506df281e] Subnet matches: ebfc8c6d-a4ed-4307-a80a-f4ec138255fd from (pid=17796) delete_csnat_router_interface_ports /opt/stack/neutron/neutron/db/l3_dvr_db.py:544
2014-10-06 09:51:25.128 DEBUG neutron.plugins.ml2.plugin [req-6a74e01a-0036-4ebd-9ae7-4e1d7c6aeb05 admin bfc91f91c80e4f64908a035506df281e] Deleting port e1ce4490-025a-4bee-9983-66bee9a5e7f4 from (pid=17796) delete_port /opt/stack/neutron/neutron/plugins/ml2/plugin.py:999
2014-10-06 09:51:25.129 DEBUG neutron.plugins.ml2.plugin [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] update_port_arp for port 7bbcedcc-77ec-496f-9a54-d0ee1c2ec94e, action del from (pid=17796) update_port_arp /opt/stack/neutron/neutron/plugins/ml2/plugin.py:825
2014-10-06 09:51:25.129 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Fanout notify l2population agents at q-agent-notifier the message remove_fdb_entries with {u'c6b32d25-8ae2-49f9-bb3b-85a7441a13ee': {'segment_id': 1002L, 'ports': {u'192.168.15.144': []}, 'network_type': u'vxlan'}} from (pid=17796) _notification_fanout /opt/stack/neutron/neutron/plugins/ml2/drivers/l2pop/rpc.py:40
2014-10-06 09:51:25.129 DEBUG neutron.common.rpc [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] neutron.plugins.ml2.drivers.l2pop.rpc.L2populationAgentNotifyAPI method fanout_cast called with arguments (<neutron.context.ContextBase object at 0x7f66c87a00d0>, {'args': {'fdb_entries': {u'c6b32d25-8ae2-49f9-bb3b-85a7441a13ee': {'ports': {u'192.168.15.144': []}, 'network_type': u'vxlan', 'segment_id': 1002L}}}, 'namespace': None, 'method': 'remove_fdb_entries'}) {'topic': 'q-agent-notifier-l2population-update'} from (pid=17796) wrapper /opt/stack/neutron/neutron/common/log.py:33
2014-10-06 09:51:25.131 DEBUG neutron.openstack.common.lockutils [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] Got semaphore "db-access" from (pid=17796) lock /opt/stack/neutron/neutron/openstack/common/lockutils.py:168
2014-10-06 09:51:25.210 DEBUG neutron.plugins.ml2.plugin [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] Calling delete_port for e1ce4490-025a-4bee-9983-66bee9a5e7f4 owned by network:router_centralized_snat from (pid=17796) delete_port /opt/stack/neutron/neutron/plugins/ml2/plugin.py:1045
2014-10-06 09:51:25.212 DEBUG neutron.plugins.ml2.plugin [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] update_port_arp for port e1ce4490-025a-4bee-9983-66bee9a5e7f4, action del from (pid=17796) update_port_arp /opt/stack/neutron/neutron/plugins/ml2/plugin.py:825
2014-10-06 09:51:25.212 DEBUG neutron.plugins.ml2.drivers.l2pop.rpc [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] Fanout notify l2population agents at q-agent-notifier the message remove_fdb_entries with {u'1aa56867-6b32-49f9-8364-106ad9c41276': {'segment_id': 1001L, 'ports': {u'192.168.15.144': [[u'fa:16:3e:a5:ae:2f', u'151.0.0.3']]}, 'network_type': u'vxlan'}} from (pid=17796) _notification_fanout /opt/stack/neutron/neutron/plugins/ml2/drivers/l2pop/rpc.py:40
2014-10-06 09:51:25.214 DEBUG neutron.common.rpc [req-f120505e-2a32-4ce5-993f-67fdaac75a29 admin bfc91f91c80e4f64908a035506df281e] neutron.plugins.ml2.drivers.l2pop.rpc.L2populationAgentNotifyAPI method fanout_cast called with arguments (<neutron.context.ContextBase object at 0x7f66c87a00d0>, {'args': {'fdb_entries': {u'1aa56867-6b32-49f9-8364-106ad9c41276': {'ports': {u'192.168.15.144': [[u'fa:16:3e:a5:ae:2f', u'151.0.0.3']]}, 'network_type': u'vxlan', 'segment_id': 1001L}}}, 'namespace': None, 'method': 'remove_fdb_entries'}) {'topic': 'q-agent-notifier-l2population-update'} from (pid=17796) wrapper /opt/stack/neutron/neutron/common/log.py:33
2014-10-06 09:51:25.237 DEBUG neutron.openstack.common.lockutils [req-4ee9dd78-0f0f-4f53-8de6-02474e6a3159 admin bfc91f91c80e4f64908a035506df281e] Got semaphore "db-access" from (pid=17796) lock /opt/stack/neutron/neutron/openstack/common/lockutils.py:168
2014-10-06 09:52:16.974 DEBUG neutron.db.l3_dvr_db [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Subnet matches: 104447e0-e3b4-4625-8722-77e94a4b6707 from (pid=17796) delete_csnat_router_interface_ports /opt/stack/neutron/neutron/db/l3_dvr_db.py:544
2014-10-06 09:52:16.976 DEBUG neutron.plugins.ml2.plugin [req-415c01b8-c3cb-4834-b9a4-04242dc9f618 admin bfc91f91c80e4f64908a035506df281e] Deleting port cf755382-31ed-463d-9d88-660506c55683 from (pid=17796) delete_port /opt/stack/neutron/neutron/plugins/ml2/plugin.py:999
2014-10-06 09:52:16.981 INFO neutron.wsgi [-] (17796) accepted ('192.168.15.144', 41344)
2014-10-06 09:52:16.982 INFO neutron.wsgi [-] (17796) accepted ('192.168.15.144', 41345)
2014-10-06 09:52:16.983 INFO neutron.wsgi [-] (17796) accepted ('192.168.15.144', 41346)
2014-10-06 09:52:16.984 INFO neutron.wsgi [-] (17796) accepted ('192.168.15.144', 41347)
2014-10-06 09:52:17.009 ERROR neutron.api.v2.resource [req-4ee9dd78-0f0f-4f53-8de6-02474e6a3159 admin bfc91f91c80e4f64908a035506df281e] remove_router_interface failed
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 87, in resource
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 200, in _handle_action
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     return getattr(self._plugin, name)(*arg_list, **kwargs)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 247, in remove_router_interface
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     context.elevated(), router, subnet_id=subnet_id)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 547, in delete_csnat_router_interface_ports
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     l3_port_check=False)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1015, in delete_port
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     port_db, binding = db.get_locked_port_and_binding(session, id)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/opt/stack/neutron/neutron/plugins/ml2/db.py", line 135, in get_locked_port_and_binding
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     with_lockmode('update').
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2310, in one
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     ret = list(self)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     return self._execute_and_instances(context)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     result = conn.execute(querycontext.statement, self._params)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     params)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     compiled_sql, distilled_params
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     context)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     six.reraise(type(newraise), newraise, sys.exc_info()[2])
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     per_fn = fn(ctx)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     context.is_disconnect)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/exc_filters.py", line 254, in _raise_operational_errors_directly_filter
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource     raise operational_error
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'SELECT ports.tenant_id AS ports_tenant_id, ports.id AS ports_id, ports.name AS ports_name, ports.network_id AS ports_network_id, ports.mac_address AS ports_mac_address, ports.admin_state_up AS ports_admin_state_up, ports.status AS ports_status, ports.device_id AS ports_device_id, ports.device_owner AS ports_device_owner \nFROM ports \nWHERE ports.id = %s FOR UPDATE' ('93378847-a7b2-45f8-9b82-1a4526342b67',)
2014-10-06 09:52:17.009 TRACE neutron.api.v2.resource 
2014-10-06 09:52:17.330 INFO neutron.wsgi [req-4ee9dd78-0f0f-4f53-8de6-02474e6a3159 admin bfc91f91c80e4f64908a035506df281e] 192.168.15.144 - - [06/Oct/2014 09:52:17] "PUT /v2.0/routers/9fbd23a8-01aa-45fe-ac5a-00f9b72a7ae9/remove_router_interface.json HTTP/1.1" 500 359 53.247294

So the above patch does not fix this problem.

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-06:

#5

When multiple back to back delete_port occurs, "get_locked_port_and_binding" does not return in time and raises the "OperationalError" during the first query for the Port with "with_lockmode('update').

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-07:

#6

When router_interface_delete is issued, it is supposed to call the "delete_csnat_router_interface" after it deletes the router interface ports, but it is delayed.
The next call to gateway_clear happens before this and during the gateway_clear operation it tries to call "delete_csnat_router_interface" and tries to delete the csnat_ports.

At this time the "router_interface_delete" finally calls "delete_csnat_router_interface" and we get this lock timeout.

There should be a way to allow the first router_interface_delete operation to complete its sequence and then call the gateway_clear.

OpenStack Infra (hudson-openstack) on 2014-10-08

Changed in neutron:
assignee:	Carl Baldwin (carl-baldwin) → Swaminathan Vasudevan (swaminathan-vasudevan)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-09: Change abandoned on neutron (master)

#7

Change abandoned by Swaminathan Vasudevan (<email address hidden>) on branch: master
Review: https://review.openstack.org/124849

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-09: Fix proposed to neutron (master)

#8

Fix proposed to branch: master
Review: https://review.openstack.org/127129

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-09:

#9

The semphore before calling the "delete_port" to remove the csnat port fixes the DB lockwait timeout issue.

Right now the "lock" on the "ml2/plugin.py - delete_port" is not sufficient to prevent this DB lockwait timeout issue.

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-10:

#10

DBLockwait-Timeout-mysql-innodboutput.docx Edit (46.1 KiB, application/vnd.openxmlformats-officedocument.wordprocessingml.document)

Some more additional information from the mysql innodb.
The attached file provides the "innodb status" when the DBlockwait happens.

If there are DB experts can you guys please take a look at it and update what is happening.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-13:

#11

Fix proposed to branch: master
Review: https://review.openstack.org/127943

Changed in neutron:
assignee:	Swaminathan Vasudevan (swaminathan-vasudevan) → Kevin Benton (kevinbenton)

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2014-10-13:

#12

Wow, I am actually spoilt for choice :)

- https://review.openstack.org/#/c/127943
- https://review.openstack.org/#/c/122880
- https://review.openstack.org/#/c/127129

Kevin's change seems promising...

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-13:

#13

This patch seems to fix the "lockwait timeout" issue that we have been seeing with our automated scripts.

The other two patches out there for the same problem can be abandoned.

https://review.openstack.org/#/c/122880
There is nothing wrong with the above patch, but Kevin's patch seems to fix the lockwait problem, but the 122880 does not fix the lockwait problem.

https://review.openstack.org/#/c/127129

All my testing was with single node devstack and the automated scripts.
The only failures that I still see is the "KeyError in dhcp_rpc.py for the network_id" and also the "DBDuplicate Error".

https://bugs.launchpad.net/bugs/1378508
https://bugs.launchpad.net/neutron/+bug/1378468

The DBDuplicateError will be handled by the patch https://review.openstack.org/#/c/126793/

If we claim that the DBDuplicateError and the KeyError in dhcp_rpc is the cause of this lockwait, then we still have not found out the root cause.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-14: Change abandoned on neutron (master)

#14

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/127129
Reason: Abandoned as per Swami's comment on:

https://review.openstack.org/#/c/127943

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-14:

#15

Armando it seems that this patch may not address the lockwait timeout issue completely. I still see this problen some times.
So as I have been saying, the main issue when gateway_clear and router_interface_delete both trying to call the "delete_csnat_router_interface" and delete the ports.

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2014-10-15:

#16

The patch I proposed is still necessary though for some lock waits because it sent RPC messages which would cause a yield during a transaction.

Do you have some debug logs that show exactly what happened leading up to the lock wait stacktrace?

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-15: Fix merged to neutron (master)

#17

Reviewed: https://review.openstack.org/127943
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4c2b42e21744be56cbf32aeac6f4b4f1c87de24e
Submitter: Jenkins
Branch: master

commit 4c2b42e21744be56cbf32aeac6f4b4f1c87de24e
Author: Kevin Benton <email address hidden>
Date: Sat Oct 11 03:42:47 2014 -0700

Call DVR VMARP notify outside of transaction

    The dvr vmarp table update notification was being called inside
    of the delete_port transaction in ML2, which can cause a yield
    and lead to the glorious mysql/eventlet deadlock.

    This patch moves it outside the transaction and adjusts it to
    use an existing port dictionary rather than re-looking it up since
    the port is now gone from the DB by the time it is called.

Closes-Bug: #1377241
Change-Id: I0b4dac61e49b2a926353f8478e421cd1a70be038

Changed in neutron:
status:	In Progress → Fix Committed

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-15:

#18

DBLockwaitEntry-Failure.txt Edit (559.1 KiB, text/plain)

Download full text (3.7 KiB)

This csnat port shown below is deleted first by the "router_gateway_clear" action and then again router_interface_delete comes in and tries to delete the same.

"705a8dcc-2448-4d1d-ba61-9d050f9b0aa0"

I have included the trace in the attached file. "DBLockwaitEntry.Failure.txt"

Also see below the console output when this failure occurs. "Internal Server Error".

stack@ubuntu:~/devstack$ ./dvs-fastClean
Found 4 subnets
Found 3 networks
Found 1 routers
Found 1 floating IPs
Found 12 ports
Found 2 security groups
Found 9 security rules
Found 3 nova VMs
Disassociated floating IP 714c30ac-41b5-4c74-9b7d-bea0672dfff2
neutron floatingip-delete 714c30ac-41b5-4c74-9b7d-bea0672dfff2
Deleted floatingip: 714c30ac-41b5-4c74-9b7d-bea0672dfff2
nova delete c0e64d5a-ec87-4305-837e-05a11aee87f4
nova delete 20c88121-87e7-4993-8d84-6f2511ab8b48
nova delete fbda9c1f-e3e5-4ee1-b12f-dc336615dd4d
Request to delete server fbda9c1f-e3e5-4ee1-b12f-dc336615dd4d has been accepted.
Request to delete server 20c88121-87e7-4993-8d84-6f2511ab8b48 has been accepted.
Request to delete server c0e64d5a-ec87-4305-837e-05a11aee87f4 has been accepted.
neutron router-port-list dvs-fip-test.rtr
delete inter sub a5f5b012-7756-436d-aa20-e638c4c25961
neutron router-interface-delete de75a152-4a73-4382-8642-c27d92c4e521 a5f5b012-7756-436d-aa20-e638c4c25961
delete inter sub 1312c18e-4fbd-4163-9137-6f96db9088cb
neutron router-interface-delete de75a152-4a73-4382-8642-c27d92c4e521 1312c18e-4fbd-4163-9137-6f96db9088cb
delete inter sub 72a54512-ff42-479f-85dc-6ea545964c46
neutron router-interface-delete de75a152-4a73-4382-8642-c27d92c4e521 72a54512-ff42-479f-85dc-6ea545964c46
neutron router-gateway-clear de75a152-4a73-4382-8642-c27d92c4e521
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| 351908de-55ce-4841-ab46-862c4f975b1d | | fa:16:3e:f5:f6:d4 | {"subnet_id": "a5f5b012-7756-436d-aa20-e638c4c25961", "ip_address": "151.0.0.1"} |
| 621c0ed9-f7d4-43da-aca3-ba882a4ad740 | | fa:16:3e:76:13:ee | {"subnet_id": "1312c18e-4fbd-4163-9137-6f96db9088cb", "ip_address": "152.2.0.1"} |
| 705a8dcc-2448-4d1d-ba61-9d050f9b0aa0 | | fa:16:3e:29:6b:92 | {"subnet_id": "1312c18e-4fbd-4163-9137-6f96db9088cb", "ip_address": "152.2.0.11"} |
| 99004bf5-1619-468a-8887-13fe87504231 | | fa:16:3e:19:cc:52 | {"subnet_id": "a5f5b012-7756-436d-aa20-e638c4c25961", "ip_address": "151.0.0.2"} |
| a5bf0dd7-d942-45ac-a6e1-292d401b51a9 | | fa:16:3e:c0:0a:47 | {"subnet_id": "8dc49a29-8078-45bd-8906-2d1bef7d2d78", "ip_address": "153.1.1.100"} |
| ad8c8d32-bb23-4d22-abff-5a9b72e7289e | | fa:16:3e:60:94:05 | {"subnet_id": "72a54512-ff42-479f-85dc-6ea545964c46", "ip_address": "152.1.0.1"} |
| aefdc820-4861-42e9-956d-63c7f99762c9 | | fa:16:3e:60:de:34 | {"subnet_id": "72a54512-ff42-479f...

This csnat port shown below is deleted first by the "router_gateway_clear" action and then again router_interface_delete comes in and tries to delete the same.
 
"705a8dcc-2448-4d1d-ba61-9d050f9b0aa0"

I have included the trace in the attached file. "DBLockwaitEntry.Failure.txt"

Also see below the console output when this failure occurs. "Internal Server Error".

stack@ubuntu:~/devstack$ ./dvs-fastClean 
Found 4 subnets
Found 3 networks
Found 1 routers
Found 1 floating IPs
Found 12 ports
Found 2 security groups
Found 9 security rules
Found 3 nova VMs
Disassociated floating IP 714c30ac-41b5-4c74-9b7d-bea0672dfff2
neutron floatingip-delete 714c30ac-41b5-4c74-9b7d-bea0672dfff2
Deleted floatingip: 714c30ac-41b5-4c74-9b7d-bea0672dfff2
nova delete c0e64d5a-ec87-4305-837e-05a11aee87f4
nova delete 20c88121-87e7-4993-8d84-6f2511ab8b48
nova delete fbda9c1f-e3e5-4ee1-b12f-dc336615dd4d
Request to delete server fbda9c1f-e3e5-4ee1-b12f-dc336615dd4d has been accepted.
Request to delete server 20c88121-87e7-4993-8d84-6f2511ab8b48 has been accepted.
Request to delete server c0e64d5a-ec87-4305-837e-05a11aee87f4 has been accepted.
neutron router-port-list dvs-fip-test.rtr
delete inter sub  a5f5b012-7756-436d-aa20-e638c4c25961
neutron router-interface-delete de75a152-4a73-4382-8642-c27d92c4e521 a5f5b012-7756-436d-aa20-e638c4c25961
delete inter sub  1312c18e-4fbd-4163-9137-6f96db9088cb
neutron router-interface-delete de75a152-4a73-4382-8642-c27d92c4e521 1312c18e-4fbd-4163-9137-6f96db9088cb
delete inter sub  72a54512-ff42-479f-85dc-6ea545964c46
neutron router-interface-delete de75a152-4a73-4382-8642-c27d92c4e521 72a54512-ff42-479f-85dc-6ea545964c46
neutron router-gateway-clear de75a152-4a73-4382-8642-c27d92c4e521
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                          |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| 351908de-55ce-4841-ab46-862c4f975b1d |      | fa:16:3e:f5:f6:d4 | {"subnet_id": "a5f5b012-7756-436d-aa20-e638c4c25961", "ip_address": "151.0.0.1"}   |
| 621c0ed9-f7d4-43da-aca3-ba882a4ad740 |      | fa:16:3e:76:13:ee | {"subnet_id": "1312c18e-4fbd-4163-9137-6f96db9088cb", "ip_address": "152.2.0.1"}   |
| 705a8dcc-2448-4d1d-ba61-9d050f9b0aa0 |      | fa:16:3e:29:6b:92 | {"subnet_id": "1312c18e-4fbd-4163-9137-6f96db9088cb", "ip_address": "152.2.0.11"}  |
| 99004bf5-1619-468a-8887-13fe87504231 |      | fa:16:3e:19:cc:52 | {"subnet_id": "a5f5b012-7756-436d-aa20-e638c4c25961", "ip_address": "151.0.0.2"}   |
| a5bf0dd7-d942-45ac-a6e1-292d401b51a9 |      | fa:16:3e:c0:0a:47 | {"subnet_id": "8dc49a29-8078-45bd-8906-2d1bef7d2d78", "ip_address": "153.1.1.100"} |
| ad8c8d32-bb23-4d22-abff-5a9b72e7289e |      | fa:16:3e:60:94:05 | {"subnet_id": "72a54512-ff42-479f-85dc-6ea545964c46", "ip_address": "152.1.0.1"}   |
| aefdc820-4861-42e9-956d-63c7f99762c9 |      | fa:16:3e:60:de:34 | {"subnet_id": "72a54512-ff42-479f-85dc-6ea545964c46", "ip_address": "152.1.0.2"}   |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
Internal Server Error (HTTP 500) (Request-ID: req-7d6bc262-31f3-4d15-8410-64aa7282aed5)

Removed interface from router de75a152-4a73-4382-8642-c27d92c4e521.
Removed interface from router de75a152-4a73-4382-8642-c27d92c4e521.
Removed gateway from router de75a152-4a73-4382-8642-c27d92c4e521
neutron router-delete de75a152-4a73-4382-8642-c27d92c4e521
Deleted router: de75a152-4a73-4382-8642-c27d92c4e521

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2014-10-15:

#19

Thanks, is it possible for me to get a copy of that script and your local.conf file for devstack so I can try to replicate it? I've poked around quite a bit and don't see anything obviously yielding with a lock anymore.

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#20

Cleanup script Edit (4.7 KiB, text/plain)

Kevin yes here are the scripts that I use for creating and cleaning up.
The issue will be seen when using the cleanup script.
You need to patiently wait until the problem shows up.

./dvs-fip-test ( create script).
./dvs-fastClean-test

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#21

create script Edit (10.9 KiB, text/plain)

Create script attached.

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#22

Kevin I have posted the test scripts in here for your reference.

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2014-10-16:

#23

Thanks, what's the required devstack topology needed for this to happen. How many L3 agents do I need?

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#24

You just need a single node devstack and set Q_DVR_MODE to dvr_snat.

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2014-10-16:

#25

Then just OVS and l2pop plugins, right?

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#26

Yes OVS and l2pop.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-16: Fix proposed to neutron (master)

#27

Fix proposed to branch: master
Review: https://review.openstack.org/128855

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2014-10-16:

#28

Hi Swami,

Can you see if you encounter the lockwait issues with this patch?
https://review.openstack.org/#/c/128855/

I ran your scripts in a loop for 30 minutes or so any didn't get any, so I think it addresses the problem.

Revision history for this message

Carl Baldwin (carl-baldwin) wrote on 2014-10-16:

#29

@Swami Could you to be a bit more specific in your descriptions? "gateway_clear" and "router_interface_delete" do not map directly to any methods that are searchable in the code. It takes time for readers to mentally map these vague references to something that can be pinpointed in the code. As far as I can tell, here is the mapping.

gateway_clear -> _delete_current_gw_port in neutron/db/l3_dvr_db.py
router_interface_delete -> remove_router_interface in the same file.
delete_csnat_router_interface -> delete_csnat_router_interface_ports (okay, maybe close enough)

Below are the ways that I see delete_csnat_router_interface_ports being called. Are you saying that a call to remove_router_interface and update_router are competing to delete the csnat router interface ports?

create_router
-> _update_router_gw_info
-> _delete_current_gw_port
-> delete_csnat_router_interface_ports

update_router
-> _update_router_db
   -> _update_router_gw_info
      -> _delete_current_gw_port
         -> delete_csnat_router_interface_ports

remove_router_interface
-> delete_csnat_router_interface_ports

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#30

Hi Carl,

Yes your understanding is correct.
Sorry if my message was not too informative.

My comment #6 states the actual functions being called. But it was not as detailed as yours.

Revision history for this message

Swaminathan Vasudevan (swaminathan-vasudevan) wrote on 2014-10-16:

#31

I agree that "gateway_clear" does not correspond directly any function in the code.

Revision history for this message

Kevin Benton (kevinbenton) wrote on 2014-10-16:

#32

To be clear, before my second patch I was able to reproduce this problem with Swami's scripts.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-24: Fix proposed to neutron (feature/lbaasv2)

#33

Fix proposed to branch: feature/lbaasv2
Review: https://review.openstack.org/130864

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-24: Fix merged to neutron (feature/lbaasv2)

#34

Download full text (72.6 KiB)

Reviewed: https://review.openstack.org/130864
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c089154a94e5872efc95eab33d3d0c9de8619fe4
Submitter: Jenkins
Branch: feature/lbaasv2

commit 62588957fbeccfb4f80eaa72bef2b86b6f08dcf8
Author: Kevin Benton <email address hidden>
Date: Wed Oct 22 13:04:03 2014 -0700

Big Switch: Switch to TLSv1 in server manager

    Switch to TLSv1 for the connections to the backend
    controllers. The default SSLv3 is no longer considered
    secure.

    TLSv1 was chosen over .1 or .2 because the .1 and .2 weren't
    added until python 2.7.9 so TLSv1 is the only compatible option
    for py26.

Closes-Bug: #1384487
Change-Id: I68bd72fc4d90a102003d9ce48c47a4a6a3dd6e03

commit 17204e8f02fdad046dabdb8b31397289d72c877b
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Oct 22 06:20:15 2014 +0000

Imported Translations from Transifex

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I58db0476c810aa901463b07c42182eef0adb5114

commit d712663b99520e6d26269b0ca193527603178742
Author: Carl Baldwin <email address hidden>
Date: Mon Oct 20 21:48:42 2014 +0000

Move disabling of metadata and ipv6_ra to _destroy_router_namespace

    I noticed that disable_ipv6_ra is called from the wrong place and that
    in some cases it was called with a bogus router_id because the code
    made an incorrect assumption about the context. In other case, it was
    never called because _destroy_router_namespace was being called
    directly. This patch moves the disabling of metadata and ipv6_ra in
    to _destroy_router_namespace to ensure they get called correctly and
    avoid duplication.

Change-Id: Ia76a5ff4200df072b60481f2ee49286b78ece6c4
Closes-Bug: #1383495

commit f82a5117f6f484a649eadff4b0e6be9a5a4d18bb
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Oct 21 12:11:19 2014 +0000

Updated from global requirements

Change-Id: Idcbd730f5c781d21ea75e7bfb15959c8f517980f

commit be6bd82d43fbcb8d1512d8eb5b7a106332364c31
Author: Angus Lees <email address hidden>
Date: Mon Aug 25 12:14:29 2014 +1000

Remove duplicate import of constants module

.. and enable corresponding pylint check now the only offending instance
is fixed.

Change-Id: I35a12ace46c872446b8c87d0aacce45e94d71bae

commit 9902400039018d77aa3034147cfb24ca4b2353f6
Author: rajeev <email address hidden>
Date: Mon Oct 13 16:25:36 2014 -0400

Fix race condition on processing DVR floating IPs

    Fip namespace and agent gateway port can be shared by multiple dvr routers.
    This change uses a set as the control variable for these shared resources
    and ensures that Test and Set operation on the control variable are
    performed atomically so that race conditions do not occur among
    multiple threads processing floating IPs.
    Limitation: The scope of this change is limited to addressing the race
    condition described in the bug report. It may not address other issues
    such as pre-existing issue wit...

Reviewed:  https://review.openstack.org/130864
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c089154a94e5872efc95eab33d3d0c9de8619fe4
Submitter: Jenkins
Branch:    feature/lbaasv2

commit 62588957fbeccfb4f80eaa72bef2b86b6f08dcf8
Author: Kevin Benton <blak111@gmail.com>
Date:   Wed Oct 22 13:04:03 2014 -0700

Big Switch: Switch to TLSv1 in server manager
    
    Switch to TLSv1 for the connections to the backend
    controllers. The default SSLv3 is no longer considered
    secure.
    
    TLSv1 was chosen over .1 or .2 because the .1 and .2 weren't
    added until python 2.7.9 so TLSv1 is the only compatible option
    for py26.
    
    Closes-Bug: #1384487
    Change-Id: I68bd72fc4d90a102003d9ce48c47a4a6a3dd6e03

commit 17204e8f02fdad046dabdb8b31397289d72c877b
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Oct 22 06:20:15 2014 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I58db0476c810aa901463b07c42182eef0adb5114

commit d712663b99520e6d26269b0ca193527603178742
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Mon Oct 20 21:48:42 2014 +0000

Move disabling of metadata and ipv6_ra to _destroy_router_namespace
    
    I noticed that disable_ipv6_ra is called from the wrong place and that
    in some cases it was called with a bogus router_id because the code
    made an incorrect assumption about the context.  In other case, it was
    never called because _destroy_router_namespace was being called
    directly.  This patch moves the disabling of metadata and ipv6_ra in
    to _destroy_router_namespace to ensure they get called correctly and
    avoid duplication.
    
    Change-Id: Ia76a5ff4200df072b60481f2ee49286b78ece6c4
    Closes-Bug: #1383495

commit f82a5117f6f484a649eadff4b0e6be9a5a4d18bb
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Oct 21 12:11:19 2014 +0000

Updated from global requirements
    
    Change-Id: Idcbd730f5c781d21ea75e7bfb15959c8f517980f

commit be6bd82d43fbcb8d1512d8eb5b7a106332364c31
Author: Angus Lees <gus@inodes.org>
Date:   Mon Aug 25 12:14:29 2014 +1000

Remove duplicate import of constants module
    
    .. and enable corresponding pylint check now the only offending instance
    is fixed.
    
    Change-Id: I35a12ace46c872446b8c87d0aacce45e94d71bae

commit 9902400039018d77aa3034147cfb24ca4b2353f6
Author: rajeev <rajeev.grover@hp.com>
Date:   Mon Oct 13 16:25:36 2014 -0400

Fix race condition on processing DVR floating IPs
    
    Fip namespace and agent gateway port can be shared by multiple dvr routers.
    This change uses a set as the control variable for these shared resources
    and ensures that Test and Set operation on the control variable are
    performed atomically so that race conditions do not occur among
    multiple threads processing floating IPs.
    Limitation: The scope of this change is limited to addressing the race
    condition described in the bug report. It may not address other issues
    such as pre-existing issue with handling of DVR floatingips on agent
    restart.
    
    closes-bug: #1381238
    
    Change-Id: I6dc2b7bad6e8ddbaa86c1f7a1e2028aeacc3afef

commit ab0992cf5303752f054ad4d47f0a949aadbea83a
Author: Bradley Jones <jones.bradley@me.com>
Date:   Mon Oct 20 12:34:55 2014 +0100

Ensure ofagent unit tests handles random hashseeds
    
    Several tests in test_ofa_neutron_agent.py and test_ofa_flows.py fail when
    running the unit tests with hashseed 2701526934.
    
    This patch fixes the failing unit tests by specifying that assert_has_calls does
    not need to assert any order.
    
    Change-Id: I48d6601130f7de1f6eeb7333a17eddd82e7acbdf
    Partial-Bug: 1348818

commit ccd650732729451aa8e5ce3401f9570c70c4f066
Author: Sylvain Afchain <sylvain.afchain@enovance.com>
Date:   Fri Sep 26 13:49:43 2014 +0000

Moves the HA resource creations outside of transaction
    
    Currently the HA resources are created in the
    _create_router_db which includes calls to
    the plugin and generates RPC calls. Even if the
    resource creations are outside of any transaction
    from the _create_router_db point of view, this
    method is  called in a transaction from the
    create_router method.
    This patch moves the resource creations to the
    create_router method outside the transaction.
    The failures are handled as previously with
    a try/expect.
    
    Change-Id: If8fcfd012f8e992175e49bbefb2ae667881a620a
    Closes-bug: #1374461

commit 5e4b600528a2b58cbb38d6b8d55b316602d4d015
Author: Peng Xiao <penxiao@cisco.com>
Date:   Mon Oct 20 14:16:36 2014 +0800

Modify docstring on send_delete_port_request in N1kv plugin
    
    N1kv plugin: Modify docstring on _send_delete_port_request method
    
    Change-Id: I64d34878ffd8f6db703e4c1d9849032fef9bae96
    Closes-Bug: #1381277

commit 9334d1c98c3abdadc12222acb3f6f4527c940895
Author: Weidong Shao <weidong.shao@huawei.com>
Date:   Thu Aug 21 20:10:47 2014 +0000

Empty files should not contain copyright or license
    
    Per instruction from
    http://docs.openstack.org/developer/hacking/#openstack-licensing
    
    Enable hacking check H104 in this CL.
    
    Change-Id: I435b9d91877499ebe1e33435f06794164a0ecc34
    Partial-Bug: #1262424

commit fd11b4264da0cef395788b0e0ee12c86cb65ec41
Author: Angus Lees <gus@inodes.org>
Date:   Mon Oct 20 11:19:02 2014 +1100

Remove superfluous except/re-raise
    
    This exception code catches all exceptions, and then always raises them
    again.
    
    More interestingly, it uses excutils.save_and_reraise_exception
    incorrectly (outside a `with` statement), which triggers the pylint test
    designed to catch exactly this.
    
    Change-Id: Iab9d69944cbae5257682ac65ff2b8fba3ef7017e

commit db5e370b0d68c3e71626c99941fe487059b3cf88
Author: Romil Gupta <romilg@hp.com>
Date:   Mon Jun 30 05:35:08 2014 -0700

Schema enhancement to support MultiSegment Network
    
    Description:
    Currently, there is nothing in the schema that ensures segments
    for a network are returned in the same order they were specified
    when the network was created, or even in a deterministic order.
    
    Solution:
    We need to add another field named 'segment_index' in
    'ml2_network_segment' table containing a numeric position index.
    With segment_index field we can retrieve the segments in the
    order in which user created.
    
    This patch set also fixes ML2 invalid unit test case in
    test_create_network_multiprovider().
    
    Closes-Bug: #1224978
    Closes-Bug: #1377346
    
    Change-Id: I560c34c6fe1c5425469ccdf9b8b4905c123d496d

commit bc4d1054c3d5125103e16cf85ef02f97987994fd
Author: armando-migliaccio <armamig@gmail.com>
Date:   Tue Oct 14 23:25:58 2014 -0700

Remove redundant initialization and check from DVR RPC mixin
    
    The initialization, as well as the recurrent checks on whether the agent
    is able to handle DVR requests, are not necessary; this is because the
    dvr_agent is going to be set at the time the L2 agent starts receiving
    RPC messages.
    
    Change-Id: I9caad965d0ff507e2ed93d6432997ad86139ac04

commit 04df85b6e5a098f8f55bb82f04d9769763beb487
Author: Kevin Benton <blak111@gmail.com>
Date:   Wed Sep 24 05:23:32 2014 -0700

Improve performance of security group DB query
    
    The _select_ips_for_remote_group method was joining the
    IP allocation, port, allowed address pair, and security group tables
    together in a single query. Additionally, it was loading all of
    the port columns and using none of them. This resulted in a
    very expensive query with no benefit.
    
    This patch eliminates the unnecessary use of the port table by joining
    the IP allocation table directly to the security groups and allowed
    address pairs tables. In local testing of the method, this sped it up
    by an order of magnitude.
    
    Closes-Bug: #1373851
    Change-Id: I12899413004838d2d22b691f1e2f3b18f7ec2c27

commit 8d430a7f2e903dda06d8d75d6abcd63423c4c0a1
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Oct 16 21:27:47 2014 -0700

Optimize query in _select_dhcp_ips_for_network_ids
    
    Only query the DB for relevant columns instead of
    all of the port columns.
    
    Partial-Bug: #1373851
    Change-Id: I32cd4a0bc6799ce77cea13188676308e3e641d19

commit d739790ba8dcdf3180e8d6d09423937aaf802a7c
Author: Ihar Hrachyshka <ihrachys@redhat.com>
Date:   Tue Oct 14 15:05:20 2014 +0200

Updated cache module and its dependencies
    
    This is to avoid cache module dependency on timeutils that are now moved
    to oslo.utils.
    
    The following changes are included:
    
     * neutron/openstack/common/cache/_backends/memory.py
      6ff6b4b Switch oslo-incubator to use oslo.utils and remove old modules
      2bedce3 Fix MemoryBackend not purging item from _keys_expired on delete
    
     * neutron/openstack/common/cache/backends.py
      39625e1 Set pbr 'warnerrors' option for doc build
    
     * neutron/openstack/common/cache/cache.py
      9c683be fix small typo
    
     * neutron/openstack/common/lockutils.py
      5d40e14 Remove code that moved to oslo.i18n
      7209975 Always log the releasing, even under failure
      bbb266c Clarify logging in lockutils
      942e1aa Use file locks by default again
      ac995be Fix E126 pep8 errors
      15b8352 Remove oslo.log from lockutils
    
    Change-Id: I02cb4b2bc4b7bcba948e67cffdb8bd0219c89a29

commit 623a30baadf24bdf672711096b1b87b60c70b038
Author: Ihar Hrachyshka <ihrachys@redhat.com>
Date:   Tue Oct 14 14:36:06 2014 +0200

Updated service.py and its dependencies
    
    This is to avoid service module dependency on importutils that are now
    moved to oslo.utils.
    
    The following changes are included:
    
     * neutron/openstack/common/eventlet_backdoor.py
      5d40e14 Remove code that moved to oslo.i18n
      90ae24b Remove redundant default=None for config options
      fcf517d Update oslo log messages with translation domains
    
     * neutron/openstack/common/loopingcall.py
      5d40e14 Remove code that moved to oslo.i18n
      e377393 Changes calcuation of variable delay
      ab5d5f1 Use timestamp in loopingcall
      bc48099 Log the function name of looping call
      fb4e863 Remove deprecated LoopingCall
      fcf517d Update oslo log messages with translation domains
    
     * neutron/openstack/common/service.py
      5d40e14 Remove code that moved to oslo.i18n
      6ede600 rpc, notifier: remove deprecated modules
    
     * neutron/openstack/common/systemd.py
      17c4e21 Fix docstring indentation in systemd
    
     * neutron/openstack/common/threadgroup.py
      5a1a016 Make stop_timers() method public
      fdc8883 Add graceful stop function to ThreadGroup.stop
      2d06d6c Simple typo correction
      4d18b57 threadgroup: use threading rather than greenthread
    
    Change-Id: I4887545f861a93223e2c7cbcdd39efe991bff547

commit c1bc6ffad8b93490df36a049bf7be0d6da0ce2b0
Author: Ihar Hrachyshka <ihrachys@redhat.com>
Date:   Tue Oct 14 14:22:09 2014 +0200

Updated fileutils and its dependencies
    
    This is to avoid fileutils dependency on excutils that are now moved to
    oslo.utils.
    
    The following changes are included:
    
    * neutron/openstack/common/__init__.py
      6b048e7 Let oslotest manage the six.move setting for mox
    
     * neutron/openstack/common/_i18n.py
      9ce1d96 Fix i18n import
      5d40e14 Remove code that moved to oslo.i18n
    
     * neutron/openstack/common/fileutils.py
      6ff6b4b Switch oslo-incubator to use oslo.utils and remove old modules
      2b966f9 Fix deletion of cached file for policy enforcer
      9c88dc3 file_open: fixed docstring to refer to open() instead of file()
      6c7407b fileutils: port to Python 3
      fcf517d Update oslo log messages with translation domains
    
     * neutron/openstack/common/log.py
      6c706c5 Delete graduated serialization files
      5d40e14 Remove code that moved to oslo.i18n
      6ff6b4b Switch oslo-incubator to use oslo.utils and remove old modules
      aa74411 log: add missing space in error message
      037dee0 Set stevedore log level to WARN by default
      37c0091 Add unicode coercion of logged messages to ContextFormatter
      6614413 Correct coercion of logged message to unicode
      1188d88 Except socket.error if syslog isn't running
      ac995be Fix E126 pep8 errors
      631f880 Set keystonemiddleware and routes.middleware to log on WARN level
      726d00a Adjust oslo logging to provide adapter is enabled for
      433fa0b Make logging_context_format_string optional in log.set_defaults
      ac92c06 Add default log level for websocket
      5fd77eb Ability to customize default_log_levels for each project
      4d9328c Python 3: enable tests/unit/test_log.py
      cb5a804 Move `mask_password` to strutils
    
    Note: cb5a804 is partially included; that's ok because we don't use the
    moved function in Neutron.
    
    Change-Id: I3bfcaff2620b368d807e9468bb7abc01d6471661

commit c97069dc9a73344ebdc7b686133269850a81b3b2
Author: Abhishek Raut <abhraut@cisco.com>
Date:   Tue Oct 7 17:06:10 2014 -0700

Cisco N1kv: Fix update network profile for add tenants
    
    This patch makes sure that while updating network profile to add new
    tenants, it does not delete the older profile bindings and maintains
    them, while adds only the new tenant ids to the bindings.
    
    Change-Id: I862eb1c400e022334a2f6a4078425448bb144843
    Closes-bug: #1379609

commit 60dd689183469d2958d6dcb60d93a8d94ef694d1
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Oct 16 21:24:07 2014 -0700

DB: Only ask for MAC instead of entire port
    
    Optimize a query in _get_lla_gateway_ip_for_subnet
    to only grab the column used instead of every column
    in the port table.
    
    Partial-Bug: #1373851
    Change-Id: I5257e1e22645f3df9a77c0967b09a0ad0cf8b251

commit 6acadab5eb8b7b627e097a638d8486bef59a7f30
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Oct 16 21:21:15 2014 -0700

Only fetch port_id from SG binding table
    
    Change a query to only retrieve the port_id instead of
    every column from the row of security group binding info.
    
    Partial-Bug: #1373851
    Change-Id: I0fba9c9623898ee52590207ebbb728503bb59a5b

commit fd37ce7d943ab1c2dbc1cf3b6f0187c227f658ad
Author: Han Zhou <zhouhan@gmail.com>
Date:   Thu Oct 16 10:43:12 2014 +0000

nsx plugin: keep old priority when reconnecting bad connection
    
    Change-Id: Id05012ec04d23a5eec8441fc85f87611e08271fd
    Closes-Bug: #1381094

commit ffcb30c4fbee334f9903d2719418bc6aa9d7721c
Author: Isaku Yamahata <isaku.yamahata@intel.com>
Date:   Thu Oct 16 15:31:03 2014 +0900

l3_agent: avoid name conflict with context
    
    module name, context, conflicts with argument name in many place in
    l3_agent. In order to avoid such conflict, import context as n_context
    following Neutron practice.
    
    Change-Id: Ic3754818f84064d2c8da04914826fc912437b2f0
    Closes-Bug: #1381900

commit 7ea605df3ac71dc568194bcd5eaf1c115008e1ee
Author: Numan Siddique <numan.siddique@enovance.com>
Date:   Sat Oct 11 17:38:05 2014 +0530

Fix KeyError in dhcp_rpc when plugin.port_update raise exception
    
    KeyError exception is seen because of following reasons
    
    * DhcpRpcCallback._port_action() is called by two functions
       -  DhcpRpcCallback.create_dchp_port()
       -  DhcpRpcCallback.update_dhcp_port()
    
    * When create_dhcp_port() function calls _port_action(), the
      function argument 'port' will have the body as
      {'port': {'network_id': foo_network_id, 'fixed_ips': [..] ...}
    
    * When update_dhcp_port() function calls _port_action(), the
      function argument 'port' will have the body as
      {'id': port_id, 'port': {{'port': {'network_id': foo_network_id,
    			            'fixed_ips': [..] ...}}
    
    * If an exception occurs when _port_action() calls plugin.create_port(),
      network id is accessed as
      net_id = port['port']['network_id']
    
    * If an exception occurs when _port_action() calls plugin.update_port(),
      network id is accessed as
      net_id = port['port']['network_id']
      which is causing the KeyError. network_id should have been accessed as
      net_id = port['port']['port']['network_id']
    
    This patch fixes the issue by making the _port_action() take the
    same port body. update_dhcp_port() insteading of passing the port_id
    and port information in a single argument, it now adds port_id
    in the port body itself.
    
    Change-Id: I70b92fa20b421b05ca2053a9a57f62db726f7625
    Closes-bug: #1378508

commit acfcb523b15fbd9ccc509e4366e4a141a66d4783
Author: Eugene Nikanorov <enikanorov@mirantis.com>
Date:   Sun Sep 28 21:56:00 2014 +0400

Refactor _make_subnet_dict to avoid issuing unnecessary queries
    
    Use joined loads for attributes dns_nameservers and host_routes.
    
    As a result, particular scenarios like restarting DHCP agent
    could benefit from improved server-side performance.
    
    Change-Id: I6470356b601e2fcf74c7e0a6df438cef7099e9fe
    Closes-Bug: #1374044

commit 105dba9addfdffecea41cce58f3646d2f2a6a6cc
Author: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
Date:   Thu Oct 9 09:04:34 2014 +0900

openvswitch: Remove no longer used options
    
    They are remainders of the recently removed monolithic OVS plugin.
    Note: This change does not touch options used by OVS agent.
    
    Partial-Bug: #1323729
    Change-Id: I6dd04c5c506ff5f97f10ebab03943cd508fbfe60

commit 06fc675928408e462f01178e0158a72f8518188a
Author: Paul Michali <pcm@cisco.com>
Date:   Tue Oct 14 18:18:15 2014 -0400

VPNaaS Cisco unit test clean-up
    
    Removed unused args to mock side-effect function and duplicate
    constant.
    
    Change-Id: I5409ce86ccaab86213d65f757f19c1bdf9a66929
    Closes-Bug: #1381221

commit 4c2b42e21744be56cbf32aeac6f4b4f1c87de24e
Author: Kevin Benton <blak111@gmail.com>
Date:   Sat Oct 11 03:42:47 2014 -0700

Call DVR VMARP notify outside of transaction
    
    The dvr vmarp table update notification was being called inside
    of the delete_port transaction in ML2, which can cause a yield
    and lead to the glorious mysql/eventlet deadlock.
    
    This patch moves it outside the transaction and adjusts it to
    use an existing port dictionary rather than re-looking it up since
    the port is now gone from the DB by the time it is called.
    
    Closes-Bug: #1377241
    Change-Id: I0b4dac61e49b2a926353f8478e421cd1a70be038

commit b4f025be065ea97710604f2f5e67fb4d9a28c16f
Author: Mark McClain <mmcclain@yahoo-inc.com>
Date:   Wed Jun 11 20:44:43 2014 -0400

remove E251 exemption from pep8 check
    
    This change removes the exemption for E251 by addressing unexpected spaces
    around keyword/parameter equals
    
    Change-Id: Iff17477e37bef2a97fc58a538d08bcfc35c67751
    Partial-Bug: 1329017

commit 3cd2163d5105faad389bee5175ef446f0bb90289
Author: Vivekanandan Narasimhan <vivekanandan.narasimhan@hp.com>
Date:   Tue Sep 23 02:25:16 2014 -0700

Race for l2pop when ports go up/down on same host
    
    With l2pop enabled, race exists in delete_port_postcommit
    when both create/update_port and delete_port deal with
    different ports on the same host, where such ports are
    either the first (or) last on same network for that host.
    This race happens outside the DB locking zones in
    the respective methods of ML2 plugin.
    
    To fix this, we have moved determination of
    fdb_entries back to delete_port_postcommit and removed
    delete_port_precommit altogether from l2pop mechanism
    driver.  In order to accomodate dvr interfaces, we
    are storing and re-using the mechanism-driver context
    which hold dvr-port-binding information while
    invoking delete_port_postcommit.  We loop through
    dvr interface bindings invoking delete_port_postcommit
    similar to delete_port_precommit.
    
    Closes-Bug: #1372438
    Change-Id: If0502f57382441fdb4510c81a89794f57a38e696

commit 79f1e8a9c1f308586077483d849e66dcdc83144f
Author: Kevin Benton <blak111@gmail.com>
Date:   Mon Sep 29 19:33:06 2014 -0700

Catch exceptions in router rescheduler
    
    Catch and log exceptions in router rescheduling loop
    rather than just dying which would stop all future
    router rescheduling attempts.
    
    This prevents transient DB connectivity issues from
    permanently breaking the rescheduler until the process
    restarts.
    
    Closes-Bug: #1375597
    Change-Id: I2ab37847074fa6bbdd2b13fd03b8742996dcfc78

commit 50d06130e1ff2a2cfb62510b52efe1537c1068b6
Author: Kevin Benton <blak111@gmail.com>
Date:   Mon Oct 13 23:40:36 2014 -0700

Minor: remove unnecessary intermediate variable
    
    Removes an unnecessary intermediary variable and an
    unnecessary list extend operation that implied previous
    list members where there weren't any. There should be no
    functional change. This just improves readability slightly.
    
    Change-Id: Ice412c29be083d82e055cc0bc45ed8b97c7628d0

commit a1fac106479f9c3c5559f8b2cfbc01fe12d3a575
Author: Elena Ezhova <eezhova@mirantis.com>
Date:   Tue Oct 14 10:12:59 2014 +0400

Handle unused set_context in L3NatTestCaseMixin.floatingip_with_assoc
    
    set_context which is passed to floatingip_with_assoc method
    is not passed further to self._make_floatingip.
    
    Change-Id: Iecf2ad88e4bad5b1f8fd60668401863bdeecce8f
    Closes-Bug: #1378756

commit 3ba06618f79fed899188aac87a8694b3344ee995
Author: Xu Han Peng <xuhanp@cn.ibm.com>
Date:   Fri Jun 20 14:59:53 2014 +0800

Use EUI64 for IPv6 SLAAC when subnet is specified
    
    This commit uses EUI64 for SLAAC and stateless IPv6 address
    when subnet id in fixed_ip is specified.
    
    After this patch, all the ports created on a subnet which has
    ipv6_address_mod=slaac or ipv6_address_mod=dhcpv6-stateless
    will use EUI64 as the address.
    This patch also checks if fixed IP address is specified
    for a IPv6 subnet with address mode slaac or dhcpv6-stateless
    during creating or updating a port. If yes, raise InvalidInput
    error to stop the port creation or update.
    
    Remove unit test test_generated_duplicate_ip_ipv6 because
    fixed_ip should not be specified for a slaac subnet.
    
    Change-Id: Ie481cfb2f4313baf44bf1a838ebda374a5c74c6a
    Closes-Bug: 1330826

commit 5fd628f123e6af1aa96b81f713867edcc67194d7
Author: Sukhdev <sukhdev@aristanetworks.com>
Date:   Wed Sep 24 15:57:15 2014 -0700

Arista L3 Ops is success if it is successful on one peer
    
    This fix checks to see if Arista HW is
    configured in MLAG (redundant) mode. If yes,
    as long as operation is successful on one of the
    paired switches, consider it successful.
    
    Closes-bug: 1373652
    
    Change-Id: If929d3fcc109b81f4cf071380a1645d403feb363

commit 65b57625c24d86ce632d475e2ba743069ae1d797
Author: rossella <rsblendido@suse.com>
Date:   Tue Sep 23 16:09:09 2014 +0000

Add unique constraints in IPAvailabilityRange
    
    first_ip, allocation_pool_id and last_ip, allocation_pool_id
    should be unique in the table.
    These constraints are essential to detect concurrent modifications
    of the IpAvailabilityRange table if the SELECT ... FOR UPDATE
    lock is removed
    
    Change-Id: Iaf2288c0b6bf27e93c03691073d7f505ef24fdd3
    Closes-bug: #1373015

commit 2c5f99a4b1376a46b8c1e1f4e82a015c2d29aa1f
Author: Andrew Boik <drew.boik@gmail.com>
Date:   Fri Oct 10 13:13:45 2014 -0400

Update VPN logging to use new i18n functions
    
    For log messages in neutron/services/vpn and neutron/db/vpn, replace
    _() marker functions with log-level-specific marker functions: _LI(),
    _LW(), _LE() from oslo.i18n.
    
    Also, remove _() functions for debug log messages as debug level log
    messages should not be translated.
    
    Change-Id: I07fcf25bb6344c47e74d6ee23f9bc08e4b560679
    Closes-Bug: #1379811

commit e40c2ed58ce998fc5aba83ffae8ebefec9403c3b
Author: Jacek Swiderski <jacek.swiderski@codilime.com>
Date:   Tue Sep 23 14:35:06 2014 +0200

mock.assert_called_once() is not a valid method
    
    mock.assert_called_once() is a no-op that tests nothing. Instead
    mock.assert_called_once_with() should be used (or use
    assertEqual(1, mock_obj.call_count) if you don't want to check
    parameters).
    
    Borrowed HACKING rule from Davanum Srinivas's nova patch to
    prevent it from appearing again.
    
    Change-Id: Idac1d3c89c07e13c9a209663f4e557fcb7547821
    Closes-Bug: #1365751
    Closes-Bug: #1300265

commit dcfcf31629706758cf7ace61dbf24e59c59f25d5
Author: vikas <vikas.d-m@hp.com>
Date:   Wed Jun 4 02:49:41 2014 -0700

Check for VPN Objects when deleting interfaces
    
    When we delete Router interfaces/gateway,
    we need to check if any VPN services are
    associated with that router.
    
    Closes-Bug:1261598
    
    Change-Id: I7df2b8b130b47ec070d0b0a36b1a62df40532760

commit 648063881ebfbb54a362ef32d58263fc0f20981c
Author: Angus Lees <gus@inodes.org>
Date:   Thu Aug 28 14:37:34 2014 +1000

Compare subnet length as well when deleting DHCP entry
    
    When searching for the DHCP subnet entry to delete, the existing code
    compares only the subnet prefix and ignores subnet length.  This could
    delete the wrong entry/entries if nested subnets are present.
    
    Change-Id: Icf079c42adeca14ef84ec57dc45a5930fde8786d
    Closes-Bug: #1362416

commit 542f5cc6277c6d66bb1f603e174f1254e823f2c9
Author: Angus Lees <gus@inodes.org>
Date:   Wed Aug 20 13:38:14 2014 +1000

Add pylint tox environment and disable all existing warnings
    
    pylintrc update disables all warnings that currently trigger on neutron
    code.  The rough plan is to slowly re-enable warning categories as we
    clean up code in question.
    
    This change also includes a few ultra-trivial syntax cleanups where it
    allowed the check to be immediately enabled for the rest of the
    codebase:
    
    - Added missing trailing newlines in several files
      (db/migration/__init__.py, nuage/{nuagedb,syncmanager,common/config}.py)
    - Renamed self to cls in @classmethods
      (cisco/db/l3/device_handling_db.py)
    - Removed whitespace around '=' in a kwarg
      (cisco/db/l3/device_handling_db.py, cisco/db/n1kv_db_v2.py)
    - Updated deprecated pylint 'disable-msg' directive to newer 'disable'
      (cisco/extensions/qos.py)
    - File-specific disable for too-many-format-args pending further
      investigation of alternatives
      (ml2/drivers/arista/arista_l3_driver.py)
    - Import module rather than object and avoid long line
      (services/l3_router/l3_arista.py)
    
    Change-Id: Ifb0a1a38e33f9073a78658ca578fbd2a42747724

commit 685f6c9a7f4bc3996bf233a51ab3968edff24681
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Sat Oct 11 22:33:01 2014 +0000

Updated from global requirements
    
    Change-Id: Ib0c8c561427f141583d677e86a76cbfe4e2be606

commit b1ca067cda3556ad2c40dce07a072f2a1044fcc1
Author: Yalei Wang <yalei.wang@intel.com>
Date:   Sun Oct 12 01:03:20 2014 +0800

update the relative path of api_extensions_path
    
    the previous path 'unit/extensions' could not be found correctly.
    
    Change-Id: Ic8e658e651e99e57d19f6c3390db89650600decd
    Closes-Bug: 1379711

commit 524981cce05a9b365036c0a1e9810036936d3d5b
Author: ZHU ZHU <zhuzhubj@cn.ibm.com>
Date:   Fri Sep 5 03:01:07 2014 -0500

Reduce security group db calls to neutron server
    
    Within ovs agent daemon loop, prepare_devices_filter will impose heavy workloads
    to neutron server in order to retrieve the security groups message to apply
    firewall rules. If agent is configured to use Noopfirewall driver or security
    groups are disabled, there is no need for loading the rules from server and
    refreshing the firewalls. This will reduce the number of db calls and improve
    performance for neutron server in this case.
    
    Change-Id: Id244aab3cac37fc6ed3dc05cbee91cdf9e34d9cc
    Closes-Bug: #1365806

commit 1423f8cd0f6ca92334660f9588a94bd0a9851b39
Author: armando-migliaccio <armamig@gmail.com>
Date:   Fri Oct 10 15:28:55 2014 -0700

NSX: drop support to deprecated dist-router extension
    
    The NSX plugin originally implemented its own 'dist-router' extension.
    During the Juno timeframe the DVR extension was introduced and the NSX
    plugin was ported to support both. At the same time 'dist-router' was
    marked for removal in Kilo.
    
    Now that Kilo opened, we can drop the deprecated one.
    
    Change-Id: I7d6d2524ef6d5cb94a9e8514425a1a288eea7210

commit f3dd0e823cbdba1cfaa37433aa56d20edd36f75e
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Fri Oct 10 05:12:43 2014 +0000

Avoid constructing a RouterInfo object to get namespace name
    
    Constructing a RouterInfo object just for a string concatenation is
    inefficient and adds more dependence on the class which needs
    refactoring.
    
    Change-Id: Ibaf369d6ebe9285a0c845802def59bfa26ac0fd5

commit 652052860f0572ce1a3e8db56a58e08aca8d9195
Author: Ihar Hrachyshka <ihrachys@redhat.com>
Date:   Fri Oct 10 20:49:57 2014 +0200

Drop sslutils and versionutils modules
    
    The modules are not used since when we've switched to oslo.messaging.
    
    Change-Id: Ife4298e0bb29ec15404fafe4b48545bd65e038e3

commit 7484c9ff93d431c3e9c99d46fea579ecb57bb10b
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Fri Oct 10 06:42:51 2014 +0000

Imported Translations from Transifex
    
    Change-Id: I674acacc96c2396c31658bab48a441dac56988b0

commit afbe668a9c1dd250be9780eb5320652ee6dec21b
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Thu Oct 9 20:33:23 2014 +0000

Remove an argument that is never used
    
    This code was creating a dict with a gw_exists key.  I was curious to
    know who was interested in receiving this information down the line.
    However, no one is ever interested in that key.  It took me some time
    to follow this through wondering what was going on and found only dead
    ends.
    
    Change-Id: I755d9628ab750b950b33f5dcf32ccf2a9b00800e

commit 5e9305a6f934549408a9c18480fc1c000126621e
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Thu Oct 2 20:35:21 2014 +0000

Refactor _process_routers to handle a single router
    
    The method _process_routers no longer handles multiple routers.  The
    only caller of this method would construct a list of exactly one
    router in order to make the call.  This made the for loop unnecessary.
    The method's logic is too heavy for its current purpose.  This commit
    removes much of the weight.
    
    The use of the sets in this method is also no longer necessary.  It
    became clear that all of it boiled down to "if the router is not
    compatible with with this agent but it is known in router_info from
    before then we need to remove it."  This is an exceptional condition
    that shouldn't be handled in this method so I raise an exception and
    handle it in process_router_update where other router removal is
    handled.  Logging was added for this exceptional condition.
    
    The eventlet pool was also obsolete.  It was used to spawn two methods
    and there was a waitall at the end.  The other refactoring made it
    clear that the two spawns were mutually exclusive.  There was only one
    thread spawned for any given invocation of the method and the eventlet
    pool is overkill.
    
    Change-Id: Ibeac591b08565d10b2a9730e25a54f2cd11fc2bc
    Closes-Bug: #1378398

commit 4e8a5b7de71ba6f8c050c424613c025310498940
Author: Mark McClain <mmcclain@yahoo-inc.com>
Date:   Thu Oct 9 13:29:48 2014 +0000

Add Juno release milestone
    
    Change-Id: Iea584b00329d9474c14847db958f8743d4058525
    Closes-Bug: #1378855

commit 93012915a3445a8ac8a0b30b702df30febbbb728
Author: Mark McClain <mmcclain@yahoo-inc.com>
Date:   Wed Oct 8 18:49:20 2014 +0000

Add database relationship between router and ports
    
    Add an explicit schema relationship between a router and its ports. This
    change ensures referential integrity among the entities and prevents orphaned
    ports.
    
    Change-Id: I09e8a694cdff7f64a642a39b45cbd12422132806
    Closes-Bug: #1378866

commit 8923c6fae229c8e3dd85972811989698e482d7f6
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Thu Oct 2 17:02:59 2014 +0000

Remove all_routers argument from _process_routers
    
    There is no code left that passes True to this argument.  It is dead
    code and it should be removed.
    
    Change-Id: I55f71a5c0b96e530e45f2a6463978e8611cbc537

commit dd35d64812fd0ad02e7a00432073d141049e7107
Author: Mark McClain <mmcclain@yahoo-inc.com>
Date:   Wed Oct 8 15:38:19 2014 -0400

update ml2_migration to reflect optional methods
    
    This change conditionally executes the schema methods for versions that
    support them.
    
    Change-Id: I8a51ac04e62dfcfe1e0a2e69a17664d154f0d1d7
    Closes-Bug: #1378732

commit 8a08a3cb47d0dd69d4aa2e8fa661d04054fe95ae
Author: Henry Gessau <gessau@cisco.com>
Date:   Tue Oct 7 20:38:38 2014 -0400

Disable PUT for IPv6 subnet attributes
    
    In Juno we are not ready for allowing the IPv6 attributes on a subnet
    to be updated after the subnet is created, because:
    - The implementation for supporting updates is incomplete.
    - Perceived lack of usefulness, no good use cases known yet.
    - Allowing updates causes more complexity in the code.
    - Have not tested that radvd, dhcp, etc. behave OK after update.
    
    Therefore, for now, we set 'allow_put' to False for the two IPv6
    attributes, ipv6_ra_mode and ipv6_address_mode. This prevents the
    modes from being updated via the PUT:subnets API.
    
    Closes-bug: #1378952
    
    Change-Id: Id6ce894d223c91421b62f82d266cfc15fa63ed0e

commit 8fac816e407dd87d4c940d2a0c873f78933b8cdc
Author: Bradley Jones <jones.bradley@me.com>
Date:   Tue Aug 5 17:55:44 2014 +0100

Do not assume order of lvm.tun_ofports set elements
    
    This fixes the test_fdb_add_flows unit test that breaks with a randomized PYTHONHASHSEED
    (see the bug report).
    
    The test assumed that the lvm.tun_ofports set had
    elements in a particular order. Found with PYTHONHASHSEED=2455351445 and
    1595538922.
    
    The fix sorts the actions output string so that it is always sorted when the
    outputs are compared.
    
    Partial-bug: #1348818
    
    Note: There are several other unrelated unit tests that also break with a
    randomized PYTHONHASHSEED, but they are not addressed here. They will be
    addressed in separate patches.
    
    Change-Id: I86b453a93f3ba09212709caf462cf3bfc5b21ee9

commit 67962943969bc737a3f680a0defc2fc9df03c429
Author: Sean M. Collins <sean_collins2@cable.comcast.com>
Date:   Mon Oct 6 15:47:24 2014 -0400

Skip IPv6 Tests in the OpenContrail plugin
    
    Similar to the way we are skipping tests in the OneConvergence plugin,
    introduced by Kevin Benton in 9294de441e684a81f6e802ba0564083f1ad319d6.
    
    Change-Id: I1650b0708af73ce63e92c55bc842607bb69efe60

commit c6a095b5a9bb6927b8f8c1380b6b4ccf310ac0ea
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Oct 8 06:10:06 2014 +0000

Imported Translations from Transifex
    
    Change-Id: If47b1705e22a7c8b6a84991d6ae0e68d419cba26

commit edb26bfcddf9d9a0e95955a6590d11fa7245ea2b
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Wed Oct 8 03:22:49 2014 +0000

Remove two sets that are not referenced
    
    The code no longer references the updated_routers and removed_routers
    sets.  This should have been cleaned up before but was missed.
    
    Change-Id: I0396e13d2f7c3789928e0c6a4c0a071b02d5ff17

commit abe30e973442271b0093eaa61c59d171a68a2028
Author: Assaf Muller <amuller@redhat.com>
Date:   Tue Oct 7 22:45:41 2014 +0300

Forbid update of HA property of routers
    
    While the HA property is update-able, and resulting router-get
    invocations suggest that the router is HA, the migration
    itself fails on the agent. This is deceiving and confusing
    and should be blocked until the migration itself is fixed
    in a future patch.
    
    Change-Id: I4171ab481e3943e0110bd9a300d965bbebe44871
    Related-Bug: #1365426
    Closes-Bug: #1378525

commit 29250949012e9c0a60b0ddb56ddbf18d7b68106b
Author: Brian Haley <brian.haley@hp.com>
Date:   Fri Oct 3 17:32:01 2014 -0400

Teach DHCP Agent about DVR router interfaces
    
    When DVR is enabled and enable_isolated_metadata=True,
    the DHCP agent should only inject a metadata host route
    when there is no port with the gateway IP address configured
    on the subnet.  Add a check for DEVICE_OWNER_DVR_INTERFACE
    when we look at each port's device_owner field, otherwise
    it will always add this route to the opts file when DVR
    is enabled.
    
    Change-Id: I3ff3bb85105b8215b36535983016d8c0ff3d8cb7
    Closes-bug: #1377307

commit 076a5f27486319139d3d1dfac592284656ef7eb7
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Oct 7 19:13:11 2014 +0000

Updated from global requirements
    
    Change-Id: I0e72933320ac6f49b55ef9782c6c19fb7e997bcb

commit 8f05708eb6d6a5a9cae7a7656873e081aad61eca
Author: Miguel Angel Ajo <mangelajo@redhat.com>
Date:   Fri Sep 19 18:59:58 2014 +0200

Modify the ProcessMonitor class to have one less config parameter
    
    It's a follow up patch, as agreed on the ProcessMonitor review
    patch to coalesce the check_child_processes parameter into
    check_child_process_interval.
    
    When this parameter is set to 0, the feature is disabled.
    
    Change-Id: I2d4d8c6a6b7c17d42d8455c98968f75bcefbb689
    Closes-Bug: 1371705

commit 24e4110eb284078775496501ff81630eb1619c11
Author: Kevin Benton <blak111@gmail.com>
Date:   Tue Oct 7 04:34:41 2014 -0700

Big Switch: Don't clear hash before sync
    
    This patch removes the step of clearing the consistency
    hash from the DB before a topology sync. This will ensure
    that inconsistency will be detected if the topology sync
    fails.
    
    This logic was originally there to make sure the hash header
    was not present on the topology sync call to the backend.
    However, the hash header is ignored by the backend in a sync
    call so it wasn't necessary.
    
    Closes-Bug: #1379510
    Change-Id: I2d58fa2aea3b692834d64192d06ace727c7df8a0

commit d81856cc608660232005b5620f2378249f25e107
Author: Julien Danjou <julien@danjou.info>
Date:   Tue Oct 7 11:16:49 2014 +0200

Remove sslutils from openstack.common
    
    This module has been imported but is not used. Let's remove it.
    
    Change-Id: I0cafdb7ddc00ce58b0724cee293f5dad6f4a1817

commit dea795263eb2968519d4235a8de98329c329e0e8
Author: John Schwarz <jschwarz@redhat.com>
Date:   Tue Sep 23 15:24:47 2014 +0300

Divide _cleanup_namespaces for easy extensibility
    
    This division of the function to 2 different functions allows for
    easier overwriting in the l3 test agent used by the HA functional
    tests, and later by the integration tests.
    
    Related-bug: #1374947
    Change-Id: I8f277759747c8a142f5c023aa3511b00a886348c

commit 467bd9476defbb028f0fbfa26f01f76c32b701cd
Author: John Schwarz <jschwarz@redhat.com>
Date:   Tue Sep 23 14:41:54 2014 +0300

L3 Agent should generate ns_name in a single place
    
    Currently the l3 agent has 2 places where it allows generating ns_name
    of specific router_ids (ie. qrouter-<router_id>): in the RouterInfo's
    constructor, and in _cleanup_namespaces. This patch proposes a
    unification of this creation code with a property which lives in
    RouterInfo's namespace. A simpler fix was also made for snat_ns_name.
    
    This patch also offers a single way to initialize a new RouterInfo.
    
    Related-bug: #1374946
    Related-bug: #1374947
    Change-Id: Ia028236b73a22ff534acee00b46c86b134dc987e

commit ea202faec972d55b169f1c596209b4129aa64caf
Author: Kevin Benton <blak111@gmail.com>
Date:   Fri May 9 21:54:39 2014 -0700

Add comments to iptables rules to help debugging
    
    Adds comments to some of the iptables rules generated
    by neutron to assist with debugging.
    
    DocImpact
    
    Partial-Bug: #1265493
    Change-Id: I7d9e37b9a43589dd7b142869b86fa15cb3fb329c

commit 54c0c950b28cb0e9857e84d7a241b7c89ebc5ba5
Author: mathieu-rohon <mathieu.rohon@gmail.com>
Date:   Thu Oct 2 17:25:16 2014 +0200

nit : missing a "%s" in a log message
    
    Change-Id: I171cc14d7d9b3917dea90cd36dfec7e58afaf626

commit 28660b19f5ef861d7d2ceba04faa7f263bdf8a37
Author: John Schwarz <jschwarz@redhat.com>
Date:   Mon Sep 29 16:28:18 2014 +0300

L3 agent should always use a unique CONF object
    
    The l3 agent accepts an oslo configuration in its constructor and uses
    it throughout the code, but there are some references to the global
    configuration object held by the oslo library. Since HA functional
    tests need to create two agents, the configuration should be consistent
    throughout the code.
    
    The important difference between the agents is their 'host' value so
    that they create different namespaces, and 'state_path' value so the
    agents get their own filesystem root.
    
    Co-Authored-By: Assaf Muller <amuller@redhat.com>
    Related-bug: #1374946
    Change-Id: I3326889482681dd631ca59271ebd6a5d5129ae70

commit e90d6d6c833db51964037afd1c4aa1f521ee380a
Author: John Schwarz <jschwarz@redhat.com>
Date:   Thu Sep 18 11:24:53 2014 +0300

Iterate over same port_id if more than one exists
    
    In certain cases where multiple ports can have the same
    external_ids:iface_id property, the ovs agent will arbitrarily choose
    one and ignore the rest. In case the chosen port isn't on the
    integration bridge the ovs agent is managing, an error is returned to
    the calling function. This is faulty since one of the other ports may
    belong to the correct bridge and it should be chosen instead.
    
    This is interesting for future L3 HA integration tests, where 2
    different instances of l3 agents are needed to run on the same machine.
    In this case, both agents will register ports which have the same
    iface_id property, but obviously only one of the ports is relevant for
    each agent.
    
    Closes-bug: #1370914
    Related-bug: #1374947
    Change-Id: I05d70417e0f78ae51a9ce377fc93a3f12046b313

commit 623aa30b33692f8b1d1d12859281d23163829835
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Oct 2 01:16:46 2014 -0700

Fix setup of Neutron core plugin in VPNaaS UT
    
    One of the VPNaaS unit test setup routines creates
    and extension manager but passes it a class but it
    should be passing an instance of that class.
    
    Change-Id: I589cdda0674fdf8fa20d92c2609e1ba6966125d8

commit 205162f58050fcb94db53cd51b674d2093dfe700
Author: Mark McClain <mmcclain@yahoo-inc.com>
Date:   Wed Sep 24 04:00:54 2014 +0000

remove openvswitch plugin
    
    This changeset removes the openvswitch plugin, but retains the agent for ML2
    The database models were not removed since operators will need to migrate the
    data.
    
    Change-Id: I8b519cb2bbebcbec2c78bb0ec9325716970736cf
    Closes-Bug: 1323729

commit 20dbbab61c1159a3dc6ef838ec65b04f7e9cf560
Author: Miguel Angel Ajo <mangelajo@redhat.com>
Date:   Wed Oct 1 15:16:19 2014 +0200

Fix pid file location to avoid I->J changes that break metadata
    
    Changes in commit 7f8ae630b87392193974dd9cb198c1165cdec93b moved
    pid files handled by agent/linux/external_process.py from
    $state_path/external/<uuid>.pid  to $state_path/external/<uuid>/pid
    that breaks the neutron-ns-metadata-proxy respawn after upgrades
    becase the l3 or dhcp agent can't find the old pid file so
    they try to start a new neutron-ns-metadata-proxy which won't
    succeed, because the old one is holding the port already.
    
    Closes-Bug: #1376128
    Change-Id: Id166ec8e508aaab8eea35d89d010a5a0b7fdba1f

commit 239b2d94339574f63afe0c6df120bfea2974ef7f
Author: Ed Bak <ed.bak2@hp.com>
Date:   Mon Sep 29 14:15:52 2014 -0600

Don't fail when trying to unbind a router
    
    If a router is already unbound from an l3 agent, don't fail.  Log
    the condition and go on.  This is harmless since it can happen
    due to a delete race condition between multiple neutron-server
    processes.  One delete request can determine that it needs to
    unbind the router.  A second process may also determine that it
    needs to unbind the router.  The exception thrown will result
    in a port delete failure and cause nova to mark a deleted instance
    as ERROR.
    
    Change-Id: Ia667ea77a0a483deff8acfdcf90ca84cd3adf44f
    Closes-Bug: 1367892

commit 91683ffeb75fe8899a2925b5490c7c57ffa5f10a
Author: Mark McClain <mmcclain@yahoo-inc.com>
Date:   Wed Sep 24 01:50:06 2014 +0000

remove linuxbridge plugin
    
    This changeset removes the linuxbridge plugin, but retains the agent for ML2.
    The database models were not removed since operators will need to migrate the
    data.
    
    Additionally, the ml2 migration script was altered to support Juno.  For
    testing, a user must either run the migration against the icehouse
    scheme or run the update, manually change alembic_version to juno and
    then run the migration script.  Once the juno migration is added, this
    manually step will not be required.
    
    Change-Id: I70689b4247947e6dc08e80fd9b31da9dc691d259
    Partial-Bug: 1323729

commit c7baaa068ed1d3c8b02717232edef60ba1b655f6
Author: Kevin Benton <blak111@gmail.com>
Date:   Wed Jun 18 12:03:01 2014 -0700

Allow reading a tenant router's external IP
    
    Adds an external IPs field to the external gateway information
    for a router so the external IP address of the router can be
    read by the tenant.
    
    DocImpact
    
    Closes-Bug: #1255142
    Change-Id: If4e77c445e9b855ff77deea6c8df4a0b3cf249d4

commit 8e36ba8d24c198cb5e6c0e4ddc29a08904e3e10c
Author: Samer Deeb <samerd@mellanox.com>
Date:   Wed Oct 1 09:55:21 2014 +0300

Fix sleep function call
    
    Remove time.sleep method reference in class and use time.sleep directly.
    
    Change-Id: Ib4c02061b29c0d584d603746a78ab50922f781c3
    Closes-Bug: 1375698

commit cf92bbead9b25ee40a3336bd76110b3733a4f7ee
Author: Erik Colnick <erik.colnick@hp.com>
Date:   Wed Jun 18 11:31:52 2014 -0600

Add admin tenant name to nova notifier
    
    This change introduces the ability to use the nova admin
    tenant name with the nova notifier in place of the nova
    admin tenant id which may not be available when the neutron
    service is being configured as is the case with tripleo
    installations where the neutron service is configured and
    started before the nova admin tenant has been configured in
    keystone and thus does not have a known id.
    
    DocImpact
    Introduces the nova_admin_tenant_name configuration entry as
    an optional configurable value in neutron.conf.  If the
    nova_admin_tenant_name is configured and the nova_admin_tenanat_id
    is not configured, a performance impact may be seen because
    keystone will become involved in communication between neutron
    and nova.
    
    Change-Id: I33701d4dc9bf61595e44a49050c405ebca779091
    Closes-Bug: #1331566

commit 36e8cbb34e78ff367cb501b8c494d9a02228251d
Author: Kevin Benton <blak111@gmail.com>
Date:   Mon Sep 29 20:21:23 2014 -0700

ML2: move L3 cleanup out of network transaction
    
    Move _process_l3_delete out of the delete_network
    transaction to eliminate the semaphore deadlock that
    occurs when it tries to delete the ports associated
    with existing floating IPs.
    
    It makes more sense to live outside of the transaction
    anyway because the operations it performs cannot be
    rolled back only in the database if the L3 plugin makes
    external calls for floating IP creation/deletion.
    e.g. if delete_floatingip is successful, it may have
    deleted external resources and restoring the DB records
    would make things inconsistent.
    
    If a failure to delete the network does occur, any cleanup
    done by _process_l3_delete will not be reversed.
    
    Closes-Bug: #1374573
    Change-Id: I3ae7bb269df9b9dcef94f48f13f1bde1e4106a80

commit 7b2ca045e846f0580de3d43ffd4595e66ebd6f4e
Author: Thierry Carrez <thierry@openstack.org>
Date:   Tue Sep 30 16:46:54 2014 +0200

Open Kilo development
    
    Bump pre-version to 2015.1 to formally open master branch to Kilo
    development.
    
    Change-Id: Ia48ad29dc1b31b1fbcb14dfffad571659ec66700

commit eacd1fc6b2cb3175f603cdd80d7b59dbe9b0719b
Author: Robert Pothier <rpothier@cisco.com>
Date:   Wed Sep 3 11:09:15 2014 -0400

ML2 Cisco Nexus MD: Fix UT to send one create vlan message
    
    With the commit of https://review.openstack.org/#/c/113009,
    test_nexus_add_trunk needs to have the device_id set to
    a unique value. Combining test_nexus_add_trunk and
    test_nexus_enable_vlan_cmd to reduce duplicate code,
    which fixes the original issue.
    
    Change-Id: I4191a6880ec0f43365aacb638a92a0dd3e1e80aa
    Closes-Bug: 1364976

commit 5247f5cdf15bad4c62bbf854e30716fcf00a1d2a
Author: Ann Kamyshnikova <akamyshnikova@mirantis.com>
Date:   Wed Feb 19 15:39:19 2014 +0400

Implement ModelsMigrationsSync test from oslo.db
    
    Add tests to verify that database migrations produce
    the same schema as the database models.
    
    Also for MySQL, check that all tables are configured to use InnoDB
    as the storage engine.
    These tests make use of the ModelsMigrationsSync test class from
    oslo.db and the load_tests protocol from Python unittest.
    
    Closes-bug: #1346444
    
    Change-Id: Ic0e7eb37c30cc5e94cbdbddf07a6dc1ebf377c17

commit 302331b2197d7adcc7eb0d501718b7c28311acbc
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Sep 30 06:49:06 2014 +0000

Imported Translations from Transifex
    
    Change-Id: Ie7ba4679ca128ff636b6bd841072dfd4b7e4236f

commit c048d3fb480cbd9033152c745802691876b7ba0d
Author: John Kasperski <jckasper@us.ibm.com>
Date:   Thu Sep 25 10:38:45 2014 -0500

Update migration scripts to support DB2
    
    Three of the migration scripts are causing failures with DB2.
    
    - DB2 doesn't support nullable column in primary key
    
    - Hard coded SQL statements which use False/True as Boolean arguments
      are not compatible with DB2. In DB2, Boolean columns are created as
      small integer with a constraint to allow only 0 & 1.
    
    - Hardcoded update rows from other table sql is not compatible with DB2
    
    - Foreign key constraints require additional handling
    
    Co-authored-by: Rahul Priyadarshi <rahul.priyadarshi@in.ibm.com>
    Change-Id: I82e2d1c522b81fed90a1e5cc6f2321f80797cf7b
    Closes-Bug: #1328019

commit 2abb77b4dadd4150f554f134ec99b07c2b28900f
Author: johndavidge <jodavidg@cisco.com>
Date:   Fri Aug 8 04:48:59 2014 -0700

Do not assume order of report list elements
    
    This fixes the test_report_multiple_services unit test that breaks with a
    randomized PYTHONHASHSEED (see the bug report).
    
    The test assumed that the report list from self.driver.report_status() had
    elements in a particular order. Found with PYTHONHASHSEED=2455351445.
    
    The fix refactors the test case to handle an unsorted report list by
    sorting it before checking equality.
    
    Partial-bug: #1348818
    
    Note: There are several other unrelated unit tests that also break with a
    randomized PYTHONHASHSEED, but they are not addressed here. They will be
    addressed in separate patches.
    
    Change-Id: I542c3818821fa2f6e460fd254a3842530ecea8d9

commit ea3a0a428fac308f9ab65d0beb733de380cace56
Author: Koteswara Rao Kelam <koteswara.kelam@hp.com>
Date:   Fri Sep 26 04:34:11 2014 -0700

Disallow unsharing used firewall policy
    
    When admin policy p1 is shared and is used by firewall f1 of different tenant,
    then updating p1 with shared=False should not be allowed as it is in use.
    
    Change-Id: I7c753f9d8a25a7edc40233316398475c8ad3efe9
    Closes-bug: #1334994

commit fd50ff4d243fbd4590b24b96039361504b45b49f
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Sep 29 06:04:31 2014 +0000

Imported Translations from Transifex
    
    Change-Id: I585e3df5504fcc071811e0ec9f925bc493584b2f

commit 9a6c073656a7e0b1a26b2bca0ba381489d04e322
Author: Eugene Nikanorov <enikanorov@mirantis.com>
Date:   Mon Sep 15 22:10:45 2014 +0400

Add missing methods to NoopFirewallDriver
    
    The fix adds missing methods into generic Firewall class
    and in NoopFirewall driver class.
    
    Change-Id: I6402448075ed414434dc007f5c403fc85b6b1456
    Closes-Bug: #1369685
    Related-Bug: #1365806

commit 0d8911115e1b722da2f1e92f444e53b22223ee32
Author: Eugene Nikanorov <enikanorov@mirantis.com>
Date:   Mon Aug 25 00:59:02 2014 +0400

Raise exception if ipv6 prefix is inappropriate for address mode
    
    Address prefix to use with slaac and stateless ipv6 address modes
    should be equal to 64 in order to work properly.
    The patch adds corresponding validation and fixes unit tests
    accordingly.
    
    Change-Id: I6c344b21a69f85f2885a72377171f70309b26775
    Closes-Bug: #1357084

commit 78a2ecb9923417f63c17c1e7055200ac97e6e947
Author: Kevin Benton <blak111@gmail.com>
Date:   Sat Sep 20 00:17:58 2014 -0700

Fix broken port query in Extraroute test case
    
    One of the queries in an extra route test case tries
    to filter based on the port owner, but the _list_ports
    method it calls doesn't take a device_owner parameter.
    This can cause failures if a DHCP port is created on
    the same subnet.
    
    Change-Id: I0e62027ae4d98944ef91a5d457d43d4224441b2f

commit 45a523681f2136f8fefb6c3da44540decd6a0fda
Author: armando-migliaccio <armamig@gmail.com>
Date:   Mon Sep 15 18:40:08 2014 -0700

Revert "Cleanup floatingips also on router delete"
    
    This reverts commit c3326996e38cb67f8d4ba3dabd829dc6f327b666.
    
    The patch being reverted here addresses an issue that can no longer be
    reproduced, in that under no circumstances, I can make the FIP lie around
    before deleting a router (which can only be done after all FIP have been
    disassociated or released).
    
    Unless we have more clarity as to what the initial commit was really meant
    to fix, there is a strong case for reverting this patch at this point.
    
    Closes-bug: #1373100
    
    Change-Id: I7e0f80e456ff4d9eb57a1d31c6ffc7cdfca5a163

commit 5be55ac161adfec9085121fe59ea3bf59daa92cf
Author: Michael Smith <michael.smith6@hp.com>
Date:   Wed Sep 24 10:20:46 2014 -0700

fix dvr snat bindings for external-gw-clear
    
    When router_gateway_clear happens, the
    schedule_router calls the unbind_snat_servicenode
    in the plugin. This will clear the agent binding
    from the binding table. But the l3-agent was
    expecting the ex_gw_port binding to be present.
    The agent needs to check its cache of the
    router['gw_host_port'] value now.
    
    Change-Id: I051fb97d802b0508b30683a33673b85f5ab24000
    Closes-bug: #1373524

commit 3ca85460a178a18dea60399fd369ed84b17721dc
Author: Elena Ezhova <eezhova@mirantis.com>
Date:   Mon Aug 25 18:44:32 2014 +0400

Fix quota limit range validator
    
    In Quota model limit is defined with Integer type which in case of
    MySQL and PostgreSQL is always stored in 4 bytes. At the same time,
    the current validator checks that the value does not exceed sys.maxsize
    which depends on whether the system is 32-bit or 64-bit based.
    
    In SQLite Integer can be stored in 1, 2, 3, 4, 6, or 8 bytes depending
    on the magnitude of the value. Nevertheless, assume that it can not
    exceed 4 bytes for consistency.
    
    Limited the upper bound of the validator with 2**31 - 1.
    
    Added a unit test.
    
    Closes-Bug: #1338479
    Change-Id: Icefa2fc228e4255a022d586cab4590607953d1ee

commit 55f6a8ac5d234f004ef06add87d16284e9f048d3
Author: shihanzhang <shihanzhang@huawei.com>
Date:   Mon Sep 22 17:28:06 2014 +0800

Fix KeyError when getting secgroup info for ports
    
    The patch fixes a regression introduced with secgroup rpc refactor by
    handling the case when security group contains rules for only IPv4 or
    IPv6.
    
    Change-Id: I02b174757bfc796a81cdb482c55ba7f9e954131d
    Closes-bug: #1372337

commit 966645538395079b5337b5ed30d597112279283c
Author: Yong Sheng Gong <gongysh@unitedstack.com>
Date:   Mon Jun 30 15:01:17 2014 +0800

Deletes floating ip related connection states
    
    When a floating ip is dissociated with a port, the current
    connection with the floating ip is still working. This patch
    will clear the connection state and cut off the connection
    immediately.
    
    Since conntrack -D will return 1, which is not an error code,
    so add extra_ok_codes argument to execute methods.
    
    Change-Id: Ia9bd7ae243a0859dcb97e2fa939f7d16f9c2456c
    Closes-Bug: 1334926

commit 4478eee9e55193746b43ad5158be5fee29aef8a3
Author: Brian Haley <brian.haley@hp.com>
Date:   Wed Sep 17 21:48:53 2014 -0400

Do not lookup l3-agent for floating IP if host=None, dvr issue
    
    If a floating IP has been associated with a port, but the port
    has not been associated with an instance, attempting to lookup
    the l3-agent hosting it will cause an AgentNotFoundByTypeHost
    exception.  Just skip it and go onto the next one.
    
    Change-Id: If3df9770fa9e2d2eada932ee5f243d3458bf7261
    Closes-Bug: #1370795

commit 411836b5411411a6046043e0264aaa7b6f5760f0
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Fri Sep 19 17:37:17 2014 +0000

Remove RPC notification from transaction in create/update port
    
    Removing notifications to the L3 agent from within the transaction in
    create_port and update_port eliminates many lock wait timeouts in the
    dvr check queue job and in scale testing locally.
    
    Since this patch leaves context unused in _process_port_binding, the
    argument is removed from the method.
    
    Closes-Bug: #1371732
    
    Change-Id: Ibd86611ad3e7eff085d769bdff777a5870f30c58

commit d5b90315d2ff647179fd91512377a7f21a57ef0a
Author: Jacek Swiderski <jacek.swiderski@codilime.com>
Date:   Wed Aug 6 11:23:16 2014 +0200

Do not assume order of body and tags elements
    
    This fixes the l2gateway unit test that breaks with a randomized PYTHONHASHSEED
    (see the bug report).
    
    The test assumed that the body dict from self._create_expected_req_body
    had elements (including contents of tags list) in a particular order.
    Found with PYTHONHASHSEED=2455351445.
    The fix ensures that body is in predictable order.
    
    Partial-bug: #1348818
    Note: There are several other unrelated unit tests that also break with a
    randomized PYTHONHASHSEED, but they are not addressed here. They will be
    addressed in separate patches.
    
    Change-Id: I423b68aff58486c113d0e5c5f4726f9eabf6920e

commit 739b3cac3210e6a1ce942238ca3efa3ddf49cb4d
Author: Steven Ren <sren@vmware.com>
Date:   Tue Sep 23 02:58:34 2014 -0700

Remove the translation tag for debug level logs in vmware plugin
    
    There is no need to translate the debug level logs in vmware plugin for neutron.
    
    Closes-bug: #1372862
    
    Change-Id: If07c06cc3a1eb3dd147191bb8ab9198d4198cd69

commit e7f0b56d74fbfbb08a3b7a0d2da4cefb6fe2aa67
Author: Derek Higgins <derekh@redhat.com>
Date:   Fri Sep 12 16:31:44 2014 +0100

Retry getting the list of service plugins
    
    On systems that start both neutron-server and neutron-l3-agent together,
    there is a chance that the first call to neutron will timeout. Retry upto
    4 more times to avoid the l3 agent exiting on startup.
    
    This should make the l3 agent a little more robust on startup but still
    not ideal, ideally it wouldn't exit and retry periodically.
    
    Change-Id: I2171a164f3f77bccd89895d73c1c8d67f7190488
    Closes-Bug: #1353953
    Closes-Bug: #1368152
    Closes-Bug: #1368795

commit 7af293ea527d5f904afcc2c3596bb151a5c3c017
Author: Ann Kamyshnikova <akamyshnikova@mirantis.com>
Date:   Tue Sep 23 12:15:05 2014 +0400

Fix entrypoint of OneConvergencePlugin plugin
    
    In setup.cfg entrypoint of OneConvergencePlugin is set incorrectly.
    Used '.' instead of ':'.
    
    Closes-bug: #1372810
    
    Change-Id: Ic1e154cfcf8b13f021b0635238e07a0459a25a9e

commit 74d10939903984d5f06c1749a8707fa3257e44ff
Author: Elena Ezhova <eezhova@mirantis.com>
Date:   Tue Aug 19 15:54:36 2014 +0400

Forbid regular users to reset admin-only attrs to default values
    
    A regular user can reset an admin-only attribute to its default
    value due to the fact that a corresponding policy rule is
    enforced only in the case when an attribute is present in the
    target AND has a non-default value.
    
    Added a new attribute "attributes_to_update" which contains a list
    of all to-be updated attributes to the body of the target that is
    passed to policy.enforce.
    
    Changed a check for whether an attribute is explicitly set.
    Now, in the case of update, the function should not pay attention
    to a default value of an attribute, but check whether it was
    explicitly marked as being updated.
    
    Added unit-tests.
    
    Closes-Bug: #1357379
    Related-Bug: #1338880
    Change-Id: I6537bb1da5ef0d6899bc71e4e949f2c760c103c2

commit 4418e80c117e896743bdd77fcc499bbd85475dc2
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Sep 11 04:26:03 2014 -0700

Finish small unit test refactor of API v2 tests
    
    Leverage a utility function for all of the tests
    that create something in the API and expect a failure.
    
    Change-Id: Iedb6ba35d637dda5ae9f553d0c7ffcb7c526f3c6

commit 32ea2e349decd750e25cb00a8c907b8f73f795f3
Author: Gary Kotton <gkotton@vmware.com>
Date:   Mon Sep 22 10:03:37 2014 -0700

Security groups: prevent race for default security group creation
    
    When a VM is booted via the Nova the client connection is created
    with an admin user. This causes problems when creating the neutron
    port. That is, there may be a race for the creation of the default
    security group for the tenant.
    The problem was introduced by commit acf44dba26ca8dca47bfb5fb2916807f9f4e2060
    
    Change-Id: Ie0199c71231a322704f1f49995facde09c92da25
    Closes-bug: #1372570

commit 2f5aca31b508da9c673c0209ad40efc2b5b2d16d
Author: Preeti Mirji <preeti.mirji@hp.com>
Date:   Wed Jul 23 03:22:30 2014 -0700

Stop admin using other tenants unshared rules
    
    If the firewall rules are not shared and if they belong to different
    tenants, then admin should not be able to create a policy using
    these rules and he should not be able to insert such rules into
    policies. An exception should be raised in such case. Added new
    exception “FirewallRuleConflict” to handle such conditions.
    
    Co-Authored-By: Koteswara Rao Kelam<koteswara.kelam@hp.com>
    Change-Id: I984eb76069bd1493a77bf523bec2bd81abb14abb
    Closes-bug: 1327057

commit c756d3c3ca9e90fab5e1d3ff37af33972747ba70
Author: Kevin L. Mitchell <kevin.mitchell@rackspace.com>
Date:   Mon Sep 22 12:18:11 2014 -0500

Eliminate OrderedDict from test_api_v2.py
    
    Neutron cannot possibly be passing tests under Python 2.6, as
    neutron/tests/unit/test_api_v2.py is referencing
    collections.OrderedDict, which does not exist in Python 2.6.
    Since there is no reason to use an OrderedDict in this case,
    this replaces it with a simple dict.
    
    Change-Id: I1b9886f508c4c8b96cf50c50f157c1960da433fc
    Closes-Bug: #1372571

commit 086496bfc45e01cd2905a074d526a7d513bf4ec2
Author: Kevin Benton <blak111@gmail.com>
Date:   Sat Sep 20 10:48:22 2014 -0700

Mock out all RPC calls with a fixture
    
    Mock out the rpc proxy calls used by various agents to
    prevent unit tests from blocking for 10+ seconds while waiting
    for a timeout. This happened with the OVS agent unit tests
    recently in Change-ID Idd770a85a9eabff112d9613e75d8bb524020234a.
    
    This change results in a reduction from 330.8 seconds to 2.7 seconds
    for the neutron.tests.unit.openvswitch.test_ovs_neutron_agent
    test module.
    
    Closes-Bug: #1372076
    Change-Id: I5e6794dc33c64c8fe309d8e72a8af3385c7d4442

commit 0cc7444f7577e53f14068a2cb35431713d6d21a0
Author: Elena Ezhova <eezhova@mirantis.com>
Date:   Tue Aug 26 19:22:20 2014 +0400

Add logging for enforced policy rules
    
    There are a lot of policy rules which should not necessarily
    be explicitly specified in policy.json to be checked while enforcement.
    There should be a way for an operator to know which policy rules are
    actually being enforced for each action.
    
    Added a unit test.
    
    Change-Id: I261d3e230eced9ea514b35cc3f5f8be04f84c751
    Closes-Bug: #1356679

commit e266c5cacf8debaf130422c7a88764c70cbef2b5
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Sep 22 06:07:23 2014 +0000

Imported Translations from Transifex
    
    Change-Id: If3c928da9856fae84b139c78ee04573daf562a95

commit d1f1722e0edb63c73b60c80abafa63749349cd8e
Author: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
Date:   Mon Sep 22 07:39:27 2014 +0900

ofagent: Drop log level of tenant-triggerable events
    
    To prevent evil tenants from flooding logs.
    
    Closes-Bug: #1372196
    Change-Id: I617e9fedf8a8c5e95c6067b716c83778d4d8cc7e

commit 271a280d0e8f86b96d24970f588184b1cfc34ef8
Author: Divya ChanneGowda <divya.hc@gmail.com>
Date:   Wed Sep 3 14:59:34 2014 -0700

Use dict_extend_functions to populate provider network attributes
    
    Use dict_extend_functions mechanism to handle populating additional
    provider network attributes into Network model in Nuage plugin.
    
    Change-Id: I72e5afe3d03ae223fcd8e75f1f4f07624c3a7daf
    Closes-Bug: #1362308

commit 9cd96fe116ac1d990410cb1135473bd15c1f7dce
Author: Vivekanandan Narasimhan <vivekanandan.narasimhan@hp.com>
Date:   Wed Sep 3 01:48:39 2014 -0700

DVR to delete router namespaces for service ports
    
    Earlier merge that enabled LBaaS in DVR with review #114141
    had not covered the removal of DVR router namespace on
    VIP Port deletion in ml2 plugin.
    
    Here we fix the ml2 plugin to attempt namespace removal for
    all dvr serviced ports.
    
    Change-Id: Ie7930ebedb12212886d45294132fefff7296e104
    Closes-Bug: #1364839

commit f92d1300e1aca68b39b927282af038acc68c5013
Author: armando-migliaccio <armamig@gmail.com>
Date:   Thu Sep 18 11:00:12 2014 -0700

Fix 500 error on retrieving metadata by invalid URI
    
    An invalid URI should return a BadRequest error rather
    than making the metadata proxy bomb out.
    
    Closes-bug: #1371160
    
    Change-Id: Ifb495f9e8929062a0c24d090c3e702109a38803a

commit 2286d220ddb2677c957837381e1827bdfe3b0c8e
Author: Paul Michali <pcm@cisco.com>
Date:   Tue Sep 16 11:22:17 2014 -0400

Rework and enable VPNaaS UT for Cisco CSR REST
    
    The Cisco CSR REST client library unit tests were developed in
    Icehouse, using the httmock library. However, the community did
    not want to add this library to global requirements, as there was
    a similar httpretty library available (albeit with some short-
    comings). As a result, the test module was renamed with a "no"
    prefix, to prevent inclusion in automated tests.
    
    Since then, a new library, requests-mock, has been added to global
    requirements, to replace httpretty, and is being used on several
    other projects.
    
    This commit reworks the unit test to use requests-mock, instead of
    httmock. The functionality is the same, but the mechanism (a
    fixture with URI registration vs context manager) is different.
    
    This commit provides coverage for the REST client code, by using a
    mock for the Cisco CSR VM. The unit test module can be subclassed,
    and used with a real CSR VM, for 3rd party CI testing, in the
    future.
    
    Change-Id: I55c8a253eb32985bc2016ae748b1ded58d021e1a
    Closes-Bug: 1358470

commit 1d4e13574574472e299296892d7c9e8f706aea67
Author: Chirag Shahani <chirag.shahani@gmail.com>
Date:   Wed Sep 10 15:47:48 2014 -0700

Fix to delete user and group association in Nuage Plugin
    
    After a router delete operation, the attached zone to that
    router is also deleted. Got rid of code that tried to do
    a get operation on the nuage_zone after router delete
    operation.
    Closes-Bug: #1367864
    
    Change-Id: I01753a472200a961cdcecee703616fd3239abd3c

commit 34cf04a0ea84f4e5b32aa8d45108bd2a38bcff58
Author: Kevin Benton <blak111@gmail.com>
Date:   Tue Sep 16 17:29:51 2014 -0700

Delete DB records instead of tables to speedup UT
    
    Now that the schema is fixed for all of the plugins,
    there isn't a need to delete and recreate the entire
    schema for every unit test.
    
    This patch clears the tables at the end of each test
    instead of deleting them. This eliminated overhead seems
    to save 10%+ execution time of the entire set of unit
    tests.
    
    Example of performance gain from tox -epy27 tests.unit.ml2:
    Before: Ran 2495 tests in 284.186s
    After: Ran 2495 tests in 223.299s
    
    Change-Id: Ic5bcbb0cf941e0745890abc776d719e58bb42e35

commit dd5c73450da908d95724e20eb3be09bc936cb551
Author: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
Date:   Fri Aug 15 14:20:00 2014 +0900

ofagent: Ignore unknown l2pop entry removals
    
    l2pop can send us entry removal without the corresponding addition.
    Gracefully ignore it instead of crashing.
    
    Closes-Bug: #1357198
    Change-Id: I5a0cc44ba62faf15d6fe3730a9532a3826647820

commit 0e4141644df45f25c229c65e66c57132a93b446c
Author: Carl Baldwin <carl.baldwin@hp.com>
Date:   Tue Feb 11 00:58:42 2014 +0000

Rename workers to api_workers and simplify code
    
    Refactor a few ugly aspects of the multiple API worker patch to make
    way for multiple rpc workers.  This came up as I was trying to add
    multiple RPC workers using similar patterns and remembering that some
    things were left in a rather awkward state.
    
    Change-Id: I549db67af4af6a2df80e12cf233109dda5213c47

commit 4e1c4cbcd71833095534bea9ff7617c582c300d2
Author: Abhishek Raut <abhraut@cisco.com>
Date:   Fri Sep 12 19:56:40 2014 -0700

Cisco N1kv: Remove vmnetwork delete REST call on last port delete
    
    Remove the vm network delete call to the VSM on final port call and
    ensure that vm network is deleted from the database when the port count
    becomes 0.
    
    Change-Id: I6c08a099adfce2fdba8eefec6aadeb68a780ac37
    Closes-Bug: 1373547

commit da4433e1af75a7ad0a03b3e48e7b12af9a4d0e55
Author: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
Date:   Thu Sep 4 13:06:21 2014 +0900

ofagent: Fix a possible crash in arp responder
    
    Be careful for exceptions when feeding packet-in data,
    which is generated by tenant VMs and thus can not be trusted,
    to Ryu packet library.
    
    Closes-Bug: #1365255
    Change-Id: Ia8bacfb55def563a1b23a47709ae72bd4fce0fce

commit c5ae5ad2637a561ccbc7484045e99d8140822b8d
Author: Eugene Nikanorov <enikanorov@mirantis.com>
Date:   Fri Sep 12 09:05:39 2014 +0400

Properly handle empty before/after notifications in l2pop code
    
    Change-Id: I8644bb7cc2afb3b181397a478f96927990c0a4ca
    Closes-Bug: #1367881

commit 5e4c14e850e2757bf74a806aaaab4c053bab741d
Author: Sam Betts <sam@code-smash.net>
Date:   Tue Aug 26 18:35:36 2014 +0100

Added TAP_DEVICE_PREFIX info to common/constants
    
    Change-Id: Ia84629732490585164237ca4f7d1db90bde9fbf2
    Closes-Bug: 1361573

commit f8e10b5f6a8b5a4d4dac845c316db23d6f67c347
Author: Assaf Muller <amuller@redhat.com>
Date:   Tue Jul 1 16:18:35 2014 +0300

Pythonified sanity_check.all_tests_passed
    
    Change-Id: I48986344b7af507e2526d4e8bd1b03277f974972

commit 424c7faa75d96950d80f49f20f5414d1a297af72
Author: Ihar Hrachyshka <ihrachys@redhat.com>
Date:   Thu Aug 7 22:27:23 2014 +0200

Removed kombu from requirements
    
    Since we've replaced oslo-incubator RPC layer with oslo.messaging, we
    don't ship any code that uses kombu.
    
    Change-Id: Ia8a74f1326ecd98c47cbe447f04d475bf61e19d3

commit b4bf668ec40ee6a329740b587fa4a651a6e0dca2
Author: Hareesh Puthalath <hareesh.puthalath@gmail.com>
Date:   Tue Sep 2 06:05:37 2014 +0200

Supply missing cisco_cfg_agent.ini file
    
    cisco_cfg_agent.ini file was missed in the initial commit and
    caused neutron startup issues. This patch supplies the proper
    ini file and adds it back to neutron setup.cfg.
    Also the introduced config options are put in a specific group
    instead of default as was in the initial commit.
    
    Change-Id: I74b3b77fe6e196524809580f522f91f3b62f5ba7
    Closes-bug: #1351466

commit fbec4d6784100214856b7500ccfb2925d8f6e6dc
Author: Jeremy Stanley <fungi@yuggoth.org>
Date:   Wed Sep 3 19:04:06 2014 +0000

Work toward Python 3.4 support and testing
    
    Change-Id: Ieae7819e0489ae19b7fcfb63497e06527c918b5c

commit 903e2a8cd1dd9169048d1ad9dd8a566b2ae52395
Author: Cédric Ollivier <ollivier.cedric@gmail.com>
Date:   Thu Aug 21 20:26:10 2014 +0200

Add unit tests covering single operations to ODL
    
    This commit adds the remaining test cases (create and update
    operations) to fully cover sync_single_resource. It also defines the
    filter_* methods as static or class methods and removes their duplicate
    arguments.
    
    Change-Id: I6b6153ea577bd685576690559b1c389490ee525d
    Closes-Bug: #1325184

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-27: Fix merged to neutron (master)

#35

Reviewed: https://review.openstack.org/128855
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f23f2ecee68ba4abd12139bbb91b77ba9410f581
Submitter: Jenkins
Branch: master

commit f23f2ecee68ba4abd12139bbb91b77ba9410f581
Author: Kevin Benton <email address hidden>
Date: Thu Oct 16 01:49:19 2014 -0700

_update_router_db: don't hold open transactions

    This patch prevents the L3 _update_router_db method from
    starting a transaction before calling the gateway interface
    removal functions. With these port changes now occuring
    outside of the L3 DB transaction, a failure to update the
    router DB information will not rollback the port deletion
    operation.

    The 'VPN in use' check had to be moved inside of the DB deletion
    transaction now that there isn't an enclosing transaction to undo
    the delete when an 'in use' error is raised.

===Details===

    The router update db method starts a transaction and calls
    the gateway update method with the transaction held open.
    This becomes a problem when the update results in an
    interface removal which uses a port table lock.

    Because the delete_port caller is still holding open a
    transaction, other sessions are blocked from getting an
    SQL lock on the same tables when delete_port starts
    performing RPC notifications, external controller calls,
    etc. During those external calls, eventlet will
    yield and another thread may try to get a lock on the
    port table, causing the infamous mysql/eventlet deadlock.

    This separation of L2/L3 transactions is similiar to change
    I3ae7bb269df9b9dcef94f48f13f1bde1e4106a80 in nature. Even
    though there is a loss in the atomic behavior of the interface
    removal operation, it was arguably incorrect to begin with.
    The restoration of port DB records during a rollback after some
    other failure doesn't undo the backend operations (e.g. REST calls)
    that happened during the original deletion. So, having a delete
    rollback without corresponding 'create_port' calls to the backend
    causes a loss in consistency.

Closes-Bug: #1377241
Change-Id: I5fdb6b24bf2fb80ac5e36a742aa7056db72c8c7d

Reviewed:  https://review.openstack.org/128855
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f23f2ecee68ba4abd12139bbb91b77ba9410f581
Submitter: Jenkins
Branch:    master

commit f23f2ecee68ba4abd12139bbb91b77ba9410f581
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Oct 16 01:49:19 2014 -0700

_update_router_db: don't hold open transactions
    
    This patch prevents the L3 _update_router_db method from
    starting a transaction before calling the gateway interface
    removal functions. With these port changes now occuring
    outside of the L3 DB transaction, a failure to update the
    router DB information will not rollback the port deletion
    operation.
    
    The 'VPN in use' check had to be moved inside of the DB deletion
    transaction now that there isn't an enclosing transaction to undo
    the delete when an 'in use' error is raised.
    
    ===Details===
    
    The router update db method starts a transaction and calls
    the gateway update method with the transaction held open.
    This becomes a problem when the update results in an
    interface removal which uses a port table lock.
    
    Because the delete_port caller is still holding open a
    transaction, other sessions are blocked from getting an
    SQL lock on the same tables when delete_port starts
    performing RPC notifications, external controller calls,
    etc. During those external calls, eventlet will
    yield and another thread may try to get a lock on the
    port table, causing the infamous mysql/eventlet deadlock.
    
    This separation of L2/L3 transactions is similiar to change
    I3ae7bb269df9b9dcef94f48f13f1bde1e4106a80 in nature. Even
    though there is a loss in the atomic behavior of the interface
    removal operation, it was arguably incorrect to begin with.
    The restoration of port DB records during a rollback after some
    other failure doesn't undo the backend operations (e.g. REST calls)
    that happened during the original deletion. So, having a delete
    rollback without corresponding 'create_port' calls to the backend
    causes a loss in consistency.
    
    Closes-Bug: #1377241
    Change-Id: I5fdb6b24bf2fb80ac5e36a742aa7056db72c8c7d

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-27: Fix proposed to neutron (stable/juno)

#36

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/131189

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-13: Fix merged to neutron (stable/juno)

#37

Reviewed: https://review.openstack.org/131189
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a575cb7cb6c30d843c3ac824e5f2f11a5055a9fb
Submitter: Jenkins
Branch: stable/juno

commit a575cb7cb6c30d843c3ac824e5f2f11a5055a9fb
Author: Kevin Benton <email address hidden>
Date: Thu Oct 16 01:49:19 2014 -0700

_update_router_db: don't hold open transactions

    This patch prevents the L3 _update_router_db method from
    starting a transaction before calling the gateway interface
    removal functions. With these port changes now occuring
    outside of the L3 DB transaction, a failure to update the
    router DB information will not rollback the port deletion
    operation.

    The 'VPN in use' check had to be moved inside of the DB deletion
    transaction now that there isn't an enclosing transaction to undo
    the delete when an 'in use' error is raised.

===Details===

    The router update db method starts a transaction and calls
    the gateway update method with the transaction held open.
    This becomes a problem when the update results in an
    interface removal which uses a port table lock.

    Because the delete_port caller is still holding open a
    transaction, other sessions are blocked from getting an
    SQL lock on the same tables when delete_port starts
    performing RPC notifications, external controller calls,
    etc. During those external calls, eventlet will
    yield and another thread may try to get a lock on the
    port table, causing the infamous mysql/eventlet deadlock.

    This separation of L2/L3 transactions is similiar to change
    I3ae7bb269df9b9dcef94f48f13f1bde1e4106a80 in nature. Even
    though there is a loss in the atomic behavior of the interface
    removal operation, it was arguably incorrect to begin with.
    The restoration of port DB records during a rollback after some
    other failure doesn't undo the backend operations (e.g. REST calls)
    that happened during the original deletion. So, having a delete
    rollback without corresponding 'create_port' calls to the backend
    causes a loss in consistency.

Conflicts:

neutron/db/l3_db.py

    Closes-Bug: #1377241
    Change-Id: I5fdb6b24bf2fb80ac5e36a742aa7056db72c8c7d
    (cherry picked from commit f23f2ecee68ba4abd12139bbb91b77ba9410f581)

Reviewed:  https://review.openstack.org/131189
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a575cb7cb6c30d843c3ac824e5f2f11a5055a9fb
Submitter: Jenkins
Branch:    stable/juno

commit a575cb7cb6c30d843c3ac824e5f2f11a5055a9fb
Author: Kevin Benton <blak111@gmail.com>
Date:   Thu Oct 16 01:49:19 2014 -0700

_update_router_db: don't hold open transactions
    
    This patch prevents the L3 _update_router_db method from
    starting a transaction before calling the gateway interface
    removal functions. With these port changes now occuring
    outside of the L3 DB transaction, a failure to update the
    router DB information will not rollback the port deletion
    operation.
    
    The 'VPN in use' check had to be moved inside of the DB deletion
    transaction now that there isn't an enclosing transaction to undo
    the delete when an 'in use' error is raised.
    
    ===Details===
    
    The router update db method starts a transaction and calls
    the gateway update method with the transaction held open.
    This becomes a problem when the update results in an
    interface removal which uses a port table lock.
    
    Because the delete_port caller is still holding open a
    transaction, other sessions are blocked from getting an
    SQL lock on the same tables when delete_port starts
    performing RPC notifications, external controller calls,
    etc. During those external calls, eventlet will
    yield and another thread may try to get a lock on the
    port table, causing the infamous mysql/eventlet deadlock.
    
    This separation of L2/L3 transactions is similiar to change
    I3ae7bb269df9b9dcef94f48f13f1bde1e4106a80 in nature. Even
    though there is a loss in the atomic behavior of the interface
    removal operation, it was arguably incorrect to begin with.
    The restoration of port DB records during a rollback after some
    other failure doesn't undo the backend operations (e.g. REST calls)
    that happened during the original deletion. So, having a delete
    rollback without corresponding 'create_port' calls to the backend
    causes a loss in consistency.
    
    Conflicts:
    
    	neutron/db/l3_db.py
    
    Closes-Bug: #1377241
    Change-Id: I5fdb6b24bf2fb80ac5e36a742aa7056db72c8c7d
    (cherry picked from commit f23f2ecee68ba4abd12139bbb91b77ba9410f581)

tags:

added: in-stable-juno

Thierry Carrez (ttx) on 2014-12-18

Changed in neutron:
milestone:	none → kilo-1
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-26: Change abandoned on neutron (master)

#38

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/128134
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Thierry Carrez (ttx) on 2015-04-30

Changed in neutron:
milestone:	kilo-1 → 2015.1.0

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Kevin Benton	neutron 2015.1.0 "kilo"
	Juno	Fix Released	Undecided	Unassigned	neutron 2014.2.1

neutron

Lock wait timeout on delete port for DVR

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches