Affects: http://jenkins-product.srt.mirantis.net:8080/view/6.0.1/job/6.0.1.test_staging_mirror/46/
6.0.1.staging.centos.bvt_1 fails due to problems with db update ports:
2015-02-10T03:40:49.832210+00:00 err: 2015-02-10 03:40:49.732 5567 ERROR oslo.messaging.rpc.dispatcher [req-8e392160-f937-4f99-9e8e-e00be2de326a ] Exception during message handling: UPDATE statement on table 'ports' expected to
update 1 row(s); 0 were matched.
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/rpc.py", line 138, in update_device_down
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher host))
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 1112, in update_port_status
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher original_port['network_id'])
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 553, in get_network
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher result = super(Ml2Plugin, self).get_network(context, id, None)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 946, in get_network
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher network = self._get_network(context, id)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 92, in _get_network
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher network = self._get_by_id(context, models_v2.Network, id)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/neutron/db/common_db_mixin.py", line 125, in _get_by_id
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher return query.filter(model.id == id).one()
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2369, in one
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher ret = list(self)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2411, in __iter__
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher self.session._autoflush()
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1198, in _autoflush
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher self.flush()
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher self._flush(objects)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher transaction.rollback(_capture_exception=True)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher flush_context.execute()
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher rec.execute(self)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher uow
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 60, in save_obj
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher mapper, table, update)
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 536, in _emit_update_statements
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher (table.description, len(update), rows))
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher StaleDataError: UPDATE statement on table 'ports' expected to update 1 row(s); 0 were matched.
2015-02-10 03:40:49.732 5567 TRACE oslo.messaging.rpc.dispatcher
2015-02-10T03:40:49.838482+00:00 err: 2015-02-10 03:40:49.835 5567 ERROR oslo.messaging._drivers.common [req-8e392160-f937-4f99-9e8e-e00be2de326a ] Returning exception UPDATE statement on table 'ports' expected to update 1 row(s
); 0 were matched. to caller
2015-02-10T03:40:49.839079+00:00 err: 2015-02-10 03:40:49.836 5567 ERROR oslo.messaging._drivers.common [req-8e392160-f937-4f99-9e8e-e00be2de326a ] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packag
es/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch\n return self._do_dispatch(
endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.6/si
te-packages/neutron/plugins/ml2/rpc.py", line 138, in update_device_down\n host))\n', ' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 1112, in update_port_status\n original_port[\'network_id\'
])\n', ' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 553, in get_network\n result = super(Ml2Plugin, self).get_network(context, id, None)\n', ' File "/usr/lib/python2.6/site-packages/neutron/d
b/db_base_plugin_v2.py", line 946, in get_network\n network = self._get_network(context, id)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 92, in _get_network\n network = self._get_by_id(context, models_v2.Network, id)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/common_db_mixin.py", line 125, in _get_by_id\n return query.filter(model.id == id).one()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2369, in one\n ret = list(self)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2411, in __iter__\n self.session._autoflush()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1198, in _autoflush\n self.flush()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1919, in flush\n self._flush(objects)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush\n flush_context.execute()\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute\n rec.execute(self)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute\n uow\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 60, in save_obj\n mapper, table, update)\n', ' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 536, in _emit_update_statements\n (table.description, len(update), rows))\n', "StaleDataError: UPDATE statement on table 'ports' expected to update 1 row(s); 0 were matched.\n"]
And:
2015-02-10 03:38:15 DEBUG (client) Error message: {"NeutronError": {"message": "Port c8e93ba7-2358-47a5-8f73-f322beba9f02 could not be found", "type": "PortNotFound", "detail": ""}}
2015-02-10 03:38:15 DEBUG (test_mixins) Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 186, in verify
result = func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/fuel_health/neutronmanager.py", line 117, in _remove_network
self.neutron_client.delete_network(network['id'])
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
ret = self.function(instance, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 359, in delete_network
return self.delete(self.network_path % (network))
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1316, in delete
headers=headers, params=params)
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
headers=headers, params=params)
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1251, in do_request
self._handle_fault_response(status_code, replybody)
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1216, in _handle_fault_response
exception_handler_v20(status_code, des_error_body)
File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 66, in exception_handler_v20
status_code=status_code)
PortNotFoundClient: Port c8e93ba7-2358-47a5-8f73-f322beba9f02 could not be found
It looks like there were two "conflicting" requests processed req-cfddc1ca- e348-44a5- 8b3e-25ba0d0fbf d7 and req-8e392160- f937-4f99- 9e8e-e00be2de32 6a with device 992e81c4- 0869-4edc- b8b5-ae01cde51 which was deleted by req-cf and failed to update by pastebin. com/9YY6nGRC
req-8e
Here is a flow: http://