Steps:
Deploy ceph HA with RadosGW for objects with Ubuntu
Scenario:
Create cluster with Neutron
Add 3 nodes with controller role
Add 3 nodes with compute and ceph-osd role
Deploy the cluster
Check ceph status
Run OSTF tests
Check the radosqw daemon is started
Actual result:
OSTF test for neutron failed on attempts to delete network.:
urllib3.connectionpool: DEBUG: "DELETE http://10.109.2.3:9696/v2.0/networks/d11d89a7-2344-45da-9b5e-156af050f142.json HTTP/1.1" 500 150
neutronclient.client: DEBUG: RESP:500 CaseInsensitiveDict({'content-length': '150', 'via': '1.1 apache_api_proxy:9696', 'server': 'Apache', 'connection': 'close', 'date': 'Mon, 19 Oct 2015 05:40:42 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-5061cc97-142e-4ba2-8917-c01b9c810d37'}) {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
neutronclient.v2_0.client: DEBUG: Error message: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
on node-2 for this network next error:
2015-10-19T05:40:42.093718+00:00 warning: Unable to retrieve the agent ip, check the agent configuration.
2015-10-19T05:40:42.149647+00:00 debug: Deleting port 61f84635-3065-4371-8d50-c212383e9627
2015-10-19T05:40:42.149647+00:00 debug: Notify callbacks for port, before_delete
2015-10-19T05:40:42.149647+00:00 debug: Calling callback neutron.db.l3_db._prevent_l3_port_delete_callback
2015-10-19T05:40:42.249992+00:00 err: Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 705, in _delete_ports\n self.delete_port(context, port.id)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1324, in delete_port\n network = self.get_network(context, port[\'network_id\'])\n', ' File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 683, in get_network\n result = super(Ml2Plugin, self).get_network(context, id, None)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 391, in get_network\n network = self._get_network(context, id)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_common.py", line 186, in _get_network\n network = self._get_by_id(context, models_v2.Network, id)\n', ' File "/usr/lib/python2.7/dist-packages/neutron/db/common_db_mixin.py", line 184, in _get_by_id\n return query.filter(model.id == id).one()\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2473, in one\n ret = list(self)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__\n return self._execute_and_instances(context)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances\n result = conn.execute(querycontext.statement, self._params)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute\n return meth(self, multiparams, params)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement\n compiled_sql, distilled_params\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context\n context)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context\n context)\n', ' File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute\n self.errorhandler(self, exc, value)\n', ' File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler\n raise errorvalue\n', "DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'SELECT networks.tenant_id AS networks_tenant_id, networks.id AS networks_id, networks.name AS networks_name, networks.status AS networks_status, networks.admin_state_up AS networks_admin_state_up, networks.mtu AS networks_mtu, networks.vlan_transparent AS networks_vlan_transparent, ipallocationpools_1.id AS ipallocationpools_1_id, ipallocationpools_1.subnet_id AS ipallocationpools_1_subnet_id, ipallocationpools_1.first_ip AS ipallocationpools_1_first_ip, ipallocationpools_1.last_ip AS ipallocationpools_1_last_ip, dnsnameservers_1.address AS dnsnameservers_1_address, dnsnameservers_1.subnet_id AS dnsnameservers_1_subnet_id, dnsnameservers_1.`order` AS dnsnameservers_1_order, subnetroutes_1.destination AS subnetroutes_1_destination, subnetroutes_1.nexthop AS subnetroutes_1_nexthop, subnetroutes_1.subnet_id AS subnetroutes_1_subnet_id, subnets_1.tenant_id AS subnets_1_tenant_id, subnets_1.id AS subnets_1_id, subnets_1.name AS subnets_1_name, subnets_1.network_id AS subnets_1_network_id, subnets_1.subnetpool_id AS subnets_1_subnetpool_id, subnets_1.ip_version AS subnets_1_ip_version, subnets_1.cidr AS subnets_1_cidr, subnets_1.gateway_ip AS subnets_1_gateway_ip, subnets_1.enable_dhcp AS subnets_1_enable_dhcp, subnets_1.ipv6_ra_mode AS subnets_1_ipv6_ra_mode, subnets_1.ipv6_address_mode AS subnets_1_ipv6_address_mode, networkrbacs_1.tenant_id AS networkrbacs_1_tenant_id, networkrbacs_1.id AS networkrbacs_1_id, networkrbacs_1.target_tenant AS networkrbacs_1_target_tenant, networkrbacs_1.action AS networkrbacs_1_action, networkrbacs_1.object_id AS networkrbacs_1_object_id, externalnetworks_1.network_id AS externalnetworks_1_network_id, qos_network_policy_bindings_1.policy_id AS qos_network_policy_bindings_1_policy_id, qos_network_policy_bindings_1.network_id AS qos_network_policy_bindings_1_network_id \\nFROM networks LEFT OUTER JOIN externalnetworks ON networks.id = externalnetworks.network_id LEFT OUTER JOIN subnets AS subnets_1 ON networks.id = subnets_1.network_id LEFT OUTER JOIN ipallocationpools AS ipallocationpools_1 ON subnets_1.id = ipallocationpools_1.subnet_id LEFT OUTER JOIN dnsnameservers AS dnsnameservers_1 ON subnets_1.id = dnsnameservers_1.subnet_id LEFT OUTER JOIN subnetroutes AS subnetroutes_1 ON subnets_1.id = subnetroutes_1.subnet_id LEFT OUTER JOIN networkrbacs AS networkrbacs_1 ON networks.id = networkrbacs_1.object_id LEFT OUTER JOIN externalnetworks AS externalnetworks_1 ON networks.id = externalnetworks_1.network_id LEFT OUTER JOIN qos_network_policy_bindings AS qos_network_policy_bindings_1 ON networks.id = qos_network_policy_bindings_1.network_id \\nWHERE networks.id = %s ORDER BY dnsnameservers_1.`order`'] [parameters: ('d11d89a7-2344-45da-9b5e-156af050f142',)]\n"]
2015-10-19T05:40:42.268628+00:00 err: delete failed
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
openstack_version: "2015.1.0-8.0"
api: "1.0"
build_number: "44"
build_id: "44"
fuel-nailgun_sha: "11da6f8d8a18f318b8b7316f29a878c0ac0bfee7"
python-fuelclient_sha: "93a2e63d2a035c3f6f8ac2f7c3ba22b89696bbef"
fuel-agent_sha: "7e30d14a59199441965879d61991e4c0a0a697b1"
fuel-nailgun-agent_sha: "08e0a11cf1f29b705e4b910d9b9db5e9b708b6e3"
astute_sha: "a090546d43c770ac27ca81c6f8c78ff0ba4a93e0"
fuel-library_sha: "a0044771e18b404de1346e2a995fd215dddb5806"
fuel-ostf_sha: "17a093c05b972c2dd47f2c57775086d96caa5dfb"
fuel-createmirror_sha: "fe81731b76071dc0a4ffdf039080732243f5a871"
fuelmain_sha: "a2d78bdf7b37d8f2658c9f0f1f43708bdf85f8cd"
https://product-ci.infra.mirantis.net/view/8.0/job/8.0.ubuntu.bvt_2/40/
This is a duplicate of bug #1503289 since deadlocks occur under the same test cases.