Build : 4.0.1.0-80
CoreLocation : /cs-shared/test_runs/nodem6/jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-170
cores : {'10.204.216.160': [], '10.204.216.98': [], '10.204.216.99': [], '10.204.216.96': [], '10.204.216.97': [], '10.204.216.95': [], '10.204.216.103': []}
LogsLocation : http://10.204.216.50/Docs/logs/4.0.1.0-80_jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-170_1505674488.87/logs/
Report : http://10.204.216.50/Docs/logs/4.0.1.0-80_jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-170_1505674488.87/junit-noframes.html
Topology :
DISTRO : "Ubuntu 14.04.5 LTS"
SKU : mitaka
Config Nodes : [u'nodem6', u'nodem7', u'nodem14']
Control Nodes : [u'nodem6', u'nodem7', u'nodem14']
Compute Nodes : [u'nodem8', u'nodem9', u'nodem10']
Openstack Node : [u'nodem6', u'nodem7', u'nodem14']
WebUI Node : [u'nodem6', u'nodem7', u'nodem14']
Analytics Nodes : [u'nodem6', u'nodem7', u'nodem14']
Database Nodes : [u'nodem6', u'nodem7', u'nodem14']
Physical Devices : [u'blr-mx1', u"'blr-mx1'"]
The testcase test_verify_object_logs failed here : http://10.204.216.50/Docs/logs/4.0.1.0-80_jenkins-SMLite_ubuntu-14-04_mitaka_Openstack_HA_Sanity-170_1505674488.87/logs//analyticsbasictestsanity.html#test_verify_object_logs
nova-api log had this :
All logs will be in http://10.204.216.50/Docs/bugs/#
2017-09-17 22:07:45.859 18370 DEBUG nova.compute.api [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] Going to run 1 instances... _provision_instances /usr/lib/python2.7/dist-packages/nova/compute/api.py:956
2017-09-17 22:07:45.870 18370 DEBUG oslo_db.sqlalchemy.engines [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
2017-09-17 22:07:45.888 18370 DEBUG nova.quota [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] Rolled back reservations [u'a9ffd834-989f-4817-84ff-94b08be2568f', u'fe712c45-4561-4a3d-b7e4-72a98231401d', u'4438b3b6-8815-4dbe-b62f-0cfd33ddadb1'] rollback /usr/lib/python2.7/dist-packages/nova/quota.py:1395
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] Unexpected exception in API method
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions **create_kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1556, in create
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1169, in _create_instance
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota, filter_properties)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1027, in _provision_instances
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions quotas.rollback()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions self.force_reraise()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 969, in _provision_instances
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions req_spec.create()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/request_spec.py", line 432, in create
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions db_spec = self._create_in_db(self._context, updates)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 709, in wrapper
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return fn(*args, **kwargs)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/request_spec.py", line 403, in _create_in_db
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions db_spec.save(context.session)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions session.flush()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2027, in flush
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions self._flush(objects)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2145, in _flush
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions transaction.rollback(_capture_exception=True)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions compat.reraise(exc_type, exc_value, exc_tb)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2109, in _flush
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions flush_context.execute()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions rec.execute(self)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions uow
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions mapper, table, insert)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 800, in _emit_insert_statements
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions execute(statement, params)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return meth(self, multiparams, params)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions return connection._execute_clauseelement(self, multiparams, params)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions compiled_sql, distilled_params
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions context)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions cursor.execute(statement, parameters)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 158, in execute
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions result = self._query(query)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 308, in _query
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions conn.query(q)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 820, in query
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1002, in _read_query_result
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions result.read()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1285, in read
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions first_packet = self.connection._read_packet()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions packet.check_error()
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions err.raise_mysql_exception(self._data)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions _check_mysql_exception(errinfo)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions raise errorclass(errno, errorvalue)
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '3' for key 'PRIMARY'") [SQL: u'INSERT INTO request_specs (created_at, updated_at, instance_uuid, spec) VALUES (%(created_at)s, %(updated_at)s, %(instance_uuid)s, %(spec)s)'] [parameters: {'instance_uuid': 'e576f405-d095-429e-b683-042fa53f214f', 'created_at': datetime.datetime(2017, 9, 17, 16, 37, 45, 874204), 'spec': '{"nova_object.version": "1.5", "nova_object.changes": ["instance_uuid", "retry", "num_instances", "pci_requests", "limits", "availability_zone", "force_nodes", "image", "instance_group", "force_hosts", "numa_topology", "flavor", "project_id", "scheduler_hints", "ignore_hosts"], "nova_object.name": "RequestSpec", "nova_object.data": {"instance_uuid": "e576f405-d095-429e-b683-042fa53f214f", "retry": null, "num_instances": 1, "pci_requests": {"nova_object.version": "1.1", "nova_object.changes": ["requests"], "nova_object.name": "InstancePCIRequests", "nova_object.data": {"requests": []}, "nova_object.namespace": "nova"}, "limits": {"nova_object.version": "1.0", "nova_object.changes": ["vcpu", "memory_mb", "numa_topology", "disk_gb"], "nova_object.name": "SchedulerLimits", "nova_object.data": {"vcpu": null, "memory_mb": null, "disk_gb": null, "numa_topology": null}, "nova_object.namespace": "nova"}, "availability_zone": "nova", "force_nodes": null, "image": {"nova_object.version": "1.8", "nova_object.changes": ["status", "name", "container_format", "created_at", "disk_format", "updated_at", "id", "owner", "min_ram", "checksum", "min_disk", "properties", "size"], "nova_object.name": "ImageMeta", "nova_object.data": {"status": "active", "created_at": "2017-09-17T16:26:25Z", "name": "ubuntu", "container_format": "bare", "min_ram": 0, "disk_format": "vmdk", "updated_at": "2017-09-17T16:26:47Z", "id": "053e2ce9-588e-49eb-867b-7a2f4edb00c7", "min_disk": 0, "checksum": "72676bb770dcc0b6cee8576518c211ad", "owner": "e775a24754c74c2c9fc7700389c36941", "properties": {"nova_object.version": "1.12", "nova_object.changes": ["hw_disk_type", "hw_disk_bus"], "nova_object.name": "ImageMetaProps", "nova_object.data": {"hw_disk_type": "sparse", "hw_disk_bus": "ide"}, "nova_object.namespace": "nova"}, "size": 948043776}, "nova_object.namespace": "nova"}, "instance_group": null, "force_hosts": ["nodem10"], "numa_topology": null, "ignore_hosts": null, "flavor": {"nova_object.version": "1.1", "nova_object.name": "Flavor", "nova_object.data": {"disabled": false, "root_gb": 1, "name": "contrail_flavor_tiny", "flavorid": "2532a7a8-a142-4227-8ff5-5d8efba77262", "deleted": false, "created_at": "2017-09-17T16:37:43Z", "ephemeral_gb": 0, "updated_at": null, "memory_mb": 512, "vcpus": 1, "extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "is_public": true, "deleted_at": null, "vcpu_weight": 0, "id": 6}, "nova_object.namespace": "nova"}, "project_id": "58b3e6cba93745189ff34bffa47d2d69", "scheduler_hints": {}}, "nova_object.namespace": "nova"}', 'updated_at': None}]
2017-09-17 22:07:45.888 18370 ERROR nova.api.openstack.extensions
2017-09-17 22:07:45.895 18370 INFO nova.api.openstack.wsgi [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBDuplicateEntry'>
2017-09-17 22:07:45.895 18370 DEBUG nova.api.openstack.wsgi [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBDuplicateEntry'> __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1070
2017-09-17 22:07:45.896 18370 INFO nova.osapi_compute.wsgi.server [req-c37664c8-88da-499a-ae87-59393a8e633b c240e61737654e99938b015cea8313fa 58b3e6cba93745189ff34bffa47d2d69 - - -] 10.10.10.14 "POST /v2/servers HTTP/1.1" status: 500 len: 415 time: 1.3934150
problem consistently seen from build79