Unexpected API error: Server fault - nova.exception.GlanceConnectionFailed

Bug #1818886 reported by Rafael Folco on 2019-03-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Arx Cruz

Bug Description

tripleo-ci-centos-7-undercloud-containers
master
gate
tempest.api.compute.admin.test_quotas_negative.QuotasAdminNegativeTest.test_create_server_when_memory_quota_is_full

http://logs.openstack.org/21/640921/2/gate/tripleo-ci-centos-7-undercloud-containers/d97eeb1/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-03-06_17_05_19

2019-03-06 17:05:15 | {0} tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_show_non_existent_security_group_rule [0.030783s] ... ok
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | ==============================
2019-03-06 17:05:19 | Failed 1 tests - output below:
2019-03-06 17:05:19 | ==============================
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | tempest.api.compute.admin.test_quotas_negative.QuotasAdminNegativeTest.test_create_server_when_memory_quota_is_full[id-6fdd7012-584d-4327-a61c-49122e0d5864,negative]
2019-03-06 17:05:19 | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | Captured traceback:
2019-03-06 17:05:19 | ~~~~~~~~~~~~~~~~~~~
2019-03-06 17:05:19 | Traceback (most recent call last):
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_quotas_negative.py", line 81, in test_create_server_when_memory_quota_is_full
2019-03-06 17:05:19 | self.create_test_server)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 422, in assertRaises
2019-03-06 17:05:19 | self.assertThat(our_callable, matcher)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
2019-03-06 17:05:19 | mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 483, in _matchHelper
2019-03-06 17:05:19 | mismatch = matcher.match(matchee)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
2019-03-06 17:05:19 | mismatch = self.exception_matcher.match(exc_info)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
2019-03-06 17:05:19 | mismatch = matcher.match(matchee)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 414, in match
2019-03-06 17:05:19 | reraise(*matchee)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
2019-03-06 17:05:19 | result = matchee()
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 969, in __call__
2019-03-06 17:05:19 | return self._callable_object(*self._args, **self._kwargs)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 259, in create_test_server
2019-03-06 17:05:19 | **kwargs)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 191, in create_test_server
2019-03-06 17:05:19 | **kwargs)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 95, in create_server
2019-03-06 17:05:19 | resp, body = self.post('servers', post_body)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 279, in post
2019-03-06 17:05:19 | return self.request('POST', url, extra_headers, headers, body, chunked)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2019-03-06 17:05:19 | method, url, extra_headers, headers, body, chunked)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 670, in request
2019-03-06 17:05:19 | self._error_checker(resp, resp_body)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 847, in _error_checker
2019-03-06 17:05:19 | message=message)
2019-03-06 17:05:19 | tempest.lib.exceptions.ServerFault: Got server fault
2019-03-06 17:05:19 | Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2019-03-06 17:05:19 | <class 'nova.exception.GlanceConnectionFailed'>
2019-03-06 17:05:19 |
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | Captured pythonlogging:
2019-03-06 17:05:19 | ~~~~~~~~~~~~~~~~~~~~~~~
2019-03-06 17:05:19 | 2019-03-06 16:57:02,586 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 200 GET http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783 0.594s
2019-03-06 17:05:19 | 2019-03-06 16:57:02,587 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: None
2019-03-06 17:05:19 | Response - Headers: {'status': '200', u'content-length': '371', 'content-location': 'http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783', u'x-compute-request-id': 'req-8b6b62a4-3db5-49f3-b137-9b5bb3e2b9fd', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:01 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-8b6b62a4-3db5-49f3-b137-9b5bb3e2b9fd'}
2019-03-06 17:05:19 | Body: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": 51200, "floating_ips": 10, "key_pairs": 100, "id": "cc54888ef0fd4005a3c5f032ba425783", "instances": 10, "security_group_rules": 20, "injected_files": 5, "cores": 20, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": 10}}
2019-03-06 17:05:19 | 2019-03-06 16:57:03,430 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 200 PUT http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783 0.838s
2019-03-06 17:05:19 | 2019-03-06 16:57:03,431 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: {"quota_set": {"ram": 0, "force": true}}
2019-03-06 17:05:19 | Response - Headers: {'status': '200', u'content-length': '325', 'content-location': 'http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783', u'x-compute-request-id': 'req-463eadb5-9a0b-4de8-a6c2-b8168c8b459d', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:02 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-463eadb5-9a0b-4de8-a6c2-b8168c8b459d'}
2019-03-06 17:05:19 | Body: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": 0, "floating_ips": 10, "key_pairs": 100, "instances": 10, "security_group_rules": 20, "injected_files": 5, "cores": 20, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": 10}}
2019-03-06 17:05:19 | 2019-03-06 16:57:04,189 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 201 POST https://192.168.24.2:13000/v3/auth/tokens
2019-03-06 17:05:19 | 2019-03-06 16:57:04,190 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
2019-03-06 17:05:19 | Body: <omitted>
2019-03-06 17:05:19 | Response - Headers: {'status': '201', u'content-length': '8731', 'content-location': 'https://192.168.24.2:13000/v3/auth/tokens', u'x-subject-token': '<omitted>', u'vary': 'X-Auth-Token', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 06 Mar 2019 16:57:03 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-9cd45c0f-1c9e-42cc-ae61-290eb9213fc1'}
2019-03-06 17:05:19 | Body: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "33c8578c9f6a45a2b12eb1a51e20f581", "name": "member"}, {"id": "bcf56553d40941cbb2d4f8823c098c79", "name": "reader"}], "expires_at": "2019-03-06T20:57:04.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cc54888ef0fd4005a3c5f032ba425783", "name": "tempest-QuotasAdminNegativeTest-1109713642"}, "catalog": [{"endpoints": [{"region_id": "regionOne", "url": "https://192.168.24.2:13005/v1", "region": "regionOne", "interface": "public", "id": "287c6788c3b148d892a24f9d8f6c9275"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8000/v1", "region": "regionOne", "interface": "admin", "id": "477e4b4284524178877cb5aae39d39aa"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8000/v1", "region": "regionOne", "interface": "internal", "id": "6756cadcfd5741628433c2cb2388686a"}], "type": "cloudformation", "id": "02141abbae704f75b9a93c988eb4b0c1", "name": "heat-cfn"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne", "interface": "internal", "id": "5db5780d156d473aa449c15a07508b37"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13774/v2.1", "region": "regionOne", "interface": "public", "id": "bc88fd9f4ccf4b7c9d4c036882a0766f"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne", "interface": "admin", "id": "e3067d071ecf41269be2897d52d4bd12"}], "type": "compute", "id": "225840c0ac7a4519b2b2a53749d4aad4", "name": "nova"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8989/v2", "region": "regionOne", "interface": "internal", "id": "a427600399064beebd483291099b75f2"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13989/v2", "region": "regionOne", "interface": "public", "id": "c02263560dca441eb87002b61432b823"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8989/v2", "region": "regionOne", "interface": "admin", "id": "cf59cfdd9c2d4dbea7f40b06cec62601"}], "type": "workflowv2", "id": "3c351c795cec49729e9da4aa0db92775", "name": "mistral"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8004/v1/cc54888ef0fd4005a3c5f032ba425783", "region": "regionOne", "interface": "admin", "id": "af2011eee7a5477ea4021c82579b01ce"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8004/v1/cc54888ef0fd4005a3c5f032ba425783", "region": "regionOne", "interface": "internal", "id": "bcf41c0482ba451ea6667f2fa599e59a"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13004/v1/cc54888ef0fd4005a3c5f032ba425783", "region": "regionOne", "interface": "public", "id": "d571ff80b7d24f5ab8946523a2a30e22"}], "type": "orchestration", "id": "4e5de06a01d24cd0bc2a158e01b7cc83", "name": "heat"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne", "interface": "internal", "id": "595e51a1293e425fb089b89850b86166"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13050", "region": "regionOne", "interface": "public", "id": "6dc82862ff034e00b5a9ec56dbeaf793"}, {"region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne", "interface": "admin", "id": "f80bb5b6dca6452da0821ae6fb6646a6"}], "type": "baremetal-introspection", "id": "79ad1bb08be1432ba69676cdd839ff2b", "name": "ironic-inspector"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne", "interface": "admin", "id": "d1d2e2f5e75e4e0c9c4456b01596952c"}, {"region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne", "interface": "internal", "id": "ec42a1f4e0644c7ca6345a88f62f6d7c"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13696", "region": "regionOne", "interface": "public", "id": "f5f8286bbdfb40a984bce226ca964d2f"}], "type": "network", "id": "8b563261f64f4e4688f4fb317e54e77d", "name": "neutron"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne", "interface": "internal", "id": "393efe9c9ce74f9580eb649edb844d90"}, {"region_id": "reg
2019-03-06 17:05:19 | 2019-03-06 16:57:04,192 66 INFO [tempest.lib.common.fixed_network] (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full) Found network None available for tenant
2019-03-06 17:05:19 | 2019-03-06 16:57:04,264 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 500 POST http://192.168.24.3:8774/v2.1/servers 0.070s
2019-03-06 17:05:19 | 2019-03-06 16:57:04,265 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: {"server": {"flavorRef": "c9b9c7d0-6aab-4832-9fc7-1346ab0711f0", "name": "tempest-QuotasAdminNegativeTest-server-839402335", "imageRef": "aded4f57-d533-4b83-8f35-bbcbe4e2f457"}}
2019-03-06 17:05:19 | Response - Headers: {'status': '500', u'content-length': '211', 'content-location': 'http://192.168.24.3:8774/v2.1/servers', u'x-compute-request-id': 'req-bc9592ab-5999-4f0b-9fb9-8b1226fa8a7e', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:04 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'req-bc9592ab-5999-4f0b-9fb9-8b1226fa8a7e'}
2019-03-06 17:05:19 | Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.GlanceConnectionFailed'>", "code": 500}}
2019-03-06 17:05:19 | 2019-03-06 16:57:04,987 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:_run_cleanups): 200 PUT http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783 0.703s
2019-03-06 17:05:19 | 2019-03-06 16:57:04,989 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: {"quota_set": {"ram": 51200}}
2019-03-06 17:05:19 | Response - Headers: {'status': '200', u'content-length': '329', 'content-location': 'http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783', u'x-compute-request-id': 'req-a56df927-09bb-457a-aa21-64bc885573dd', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:04 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-a56df927-09bb-457a-aa21-64bc885573dd'}
2019-03-06 17:05:19 | Body: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": 51200, "floating_ips": 10, "key_pairs": 100, "instances": 10, "security_group_rules": 20, "injected_files": 5, "cores": 20, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": 10}}
2019-03-06 17:05:19 |
2019-03-06 17:05:19 |
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | ======
2019-03-06 17:05:19 | Totals
2019-03-06 17:05:19 | ======
2019-03-06 17:05:19 | Ran: 370 tests in 551.0000 sec.
2019-03-06 17:05:19 | - Passed: 333
2019-03-06 17:05:19 | - Skipped: 36
2019-03-06 17:05:19 | - Expected Fail: 0
2019-03-06 17:05:19 | - Unexpected Success: 0
2019-03-06 17:05:19 | - Failed: 1
2019-03-06 17:05:19 | Sum of execute time for each test: 914.5143 sec.
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | ==============
2019-03-06 17:05:19 | Worker Balance
2019-03-06 17:05:19 | ==============
2019-03-06 17:05:19 | - Worker 0 (97 tests) => 0:08:47.421099
2019-03-06 17:05:19 | - Worker 1 (116 tests) => 0:08:24.240674
2019-03-06 17:05:19 | - Worker 2 (80 tests) => 0:07:15.029259
2019-03-06 17:05:19 | - Worker 3 (77 tests) => 0:05:35.882546

Tags: ci Edit Tag help

Reviewed: https://review.openstack.org/641455
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=2c655258ed600d182870fbe78f610c6c087b4ec6
Submitter: Zuul
Branch: master

commit 2c655258ed600d182870fbe78f610c6c087b4ec6
Author: Rafael Folco <email address hidden>
Date: Wed Mar 6 15:06:11 2019 -0300

    Add test_create_server_when_memory_quota_is_full to skip list

    While we trace root cause on Bug #1818886, we need this test in skip
    list as it is failing on gate pipeline.

    Change-Id: I9add83a5ef0f3c46835d6bf6944483cb818ad87a
    Related-Bug: #1818886

Changed in tripleo:
milestone: stein-3 → stein-rc1
wes hayutin (weshayutin) on 2019-03-18
Changed in tripleo:
status: Triaged → Invalid
status: Invalid → Triaged
Rafael Folco (rafaelfolco) wrote :

Arx, can you please check if we want to re-enable this test from skip_list ?
Thanks

Changed in tripleo:
milestone: stein-rc1 → train-1

Reviewed: https://review.openstack.org/651730
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=3dabc41d032eeb8ab23b1f36d099b1757f2259d2
Submitter: Zuul
Branch: master

commit 3dabc41d032eeb8ab23b1f36d099b1757f2259d2
Author: Arx Cruz <email address hidden>
Date: Thu Apr 11 09:01:39 2019 +0000

    Remove test_create_server_when_memory_quota_is_full from skiplist

    This test is passing in latest featureset021 run

    Change-Id: Ie03687ac9bf75453a20e97e31665d57308cc7dd4
    Related-Bug: 1818886

Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers