tripleo

Unexpected API error: Server fault - nova.exception.GlanceConnectionFailed

Bug #1818886 reported by Rafael Folco
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Arx Cruz
tripleo train-rc1

Bug Description

tripleo-ci-centos-7-undercloud-containers
master
gate
tempest.api.compute.admin.test_quotas_negative.QuotasAdminNegativeTest.test_create_server_when_memory_quota_is_full

http://logs.openstack.org/21/640921/2/gate/tripleo-ci-centos-7-undercloud-containers/d97eeb1/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-03-06_17_05_19

2019-03-06 17:05:15 | {0} tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_show_non_existent_security_group_rule [0.030783s] ... ok
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | ==============================
2019-03-06 17:05:19 | Failed 1 tests - output below:
2019-03-06 17:05:19 | ==============================
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | tempest.api.compute.admin.test_quotas_negative.QuotasAdminNegativeTest.test_create_server_when_memory_quota_is_full[id-6fdd7012-584d-4327-a61c-49122e0d5864,negative]
2019-03-06 17:05:19 | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | Captured traceback:
2019-03-06 17:05:19 | ~~~~~~~~~~~~~~~~~~~
2019-03-06 17:05:19 | Traceback (most recent call last):
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/api/compute/admin/test_quotas_negative.py", line 81, in test_create_server_when_memory_quota_is_full
2019-03-06 17:05:19 | self.create_test_server)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 422, in assertRaises
2019-03-06 17:05:19 | self.assertThat(our_callable, matcher)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
2019-03-06 17:05:19 | mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 483, in _matchHelper
2019-03-06 17:05:19 | mismatch = matcher.match(matchee)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
2019-03-06 17:05:19 | mismatch = self.exception_matcher.match(exc_info)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
2019-03-06 17:05:19 | mismatch = matcher.match(matchee)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 414, in match
2019-03-06 17:05:19 | reraise(*matchee)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
2019-03-06 17:05:19 | result = matchee()
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 969, in __call__
2019-03-06 17:05:19 | return self._callable_object(*self._args, **self._kwargs)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 259, in create_test_server
2019-03-06 17:05:19 | **kwargs)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 191, in create_test_server
2019-03-06 17:05:19 | **kwargs)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 95, in create_server
2019-03-06 17:05:19 | resp, body = self.post('servers', post_body)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 279, in post
2019-03-06 17:05:19 | return self.request('POST', url, extra_headers, headers, body, chunked)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2019-03-06 17:05:19 | method, url, extra_headers, headers, body, chunked)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 670, in request
2019-03-06 17:05:19 | self._error_checker(resp, resp_body)
2019-03-06 17:05:19 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 847, in _error_checker
2019-03-06 17:05:19 | message=message)
2019-03-06 17:05:19 | tempest.lib.exceptions.ServerFault: Got server fault
2019-03-06 17:05:19 | Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2019-03-06 17:05:19 | <class 'nova.exception.GlanceConnectionFailed'>
2019-03-06 17:05:19 |
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | Captured pythonlogging:
2019-03-06 17:05:19 | ~~~~~~~~~~~~~~~~~~~~~~~
2019-03-06 17:05:19 | 2019-03-06 16:57:02,586 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 200 GET http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783 0.594s
2019-03-06 17:05:19 | 2019-03-06 16:57:02,587 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: None
2019-03-06 17:05:19 | Response - Headers: {'status': '200', u'content-length': '371', 'content-location': 'http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783', u'x-compute-request-id': 'req-8b6b62a4-3db5-49f3-b137-9b5bb3e2b9fd', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:01 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-8b6b62a4-3db5-49f3-b137-9b5bb3e2b9fd'}
2019-03-06 17:05:19 | Body: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": 51200, "floating_ips": 10, "key_pairs": 100, "id": "cc54888ef0fd4005a3c5f032ba425783", "instances": 10, "security_group_rules": 20, "injected_files": 5, "cores": 20, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": 10}}
2019-03-06 17:05:19 | 2019-03-06 16:57:03,430 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 200 PUT http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783 0.838s
2019-03-06 17:05:19 | 2019-03-06 16:57:03,431 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: {"quota_set": {"ram": 0, "force": true}}
2019-03-06 17:05:19 | Response - Headers: {'status': '200', u'content-length': '325', 'content-location': 'http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783', u'x-compute-request-id': 'req-463eadb5-9a0b-4de8-a6c2-b8168c8b459d', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:02 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-463eadb5-9a0b-4de8-a6c2-b8168c8b459d'}
2019-03-06 17:05:19 | Body: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": 0, "floating_ips": 10, "key_pairs": 100, "instances": 10, "security_group_rules": 20, "injected_files": 5, "cores": 20, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": 10}}
2019-03-06 17:05:19 | 2019-03-06 16:57:04,189 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 201 POST https://192.168.24.2:13000/v3/auth/tokens
2019-03-06 17:05:19 | 2019-03-06 16:57:04,190 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
2019-03-06 17:05:19 | Body: <omitted>
2019-03-06 17:05:19 | Response - Headers: {'status': '201', u'content-length': '8731', 'content-location': 'https://192.168.24.2:13000/v3/auth/tokens', u'x-subject-token': '<omitted>', u'vary': 'X-Auth-Token', u'server': 'Apache', u'connection': 'close', u'date': 'Wed, 06 Mar 2019 16:57:03 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-9cd45c0f-1c9e-42cc-ae61-290eb9213fc1'}
2019-03-06 17:05:19 | Body: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "33c8578c9f6a45a2b12eb1a51e20f581", "name": "member"}, {"id": "bcf56553d40941cbb2d4f8823c098c79", "name": "reader"}], "expires_at": "2019-03-06T20:57:04.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cc54888ef0fd4005a3c5f032ba425783", "name": "tempest-QuotasAdminNegativeTest-1109713642"}, "catalog": [{"endpoints": [{"region_id": "regionOne", "url": "https://192.168.24.2:13005/v1", "region": "regionOne", "interface": "public", "id": "287c6788c3b148d892a24f9d8f6c9275"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8000/v1", "region": "regionOne", "interface": "admin", "id": "477e4b4284524178877cb5aae39d39aa"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8000/v1", "region": "regionOne", "interface": "internal", "id": "6756cadcfd5741628433c2cb2388686a"}], "type": "cloudformation", "id": "02141abbae704f75b9a93c988eb4b0c1", "name": "heat-cfn"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne", "interface": "internal", "id": "5db5780d156d473aa449c15a07508b37"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13774/v2.1", "region": "regionOne", "interface": "public", "id": "bc88fd9f4ccf4b7c9d4c036882a0766f"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne", "interface": "admin", "id": "e3067d071ecf41269be2897d52d4bd12"}], "type": "compute", "id": "225840c0ac7a4519b2b2a53749d4aad4", "name": "nova"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8989/v2", "region": "regionOne", "interface": "internal", "id": "a427600399064beebd483291099b75f2"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13989/v2", "region": "regionOne", "interface": "public", "id": "c02263560dca441eb87002b61432b823"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8989/v2", "region": "regionOne", "interface": "admin", "id": "cf59cfdd9c2d4dbea7f40b06cec62601"}], "type": "workflowv2", "id": "3c351c795cec49729e9da4aa0db92775", "name": "mistral"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8004/v1/cc54888ef0fd4005a3c5f032ba425783", "region": "regionOne", "interface": "admin", "id": "af2011eee7a5477ea4021c82579b01ce"}, {"region_id": "regionOne", "url": "http://192.168.24.3:8004/v1/cc54888ef0fd4005a3c5f032ba425783", "region": "regionOne", "interface": "internal", "id": "bcf41c0482ba451ea6667f2fa599e59a"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13004/v1/cc54888ef0fd4005a3c5f032ba425783", "region": "regionOne", "interface": "public", "id": "d571ff80b7d24f5ab8946523a2a30e22"}], "type": "orchestration", "id": "4e5de06a01d24cd0bc2a158e01b7cc83", "name": "heat"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne", "interface": "internal", "id": "595e51a1293e425fb089b89850b86166"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13050", "region": "regionOne", "interface": "public", "id": "6dc82862ff034e00b5a9ec56dbeaf793"}, {"region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne", "interface": "admin", "id": "f80bb5b6dca6452da0821ae6fb6646a6"}], "type": "baremetal-introspection", "id": "79ad1bb08be1432ba69676cdd839ff2b", "name": "ironic-inspector"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne", "interface": "admin", "id": "d1d2e2f5e75e4e0c9c4456b01596952c"}, {"region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne", "interface": "internal", "id": "ec42a1f4e0644c7ca6345a88f62f6d7c"}, {"region_id": "regionOne", "url": "https://192.168.24.2:13696", "region": "regionOne", "interface": "public", "id": "f5f8286bbdfb40a984bce226ca964d2f"}], "type": "network", "id": "8b563261f64f4e4688f4fb317e54e77d", "name": "neutron"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne", "interface": "internal", "id": "393efe9c9ce74f9580eb649edb844d90"}, {"region_id": "reg
2019-03-06 17:05:19 | 2019-03-06 16:57:04,192 66 INFO [tempest.lib.common.fixed_network] (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full) Found network None available for tenant
2019-03-06 17:05:19 | 2019-03-06 16:57:04,264 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:test_create_server_when_memory_quota_is_full): 500 POST http://192.168.24.3:8774/v2.1/servers 0.070s
2019-03-06 17:05:19 | 2019-03-06 16:57:04,265 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: {"server": {"flavorRef": "c9b9c7d0-6aab-4832-9fc7-1346ab0711f0", "name": "tempest-QuotasAdminNegativeTest-server-839402335", "imageRef": "aded4f57-d533-4b83-8f35-bbcbe4e2f457"}}
2019-03-06 17:05:19 | Response - Headers: {'status': '500', u'content-length': '211', 'content-location': 'http://192.168.24.3:8774/v2.1/servers', u'x-compute-request-id': 'req-bc9592ab-5999-4f0b-9fb9-8b1226fa8a7e', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:04 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'req-bc9592ab-5999-4f0b-9fb9-8b1226fa8a7e'}
2019-03-06 17:05:19 | Body: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.GlanceConnectionFailed'>", "code": 500}}
2019-03-06 17:05:19 | 2019-03-06 16:57:04,987 66 INFO [tempest.lib.common.rest_client] Request (QuotasAdminNegativeTest:_run_cleanups): 200 PUT http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783 0.703s
2019-03-06 17:05:19 | 2019-03-06 16:57:04,989 66 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
2019-03-06 17:05:19 | Body: {"quota_set": {"ram": 51200}}
2019-03-06 17:05:19 | Response - Headers: {'status': '200', u'content-length': '329', 'content-location': 'http://192.168.24.3:8774/v2.1/os-quota-sets/cc54888ef0fd4005a3c5f032ba425783', u'x-compute-request-id': 'req-a56df927-09bb-457a-aa21-64bc885573dd', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Wed, 06 Mar 2019 16:57:04 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-a56df927-09bb-457a-aa21-64bc885573dd'}
2019-03-06 17:05:19 | Body: {"quota_set": {"injected_file_content_bytes": 10240, "metadata_items": 128, "server_group_members": 10, "server_groups": 10, "ram": 51200, "floating_ips": 10, "key_pairs": 100, "instances": 10, "security_group_rules": 20, "injected_files": 5, "cores": 20, "fixed_ips": -1, "injected_file_path_bytes": 255, "security_groups": 10}}
2019-03-06 17:05:19 |
2019-03-06 17:05:19 |
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | ======
2019-03-06 17:05:19 | Totals
2019-03-06 17:05:19 | ======
2019-03-06 17:05:19 | Ran: 370 tests in 551.0000 sec.
2019-03-06 17:05:19 | - Passed: 333
2019-03-06 17:05:19 | - Skipped: 36
2019-03-06 17:05:19 | - Expected Fail: 0
2019-03-06 17:05:19 | - Unexpected Success: 0
2019-03-06 17:05:19 | - Failed: 1
2019-03-06 17:05:19 | Sum of execute time for each test: 914.5143 sec.
2019-03-06 17:05:19 |
2019-03-06 17:05:19 | ==============
2019-03-06 17:05:19 | Worker Balance
2019-03-06 17:05:19 | ==============
2019-03-06 17:05:19 | - Worker 0 (97 tests) => 0:08:47.421099
2019-03-06 17:05:19 | - Worker 1 (116 tests) => 0:08:24.240674
2019-03-06 17:05:19 | - Worker 2 (80 tests) => 0:07:15.029259
2019-03-06 17:05:19 | - Worker 3 (77 tests) => 0:05:35.882546

Tags: ci
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/641455

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/641455
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=2c655258ed600d182870fbe78f610c6c087b4ec6
Submitter: Zuul
Branch: master

commit 2c655258ed600d182870fbe78f610c6c087b4ec6
Author: Rafael Folco <email address hidden>
Date: Wed Mar 6 15:06:11 2019 -0300

    Add test_create_server_when_memory_quota_is_full to skip list

    While we trace root cause on Bug #1818886, we need this test in skip
    list as it is failing on gate pipeline.

    Change-Id: I9add83a5ef0f3c46835d6bf6944483cb818ad87a
    Related-Bug: #1818886

Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: stein-3 → stein-rc1
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Invalid
status: Invalid → Triaged
Revision history for this message
Rafael Folco (rafaelfolco) wrote :

Arx, can you please check if we want to re-enable this test from skip_list ?
Thanks

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/651730

Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: stein-rc1 → train-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/651730
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=3dabc41d032eeb8ab23b1f36d099b1757f2259d2
Submitter: Zuul
Branch: master

commit 3dabc41d032eeb8ab23b1f36d099b1757f2259d2
Author: Arx Cruz <email address hidden>
Date: Thu Apr 11 09:01:39 2019 +0000

    Remove test_create_server_when_memory_quota_is_full from skiplist

    This test is passing in latest featureset021 run

    Change-Id: Ie03687ac9bf75453a20e97e31665d57308cc7dd4
    Related-Bug: 1818886

Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: train-1 → train-2
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: train-2 → train-3
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: train-3 → train-rc1
Alex Schultz (alex-schultz)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.