Mirantis OpenStack

Image creation failed with 500 server error after adding 2 controllers to cluster

Bug #1557494 reported by Andrey Sledzinskiy on 2016-03-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Invalid	High	Kairat Kushaev	Mirantis OpenStack 9.0

Bug Description

iso - 9.0-59
fuel version - http://paste.openstack.org/show/490488/

Steps:
1. Install fuel
2. Create and deploy next cluster - Neutron tun, default storages, 1 controller
3. After deployment add 2 controllers and deploy again
4. After deployment open Health Check tab
5. Run test - Check create, update and delete image actions using Glance v2

Actual result - it failed on Send request to create image
from glance-api.log from node-5:

2016-03-14 23:42:28 ERR glance.common.wsgi [req-7be8a2eb-70e6-4a8d-bfcc-2d97db32fb14 cc495cbaece04bc2b2a0c5ff1074955a 7ae4bbc69c2d434786ebee7336977d77 - - -] Caught error: Failed to add object to Swift.
Got error from Swift: Object PUT failed: http://10.109.6.3:8080/v1/AUTH_1ddb7983c7374c1c96cfa0c421f81401/glance/59c1ec23-4349-4e15-9da8-819ba5b04bd9 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<.
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi Traceback (most recent call last):
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 885, in __call__
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi request, **action_args)
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 919, in dispatch
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi return method(*args, **kwargs)
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 423, in wrapped
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/v2/image_data.py", line 249, in upload
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi self._restore(image_repo, image)
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi self.force_reraise()
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-14 23:42:28.602 15655 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2016-03-14 23:42:28.602 1

Tags:

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2016-03-15:

fail_error_neutron_tun_scalability-fuel-snapshot-2016-03-14_23-42-58.tar.xz Edit (64.3 MiB, application/octet-stream)

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2016-03-15:

might be related - https://bugs.launchpad.net/mos/+bug/1555004

Kairat Kushaev (kkushaev) on 2016-03-16

Changed in mos:
assignee:	MOS Glance (mos-glance) → Kairat Kushaev (kkushaev)

Revision history for this message

Kairat Kushaev (kkushaev) wrote on 2016-03-16:

Not sure it is related to that issue. Glance v2 doesn't use registry in MOS so it should not be affected by registry configuration.
What I have found is:
https://paste.mirantis.net/show/1987/
Looks like 2 parallel requests came to Glance:
https://paste.mirantis.net/show/1989/

Revision history for this message

Kairat Kushaev (kkushaev) wrote on 2016-03-16:

I have found some error in Swift logs but not sure it is related to the problem. Unfortunately, no useful info in Glance logs.
It looks like glance passed two simultaneous requests for object upload and they failed somewhere in Swift because of some reason (Container does exist before object upload according to logs).
Here is haproxy:
./node-1.test.domain.local/haproxy.log:2016-03-14T23:42:28.492380+00:00 info: 10.109.6.5:49807 [14/Mar/2016:23:42:28.414] object-storage object-storage/node-4 0/0/0/78/78 404 260 - - ---- 90/0/0/0/0 0/0 "PUT /v1/AUTH_1ddb7983c7374c1c96cfa0c421f81401/glance/59c1ec23-4349-4e15-9da8-819ba5b04bd9 HTTP/1.1"
./node-1.test.domain.local/haproxy.log:2016-03-14T23:42:28.492539+00:00 info: 10.109.6.5:49807 [14/Mar/2016:23:42:28.414] object-storage object-storage/node-4 0/0/0/78/78 404 260 - - ---- 90/0/0/0/0 0/0 "PUT /v1/AUTH_1ddb7983c7374c1c96cfa0c421f81401/glance/59c1ec23-4349-4e15-9da8-819ba5b04bd9 HTTP/1.1"