OpenStack Masakari Monitors Charm

[focal][ussuri] zaza test fails at downloading cirus image due to 500 server error on glance

Bug #1872766 reported by Alex Kavanagh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Masakari Monitors Charm
In Progress
High
Liam Young

Bug Description

2020-04-13 13:58:18 [INFO] Using keystone API V3 (or later) for overcloud auth
2020-04-13 13:58:23 [INFO] Downloading image cirros
Traceback (most recent call last):
  File "/tmp/tmp.OUBeH9AxeY/func/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 191, in main
    force=args.force)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 128, in func_test_runner
    force=force)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 74, in run_env_deployment
    config_steps.get(deployment.model_alias, []))
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 48, in configure
    run_configure_list(functions)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/glance/setup.py", line 77, in add_cirros_image
    image_name=image_name)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/glance/setup.py", line 62, in add_image
    tags=tags)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 2034, in create_image
    image = upload_image_to_glance(glance, local_path, image_name)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 1991, in upload_image_to_glance
    glance.images.upload(image.id, open(local_path, 'rb'))
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/glanceclient/common/utils.py", line 598, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/glanceclient/v2/images.py", line 288, in upload
    resp, body = self.http_client.put(url, headers=hdrs, data=body)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 395, in put
    return self.request(url, 'PUT', **kwargs)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/glanceclient/common/http.py", line 378, in request
    return self._handle_response(resp)
  File "/tmp/tmp.OUBeH9AxeY/func/lib/python3.5/site-packages/glanceclient/common/http.py", line 126, in _handle_response
    raise exc.from_response(resp, resp.content)
glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.
Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/base_events.py", line 431, in __del__
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 58, in close
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 139, in remove_signal_handler
  File "/usr/lib/python3.5/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object
ERROR: InvocationError: '/tmp/tmp.OUBeH9AxeY/func/bin/functest-run-suite --keep-model'
___________________________________ summary ____________________________________
ERROR: func: commands failed

Error from the glance unit:

2020-04-13 13:58:23.584 4120 INFO eventlet.wsgi.server [req-601ed1d8-16d7-49bf-815f-f0ff1b505341 d025ebad2f3345e7896e95c5a3025f54 59a2a44bdd6d40a2b7801ebff7861dbd - e334ae3bdb824fb8bbdda4728f5a0934 e334ae3bdb824fb8bbdda4728f5a0934] 172.17.112.52,127.0.0.1 - - [13/Apr/2020 13:58:23] "POST /v2/images HTTP/1.1" 201 992 0.076205
2020-04-13 13:58:23.725 4120 WARNING glance_store._drivers.rbd [req-35ef9a99-d128-463b-abcd-234caebbc189 d025ebad2f3345e7896e95c5a3025f54 59a2a44bdd6d40a2b7801ebff7861dbd - e334ae3bdb824fb8bbdda4728f5a0934 e334ae3bdb824fb8bbdda4728f5a0934] since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal
2020-04-13 13:58:25.375 4120 ERROR glance_store._drivers.rbd [req-35ef9a99-d128-463b-abcd-234caebbc189 d025ebad2f3345e7896e95c5a3025f54 59a2a44bdd6d40a2b7801ebff7861dbd - e334ae3bdb824fb8bbdda4728f5a0934 e334ae3bdb824fb8bbdda4728f5a0934] Failed to store image 40a86597-0640-4fc5-a21b-19689428a38d Store Exception [errno 95] RBD operation not supported (error opening image b'40a8659
7-0640-4fc5-a21b-19689428a38d' at snapshot None): rbd.OperationNotSupported: [errno 95] RBD operation not supported (error opening image b'40a86597-0640-4fc5-a21b-19689428a38d' at snapshot None)
2020-04-13 13:58:25.411 4120 ERROR glance.api.v2.image_data [req-35ef9a99-d128-463b-abcd-234caebbc189 d025ebad2f3345e7896e95c5a3025f54 59a2a44bdd6d4
0a2b7801ebff7861dbd - e334ae3bdb824fb8bbdda4728f5a0934 e334ae3bdb824fb8bbdda4728f5a0934] Failed to upload image data due to internal error: rbd.OperationNotSupported: [errno 95] RBD operation not supported (error opening image b'40a86597-0640-4fc5-a21b-19689428a38d' at snapshot None)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi [req-35ef9a99-d128-463b-abcd-234caebbc189 d025ebad2f3345e7896e95c5a3025f54 59a2a44bdd6d40a2b7801ebff7861dbd - e334ae3bdb824fb8bbdda4728f5a0934 e334ae3bdb824fb8bbdda4728f5a0934] Caught error: [errno 95] RBD operation not supported (error opening image b'40a86597-0640-4fc5-a21b-19689428a38d' at snapshot None): rbd.OperationNotSupported: [errno 95] RBD operation not supported (error opening image b'40a86597-0640-4fc5-a21b-19689428a38d' at snapshot None)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi Traceback (most recent call last):
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1347, in __call__
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi request, **action_args)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1390, in dispatch
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi return method(*args, **kwargs)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/utils.py", line 416, in wrapped
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line 299, in upload
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi self._restore(image_repo, image)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi self.force_reraise()
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi raise value
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line 164, in upload
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/domain/proxy.py", line 198, in set_data
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/notifier.py", line 501, in set_data
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi self.force_reraise()
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi raise value
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/notifier.py", line 448, in set_data
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi set_active=set_active)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/policy.py", line 204, in set_data
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/quota/__init__.py", line 319, in set_data
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi set_active=set_active)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/location.py", line 559, in set_data
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/location.py", line 486, in _upload_to_store
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi verifier=verifier)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/backend.py", line 489, in add_to_backend_with_multihash
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/backend.py", line 466, in store_add_to_backend_with_multihash
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/driver.py", line 279, in add_adapter
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/capabilities.py", line 176, in op_checker
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py", line 578, in add
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi loc.snapshot)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py", line 422, in _delete_image
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi with rbd.Image(ioctx, image_name) as image:
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi File "rbd.pyx", line 3369, in rbd.Image.__init__
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi rbd.OperationNotSupported: [errno 95] RBD operation not supported (error opening image b'40a86597-0640-4fc5-a21b-19689428a38d' at snapshot None)
2020-04-13 13:58:25.444 4120 ERROR glance.common.wsgi
2020-04-13 13:58:25.510 4120 INFO eventlet.wsgi.server [req-35ef9a99-d128-463b-abcd-234caebbc189 d025ebad2f3345e7896e95c5a3025f54 59a2a44bdd6d40a2b7801ebff7861dbd - e334ae3bdb824fb8bbdda4728f5a0934 e334ae3bdb824fb8bbdda4728f5a0934] 172.17.112.52,127.0.0.1 - - [13/Apr/2020 13:58:25] "PUT /v2/images/40a86597-0640-4fc5-a21b-19689428a38d/file HTTP/1.1" 500 454 1.911396

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :
Download full text (6.9 KiB)

From the glance-api.log:

2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi [req-b0392b2c-365a-4e34-9455-daffc9d8c070 012cb8b9620c478294f788520f712126 f959d620f85441949ce
accdb3ddac018 - 672ddb978f364fc485406ce58ba2edea 672ddb978f364fc485406ce58ba2edea] Caught error: [errno 95] RBD operation not supported (error opening image b'742c5de9-e9d0-43ab-b221-a02dd07bb0af' at snapshot None): rbd.OperationNotSupported: [errno 95] RBD operation not supported (error opening image b'742c5de9-e9d0-43ab-b221-a02dd07bb0af' at snapshot None)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi Traceback (most recent call last):
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1347, in __call__
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi request, **action_args)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1390, in dispatch
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi return method(*args, **kwargs)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/utils.py", line 416, in wrapped
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line 299, in upload
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi self._restore(image_repo, image)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi self.force_reraise()
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi raise value
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line 164, in upload
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/domain/proxy.py", line 198, in set_data
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/notifier.py", line 501, in set_data
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-17 07:25:25.735 4119 ERROR glance.common.wsgi self.force_reraise()
2020-04-17 0...

Read more...

Liam Young (gnuoy)
Changed in charm-masakari-monitors:
assignee: nobody → Liam Young (gnuoy)
status: New → In Progress
milestone: none → 20.05
importance: Undecided → High
David Ames (thedac)
Changed in charm-masakari-monitors:
milestone: 20.05 → 20.08
James Page (james-page)
Changed in charm-masakari-monitors:
milestone: 20.08 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.