OpenStack Glance Charm

Cannot upload images into Glance

Bug #1829241 reported by Greg Zavertnik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Glance Charm
Expired
Undecided
Unassigned

Bug Description

From the Command line or from Horizon dashboard. Every time I try to create any image, LXD of KVM, I get an error. In Horizon it just stays locked in a queing state and never completes the upload.

$ openstack image create bionic_lxd --file bionic.squashfs --disk-format=raw --container-format=bare --public --property architecture=x86_64

500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

In the glance-registry.log I see:

2019-05-14 17:36:31.604 43590 CRITICAL glance [-] Unhandled error: TypeError: 'OSError' object is not subscriptable
2019-05-14 17:36:31.604 43590 ERROR glance Traceback (most recent call last):
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 704, in run_server
2019-05-14 17:36:31.604 43590 ERROR glance socket_timeout=self.client_socket_timeout)
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 880, in server
2019-05-14 17:36:31.604 43590 ERROR glance client_socket = sock.accept()
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 214, in accept
2019-05-14 17:36:31.604 43590 ERROR glance res = socket_accept(fd)
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 56, in socket_accept
2019-05-14 17:36:31.604 43590 ERROR glance return descriptor.accept()
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3.6/socket.py", line 205, in accept
2019-05-14 17:36:31.604 43590 ERROR glance fd, addr = self._accept()
2019-05-14 17:36:31.604 43590 ERROR glance OSError: [Errno 22] Invalid argument
2019-05-14 17:36:31.604 43590 ERROR glance
2019-05-14 17:36:31.604 43590 ERROR glance During handling of the above exception, another exception occurred:
2019-05-14 17:36:31.604 43590 ERROR glance
2019-05-14 17:36:31.604 43590 ERROR glance Traceback (most recent call last):
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/bin/glance-registry", line 10, in <module>
2019-05-14 17:36:31.604 43590 ERROR glance sys.exit(main())
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/cmd/registry.py", line 79, in main
2019-05-14 17:36:31.604 43590 ERROR glance server.wait()
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 655, in wait
2019-05-14 17:36:31.604 43590 ERROR glance self.wait_on_children()
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 592, in wait_on_children
2019-05-14 17:36:31.604 43590 ERROR glance self._verify_and_respawn_children(pid, status)
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 584, in _verify_and_respawn_children
2019-05-14 17:36:31.604 43590 ERROR glance self.run_child()
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 680, in run_child
2019-05-14 17:36:31.604 43590 ERROR glance self.run_server()
2019-05-14 17:36:31.604 43590 ERROR glance File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 706, in run_server
2019-05-14 17:36:31.604 43590 ERROR glance if err[0] != errno.EINVAL:
2019-05-14 17:36:31.604 43590 ERROR glance TypeError: 'OSError' object is not subscriptable
2019-05-14 17:36:31.604 43590 ERROR glance

In the glance-api.log I see:

2019-05-15 13:39:37.089 473 ERROR glance.api.v2.image_data [req-8399154e-7c67-465b-93a4-ffeff458d2a9 880419d412b248ad9bf39c015142f42b d4794951b7734c7486df39f91cf0282b - 463cfb2f19c94a04a48faaa20258b74c 463cfb2f19c94a04a48faaa20258b74c] Failed to upload image data due to internal error: rados.ObjectNotFound: [errno 2] error opening pool 'b'glance''
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi [req-8399154e-7c67-465b-93a4-ffeff458d2a9 880419d412b248ad9bf39c015142f42b d4794951b7734c7486df39f91cf0282b - 463cfb2f19c94a04a48faaa20258b74c 463cfb2f19c94a04a48faaa20258b74c] Caught error: [errno 2] error opening pool 'b'glance'': rados.ObjectNotFound: [errno 2] error opening pool 'b'glance''
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi Traceback (most recent call last):
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1274, in __call__
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi request, **action_args)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1317, in dispatch
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi return method(*args, **kwargs)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/utils.py", line 417, in wrapped
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line 279, in upload
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self._restore(image_repo, image)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self.force_reraise()
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi raise value
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/image_data.py", line 144, in upload
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi image.set_data(data, size, backend=backend)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/domain/proxy.py", line 198, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/notifier.py", line 485, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self.force_reraise()
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi raise value
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/notifier.py", line 432, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self.force_reraise()
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi raise value
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/notifier.py", line 432, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/policy.py", line 193, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/quota/__init__.py", line 310, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/location.py", line 483, in set_data
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi verifier=verifier)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/backend.py", line 541, in add_to_backend_with_multihash
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/backend.py", line 518, in store_add_to_backend_with_multihash
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/driver.py", line 274, in add_adapter
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/capabilities.py", line 225, in op_checker
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/_drivers/rbd.py", line 473, in add
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi with conn.open_ioctx(self.pool) as ioctx:
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "rados.pyx", line 517, in rados.requires.wrapper.validate_func
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi File "rados.pyx", line 1212, in rados.Rados.open_ioctx
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi rados.ObjectNotFound: [errno 2] error opening pool 'b'glance''
2019-05-15 13:39:37.112 473 ERROR glance.common.wsgi

Revision history for this message
David Ames (thedac) wrote :

Greg,

It seems there is a problem with ceph. The "glance" pool has not been created yet.

"Failed to upload image data due to internal error: rados.ObjectNotFound: [errno 2] error opening pool 'b'glance''"

The problem is not likely to be glance itself.

Can you provide a sanitized bundle you used to deploy? As well as the "juju status --format yaml --relations" output?

Changed in charm-glance:
status: New → Incomplete
Revision history for this message
Greg Zavertnik (quartzeye) wrote : Re: [Bug 1829241] Re: Cannot upload images into Glance
Download full text (11.9 KiB)

Yes, but not until Friday as I am away for a business trip.

On Wed, May 15, 2019 at 12:25 PM David Ames <email address hidden>
wrote:

> Greg,
>
> It seems there is a problem with ceph. The "glance" pool has not been
> created yet.
>
> "Failed to upload image data due to internal error:
> rados.ObjectNotFound: [errno 2] error opening pool 'b'glance''"
>
> The problem is not likely to be glance itself.
>
> Can you provide a sanitized bundle you used to deploy? As well as the
> "juju status --format yaml --relations" output?
>
> ** Changed in: charm-glance
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1829241
>
> Title:
> Cannot upload images into Glance
>
> Status in OpenStack glance charm:
> Incomplete
>
> Bug description:
> From the Command line or from Horizon dashboard. Every time I try to
> create any image, LXD of KVM, I get an error. In Horizon it just
> stays locked in a queing state and never completes the upload.
>
> $ openstack image create bionic_lxd --file bionic.squashfs --disk-
> format=raw --container-format=bare --public --property
> architecture=x86_64
>
> 500 Internal Server Error: The server has either erred or is incapable
> of performing the requested operation. (HTTP 500)
>
> In the glance-registry.log I see:
>
> 2019-05-14 17:36:31.604 43590 CRITICAL glance [-] Unhandled error:
> TypeError: 'OSError' object is not subscriptable
> 2019-05-14 17:36:31.604 43590 ERROR glance Traceback (most recent call
> last):
> 2019-05-14 17:36:31.604 43590 ERROR glance File
> "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 704, in
> run_server
> 2019-05-14 17:36:31.604 43590 ERROR glance
> socket_timeout=self.client_socket_timeout)
> 2019-05-14 17:36:31.604 43590 ERROR glance File
> "/usr/lib/python3/dist-packages/eventlet/wsgi.py", line 880, in server
> 2019-05-14 17:36:31.604 43590 ERROR glance client_socket =
> sock.accept()
> 2019-05-14 17:36:31.604 43590 ERROR glance File
> "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 214, in
> accept
> 2019-05-14 17:36:31.604 43590 ERROR glance res = socket_accept(fd)
> 2019-05-14 17:36:31.604 43590 ERROR glance File
> "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 56, in
> socket_accept
> 2019-05-14 17:36:31.604 43590 ERROR glance return descriptor.accept()
> 2019-05-14 17:36:31.604 43590 ERROR glance File
> "/usr/lib/python3.6/socket.py", line 205, in accept
> 2019-05-14 17:36:31.604 43590 ERROR glance fd, addr = self._accept()
> 2019-05-14 17:36:31.604 43590 ERROR glance OSError: [Errno 22] Invalid
> argument
> 2019-05-14 17:36:31.604 43590 ERROR glance
> 2019-05-14 17:36:31.604 43590 ERROR glance During handling of the above
> exception, another exception occurred:
> 2019-05-14 17:36:31.604 43590 ERROR glance
> 2019-05-14 17:36:31.604 43590 ERROR glance Traceback (most recent call
> last):
> 2019-05-14 17:36:31.604 43590 ERROR glance File
> "/usr/bin/glance-registry", line 10, in <module>
> 2019-05-14 17:36:31.604 43590 ERROR glance s...

Revision history for this message
Greg Zavertnik (quartzeye) wrote :

Here is my YAML file I deployed.

https://pastebin.com/4LyNMug6

Here are my build steps.

https://pastebin.com/vviq1UM7

I skipped the erasure coding steps in order to test the environment
immediately after the deploy was completed and before making changes.

Here is the output from  "juju status --format yaml --relations"

https://pastebin.com/JwFGcH8s

On 05/15/2019 12:18 PM, David Ames wrote:
> Greg,
>
> It seems there is a problem with ceph. The "glance" pool has not been
> created yet.
>
> "Failed to upload image data due to internal error:
> rados.ObjectNotFound: [errno 2] error opening pool 'b'glance''"
>
> The problem is not likely to be glance itself.
>
> Can you provide a sanitized bundle you used to deploy? As well as the
> "juju status --format yaml --relations" output?
>
> ** Changed in: charm-glance
> Status: New => Incomplete
>

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack glance charm because there has been no activity for 60 days.]

Changed in charm-glance:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.