glance_store

ceph jobs failing to upload images in pike due to "AttributeError: 'NoneType' object has no attribute 'Rados'" on py35

Bug #1706405 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
Invalid
Undecided
Unassigned

Bug Description

http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv/6066e44/logs/screen-g-api.txt.gz?level=TRACE#_Jul_25_14_37_19_667434

Jul 25 14:37:19.667434 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.api.v2.image_data [None req-19231148-54d7-4337-be6b-a9a73c34b617 admin admin] Failed to upload image data due to internal error
Jul 25 14:37:19.704732 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi [None req-19231148-54d7-4337-be6b-a9a73c34b617 admin admin] Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'
Jul 25 14:37:19.704898 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi Traceback (most recent call last):
Jul 25 14:37:19.705014 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/common/wsgi.py", line 1155, in __call__
Jul 25 14:37:19.705130 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi request, **action_args)
Jul 25 14:37:19.705242 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/common/wsgi.py", line 1194, in dispatch
Jul 25 14:37:19.705345 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi return method(*args, **kwargs)
Jul 25 14:37:19.705445 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/common/utils.py", line 363, in wrapped
Jul 25 14:37:19.705546 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
Jul 25 14:37:19.705646 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 269, in upload
Jul 25 14:37:19.705751 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi self._restore(image_repo, image)
Jul 25 14:37:19.705850 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 25 14:37:19.705960 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi self.force_reraise()
Jul 25 14:37:19.706062 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 25 14:37:19.706174 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Jul 25 14:37:19.706274 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/six.py", line 686, in reraise
Jul 25 14:37:19.706373 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi raise value
Jul 25 14:37:19.706472 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 134, in upload
Jul 25 14:37:19.706572 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi image.set_data(data, size)
Jul 25 14:37:19.706670 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/domain/proxy.py", line 195, in set_data
Jul 25 14:37:19.706787 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi self.base.set_data(data, size)
Jul 25 14:37:19.706914 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/notifier.py", line 480, in set_data
Jul 25 14:37:19.707029 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
Jul 25 14:37:19.707133 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 25 14:37:19.707233 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi self.force_reraise()
Jul 25 14:37:19.707345 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 25 14:37:19.707459 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Jul 25 14:37:19.707554 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/six.py", line 686, in reraise
Jul 25 14:37:19.707649 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi raise value
Jul 25 14:37:19.707752 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/notifier.py", line 427, in set_data
Jul 25 14:37:19.707848 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi self.repo.set_data(data, size)
Jul 25 14:37:19.707942 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/api/policy.py", line 192, in set_data
Jul 25 14:37:19.708036 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
Jul 25 14:37:19.708138 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/quota/__init__.py", line 304, in set_data
Jul 25 14:37:19.708238 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi self.image.set_data(data, size=size)
Jul 25 14:37:19.708333 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/opt/stack/new/glance/glance/location.py", line 439, in set_data
Jul 25 14:37:19.708432 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi verifier=verifier)
Jul 25 14:37:19.708536 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/glance_store/backend.py", line 453, in add_to_backend
Jul 25 14:37:19.708635 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi verifier)
Jul 25 14:37:19.708734 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/glance_store/backend.py", line 426, in store_add_to_backend
Jul 25 14:37:19.708833 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi verifier=verifier)
Jul 25 14:37:19.708938 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/glance_store/capabilities.py", line 225, in op_checker
Jul 25 14:37:19.709037 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
Jul 25 14:37:19.709142 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/glance_store/_drivers/rbd.py", line 454, in add
Jul 25 14:37:19.709237 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi rados_id=self.user) as conn:
Jul 25 14:37:19.709337 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
Jul 25 14:37:19.709436 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi return next(self.gen)
Jul 25 14:37:19.709531 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi File "/usr/local/lib/python3.5/dist-packages/glance_store/_drivers/rbd.py", line 268, in get_connection
Jul 25 14:37:19.709625 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi client = rados.Rados(conffile=conffile, rados_id=rados_id)
Jul 25 14:37:19.709719 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi AttributeError: 'NoneType' object has no attribute 'Rados'
Jul 25 14:37:19.709869 ubuntu-xenial-ovh-bhs1-10067578 <email address hidden>[21781]: ERROR glance.common.wsgi

Tags: ceph rbd
Revision history for this message
Matt Riedemann (mriedem) wrote :

The py27 ceph job passed on the same change:

http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/db9b1d6/

So maybe this is just a py35 issue with librados or the glance-store rbd driver code?

tags: added: ceph rbd
no longer affects: glance
summary: ceph jobs failing to upload images in pike due to "AttributeError:
- 'NoneType' object has no attribute 'Rados'"
+ 'NoneType' object has no attribute 'Rados'" on py35
Revision history for this message
Matt Riedemann (mriedem) wrote :

rados is imported here and set to None on ImportError:

https://github.com/openstack/glance_store/blob/0.21.0/glance_store/_drivers/rbd.py#L41

If you look at pip freeze for the py27 job, rados is there:

http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/db9b1d6/logs/pip2-freeze.txt.gz

rados==0
rbd==0

But not for the py35 job:

http://logs.openstack.org/56/477556/5/experimental/gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv/6066e44/logs/pip3-freeze.txt.gz

Changed in glance-store:
status: New → Confirmed
Revision history for this message
Matt Riedemann (mriedem) wrote :

Note that we're using python-rados 10.2.7 from the xenial-updates archive.

Revision history for this message
Matt Riedemann (mriedem) wrote :

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'NoneType'%20object%20has%20no%20attribute%20'Rados'%5C%22%20AND%20tags%3A%5C%22screen-g-api.txt%5C%22&from=7d

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

We're not seeing this on current releases, maybe it's a packaging problem with a (by now) older distribution, as Matt suggests.

Changed in glance-store:
status: Confirmed → Incomplete
Revision history for this message
Erno Kuvaja (jokke) wrote :

Nothing about this for over half a year. Will close it as Invalid. Feel free to reopen if still affected.

Changed in glance-store:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.