GoodnessWeigher alone fails to schedule a non-typed volume

Bug #1766966 reported by Eric M Gonzalez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Eric M Gonzalez

Bug Description

According to this documentation, https://docs.openstack.org/cinder/pike/admin/blockstorage-driver-filter-weighing.html, the GoodnessWeigher can be configured to operate independently of other Filters and Weights ( there is an example at the bottom of the page ). After configuring my scheduler to have:

 scheduler_default_weighers = GoodnessWeigher

and my volume-types to be configured:

 enabled_backends = volumes_one,volumes_two

 [volumes_one]
 goodness_function = "100"

 [volumes_two]
 goodness_function = "0"

If a volume is created without a volume-type, the scheduler fails with the following exception:

 Traceback (most recent call last):
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
     res = self.dispatcher.dispatch(message)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
     return self._do_dispatch(endpoint, method, ctxt, args)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
     result = func(ctxt, **new_args)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/manager.py", line 400, in create_volume
     filter_properties=filter_properties, volume=volume)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/manager.py", line 184, in create_volume
     flow_engine.run()
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
     for _state in self.run_iter(timeout=timeout):
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
     failure.Failure.reraise_if_any(er_failures)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
     failures[0].reraise()
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
     six.reraise(*self._exc_info)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
     result = task.execute(**arguments)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 144, in execute
     common.error_out(volume, reason=e)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
     self.force_reraise()
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
     six.reraise(self.type_, self.value, self.tb)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 126, in execute
     filter_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 108, in schedule_create_volume
     filter_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 607, in _schedule
     filter_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 337, in _get_weighted_candidates
     filter_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/host_manager.py", line 432, in get_weighed_hosts
     weight_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/base_weight.py", line 134, in get_weighed_objects
     weights = weigher.weigh_objects(weighed_objs, weighing_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/base_weight.py", line 102, in weigh_objects
     weight = self._weigh_object(obj.obj, weight_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/weights/goodness.py", line 45, in _weigh_object
     stats = self._generate_stats(host_state, weight_properties)
   File "/openstack/venvs/cinder-14.2.14/lib/python2.7/site-packages/cinder/scheduler/weights/goodness.py", line 133, in _generate_stats
     extra_specs = volume_type.get('extra_specs', {})
 AttributeError: 'NoneType' object has no attribute 'get'

The offending lines are:

        volume_type = weight_properties.get('volume_type', {})
        extra_specs = volume_type.get('extra_specs', {})

where weight_properties["volume_type"] is a NoneType and this is being assigned to 'volume_type'. When the next line executes, the AttributeError is thrown.

I've patched the _generate_stats() function to:

        volume_type = weight_properties.get('volume_type', {}) or {}
        extra_specs = volume_type.get('extra_specs', {})

and I can now schedule non-typed volumes. I will have a patch for review shortly.

My deployment is OpenStack Newton, but the goodness module has not been updated up to current master ( 2018-04-15 ), which suggets that this problem exists in versions up to Rocky.

Revision history for this message
Eric M Gonzalez (egrh3) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/564339

Changed in cinder:
assignee: nobody → Eric M Gonzalez (egrh3)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/564339
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=34d75ca1a6eea03bc0fe856bf1a63e7d8d60f024
Submitter: Zuul
Branch: master

commit 34d75ca1a6eea03bc0fe856bf1a63e7d8d60f024
Author: Eric M Gonzalez <email address hidden>
Date: Wed Apr 25 15:31:27 2018 -0500

    GoodnessWeigher schedules non-type volumes

    If the GoodnessWeigher is configured to be the only active weigher within
    the scheduler, it cannot schedule a non-typed volumed for creation. This
    patch adds a safety assignement to the weigher stats generation to allow
    for scheduling untyped volumes.

    This patch safely assigns a dictionary data-type to "volume_type" in the
    event that:

        weight_properties = {
            "volume_type": None,
            etc..
        }

    Volumes created with a volume-type will not cause an exception and will
    be scheduled with the volume-type and back-end as configured.

    Closes-Bug: 1766966

    Change-Id: I7bc77ab02154aee80282e54b19a242bdea0a78ec
    Signed-off-by: Eric M Gonzalez <email address hidden>

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 13.0.0.0b2

This issue was fixed in the openstack/cinder 13.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/574922

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/queens)

Reviewed: https://review.openstack.org/574922
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a02126af7933ae4d5356bc6fa2e3422854ba2838
Submitter: Zuul
Branch: stable/queens

commit a02126af7933ae4d5356bc6fa2e3422854ba2838
Author: Eric M Gonzalez <email address hidden>
Date: Wed Apr 25 15:31:27 2018 -0500

    GoodnessWeigher schedules non-type volumes

    If the GoodnessWeigher is configured to be the only active weigher within
    the scheduler, it cannot schedule a non-typed volumed for creation. This
    patch adds a safety assignement to the weigher stats generation to allow
    for scheduling untyped volumes.

    This patch safely assigns a dictionary data-type to "volume_type" in the
    event that:

        weight_properties = {
            "volume_type": None,
            etc..
        }

    Volumes created with a volume-type will not cause an exception and will
    be scheduled with the volume-type and back-end as configured.

    Closes-Bug: 1766966

    Change-Id: I7bc77ab02154aee80282e54b19a242bdea0a78ec
    Signed-off-by: Eric M Gonzalez <email address hidden>
    (cherry picked from commit 34d75ca1a6eea03bc0fe856bf1a63e7d8d60f024)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/578860

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/580686

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/580686
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0d938d7b41f60a4c0bc28c3cdb361609f346758b
Submitter: Zuul
Branch: stable/pike

commit 0d938d7b41f60a4c0bc28c3cdb361609f346758b
Author: Eric M Gonzalez <email address hidden>
Date: Wed Apr 25 15:31:27 2018 -0500

    GoodnessWeigher schedules non-type volumes

    If the GoodnessWeigher is configured to be the only active weigher within
    the scheduler, it cannot schedule a non-typed volumed for creation. This
    patch adds a safety assignement to the weigher stats generation to allow
    for scheduling untyped volumes.

    This patch safely assigns a dictionary data-type to "volume_type" in the
    event that:

        weight_properties = {
            "volume_type": None,
            etc..
        }

    Volumes created with a volume-type will not cause an exception and will
    be scheduled with the volume-type and back-end as configured.

    Closes-Bug: 1766966

    Change-Id: I7bc77ab02154aee80282e54b19a242bdea0a78ec
    Signed-off-by: Eric M Gonzalez <email address hidden>
    (cherry picked from commit 34d75ca1a6eea03bc0fe856bf1a63e7d8d60f024)
    (cherry picked from commit a02126af7933ae4d5356bc6fa2e3422854ba2838)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/578860
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=57804eccf08a563aea19604fe00c08fe68fbf242
Submitter: Zuul
Branch: stable/ocata

commit 57804eccf08a563aea19604fe00c08fe68fbf242
Author: Eric M Gonzalez <email address hidden>
Date: Wed Apr 25 15:31:27 2018 -0500

    GoodnessWeigher schedules non-type volumes

    If the GoodnessWeigher is configured to be the only active weigher within
    the scheduler, it cannot schedule a non-typed volumed for creation. This
    patch adds a safety assignement to the weigher stats generation to allow
    for scheduling untyped volumes.

    This patch safely assigns a dictionary data-type to "volume_type" in the
    event that:

        weight_properties = {
            "volume_type": None,
            etc..
        }

    Volumes created with a volume-type will not cause an exception and will
    be scheduled with the volume-type and back-end as configured.

    Closes-Bug: 1766966

    Change-Id: I7bc77ab02154aee80282e54b19a242bdea0a78ec
    Signed-off-by: Eric M Gonzalez <email address hidden>
    (cherry picked from commit 34d75ca1a6eea03bc0fe856bf1a63e7d8d60f024)
    (cherry picked from commit a02126af7933ae4d5356bc6fa2e3422854ba2838)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.8

This issue was fixed in the openstack/cinder 10.0.8 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.4

This issue was fixed in the openstack/cinder 12.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/newton)

Fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/633031

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (driverfixes/newton)

Change abandoned by Keith Berger (<email address hidden>) on branch: driverfixes/newton
Review: https://review.openstack.org/633031
Reason: fix not eligible for driversfixes/newton based on EOL policy

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 11.2.0

This issue was fixed in the openstack/cinder 11.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.