neutron

Hit internal error when using StandardAttribute as sort-key

Bug #1659175 reported by zhaobo
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
pandatt

Bug Description

Maybe our server not support query resources with created_at. This is a standard requirement.

cmd:
neutron port-list --sort-key created_at --sort-dir asc
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-2da26481-ef77-4bf8-a83c-f0c412d2936c']

2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return f(*args, **kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m ectxt.value = e.inner_exc
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m self.force_reraise()
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return f(*args, **kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 128, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m traceback.format_exc())
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m self.force_reraise()
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 123, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return f(*dup_args, **dup_kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/api/v2/base.py", line 369, in index
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return self._items(request, True, parent_id)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/api/v2/base.py", line 308, in _items
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m obj_list = obj_getter(request.context, **kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 163, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return method(*args, **kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 92, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m setattr(e, '_RETRY_EXCEEDED', True)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m self.force_reraise()
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return f(*args, **kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m ectxt.value = e.inner_exc
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m self.force_reraise()
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return f(*args, **kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 128, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m traceback.format_exc())
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m self.force_reraise()
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m six.reraise(self.type_, self.value, self.tb)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/api.py", line 123, in wrapped
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m return f(*dup_args, **dup_kwargs)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1333, in get_ports
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m page_reverse=page_reverse)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1317, in _get_ports_query
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m sort_keys = db_utils.get_and_validate_sort_keys(sorts, Port)
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/neutron_lib/db/utils.py", line 43, in get_and_validate_sort_keys
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m if isinstance(sort_key_attr.property,
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00mAttributeError: 'AssociationProxy' object has no attribute 'property'
2017-01-25 09:42:59.830 TRACE neutron.api.v2.resource ^[[01;35m^[[00m

zhaobo (zhaobo6)
Changed in neutron:
assignee: nobody → zhaobo (zhaobo6)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-lib (master)

Fix proposed to branch: master
Review: https://review.openstack.org/424948

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/424949

Hirofumi Ichihara (ichihara-hirofumi)
tags: added: api
Hirofumi Ichihara (ichihara-hirofumi)
summary: - Hit internal error when query port with sort-key created_at
+ Hit internal error when using StandardAttribute as sort-key
Changed in neutron:
status: New → Confirmed
importance: Undecided → High
OpenStack Infra (hudson-openstack)
Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/424949
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron-lib (master)

Change abandoned by boden (<email address hidden>) on branch: master
Review: https://review.openstack.org/424948
Reason: Abandoning this one as it's been idle for more than 4 weeks. Please push an update if it's still active. Thanks

Revision history for this message
Kevin Benton (kevinbenton) wrote :

unassigning current since patches have been abandoned so someone else can pick this up

Changed in neutron:
assignee: zhaobo (zhaobo6) → nobody
status: In Progress → Confirmed
pandatt (pandatt)
Changed in neutron:
assignee: nobody → pandatt (pandatt)
Revision history for this message
pandatt (pandatt) wrote :

(1) Use any invalid sort key and standard attribute of `Port` resource to reproduce this issue:
[root@master ~]# neutron net-list --sort-key fake_key --sort-dir asc
[u'fake_key'] is invalid attribute for sort_keys
Neutron server returns request_ids: ['req-8a0c2434-18fa-44ee-910d-da299c4b4578']
[root@master ~]#
[root@master ~]# neutron net-list --sort-key created_at --sort-dir asc
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-9b841cd8-82bd-4bc6-9df3-b39e9a892148']

(2) All supported sort keys are documented here: https://developer.openstack.org/api-ref/network/v2/index.html?expanded=list-ports-detail

(3) Common Neutron resource consists of resource attributes and standard attributes.
e.g: `Port` resource with properties:
i. resource attributes: admin_state_up, device_id,device_owner, id,ip_allocation,mac_address,name
ii.standard attributes: created_at,updated_at,description,revision_number

(4) Standard attributes are shared among all neutron resources and not supported to server as sort keys.
    However, they're not considered while `get_sorts` for neutron resource in `neutron/api/api_common.py`.

(5) By handling standard attributes in advance, we can avoid triggering the AttributeError later:
    resource attributes of neutron.db.models_v2.Port has `property` property, and standard attributes not.

Revision history for this message
pandatt (pandatt) wrote :

... sorry, hava pasted wrong scripts and results.

(1) Use any invalid sort key and standard attribute of `Port` resource to reproduce this issue:
[root@master ~]# neutron port-list --sort-key fake_key --sort-dir asc
[u'fake_key'] is invalid attribute for sort_keys
Neutron server returns request_ids: ['req-8fee06c7-68b1-4e1b-91cc-50f5b9d19f44']

[root@master ~]# neutron port-list --sort-key created_at --sort-dir asc
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-9b9fd387-7d7b-4741-86a4-15bba1191812']

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/653903

Revision history for this message
pandatt (pandatt) wrote :

Seems duplicated with issue: https://bugs.launchpad.net/neutron/+bug/1804259

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/657913
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=65264a936a6c9d0a8c4eae69acaee2c3ee54b5d6
Submitter: Zuul
Branch: master

commit 65264a936a6c9d0a8c4eae69acaee2c3ee54b5d6
Author: pandatt <guojy8993@163.com>
Date: Thu May 9 10:48:48 2019 +0800

    Add sort_keys retrieving function

    This patch provides a tool function to retrieve valid sort keys
    from `attr_info` dict. And it's necessary prerequisite to fix
    bug-1659175.

    Related-bug: #1659175
    Change-Id: I5ef7a0730122011f5ef18c1b4e2c0bfe82ed585b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/653903
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=335ac4e2d95c164d66a3b09d9fd08e9c563edfc2
Submitter: Zuul
Branch: master

commit 335ac4e2d95c164d66a3b09d9fd08e9c563edfc2
Author: pandatt <guojy8993@163.com>
Date: Fri Apr 19 11:29:54 2019 +0800

    Fix bug: AttributeError arises while sorting with standard attributes

    Common neutron resource(e.g, Port) consists of:
    1. Resource Attributes, e.g: Port.mac_address, etc.
    2. Standard Attributes, e.g: created_at, and are shared among all
       neutron resources.
    The `sort` opt only supports limited attributes. We need to filter
    attributes that are defined with `is_sort_key=True` and it's preferred
    to explicitly warn CLI & API users of illegal sort keys rather than
    just accept without check, pass forward and then hit a internal error
    which's quite confusing.

    Depends-on: https://review.opendev.org/#/c/660097/
    Change-Id: I8d206f909b09f1279dfcdc25c39989a67bff93d5
    Closes-Bug: #1659175

Changed in neutron:
status: In Progress → Fix Released
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/705198

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/705199

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/705198
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b99298dfad1611294bf6dfa41712388af78c7197
Submitter: Zuul
Branch: stable/stein

commit b99298dfad1611294bf6dfa41712388af78c7197
Author: pandatt <guojy8993@163.com>
Date: Fri Apr 19 11:29:54 2019 +0800

    Fix bug: AttributeError arises while sorting with standard attributes

    Common neutron resource(e.g, Port) consists of:
    1. Resource Attributes, e.g: Port.mac_address, etc.
    2. Standard Attributes, e.g: created_at, and are shared among all
       neutron resources.
    The `sort` opt only supports limited attributes. We need to filter
    attributes that are defined with `is_sort_key=True` and it's preferred
    to explicitly warn CLI & API users of illegal sort keys rather than
    just accept without check, pass forward and then hit a internal error
    which's quite confusing.

    Depends-on: https://review.opendev.org/#/c/660097/
    Change-Id: I8d206f909b09f1279dfcdc25c39989a67bff93d5
    Closes-Bug: #1659175
    (cherry picked from commit 335ac4e2d95c164d66a3b09d9fd08e9c563edfc2)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.1.0

This issue was fixed in the openstack/neutron 14.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/705199
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3750587833310aafdc0f6948a630ca3afe6bea0a
Submitter: Zuul
Branch: stable/rocky

commit 3750587833310aafdc0f6948a630ca3afe6bea0a
Author: pandatt <guojy8993@163.com>
Date: Fri Apr 19 11:29:54 2019 +0800

    Fix bug: AttributeError arises while sorting with standard attributes

    Common neutron resource(e.g, Port) consists of:
    1. Resource Attributes, e.g: Port.mac_address, etc.
    2. Standard Attributes, e.g: created_at, and are shared among all
       neutron resources.
    The `sort` opt only supports limited attributes. We need to filter
    attributes that are defined with `is_sort_key=True` and it's preferred
    to explicitly warn CLI & API users of illegal sort keys rather than
    just accept without check, pass forward and then hit a internal error
    which's quite confusing.

    Depends-on: https://review.opendev.org/#/c/660097/
    Change-Id: I8d206f909b09f1279dfcdc25c39989a67bff93d5
    Closes-Bug: #1659175
    (cherry picked from commit 335ac4e2d95c164d66a3b09d9fd08e9c563edfc2)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.7

This issue was fixed in the openstack/neutron 13.0.7 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.