neutron

Neutron returns HttpException: 500 on certain operations with modified list of policies for non-admin users

Bug #1829304 reported by Nate Johnston on 2019-05-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Undecided	Nate Johnston

Bug Description

Description of problem:

When deploying with a modified list of Neutron API policies, post deployment, policies which worked on previous versions will result in Neutron API Server returning 'HttpException: 500' when using the API with non admin users.

Additional API policies which were passed during deployment: http://paste.openstack.org/show/751347/

Example:

1. Source credentials with non admin user
[stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant
2. Query port list with as non admin user
(overcloud) [stack@undercloud-0 ~]$ openstack port list

At this point, neutron will return:
HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error

And the following exception will be generated inside server.log on controller nodes:
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return request.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self._app)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = self.app(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.application(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).__call__(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors state
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).handle_hooks(hooks, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors result = getattr(hook, hook_type)(*args)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for item in to_process
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp>
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors pluralized=collection))]
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors neutron_context, controller, resource, collection, data)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for attr_name in data.keys():
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors
}

Version-Release number of selected component (if applicable):
Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1
rpm -qa | grep neutron
puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch
python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy Overcloud with modified Neutron APIs
2. Create non admin user/tenant
3. Attempt to list ports

Actual results:
Fail to retrieve ports and receive python exceptions

Expected results:
List of ports is returned

Additional info:

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-15: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/659396

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-15: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/659397

Changed in neutron:
status:	New → In Progress

Revision history for this message

Nate Johnston (nate-johnston) wrote on 2019-05-15:

I did not realize the local repo was on stable/rocky when I pushed the first change; I marked it -W until the master version is merged.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-16: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/659397
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2c1098b3afb4f5d7cd1d10eaa722652624e6bb74
Submitter: Zuul
Branch: master

commit 2c1098b3afb4f5d7cd1d10eaa722652624e6bb74
Author: Nate Johnston <email address hidden>
Date: Wed May 15 19:18:52 2019 -0400

Use six.viewkeys instead of dict.keys to avoid py2 to py3 problems

    This change fixes an 'RuntimeError: dictionary changed size during
    iteration' error that is raised because of different behaviour between
    python2 and python3. We use the six library to ensure that the behavior
    is compatible across versions.

Change-Id: I0723ae10825e1e2d86789627895e3286d8c97602
Resolves-Bug: #1829304

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-17: Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/659790

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-21: Change abandoned on neutron (stable/rocky)

Change abandoned by Nate Johnston (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/659396
Reason: This doesn't look like a cherry pick because it isn't, it's my initial push that I did without noticing I was on a stable branch. Abandoning this and cherry-picking normally.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-21: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/660427

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-21: Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/660428

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-21: Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/659790
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=db2d3895d1c5051d4b8da6569bf164993c67ad74
Submitter: Zuul
Branch: stable/stein

commit db2d3895d1c5051d4b8da6569bf164993c67ad74
Author: Nate Johnston <email address hidden>
Date: Wed May 15 19:18:52 2019 -0400

Use six.viewkeys instead of dict.keys to avoid py2 to py3 problems

    Change-Id: I0723ae10825e1e2d86789627895e3286d8c97602
    Resolves-Bug: #1829304
    (cherry picked from commit 2c1098b3afb4f5d7cd1d10eaa722652624e6bb74)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-22: Fix merged to neutron (stable/queens)

#10

Reviewed: https://review.opendev.org/660428
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8bff653d903223014f15e6af05b484bcd2b4a026
Submitter: Zuul
Branch: stable/queens

commit 8bff653d903223014f15e6af05b484bcd2b4a026
Author: Nate Johnston <email address hidden>
Date: Wed May 15 19:18:52 2019 -0400

Use six.viewkeys instead of dict.keys to avoid py2 to py3 problems

    Change-Id: I0723ae10825e1e2d86789627895e3286d8c97602
    Resolves-Bug: #1829304
    (cherry picked from commit 2c1098b3afb4f5d7cd1d10eaa722652624e6bb74)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-22: Fix merged to neutron (stable/rocky)

#11

Reviewed: https://review.opendev.org/660427
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d9082270504728f45994cfd3b79f4ab8ebe53473
Submitter: Zuul
Branch: stable/rocky

commit d9082270504728f45994cfd3b79f4ab8ebe53473
Author: Nate Johnston <email address hidden>
Date: Wed May 15 19:18:52 2019 -0400

Use six.viewkeys instead of dict.keys to avoid py2 to py3 problems

    Change-Id: I0723ae10825e1e2d86789627895e3286d8c97602
    Resolves-Bug: #1829304
    (cherry picked from commit 2c1098b3afb4f5d7cd1d10eaa722652624e6bb74)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-31: Fix merged to neutron (master)

#12

Reviewed: https://review.opendev.org/662203
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a65ae9c81eea909133446c0be4d54ce389f6e0e1
Submitter: Zuul
Branch: master

commit a65ae9c81eea909133446c0be4d54ce389f6e0e1
Author: Nate Johnston <email address hidden>
Date: Thu May 30 08:53:31 2019 -0400

Use list instead of six.viewkeys to avoid py2 to py3 problems

    This change fixes an 'RuntimeError: dictionary changed size during
    iteration' error that is raised because of different behaviour between
    python2 and python3. An earlier attempt [1] to use the six library did
    not fix the issue.

[1] https://review.opendev.org/660427

Resolves-Bug: #1829304
Change-Id: Ic4c5790a5d66e10a606327f4f4e4c92611f161d8

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-31: Fix proposed to neutron (stable/stein)

#13

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/662461

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-31: Fix proposed to neutron (stable/rocky)

#14

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/662462

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-31: Fix proposed to neutron (stable/queens)

#15

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/662464

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-01: Fix merged to neutron (stable/rocky)

#16

Reviewed: https://review.opendev.org/662462
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5424bdc38b1c39ec9cd4aeac590f1ce86a5804be
Submitter: Zuul
Branch: stable/rocky

commit 5424bdc38b1c39ec9cd4aeac590f1ce86a5804be
Author: Nate Johnston <email address hidden>
Date: Thu May 30 08:53:31 2019 -0400

Use list instead of six.viewkeys to avoid py2 to py3 problems

[1] https://review.opendev.org/660427

    Resolves-Bug: #1829304
    Change-Id: Ic4c5790a5d66e10a606327f4f4e4c92611f161d8
    (cherry picked from commit a65ae9c81eea909133446c0be4d54ce389f6e0e1)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-01: Fix merged to neutron (stable/queens)

#17

Reviewed: https://review.opendev.org/662464
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=aa0d4f5f641fb8fd88616b1b291bcff826d9c0ac
Submitter: Zuul
Branch: stable/queens

commit aa0d4f5f641fb8fd88616b1b291bcff826d9c0ac
Author: Nate Johnston <email address hidden>
Date: Thu May 30 08:53:31 2019 -0400

Use list instead of six.viewkeys to avoid py2 to py3 problems

[1] https://review.opendev.org/660427

    Resolves-Bug: #1829304
    Change-Id: Ic4c5790a5d66e10a606327f4f4e4c92611f161d8
    (cherry picked from commit a65ae9c81eea909133446c0be4d54ce389f6e0e1)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-04: Fix merged to neutron (stable/stein)

#18

Reviewed: https://review.opendev.org/662461
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=690e1824a03c2f6d8a4c84efda46f3fa376ca508
Submitter: Zuul
Branch: stable/stein

commit 690e1824a03c2f6d8a4c84efda46f3fa376ca508
Author: Nate Johnston <email address hidden>
Date: Thu May 30 08:53:31 2019 -0400

Use list instead of six.viewkeys to avoid py2 to py3 problems

[1] https://review.opendev.org/660427

    Resolves-Bug: #1829304
    Change-Id: Ic4c5790a5d66e10a606327f4f4e4c92611f161d8
    (cherry picked from commit a65ae9c81eea909133446c0be4d54ce389f6e0e1)

Bernard Cafarelli (bcafarel) on 2019-06-19

tags:

added: neutron-proactive-backport-potential

Bernard Cafarelli (bcafarel) on 2020-01-14

tags:

removed: neutron-proactive-backport-potential

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.