Some API requests (GET networks) fail with "Accept: application/json; charset=utf-8" header and WebOb>=1.8.0

[Bernard Cafarelli]

Original downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=1706222

On versions newer than Rocky, we have WebOb 1.8 in requirements. This causes the following API calls to end with 500 error:
GET http://localhost:9696/v2.0/ports
GET http://localhost:9696/v2.0/subnets
GET http://localhost:9696/v2.0/networks

when setting an Accept header with charset like "Accept: application/json; charset=utf-8"

These calls do not go through neutron.api.v2 and wsgi.request as other resources, is it something that should be fixed too?

To reproduce (on master too):
$ curl -s -H "Accept: application/json; charset=utf-8" -H "X-Auth-Token: $OS_TOKEN" "http://localhost:9696/v2.0/ports" | python -mjson.tool
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"
    }
}

mai 14 18:16:19 devstack neutron-server[1519]: DEBUG neutron.wsgi [-] (1533) accepted ('127.0.0.1', 47790) {{(pid=1533) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:956}}
mai 14 18:16:19 devstack neutron-server[1519]: ERROR pecan.core [None req-0848fbc9-5c8a-4713-b436-029814f89a32 None demo] content type None
mai 14 18:16:19 devstack neutron-server[1519]: ERROR pecan.core [None req-0848fbc9-5c8a-4713-b436-029814f89a32 None demo] Controller 'index' defined does not support content_type 'None'. Supported type(s): ['application/json']
mai 14 18:16:19 devstack neutron-server[1519]: INFO neutron.pecan_wsgi.hooks.translation [None req-0848fbc9-5c8a-4713-b436-029814f89a32 None demo] GET failed (client error): The server could not comply with the request since it is either malformed or otherwise incorrect.
mai 14 18:16:19 devstack neutron-server[1519]: INFO neutron.wsgi [None req-0848fbc9-5c8a-4713-b436-029814f89a32 None demo] 127.0.0.1 "GET /v2.0/ports HTTP/1.1" status: 406 len: 360 time: 0.2243972

Relevant WebOb warning: https://github.com/Pylons/webob/blob/master/docs/whatsnew-1.8.txt#L24

[Nate Johnston (nate-johnston)]

I believe we have conclusively determined in [1] that this is not a WebOb issue. I believe that there may be different codepaths between the neutron API methods that see this issue and those that do not. I will note that the error seen in the Neutron server.log when this happens tells me that it is not handling parsing the string correctly.

% curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/networks"
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"
    }
}

The logs:

2019-05-07 14:47:47.341 77 DEBUG neutron.wsgi [-] (77) accepted ('172.17.1.31', 36530) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2019-05-07 14:47:47.345 77 ERROR pecan.core [req-58872c3a-e917-4e3e-82cf-d3dbd931b6b2 21ec7039924a46e9aec6a9fa490da066 fb53e9370fb0427fadb680238d90987a - default default] Controller 'index' defined does not support content_type 'None'. Supported type(s): ['application/json']
2019-05-07 14:47:47.346 77 INFO neutron.pecan_wsgi.hooks.translation [req-58872c3a-e917-4e3e-82cf-d3dbd931b6b2 21ec7039924a46e9aec6a9fa490da066 fb53e9370fb0427fadb680238d90987a - default default] GET failed (client error): The server could not comply with the request since it is either malformed or otherwise incorrect.
2019-05-07 14:47:47.347 77 INFO neutron.wsgi [req-58872c3a-e917-4e3e-82cf-d3dbd931b6b2 21ec7039924a46e9aec6a9fa490da066 fb53e9370fb0427fadb680238d90987a - default default] 172.17.1.31 "GET /v2.0/networks HTTP/1.1" status: 406 len: 360 time: 0.0057609

It gets weirder. This is only a problem for certain methods.

/v2.0/address-scopes is fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/address-scopes" | python -mjson.tool
{
    "address_scopes": []
}

/v2.0/ports is no good:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfUtoM6_4IUzDPKnQ" "http://192.168.24.1:9696/v2.0/ports" | python -mjson.tool
{
    "NeutronError": {
        "detail": "",
        "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.",
        "type": "HTTPNotAcceptable"
    }
}

/v2.0/flavors is fine:

[stack@undercloud-0 ~]$ curl -s -H "Accept: application/json;charset=utf-8" -H "X-Auth-Token: gAAAAABc0cHuHmvz0YFb0uuPv5ADSxi5Vda_pi-Z8KwIeqjfIu2MtjBLA6NaoRorAjS8HrPzbmrby0K28uajyYj5GiKIJcMCDDYRhN1Hs_2Bfb-4ORbBBj2_X7o2gnybxaBak-TVdrQWdWsHyYnBfTq5CCT-K_mCwPd4KgQfU...

[Dan Radez (dradez)]

Further testing would indicate this is not a WebOb issue.

The value that is being passed by the client conforms to RFC7231: https://tools.ietf.org/html/rfc7231#section-3.1.1

According to the WebOb issue that was created: https://github.com/Pylons/webob/issues/403
if we want to match on a header that possibly includes key=value pairs behind the content type we have to specifically match on the full value 'application/json;charset=utf-8'

An initial fix would be to split on ; and only pass */* part of the Accept header. We don't seem to care about any other values passed.

A longer term fix would be to compare why certain methods work and others don't and make sure they're all being call consistently.

[Dan Radez (dradez)]

More explanation:
The following Accept Header Values are not valid and and generate a AcceptInvalidHeader object in WebOb. Being invalid WebOb discards the header and ends up defaulting the content type to application/json which then succeeds to process.

application/json;charset=
application/json;charset
application/json,charset=utf-8
charset=utf-8;application/json

When we pass valid header syntax application/json;charset=utf-8 We get a AcceptValidHeader object in WebOb. Because this is valid syntax WebOb tries to match our explicitly defined content type application/json to the passed content type application/json;charset=utf-8. This matching happens without any processing on the header string.

application/json != application/json;charset=utf-8

Since these don't match we get exceptions that state we have passed an invalid content type and the request bails.

[Dan Radez (dradez)]

Attempting to fix this in pecan:
https://github.com/pecan/pecan/pull/119

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744035
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e92193d246c8ea30d42a064297bb7a1955c62a98
Submitter: Zuul
Branch: master

commit e92193d246c8ea30d42a064297bb7a1955c62a98
Author: Dan Radez <email address hidden>
Date: Thu Jul 30 15:21:34 2020 -0400

    Updating the pecan version requirement

    There's a fix in pecan 1.4 to handle accept headers that have
    extra parameters included.

    Closes-Bug: #1829042
    Change-Id: Id7d78d77da8dfd1620936e437d862c1d60e8eb25

[Minho Ban (mhban)]

Unfortunately the fixup was reverted by the https://review.opendev.org/c/openstack/neutron/+/752058/
It seems it was reverted just to pass the gate-failure but after that no one followed up.

[Minho Ban (mhban)]

somebody must reopen this

[Bernard Cafarelli (bcafarel)]

Follow-up is waiting for other packages that are apparently broken by newer pecan (but did not receive many patches this cycle...). We are waiting for https://review.opendev.org/c/openstack/requirements/+/747419

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/865575

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865575
Committed: https://opendev.org/openstack/neutron/commit/499ae9588141b04ab9c6e1ecd421f5cd9af5f447
Submitter: "Zuul (22348)"
Branch: master

commit 499ae9588141b04ab9c6e1ecd421f5cd9af5f447
Author: Bence Romsics <email address hidden>
Date: Thu Nov 24 16:27:36 2022 +0100

    Bump pecan to fix #1829042

    This has a long history [1][2] but at this point we are already using
    1.4.0+ versions of pecan so this change is proven to be safe.

    [1] https://review.opendev.org/c/openstack/neutron/+/752058
    [2] https://review.opendev.org/c/openstack/requirements/+/747419

    Change-Id: I9fa0ed8c2170baa98e4dad868e0f614f92f6c7ab
    Closes-Bug: #1829042

[Srujana Bandari (sbandari)]

When can we have these fixes be backported to Openstack Victoria neutron

https://review.opendev.org/c/openstack/neutron/+/865575

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.