Swift max header size too small

[p.cipriano]

Hi,

After enabling swift and follow the guide.

I have ann error

Error: Unable to get the Swift container listing via Horizon

via CLI

[root@controller01 ~]# openstack container list
Bad Request (HTTP 400) (Request-ID: tx0ec699f30c45401b99c16-005b21fab0)

==> horizon.log <==
[Thu Jun 14 01:19:04.232688 2018] [:error] [pid 80] REQ: curl -i http://107.105.135.163:8080/info -X GET -H "Accept-Encoding: gzip"
[Thu Jun 14 01:19:04.232825 2018] [:error] [pid 80] RESP STATUS: 401 Unauthorized
[Thu Jun 14 01:19:04.233119 2018] [:error] [pid 80] RESP HEADERS: {u'Content-Length': u'114', u'WWW-Authenticate': u"Keystone uri='http://107.105.135.163:5000'", u'X-Trans-Id': u'tx045f860b4dfc41e394cec-005b21fac8', u'Date': u'Thu, 14 Jun 2018 05:19:04 GMT', u'Content-Type': u'application/json', u'X-Openstack-Request-Id': u'tx045f860b4dfc41e394cec-005b21fac8'}
[Thu Jun 14 01:19:04.233413 2018] [:error] [pid 80] RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
[Thu Jun 14 01:19:04.245298 2018] [:error] [pid 81] REQ: curl -i http://107.105.135.163:8080/v1/AUTH_04eabf5e839a4476a4d312e7941a1e4f?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABbIfq-DQLK..."
[Thu Jun 14 01:19:04.245414 2018] [:error] [pid 81] RESP STATUS: 400 Header Line Too Long
[Thu Jun 14 01:19:04.245680 2018] [:error] [pid 81] RESP HEADERS: {u'Date': u'Thu, 14 Jun 2018 05:19:04 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx455f6eccb7d94a22b426a-005b21fac8', u'X-Trans-Id': u'tx455f6eccb7d94a22b426a-005b21fac8'}
[Thu Jun 14 01:19:04.282315 2018] [:error] [pid 79] REQ: curl -i http://107.105.135.163:8080/v1/AUTH_04eabf5e839a4476a4d312e7941a1e4f?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABbIfq-DQLK..."
[Thu Jun 14 01:19:04.282418 2018] [:error] [pid 79] RESP STATUS: 400 Header Line Too Long
[Thu Jun 14 01:19:04.282570 2018] [:error] [pid 79] RESP HEADERS: {u'Date': u'Thu, 14 Jun 2018 05:19:04 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'txa76e16eae0b341dcadb9a-005b21fac8', u'X-Trans-Id': u'txa76e16eae0b341dcadb9a-005b21fac8'}

[Eduardo Gonzalez (egonzalez90)]

This is caused because keystone v3 tokens exceeds the max header size allowed in swift.

We need to add into swift.conf with a higher value than default (8192)

[swift-constraints]
max_header_size=32768

Regards

[p.cipriano (ctpeter)]

The problem still exist

==> horizon.log <==
[Thu Jun 14 05:00:23.287534 2018] [:error] [pid 83] REQ: curl -i http://107.105.135.163:8080/info -X GET -H "Accept-Encoding: gzip"
[Thu Jun 14 05:00:23.287625 2018] [:error] [pid 83] RESP STATUS: 401 Unauthorized
[Thu Jun 14 05:00:23.287768 2018] [:error] [pid 83] RESP HEADERS: {u'Content-Length': u'114', u'WWW-Authenticate': u"Keystone uri='http://107.105.135.163:5000'", u'X-Trans-Id': u'txa31af40a16a74329b56de-005b222ea7', u'Date': u'Thu, 14 Jun 2018 09:00:23 GMT', u'Content-Type': u'application/json', u'X-Openstack-Request-Id': u'txa31af40a16a74329b56de-005b222ea7'}
[Thu Jun 14 05:00:23.287886 2018] [:error] [pid 83] RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
[Thu Jun 14 05:00:23.308937 2018] [:error] [pid 82] REQ: curl -i http://107.105.135.163:8080/v1/AUTH_04eabf5e839a4476a4d312e7941a1e4f?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABbIi6hT7D_..."
[Thu Jun 14 05:00:23.309039 2018] [:error] [pid 82] RESP STATUS: 400 Header Line Too Long
[Thu Jun 14 05:00:23.309215 2018] [:error] [pid 82] RESP HEADERS: {u'Date': u'Thu, 14 Jun 2018 09:00:23 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx558f165204f24ce5820bd-005b222ea7', u'X-Trans-Id': u'tx558f165204f24ce5820bd-005b222ea7'}
[Thu Jun 14 05:00:23.350241 2018] [:error] [pid 81] REQ: curl -i http://107.105.135.163:8080/v1/AUTH_04eabf5e839a4476a4d312e7941a1e4f?format=json&limit=1001 -X GET -H "Accept-Encoding: gzip" -H "X-Auth-Token: gAAAAABbIi6hT7D_..."
[Thu Jun 14 05:00:23.350337 2018] [:error] [pid 81] RESP STATUS: 400 Header Line Too Long
[Thu Jun 14 05:00:23.350470 2018] [:error] [pid 81] RESP HEADERS: {u'Date': u'Thu, 14 Jun 2018 09:00:23 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6c9f3f34367b4069a2348-005b222ea7', u'X-Trans-Id': u'tx6c9f3f34367b4069a2348-005b222ea7'}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/576412

[alpha23 (alpha23)]

Still and issue as of today in Rocky.

[alpha23 (alpha23)]

Does this config change need to be included in all of the below?

./swift-container-replicator/swift.conf
./swift-object-server/swift.conf
./swift-account-server/swift.conf
./swift-object-expirer/swift.conf
./swift-account-reaper/swift.conf
./swift-object-auditor/swift.conf
./swift-object-updater/swift.conf
./swift-container-updater/swift.conf
./swift-container-auditor/swift.conf
./swift-account-replicator/swift.conf
./swift-object-replicator/swift.conf
./swift-account-auditor/swift.conf
./swift-container-server/swift.conf
./swift-proxy-server/swift.conf

[Mark Goddard (mgoddard)]

The discussion in the proposed patch suggests more work may be required than just adding that config option. If you can verify that it works, we could accept it.

[alpha23 (alpha23)]

Still an issue in Stein. While my previous suggested fix above works, there is another issue in that creating a conf file in the /etc/kolla/config/swift/ folder, e.g. swift-account-auditor.conf, does not copy the settings to, e.g, /etc/kolla/swift-account-auditor/swift.conf, when upgrading. I therefore manually copied to the settings to each file. Including a swift.conf file in /etc/kolla/config/ with the same settings does not resolve the issue.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Michal Nasiadka <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/576412
Reason: No updates since 2018

[alpha23 (alpha23)]

Still an issue in Ussuri. When will this be fixed?

As a workaround, after upgrading, from /etc/kolla/ folder (assuming standard config) on each swift node:

1. sudo printf "\n[swift-constraints]\nmax_header_size=32768\n" | sudo tee -a swift-{account,container,object,proxy}-{auditor,replicator,replication-server,reaper,server,updater,expirer}/swift.conf > /dev/null

2. sudo docker restart $(sudo docker ps -aq --filter name=swift)

[alpha23 (alpha23)]

Still and issue in Victoria.

[alpha23 (alpha23)]

Still an issue in Wallaby and Xena. What I don't understand is why the configuration settings in /etc/kolla/config/swift do not correctly copy over, such that I have to manually run the command I mentioned on 2023-02-10. If that was done, I at least would not have to manually update the swift configuration on each upgrade and on each reconfiguration.

[Roman Krček (r-krcek)]

Hello, I have tried to reproduce this issue, however I was unsuccessful. I was able to install swift and run "openstack container list" without any problems. Could you share more info about your deployment and configuration?