Keystone returns 500 for v3 api with PKI tokens

[Dmitry Ukov]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"
  nailgun_sha: "4162b0c15adb425b37608c787944d1983f543aa8"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"

Steps to reproduce
1. Deploy fuel 7.0
2. Create environment (1 controller, 1 compute)
3. Deploy environment
4. Edit keystone configuration file. replace
    provider = keystone.token.providers.uuid.Provider
    with
    provider = keystone.token.providers.pki.Provider
5. Restart apache service
6. Create openrcv3 file
    # cat openrcv3
    export OS_USERNAME=admin
    export OS_PASSWORD=admin
    export OS_TENANT_NAME=admin
    export OS_AUTH_URL=http://localhost:5000/v3
    export OS_VOLUME_API_VERSION=2
    export OS_IDENTITY_API_VERSION=3
7. Execute # source openrcv3
8. Execute
    # openstack domain list
    Expected result
      - list of keystone domains
    Actual result
      - ERROR: openstack Internal Server Error (HTTP 500)

Diagnostic Snapshot
https://drive.google.com/file/d/0B0kV2KAlVj3Na2RZeHhuWHl1eHc/view?usp=sharing

[Dmitry Ukov (dukov)]

I see following error in Apache logs
[Tue Oct 27 10:23:03.955795 2015] [core:error] [pid 3968:tid 139918802110208] [client ::1:35725] Premature end of script headers: main
[Tue Oct 27 10:23:03.962124 2015] [:error] [pid 3964:tid 139918953936640] [remote ::1:33394] mod_wsgi (pid=3964): Exception occurred processing WSGI script '/usr/lib/cgi-bin/keystone/main'.
[Tue Oct 27 10:23:03.962346 2015] [:error] [pid 3964:tid 139918953936640] [remote ::1:33394] IOError: failed to write data

[Dmitry Ukov (dukov)]

It seems that issue is related to mod_wsgi. There is no way to configure response headers (i guess 8KB is maximum) in version 3.4 which is used in MOS 7.0 . Version 4.3 has additional option header-buffer-size which can be used to configure max response header

[Dmitry Ukov (dukov)]

UPD: version with header-buffer-size is 4.1

[Roman Podoliaka (rpodolyaka)]

Do we support Fernet tokens for 7.0 at all?

[Roman Podoliaka (rpodolyaka)]

They are not enabled for 8.0 either.

[Dmitry Ukov (dukov)]

You right but we have to support everything which suppose to work in vanilla OpenStack. Moreover MOS users require PKI tokens to be enabled. Therefore we need this issue fixed for all releases which utilize Apache for Keystone

[Alexander Makarov (amakarov)]

Thanks for heads-up, colleagues.

On Fri, Nov 13, 2015 at 11:01 AM, Dmitry Ukov
<email address hidden> wrote:
> You right but we have to support everything which suppose to work in
> vanilla OpenStack. Moreover MOS users require PKI tokens to be enabled.
> Therefore we need this issue fixed for all releases which utilize Apache
> for Keystone
>
> --
> You received this bug notification because you are a member of MOS
> Keystone, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1510446
>
> Title:
> Keystone returns 500 for v3 api with PKI tokens
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1510446/+subscriptions

--
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: [phone]
Tel.: [phone]

Skype: MAKAPOB.AJIEKCAHDP

[Alexander Makarov (amakarov)]

From Dims' letter:

Thomas,

Looks like we may have to try latest Debian mod_wsgi for [1]. What's the current level? We'll need at least 4.4.15 for the graceful restart bug.

Thanks,
Dims

[1] https://bugs.launchpad.net/mos/+bug/1510446
[2] https://github.com/GrahamDumpleton/mod_wsgi/blob/5458417ac6f645ab6943c35f5d6a424d3134b123/docs/release-notes/version-4.4.15.rst

Need to try with fixed mod_wsgi

[Davanum Srinivas (DIMS) (dims-v)]

Sasha,

Are we going to update to newer mod_wsgi in MOS 8.0? Are we going to update mod_wsgi in a 7.0 MU update as well?

[Alexander Makarov (amakarov)]

Looks like we have to wait for fix to get into the ubuntu repository

[Davanum Srinivas (DIMS) (dims-v)]

Should we merge this for 8.0? https://review.fuel-infra.org/#/c/14067/

[Alexander Makarov (amakarov)]

We finally came to agreement with Igor Yozhikov that we need this patch in 8.0

[Davanum Srinivas (DIMS) (dims-v)]

Igor, I still see the following reviews not merged:

https://review.fuel-infra.org/#/c/14067/
https://review.fuel-infra.org/#/c/15652/

Can you please take a look to see what needs to be done?

[Davanum Srinivas (DIMS) (dims-v)]

Looks like https://review.fuel-infra.org/#/c/14067/ merged.

[Alexander Petrov (apetrov-n)]

Not reproduced on the latest MOS 8.0 ISO on my environment.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "481"
  build_id: "481"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

[Denis Meltsaykin (dmeltsaykin)]

Dmirty, could you please confirm that the fix is still needed in 7.0?