Doc error in Auth Overview for specifying keystone domain

[James Clark]

The documentation in the Swift tree under doc/source/overview_auth.rst contains incorrect instructions for specifying the Keystone domain. This content is also displayed at: http://docs.openstack.org/developer/swift/overview_auth.html

Under "Configuring Swift to use Keystone" the doc shows:

  project_domain_id = default
  user_domain_id = default

Where the correct information is:

  project_domain_name = default
  user_domain_name = default

Specifying 'default' for the domain_id in both cases causes swift proxy to be unable to validate tokens against keystone.

See http://lists.openstack.org/pipermail/openstack/2016-July/016935.html

This error is present in the master branch at the time of bug filing:

$ git grep -n "domain_id = default" doc etc
doc/manpages/proxy-server.conf.5:281:.IP "project_domain_id = default"
doc/manpages/proxy-server.conf.5:282:.IP "user_domain_id = default"
doc/source/overview_auth.rst:160: project_domain_id = default
doc/source/overview_auth.rst:161: user_domain_id = default
etc/proxy-server.conf-sample:306:# project_domain_id = default
etc/proxy-server.conf-sample:307:# user_domain_id = default
etc/proxy-server.conf-sample:371:# default_domain_id = default

Users who copy the sample configs will also encounter the error.

Correct documentation is in the install guide:

http://docs.openstack.org/mitaka/install-guide-ubuntu/swift-controller-install.html

[Alistair Coles (alistair-coles)]

My recollection is that the keystone default domain was at some point in time named "Default" and had id "default", hence the config options in swift. I can see from the links to the install guide that this has perhaps changed. It would be great to know when/which release this changed. I'm concerned that we may have users with existing keystone installations with default domain id = "default".

[James Clark (jamiec)]

Aha, that confounds the problem a little; didn't realise that was once true.

current keystone guide uses "openstack domain create ..." which will generate an ID.

My thought would be it's better to have the current documentation consistent for those who follow it. Would the _domain_name parameters be safer in both cases? The _name parameters aren't case sensitive.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/372573

[Alistair Coles (alistair-coles)]

Adding some notes:

Here [1][2] we have a description of a default (i.e. installed by default) domain with id=default and name=Default. Before Mitaka, this was the domain referenced in the install guides [3] for use in the authtoken credentials. The description of the default domain still seems to be valid, being the destination for v2 API requests in absence of any known domain.

However, here [4], since Mitaka, we have a domain with name=default being created where the service users are created.

[1] https://github.com/openstack/keystone/blob/master/etc/keystone.conf.sample#L934
[2] http://docs.openstack.org/developer/keystone/services.html#the-default-domain
[3] http://docs.openstack.org/liberty/install-guide-ubuntu/keystone-users.html
[4] http://docs.openstack.org/mitaka/install-guide-ubuntu/keystone-users.html

[Alistair Coles (alistair-coles)]

Fix proposed here https://review.openstack.org/#/c/372573

[Alistair Coles (alistair-coles)]

A couple more links to various authtoken config examples:

here [5] we see user_domain_name =Default

here [6] we see user_domain_id = default

[5] http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html
[6] https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L198

[Guang Yee (guang-yee)]

I think Keystone doc is confusing at the very least. We should not recommend, or at least WARN, deployers about changing the default domain ID to anything other than 'default' as it is assumed in many places. Keystone default domain should be bootstrapped via "keystone-manage bootstrap" CLI.

[James Clark (jamiec)]

We're not suggesting changing the default domain ID at all. The main problem being that an old assumption on the ID no longer appears to hold true, and there is now a documentation inconsistency.

So it is now safer to refer to the default domain by *name* "Default" than by *id* which can be something other than "default". This is already partially addressed in the Swift documentation.

Looking at the current Keystone install guide shows that the default domain ID is now likely to be something other than "default": http://docs.openstack.org/mitaka/install-guide-ubuntu/keystone-users.html

[Alistair Coles (alistair-coles)]

For the time being I have made this patch https://review.openstack.org/#/c/372573 simply add some more text to the comments to draw attention to the need to get the correct options. I'm still not convinced how to best align with the various ways to install keystone, and an install guide that is questioned (e.g. Guang's comment at #7).

More notes, with *some* more clarity:

With Mitaka release, keystone-manage db_sync no longer creates the Default domain:
http://docs.openstack.org/releasenotes/keystone/mitaka.html

Instead keystone-manage bootstrap creates the default domain - usage described here:
http://docs.openstack.org/developer/keystone/configuringservices.html

OR it is created when any v2 API request is received.

The default domain is defined here:
https://github.com/openstack/keystone/blob/master/keystone/cmd/cli.py#L181

with reference to:
https://github.com/openstack/keystone/blob/d32910b29dff9c1982a5fc064aad1dcf31a6cced/keystone/conf/identity.py#L19

i.e. the default domain has {id=default, name=Default}

Now, the Mitaka install guide does not AFAICT specify using keystone-manage bootstrap, but does instruct to create a domain with name 'default' for which a random ID will be generated:
http://docs.openstack.org/mitaka/install-guide-ubuntu/keystone-users.html

Clearly this domain with name=default is then used by install guide for the service users and project. I do not know if it somehow also becomes the domain which handles any v2 API requests or whether the domain with id=default is still autocreated when a v2API request is received. I assume the latter, i.e. that although this domain has name=default, it is not equivalent to the "default domain". But that uncertainty is not the scope of this bug.

It would seem that if you follow the install guide you end up with a domain with {name=default, id=UUID} in which the swift service is created, so the swift docs could use that as an example.

However if you follow the instructions to install keystone here http://docs.openstack.org/developer/keystone/configuringservices.html then you will use 'keystone-manage bootstrap' which will create a domain with {name=Default, id=default} and then create a swift user in that domain, which is what our docs currently reflect.

I don't see the answer to be simply changing swift docs to use *_domain_name=Default as suggested in #8 because that is inconsistent with the install guide that says *_domain_name=default (so it's just plain confusing to the user). It *may* work if you install keystone with a case insensitive backend, or if you used 'keystone-manage bootstrap', but in that case what we have already works (*_domain_id=default).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/372573
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=18bb99971f1a793dc75b6b3cb393d5503be43575
Submitter: Jenkins
Branch: master

commit 18bb99971f1a793dc75b6b3cb393d5503be43575
Author: Alistair Coles <email address hidden>
Date: Mon Sep 19 16:06:18 2016 +0100

    Add more comment to authtoken sample options

    Prior to the Mitaka release the install guides showed
    services (including Swift) being in a default Keystone
    domain which existed by default and has id=default. This
    domain id is reflected in the proxy-server.conf-sample
    authtoken options and also shown in man page and auth docs.

    The Mitaka install guide shows a domain with *name* default
    being created, and having a random UUID assigned, in which
    services are created. This has caused confusion (see
    discussion on linked bug report).

    This patch does not change the sample options but does
    add to the comments in order to emphasize that a user
    may need to alter the options to match their Keystone
    configuration.

    Change-Id: I17bfcdbd983402eeb561bb704b8b1f1e27547c7d
    Partial-Bug: #1604674

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/400985

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/400985
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0c3f8f87104af8717115c5badffd243dbaa1c430
Submitter: Jenkins
Branch: feature/hummingbird

commit 2d25fe6ad3573b2a06b6b3e5e66493d7b0c55693
Author: Tim Burke <email address hidden>
Date: Mon Jul 25 15:06:23 2016 -0700

    Reduce backend requests for SLO If-Match / HEAD requests

    ... by storing SLO Etag and size in sysmeta.

    Previously, we had to GET the manifest for every HEAD or conditional
    request to an SLO. Worse, since SLO PUTs require that we HEAD every
    segment, we'd GET all included sub-SLO manifests. This was necessary so
    we could recompute the large object's Etag and content-length.

    Since we already know both of those during PUT, we'll now store it in
    object sysmeta. This allows us to:

     * satisfy HEAD requests based purely off of the manifest's HEAD
       response, and
     * perform the If-(None-)Match comparison on the object server, without
       any additional subrequests.

    Note that the large object content-length can't just be parsed from
    content-type -- with fast-POST enabled, the content-type coming out of
    the object-server won't necessarily include swift_bytes.

    Also note that we must still fall back to GETting the manifest if the
    sysmeta headers were not found. Otherwise, we'd break existing large
    objects.

    Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9

commit ae7dddd801e28217d7dc46bd45cd6b621f29340c
Author: Ondřej Nový <email address hidden>
Date: Mon Nov 21 22:13:11 2016 +0100

    Added comment for "user" option in drive-audit config

    Change-Id: I24362826bee85ac3304e9b63504c9465da673014

commit c3e1d847f4b9d6cc6212aae4dc1b1e6dff45fb40
Author: Thiago da Silva <email address hidden>
Date: Thu Nov 17 17:17:00 2016 -0500

    breaking down tests.py into smaller pieces

    tests.py is currently at ~5500 lines of code, it's
    time to break it down into smaller files.

    I started with an easy middleware set of tests
    (i.e., versioned writes, ~600 lines of code ) so I can get
    some feedback. There are more complicated tests that cover
    multiple middlewares for example, it is not so clear where
    those should go.

    Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
    Signed-off-by: Thiago da Silva <email address hidden>

commit 5d7a3a4172f0f11ab870252eec784cf24b247dea
Author: Ondřej Nový <email address hidden>
Date: Sat Nov 19 23:24:30 2016 +0100

    Removed "in-process-" from func env tox name

    This shorten shebang in infra, because we are hitting 128 bytes limit.

    Change-Id: I02477d81b836df71780942189d37d616944c4dce

commit 9ea340256996a03c8c744201297b47a0e91fe65b
Author: Kota Tsuyuzaki <email address hidden>
Date: Fri Nov 18 01:50:11 2016 -0800

    Don't overwrite built-in 'id'

    This is a follow up for https://review.openstack.org/#/c/399237

    'id' is assigned as a builtin function so that we should not use 'id'
    for the local variable name.

    Change-Id: Ic27460d49e68f6cd50bda1d5b3810e01ccb07a37

commit bf...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/443750

[Matthew Oliver (matt-0)]

In my playing with patch https://review.openstack.org/#/c/454386, I investigated this *_id = default and *_name = Default and it depends on how keystone is storing your default domain, but by default:

<snip from review>
From my look, while reviewing this patch:

https://docs.openstack.org/developer/keystone/devref/services.html#the-default-domain

So params ending in _id are default and the -name are Default.. which I find annoying. But there you go. Our sample doc uses _id = default which "should" be correct.

What do others use, well according to https://docs.openstack.org/developer/keystonemiddleware/api/keystonemiddleware.auth_token.html
it gives a sample, apparently from nova that:

  [keystone_authtoken]
  auth_plugin = password
  auth_url = http://keystone:35357/
  username = nova
  user_domain_id = default
  password = whyarewestillusingpasswords
  project_name = service
  project_domain_id = default

Which using using "default" against the *_id versions of the configuration.

Now if I pop into my test keystone server, and take a look:

  (openstack) domain list
  +---------+---------+---------+--------------------+
  | ID | Name | Enabled | Description |
  +---------+---------+---------+--------------------+
  | default | Default | True | The default domain |
  +---------+---------+---------+--------------------+
  (openstack) domain show default
  +-------------+--------------------+
  | Field | Value |
  +-------------+--------------------+
  | description | The default domain |
  | enabled | True |
  | id | default |
  | name | Default |
  +-------------+--------------------+

Now lets do some keystone config tests.
First using:

  [filter:authtoken]
  paste.filter_factory = keystonemiddleware.auth_token:filter_factory
  auth_url = http://localhost:35357
  auth_uri = http://localhost:5000
  auth_plugin = password
  project_name = service
  username = swift
  password = Sekr3tPass
  delay_auth_decision = True
  user_domain_id = default
  project_domain_id = default

It works:

  $ swift stat
                 Account: KEY_fce1546e0675483bb3862a54a2ed68f6
              Containers: 0
                 Objects: 0
                   Bytes: 0
         X-Put-Timestamp: 1491904442.67731
             X-Timestamp: 1491904442.67731
              X-Trans-Id: tx0814c8d8f3694f47a13d9-0058eca7ba
            Content-Type: text/plain; charset=utf-8
  X-Openstack-Request-Id: tx0814c8d8f3694f47a13d9-0058eca7ba

Change to _id = Default:

  [filter:authtoken]
  paste.filter_factory = keystonemiddleware.auth_token:filter_factory
  auth_url = http://localhost:35357
  auth_uri = http://localhost:5000
  auth_plugin = password
  project_name = service
  username = swift
  password = Sekr3tPass
  delay_auth_decision = True
  user_domain_id = Default
  project_domain_id = Default

And:

  $ swift stat
  Account HEAD failed: http://localhost:8080/v1/KEY_fce1546e0675483bb3862a54a2ed68f6 503 Service Unavailable
  Failed Transaction ID: tx4ead4df812e2407f95b98-0058eca84d

And in the keystone log you get:

  2017-04-11 09:56:05.681 20254 INFO keystone.common.wsgi [req-cb4...

[Boris Bobrov (bbobrov)]

Today in most major OpenStack installers the following convention is true:

user_domain_id = default
project_domain_id = default

user_domain_name = Default
project_domain_name = Default

Domain "Default" is created by keystone-manage bootstrap and in some cases when v2 API is used. In keystone it is there only to support legacy v2 api. Name "Default" is not configurable. ID "default" is configurable in keystone.conf:[identity]default_domain_id.

In a custom installation deployers might choose to have other id for default domain or not to have default domain at all.

[Matthew Oliver (matt-0)]

Wow, right, so bootstrap will create id=default, name=Default. And that's what I did, so now that makes sense.

However, from Newton+ ubuntu guide now does point to using bootstrap (see step 5 of install and configure components https://docs.openstack.org/ocata/install-guide-ubuntu/keystone-install.html). And mitaka is about to be EOLed.

So from newton onwards its pretty safe to say {id=default, name=Default} is a pretty safe example.

Although from the sound of what Boris is saying, name=Domain is not configurable so might be ever more correct, so maybe:

  user_domain_name = Default
  project_domain_name = Default

Is the best combination now?

[Alistair Coles (alistair-coles)]

Note Related bug [4]

AFAICT there was a change to how the 'default' keystone domain was created in Mitaka - that required a domain with name default and random id to be manually created [0] vs previously there being a domain with id 'default' being automatically created during keystone bootstrap.

Since Newton the keystone install seems to have reverted, so bootstrap is used again to create the domain with id 'default' [1]. Note here that the id of the project parent is 'default'. Which implies that the config we have on master in docs sample config files and docs [3] is correct. But at time of writing the install guide is wrong [4].

[0] https://docs.openstack.org/mitaka/install-guide-ubuntu/keystone-users.html
[1] https://docs.openstack.org/developer/keystone/devref/services.html#the-default-domain
[2] https://docs.openstack.org/newton/install-guide-ubuntu/keystone-users.html
[3] https://docs.openstack.org/developer/swift/overview_auth.html#configuring-swift-to-use-keystone
[4] https://launchpad.net/bugs/1682059

[Alistair Coles (alistair-coles)]

As per [1] we're now using default domain with id=default and name=Default, so I'm making this bug as Won't Fix, for want of a better description. The bug/discrepancy seems to have been a transient confusion in the mItaka install guide. The keystone install guide has now reverted to using a default domain with id=default, which is what swift has always documented.

Also note that the install guide was added to the swift repo [2] after the Mitaka release so any fix for the mitaka install guide would need to be backported to a different project (openstack-docs??)

[1] https://bugs.launchpad.net/swift/+bug/1682059
[2] commit cf8b93918cc8a46ebd9035028847d8009d362794 https://review.openstack.org/#/c/330070/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alistair Coles (<email address hidden>) on branch: master
Review: https://review.openstack.org/443750