Cannot deploy stable/ocata keystone due to missing policy.json

[Yushiro FURUKAWA]

I tried to deploy stable/ocata environment for following 2 ways in Ubuntu 16.04.2 LTS. Both ways were failed to deploy. Am I missing something?

Pattern A: using master devstack and following local.conf

  REQUIREMENTS_BRANCH=stable/ocata
  KEYSTONE_BRANCH=stable/ocata
  NOVA_BRANCH=stable/ocata
  NEUTRON_BRANCH=stable/ocata
  GLANCE_BRANCH=stable/ocata
  CINDER_BRANCH=stable/ocata
  IRONIC_BRANCH=stable/ocata
  SWIFT_BRANCH=stable/ocata

  disable_service n-net
  disable_service horizon
  disable_service tempest
  disable_service c-api
  disable_service c-vol
  disable_service c-sch
  enable_service neutron
  enable_plugin ironic https://git.openstack.org/openstack/ironic stable/ocata
  enable_service s-proxy
  enable_service s-object
  enable_service s-container
  enable_service s-account
  ..(snip)...

Pattern B: using stable/ocata devstack and same local.conf with above definition.

[Error for Pattern A] /opt/stack/logs/stack.sh.log

     ...(snip)...
    2017-06-12 13:21:57.118 | ++lib/keystone:create_keystone_accounts:330 openstack project show admin -f value -c id
    2017-06-12 13:22:00.598 | You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-55f243e3-8720-4cc2-a63d-8c5dfcfa269d)

    I executed 'source devstack/openrc admin admin; openstack --debug endpoint list' and got an error:
        ...(snip)...
        REQ: curl -g -i -X GET http://192.168.122.198/identity/v3/auth/tokens -H "X-Subject-Token: {SHA1}23dde272ead75b0e520d229864a9fb9931aeabce" -H "User-Agent: python-keystoneclient"
 -H "Accept: application/json" -H "X-Auth-Token: {SHA1}23dde272ead75b0e520d229864a9fb9931aeabce"
        Resetting dropped connection: 192.168.122.198 http://192.168.122.198:80 "GET /identity/v3/auth/tokens
 HTTP/1.1" 403 141
        RESP: [403] Date: Mon, 12 Jun 2017 13:22:54 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token Content-Type: application/json Content-Length: 141 x-openstack-request-id: req-bb143aa4-e31a-46f6-91e2-89984a512ad4 Connection: close
        RESP BODY: {"error": {"message": "You are not authorized to perform the requested action: identity:validate_token.", "code": 403, "title": "Forbidden"}}
        ...(snip)...

[Error for Pattern B] /opt/stack/logs/stack.sh.log
    2017-06-12 13:52:53.474 | ++:: curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}' http://192.168.122.198/identity/v3/
    2017-06-12 13:52:53.498 | +:: [[ 503 == 503 ]]
    2017-06-12 13:52:53.505 | +:: sleep 1
    2017-06-12 13:52:54.517 | ++:: curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}' http://192.168.122.198/identity/v3/
    2017-06-12 13:52:54.537 | +:: [[ 503 == 503 ]]
    2017-06-12 13:52:54.544 | +:: sleep 1
    ...(snip)...
    2017-06-12 13:52:55.363 | [ERROR] /home/stack/devstack/lib/keystone:615 keystone did not start
    2017-06-12 13:52:56.371 | Error on exit

  I also checked /var/log/apache2/error.log

    [Mon Jun 12 22:56:01.868120 2017] [proxy:error] [pid 32263:tid 140048708118272] (111)Connection refused: AH02454: uwsgi: attempt to connect to Unix domain socket /var/run/uwsgi/keystone-wsgi-public.socket (uwsgi-uds-keystone-wsgi-public) failed
    [Mon Jun 12 22:56:01.868214 2017] [proxy:error] [pid 32263:tid 140048708118272] AH00959: ap_proxy_connect_backend disabling worker for (uwsgi-uds-keystone-wsgi-public) for 0s
    [Mon Jun 12 22:56:01.868232 2017] [:error] [pid 32263:tid 140048708118272] [client 192.168.122.198:36640] failed to make connection to backend: httpd-UDS:0

[Lance Bragstad (lbragstad)]

The error from pattern A is certainly a policy traceback. I'd be curious to know what user or what the state of the identity/assignment tables are when devstack is making that call.

The error from pattern B looks like keystone is having a hard time connecting to the database or backend store. Are you able to double check the keystone configuration files to ensure it can actually talk to the database?

Both of these sound like issues related to how keystone was configured, which means it could be something with devstack causing this and not keystone.

[Kristi Nikolla (knikolla)]

I was unable to reproduce pattern B with stable/ocata on ubuntu 16.04.2.

[Lance Bragstad (lbragstad)]

Marking this as invalid for now. If the issue resurfaces, please feel free to reopen this issue.

[Silvan Kaiser (2-silvan)]

Hi! I'm hitting pattern A with a similar setup (Ubuntu Xenial 16.04, stable ocata branches for Keystone, Nova, Cinder, Neutron, Glance and requirements repos.).

DevStack creation fails with:
[..]
2017-07-26 09:56:40.809 | 2017-07-26 09:56:40.809 32617 WARNING py.warnings [-] /usr/local/lib/python2.7/dist-packages/pycadf/identifier.py:60: UserWarning: Invalid uuid. To ensure interoperability, identifiers should be a valid uuid.
2017-07-26 09:56:40.809 | warnings.warn('Invalid uuid. To ensure interoperability, identifiers '
2017-07-26 09:56:40.810 |
2017-07-26 09:56:40.847 | 2017-07-26 09:56:40.846 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created domain default
2017-07-26 09:56:40.872 | 2017-07-26 09:56:40.872 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created project admin
2017-07-26 09:56:40.889 | 2017-07-26 09:56:40.888 32617 DEBUG passlib.registry [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] registered 'sha512_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler /usr/local/lib/python2.7/dist-packages/passlib/registry.py:294
2017-07-26 09:56:40.910 | 2017-07-26 09:56:40.910 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created user admin
2017-07-26 09:56:40.923 | 2017-07-26 09:56:40.923 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created role admin
2017-07-26 09:56:40.937 | 2017-07-26 09:56:40.937 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Granted admin on admin to user admin.
2017-07-26 09:56:40.971 | 2017-07-26 09:56:40.971 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created region RegionOne
2017-07-26 09:56:41.013 | 2017-07-26 09:56:41.013 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created admin endpoint http://127.0.0.1/identity
2017-07-26 09:56:41.068 | 2017-07-26 09:56:41.068 32617 INFO keystone.cmd.cli [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Created public endpoint http://127.0.0.1/identity
2017-07-26 09:56:41.071 | 2017-07-26 09:56:41.071 32617 INFO keystone.assignment.core [req-e97f7067-fef4-40e4-b4c2-2959bc86fd09 - -] Creating the default role 9fe2ff9ee4384b1894a90878d3e92bab because it does not exist.
2017-07-26 09:56:41.180 | + ./stack.sh:main:1128 : create_keystone_accounts
2017-07-26 09:56:41.181 | + lib/keystone:create_keystone_accounts:336 : local admin_project
2017-07-26 09:56:41.183 | ++ lib/keystone:create_keystone_accounts:337 : oscwrap project show admin -f value -c id
2017-07-26 09:56:41.184 | ++ functions-common:oscwrap:2530 : local out
2017-07-26 09:56:41.186 | ++ functions-common:oscwrap:2531 : local rc
2017-07-26 09:56:41.187 | ++ functions-common:oscwrap:2532 : local start
2017-07-26 09:56:41.188 | ++ functions-common:oscwrap:2533 : local end
2017-07-26 09:56:41.190 | +++ functions-common:oscwrap:2537 : date +%s%3N
2017-07-26 09:56:41.192 | ++ functions-common:oscwrap:2537 : start=1501063001191
2017-07-26 09:56:41.194 | +++ functions-common:oscwra...

[Silvan Kaiser (2-silvan)]

I'm not sure why but this issue can be worked around by manually copying /opt/stack/keystone/etc/policy.json to /etc/keystone/policy.json . Does stable/ocata Keystone require that file? Because master branch Keystone works fine without that file in that location.

I do hit another stack issue later on with that setup (Nova/WSGI does not work) but the keystone setup looks fine to me at that point.

[Lance Bragstad (lbragstad)]

As of the pike release, keystone no longer requires the policy.json file. The policy file is used to describe which roles are required for specific APIs. In pike we moved the management and documentation of policy into code and treat it like configuration [0][1]. These changes were limited to pike, so ocata will still require the policy.json file to be on the host at run time.

[0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/policy-in-code.html
[1] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/policy-docs.html

[Lance Bragstad (lbragstad)]

Did something in devstack change that made it so the policy.json file is no longer deployed for stable/ocata? Keystone's stable/ocata branch didn't receive any backports that would have allowed it to run without a default policy file.

[Gage Hugo (gagehugo)]

stable/ocata devstack worked for me as of today, it successfully copied the policy.json into /etc/keystone, so I was unable to replicate this.

I wonder if something changed in devstack since then?

[Dr. Jens Harbott (j-harbott)]

If the issue still exists, please detail the steps to reproduce and include your local.conf and a full stack.sh log.

[Lance Bragstad (lbragstad)]

I'm going to mark this as Invalid since keystone's stable/ocata branch includes the policy.json file [0]. All the work to move policy into code was done in Pike. Per comment #9, this should be reopened if there are steps provided to recreate the issue.

[0] https://github.com/openstack/keystone/tree/stable/ocata/etchttps://github.com/openstack/keystone/tree/stable/ocata/etc

[Silvan Kaiser (2-silvan)]

I could not reproduce this issue in several attempts this week, seems to be solved. :-)

[Silvan Kaiser (2-silvan)]

I'll mark this as invalid. Please reopen if somebody else is still hitting this.