Instance creation fails with keystone v3 and 'master' charms

[Liam Young]

Creating an instance using the master charms to deploy xenial/newton fails, the instance goes from PENDING to ERROR state after a wait of a few minutes.

The neutron-server logs report it got a 403 back from the nova-cloud-controller. The nova-api-os-compute.log has this entry:

Returning 403 to user: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. On the nova-cloud-controller the /etc/nova/policy.json contains:

"admin_api": "is_admin:True"
"os_compute_api:os-server-external-events:create": "rule:admin_api"

But the service user does not have the admin on the service project in the service domain.

Granting this role allows the VM to be created (but with not network atm).

openstack domain list
+----------------------------------+----------------+---------+-----------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------+---------+-----------------+
| 539d91217e4e4272a9a74907f56d6847 | norwich | True | |
| 88f4d246106d4da0a60fa7603e43507b | default | True | Created by Juju |
| 89c5830014214a0e9c8d87adea2c715b | service_domain | True | Created by Juju |
| c2959b71ab0e4b478133ddd0c99df687 | admin_domain | True | Created by Juju |
| eda4a4cdef904b12926e447bdde7c7f0 | leeds | True | |
+----------------------------------+----------------+---------+-----------------+

openstack role add --user neutron --user-domain 89c5830014214a0e9c8d87adea2c715b --project services --project-domain 89c5830014214a0e9c8d87adea2c715b Admin

[Liam Young (gnuoy)]

My first thoughts for fixing it are:

1) Changing admin_api rule from

"admin_api": "is_admin:True"
to
"admin_api": "rule:service_or_admin"

2) Changing the charm to grant admin to the service users on the service project in the service domain.

[Liam Young (gnuoy)]

Just to be clear, option 1 is a change to the policy.json on the nova-cloud-controller charm and option 2 is a change to the keystone charm.

[Frode Nordahl (fnordahl)]

Reverted the summary change. This does indeed affect Mitaka as well, was not visible in tests at time of merge due to timing of merge of charm-helers sync for other charms.

[Frode Nordahl (fnordahl)]

A few in-flight thoughts:
Nova is the only service that accesses Neutron directly using its service account.

Other inter-service RPC either uses the end-users token or the Message Queue. We might want to consider just granting the nova service account elevated access to Neutron and not all service accounts in general.

I will verify, test and propose fixes along these lines ASAP.

[Frode Nordahl (fnordahl)]

Neutron also has calls back to to Nova API.
Nova has calls to Cinder API.

[Frode Nordahl (fnordahl)]

Current working theory
----------------------
charm-keystone:
  - Add 'service_user_id' to IdentityServiceContext
charm-helpers:
  - Add 'service_user_id' to IdentityServiceContext
  - Add 'neutron_user_id' to NeutronContext
charm-nova-cloud-controller:
  - Expose 'nova_user_id' through relation data in relations:
    - 'neutron_api'
    - 'cinder-volume-service'
  - Add 'neutron_user_id' to rule 'admin_api' in policy.json.
charm-neutron-api:
  - Expose 'neutron_user_id' through relation data in relations:
    - 'neutron_api'
  - Add 'nova_user_id' to rule X in policy.json.
charm-cinder:
  - Add 'nova_user_id' to rule 'admin_api' in policy.json.

[Liam Young (gnuoy)]

Can I confirm that you are proposing that the keystone charm would pass the service_user_id along with the other service_* values along the identity-service relation ? If so I think this looks good but I'd also like to give jamespage a chance to comment too.

[Frode Nordahl (fnordahl)]

Yes, the keystone charm will expose the service_user_id in the identity-service relation for all services.

The charms for services that do inter-service RPC will have to excplicitly expose/consume their peers service_user_id in the inter-service relation. This is necessary because any given service does only have access to their own credentials and they have no access to obtain information about other services or users.

[Frode Nordahl (fnordahl)]

To be more clear: keystone would expose the service_user_id for that individual relation only, not listing information for multiple services in one relation.

[James Page (james-page)]

Reading this bug report, it sounds like the change to remove the admin grant from service accounts in the changes to support telemetry in effect caused this change.

I'm getting a bit uncomfortable that we're diverging from what's considered 'reference' by openstack projects with the number of files we're having to touch across all openstack charms to implement the keystone v3 policy that we current have in master branch.

[Frode Nordahl (fnordahl)]

I do not disagree with that and it is a perfectly valid stance. The intention was to attempt at more fine grained RBAC for service accounts rather than just granting them all Admin access. However as you point out such a model may not have received enough scrutiny as it is currently not implemented that way in the reference implementation.

I am perfectly fine with granting the service accounts the Admin role for now and saving fine grained RBAC for service accounts for later.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418973

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418975

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418973
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=dd65408d945c10ef383fb59b0c2177fdb1e07e09
Submitter: Jenkins
Branch: master

commit dd65408d945c10ef383fb59b0c2177fdb1e07e09
Author: Frode Nordahl <email address hidden>
Date: Wed Jan 11 14:37:21 2017 +0100

    Revert change of role for v3 service accounts

    More work is needed on policy changes before we can have fine
    grained RBAC for service accounts.

    Add service project to cloud_admin rule to maintain service access
    to admin-only calls.

    Change-Id: I3d6776ec821e97353d63d2709b36efd9091f0123
    Closes-Bug: 1655028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418975
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=2eef644a5c1acb2675e94908c88182658fec4ac5
Submitter: Jenkins
Branch: master

commit 2eef644a5c1acb2675e94908c88182658fec4ac5
Author: Frode Nordahl <email address hidden>
Date: Wed Jan 11 14:51:06 2017 +0100

    Revert "Update policy.json to give service role access"

    This reverts commit ec2579a8448281cdf1154018c0a56c9d4b174e36.

    We are not quite ready for fine grained RBAC for service
    accounts.

    Change-Id: I210685d7b9036abb191073d512f8a65ebff30613
    Closes-Bug: 1655028