Ceilometer is unable to get meters from nova when using Keystone v3 authentication/authorization

[Frode Nordahl]

When using Ceilometer in a deployment with Keystone v3 authentication enabled the current policy and role assignments does not give ceilometer-agent-central access to list projects from Keystone. Consequently Ceilometer is not retrieving statistics from instances in Nova.

Symptoms:
- Ceilometer meters from nova are not populated/updated
- This ERROR-message appears in ceilometer-agent-central.log:

    ERROR ceilometer.agent.manager [req-XXX admin - - - -] Skipping tenant, keystone issue: You are not authorized to perform the requested action: identity:list_projects (HTTP 403)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/390200

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/390202

[Frode Nordahl (fnordahl)]

When using charm-keystone with preferred-api-version=3 the policy installed denies service accounts access to identity:list_projects. The current policy specifically requires the user to be authorized using a domain scoped token, and explicitly denies users authorized with a project scoped token.

Ceilometer depends on listing projects to know where to gather Nova statistics from. Ceilometer also depends on listing the service catalog to operate. Under current policy Ceilometer would need to authorize with a domain scoped token to get the list of projects, but it would need to authorize with a project scoped token to be allowed to list the service catalog. Ceilometer is currently not designed this way and we need to change the policy to allow it to operate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/395101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/395101
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=10e3d84effcadd9c527d5d18f8bdd0d25003b85a
Submitter: Jenkins
Branch: master

commit 10e3d84effcadd9c527d5d18f8bdd0d25003b85a
Author: Frode Nordahl <email address hidden>
Date: Tue Nov 8 08:49:36 2016 +0100

    Refresh keystone.conf and policy.json for Mitaka and Newton

    keystone.conf:
    - Change log_config to log_config_append DEPRECATED
    - Remove verbose DEPRECATED
    - Remove eventlet_server section DEPRECATED
    - Remove ec2 section, no longer available in Keystone
      It has been moved to the keystonemiddleware package
    - Update driver names. Using full module path is DEPRECATED
    - Add resource section and specify admin_project_domain_name
      and admin_project_name

    mitaka/policy.json:
    - Refresh from upstream stable/mitaka
    - Apply stricter rule:service_role
    - Allow identity:list_projects to rule:service_role

    newton/policy.json:
    - Refresh from upstream stable/newton
    - Apply stricter rule:service_role
    - Allow identity:list_projects to rule:service_role

    hooks/keystone_context.py:
    - Add admin_domain_name to Keystone context

    tests/basic_deployment.py:
    - Add config check for changes for Mitaka and newer releases

    Partial-Bug: 1636098
    Change-Id: Ib267418f34066eaf6e4885627010d2a18e312192

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/392676
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=5de1770931e886732870da1909f08279a0b804b4
Submitter: Jenkins
Branch: master

commit 5de1770931e886732870da1909f08279a0b804b4
Author: Frode Nordahl <email address hidden>
Date: Wed Nov 2 13:21:07 2016 +0100

    Create service credentials in SERVICE_DOMAIN

    Cleanup code that references users, projects or domains without
    necessary scoping or filtering throughout the charm. Add logging
    of domain name in contexts where this is relevant.

    Tighten rule:service_role to require role:service and token scoped
    to project config('service-tenant') created in SERVICE_DOMAIN. This
    ensures that if you have a deployment with end-user access to assign
    roles within their own domains they will not gain privileged access
    simply by assigning the service role to one of their own users.

    Allow users authorized by rule:service_role to perform
    identity:list_projects. This is required to allow Ceilometer
    to operate without Admin privileges.

    Services are given a user in project config('service-tenant') in
    SERVICE_DOMAIN for v3 authentication / authorization. As of Mitaka
    Keystone v3 policy the 'service' role is sufficient for services to
    validate tokens.

    Services are also given a user in project config('service-tenant') in
    DEFAULT_DOMAIN to support services still configured with v2.0
    authentication / authorization.

    This will allow us to transition from v2.0 based authentication /
    authorization and existing services and charms will continue to
    operate as before. This will also allow the end-user to roll their
    deployment up to api_version 3 and back to api_version 2 as needed.

    Services and charms that has made the transition to fully use the
    v3 API for authentication and authorization will gain full access to
    domains and projects across the deployment. The first charm to make
    use of this is charm-ceilometer.

    Closes-Bug: 1636098
    Change-Id: If1518029c43476a5e14bf94596197eabe663499c

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/397388
Reason: Abandon this in favor of https://review.openstack.org/#/c/401386

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/397389
Reason: Abandon in favor of https://review.openstack.org/401387

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/390200
Reason: Abandoned in favor of https://review.openstack.org/#/c/395101/ (Merged)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/390202
Reason: Abandoned in favor of https://review.openstack.org/395101 (Merged) https://review.openstack.org/392676 (Merged) and https://review.openstack.org/392677 (In review)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/397388
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=dae097c7dc86f1bd55d9bf92ea5683746ff9a966
Submitter: Jenkins
Branch: master

commit dae097c7dc86f1bd55d9bf92ea5683746ff9a966
Author: Frode Nordahl <email address hidden>
Date: Mon Nov 14 22:53:08 2016 +0100

    Sync charmhelpers for Keystone v3 fixes

    Needed to get admin_domain_name relation data and updated version of
    section-keystone-authtoken-mitaka

    Partial-Bug: 1636098
    Change-Id: If2be2265530d8f9535fc15e29a1a07748943d189

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/392677
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=03f162e84bc3aaef50c90305025ab735e82a57bc
Submitter: Jenkins
Branch: master

commit 03f162e84bc3aaef50c90305025ab735e82a57bc
Author: Frode Nordahl <email address hidden>
Date: Wed Nov 2 15:17:01 2016 +0100

    Fix support for Keystone v3 domain auth

    When using v3 authentication and authorization Ceilometer must
    authenticate to services project in service_domain.

    Closes-Bug: 1636098
    Change-Id: I23354e7db0eef83de31f703fdf754ca3ae2db92d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/400357
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=ec2579a8448281cdf1154018c0a56c9d4b174e36
Submitter: Jenkins
Branch: master

commit ec2579a8448281cdf1154018c0a56c9d4b174e36
Author: Frode Nordahl <email address hidden>
Date: Mon Nov 21 09:52:06 2016 +0100

    Update policy.json to give service role access

    Role check is scoped to the configured service project and access
    is granted to os_compute_api:servers:detail:get_all_tenants.

    This change is required to allow Ceilometer to operate using a
    non-Admin user.

    Change-Id: I17729e2089cc658588eaea93f8de5051369e5dff
    Closes-Bug: 1636098

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/397389
Reason: Abandon in favor of already merged https://review.openstack.org/#/c/401387/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/395105
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer-agent/commit/?id=a904456fe3d353bbd972d7a3af91df9f49fbec5b
Submitter: Jenkins
Branch: master

commit a904456fe3d353bbd972d7a3af91df9f49fbec5b
Author: Frode Nordahl <email address hidden>
Date: Tue Nov 8 18:45:32 2016 +0100

    Fix support for Keystone v3 domain auth

    When using v3 authentication and authorization Ceilometer must
    authenticate to services project in service_domain.

    Change-Id: Ibb19fa8476cf712ccb4bc524a48d3501f42fc62c
    Closes-Bug: 1636098