All users with the admin role are cloud admins

[Liam Young]

If a user is given the admin role against any project or domain then neutron interprets this as the user having global admin rights. To reproduce this run the first set of commands as cloud admin to set up a new domain, project and user. Then grant admin to the user over the new project. Create a network as the cloud admin user. Finally switch to the new user and disable the cloud admins network:

openstack domain create --or-show --enable bug-domain
openstack project create --or-show --enable --domain bug-domain bug-project
openstack user create --or-show --enable --domain bug-domain \
                      --project-domain bug-domain --project bug-project \
                      --password buguserpass bug-user
openstack role add --user-domain bug-domain --user bug-user \
                   --project-domain bug-domain --project bug-project Admin
openstack network create cloudadmin-owned-net
openstack network show cloudadmin-owned-net -c admin_state_up -f value
source unset_all
env | grep OS_
openstack --os-region-name RegionOne --os-auth-url http://10.5.0.4:5000/v3 \
          --os-project-name bug-project --os-project-domain-name bug-domain \
          --os-username bug-user --os-user-domain-name bug-domain \
          --os-password buguserpass network set --disable cloudadmin-owned-net
openstack --os-region-name RegionOne --os-auth-url http://10.5.0.4:5000/v3 \
          --os-project-name bug-project --os-project-domain-name bug-domain \
          --os-username bug-user --os-user-domain-name bug-domain \
          --os-password buguserpass network show cloudadmin-owned-net \
          -c admin_state_up -f value

(Output can be seen here: https://paste.ubuntu.com/p/9zbgTFHMBQ/)

This appears to be due to the default policy.json that ships with the neutron server.

I think this can be fixed by updating the policy.json and changing the context_is_admin rule from:

"context_is_admin": "role:admin",

to

"context_is_admin": "role:admin and is_admin_project:True"

[Liam Young (gnuoy)]

Related bugs:
https://bugs.launchpad.net/keystone/+bug/968696
https://bugs.launchpad.net/glance/+bug/1602081

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671721

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671721
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=e03501dee1a43fd264780966e7831c96c0279232
Submitter: Zuul
Branch: master

commit e03501dee1a43fd264780966e7831c96c0279232
Author: Liam Young <email address hidden>
Date: Fri Jul 19 13:00:59 2019 +0000

    Ensure that only cloud admins are neutron admins

    When determining if a user is an admin the default neutron policy
    file only checks if a user has the 'admin' role. It does not check
    what that role is applied to.

    The problem is illustrated by the following scenario: A cloud
    admin creates a new domain, then creates a new project within that
    domain. The cloud admin wants to delegate the maintenance of the
    new project to userA so she grants them admin on the new project.
    UserA is now a cloud admin from Neutrons pov.

    To fix this issue a policy override file is added which checks that
    the user is admin either against the admin project (as defined by
    keystone) or the service project.

    Change-Id: If4c5b0c1ab7bf2c75e911e77531d442d417a1231
    Closes-Bug: 1830536

[Sahid Orentino (sahid-ferdjaoui)]

The change is creating a regression. Two tempest tests are failing:

 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
 tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes

Basically, it seems that the os_admin is not able to list the networks of all tenants anymore. I'm trying to investigate what could create this bad effect but at the same time, we should also consider reverting the change since as pointed by Ryan, [0] may also resolve the issue.

[0] https://review.opendev.org/#/c/677004/

[Sahid Orentino (sahid-ferdjaoui)]

I was thinking a new time at this change and I think it is not right we can't add a condition to the context_is_admin without to create a regression. To resolve this issue I think a new role should be used or basically this change [0] is really what we need so we can scope an admin to a specific tenant.

I'm going to propose a change to remove that policy added and I guess we will have to backport it.

[0] https://review.opendev.org/#/c/677004/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/680956

[Amad Ali (amad)]

Another customer is facing regression due to this fix. Previously they have been managing network creation with LDAP authenticated users with admin role against the project or domain. Switching to local users in default domain with global admin access is against their security policy.

[Amad Ali (amad)]

Adding more details about the regression experienced.

The issue is affecting all users assigned "Admin" role in all domains and projects apart from the admin project.

Here is an extract of novarc file of the user with a token scoped to a project in LDAP domain.

#!/usr/bin/env bash
export OS_AUTH_URL=https://xxx.xxx.xxx:5000/v3
export OS_PROJECT_ID=<project_in_ldap_domain>
export OS_PROJECT_NAME="xxxx"
export OS_USER_DOMAIN_NAME="<ldap_domain_name>"
export OS_PROJECT_DOMAIN_ID="<ldap_domain_id>"
export OS_USERNAME=xxxx
export OS_PASSWORD=xxxx
export OS_REGION_NAME="xxxx"
export OS_IDENTITY_API_VERSION=3
export OS_CACERT=<path>

This user could create provider network before this fix but cannot anymore. There are NO issues creating internal networks by such a user.

Here is the error when trying to create provider network with such a user.

$ openstack network create --provider-network-type vlan --provider-segment 1111 --provider-physical-network physnet1 Test
Error while executing command: HttpException: 403, (((rule:create_network and rule:create_network:provider:physical_network) and rule:create_network:provider:network_type) and rule:create_network:provider:segmentation_id) is disallowed by policy

[Amad Ali (amad)]

To add more context, as a work around we tried to create networks and subnets as admin users with --project <project_in_ldap_domain> and also with rbac policy to share the network with <project_in_ldap_domain>. In both case the network and subnet is visible to the <project_in_ldap_domain> but instance creation fails with the following error.

Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance <<instance_uuid1>>. Last exception: Binding failed for port <<port_uuid1>>, please check neutron logs for more information.].

Then we tried to create a port and then create an instance with the port with the user with admin role against the project in ldap domain. Port creation was successful but instance creation failed with the following error:

Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance <uuid1>. Last exception: Port <<port_uuid>> not usable for instance <<instance_uuid>>.].

I think the problem is with binding rule in default policy.json

"create_port:binding:host_id": "rule:admin_only"

[Sahid Orentino (sahid-ferdjaoui)]

Basically, we should make this bug as INVALID and revert the patch proposed. To achieve the scenario indicated in the bug description we should consider using keystone trust API.

In this example, I will give Admin role to the demo user for the admin project so it will be able to execute some privileged operations.

# admin env context
$ openstack trust create --project admin --role Admin admin demo
+--------------------+----------------------------------+
| Field | Value |
+--------------------+----------------------------------+
| deleted_at | None |
| expires_at | None |
| id | 67081c80ccab4e20834a48ce002b9408 |
| impersonation | False |
| project_id | 5c2157c6ff4849faa2e586156acbb7d9 |
| redelegation_count | 0 |
| remaining_uses | None |
| roles | Admin |
| trustee_user_id | f110c3e37099409f8b3c1867840a79c7 |
| trustor_user_id | 06b930776bfd4fa098867ca5fd8c9804 |
+--------------------+----------------------------------+

# demo env context
declare -x OS_AUTH_TYPE="password"
declare -x OS_AUTH_URL="http://10.5.0.3:5000/v3"
declare -x OS_AUTH_VERSION="3"
declare -x OS_IDENTITY_API_VERSION="3"
declare -x OS_PASSWORD="pass"
declare -x OS_PROJECT_DOMAIN_NAME="admin_domain"
declare -x OS_PROJECT_NAME="admin"
declare -x OS_REGION_NAME="RegionOne"
declare -x OS_USERNAME="demo"
declare -x OS_USER_DOMAIN_NAME="admin_domain"

$ openstack network list
The request you have made requires aut...

[Sahid Orentino (sahid-ferdjaoui)]

Oops I made a mistake in the last line... please read:

$ openstack --os-trust-id 67081c80ccab4e20834a48ce002b9408 network set --disable private
$ openstack --os-trust-id 67081c80ccab4e20834a48ce002b9408 network show private -c admin_state_up -f value
DOWN

[Sahid Orentino (sahid-ferdjaoui)]

As a side note instead of passing --os-trust-id someone could use the env variable OS_TRUST_ID

e.g.
  $ export OS_TRUST_ID=67081c80ccab4e20834a48ce002b9408

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/680956
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=2ad425b1cdd8093b40f4cd57ba59b0c3b4eb8e6a
Submitter: Zuul
Branch: master

commit 2ad425b1cdd8093b40f4cd57ba59b0c3b4eb8e6a
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Sep 9 07:32:10 2019 +0000

    revert "Ensure that only cloud admins are neutron admins"

    We can't add constraints to admin role without consider
    regressions. It happens that two tempest scenarios are now failling:

     tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
     tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes

    If admin wants to give role (even Admin role) to an user for a tenant,
    the right way is to use keystone trust API.

    Change-Id: I161ea7d1aec5e5784455b5bce4605b2f9143daa2
    Related-Bug: #1830536
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/19.07
Review: https://review.opendev.org/683057

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/683057
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=f33b6789acac63ebb15e6a01a2a0c60e62d9d076
Submitter: Zuul
Branch: stable/19.07

commit f33b6789acac63ebb15e6a01a2a0c60e62d9d076
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Sep 9 07:32:10 2019 +0000

    revert "Ensure that only cloud admins are neutron admins"

    We can't add constraints to admin role without consider
    regressions. It happens that two tempest scenarios are now failling:

     tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
     tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes

    If admin wants to give role (even Admin role) to an user for a tenant,
    the right way is to use keystone trust API.

    Change-Id: I161ea7d1aec5e5784455b5bce4605b2f9143daa2
    Related-Bug: #1830536
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>
    (cherry picked from commit 2ad425b1cdd8093b40f4cd57ba59b0c3b4eb8e6a)