Upstream update for h.2 causes keystone error

[Mark T. Voelker]

After h.1 was cut we pulled in updates to several puppet modules from upstream during feature development work. AIO nodes (at least) now throw a keystone error near the end of the catalog run:

Debug: Executing '/usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ user-list --tenant-id '
Debug: Executing '/usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ user-role-list --user-id 81b61a28cb6b445ca3c9160f23c48236 --tenant-id '
Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@openstack]: Could not evaluate: Execution of '/usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ user-role-list --user-id 81b61a28cb6b445ca3c9160f23c48236 --tenant-id ' returned 1: 'Client' object has no attribute 'auth_tenant_id'

At first glance it looks as if this command is incomplete (no tenant-id specified), but needs a closer look to see why that might be happening. The command originates in puppet-keystone's lib/puppet/provider/keystone_user_role/keystone.rb file (e.g. the keystone provider) in the list_user_roles function.

[Mark T. Voelker (mvoelker)]

A few notes:

1.) Ignore the bit above about no tenant-id being specified. I think that's just log truncation, as the error would be very different if that were what was actually going on:

root@control01:/usr/share/puppet/modules/keystone# /usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ user-role-list --user-id 81b61a28cb6b445ca3c9160f23c48236 --tenant-id
usage: keystone user-role-list [--user <user>] [--tenant <tenant>]
keystone user-role-list: error: argument --tenant/--tenant-id: expected one argument
root@control01:/usr/share/puppet/modules/keystone#

2.) This could be related to an old-but-not-yet-fixed python-keystoneclient bug:

https://bugs.launchpad.net/python-keystoneclient/+bug/1058750

3.) Digging a bit, the --user-id argument in the command is the admin user (which the error message itself also indicates). The missing tenant_id that got truncated out of the message could be for either the 'openstack' or 'services' tenants (the only two that exist on the system at the time). From the error message we know it's the former. Note also that both the admin and service tenants do exist, as does the admin user, and the admin user is assigned to the _member_ role:

root@control01:~# /usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ user-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+------------+---------+--------------------+
| id | name | enabled | email |
+----------------------------------+------------+---------+--------------------+
| 5d4c0ea62e1e403595d1cd109c9db441 | admin | True | root@localhost |
| fafbfe1716ba4d2c90e0c51c8bdf3827 | ceilometer | True | root@localhost |
| e6184021519f4574b6864abb63f02224 | cinder | True | cinder@localhost |
| efa487e0eb3e4bf3991de927237a8a36 | glance | True | glance@localhost |
| 431e72e0982f485bbc512e92c44ca75a | heat | True | heat@localhost |
| b914db6256f0493694d1859e20bbd65f | heat-cfn | True | heat-cfn@localhost |
| 0e717c467ecb4741aabff86f860122c5 | neutron | True | neutron@localhost |
| 78cb3f44f1e24347b4a662c55ee6b1a4 | nova | True | nova@localhost |
| e8fcb0bf580e4b609c1ce79fed5fe232 | swift | True | swift@localhost |
+----------------------------------+------------+---------+--------------------+
root@control01:~# /usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ tenant-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+-----------+---------+
| id | name | enabled |
+----------------------------------+-----------+---------+
| 6b250e187fd44d84a3267ea96298d2bb | openstack | True |
| 446d49a370fc4814a0d33a48712eb82e | services | True |
+----------------------------------+-----------+---------+
root@control01:~# /usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ role-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----...

[Mark T. Voelker (mvoelker)]

New theory: it's a lazy retrival thing. Since h.1 this patch was introduced to puppet-keystone:

https://github.com/stackforge/puppet-keystone/commit/a0a359801e7248b24d15b3acef3b398d5a2fb65f

That appears to be the source of the borkage, because it breaks if tenant is created after prefetching, a keystone lookup isn't performed. There's a patch out for review on stackforge that fixes the problem:

https://review.openstack.org/#/c/73913/

[Mark T. Voelker (mvoelker)]

https://review.openstack.org/75628 has been pulled in.