keystone v3 :: kilo HA setup token authentication failure while listing projects and running service commands.

[Ritam Gangopadhyay]

        I have 2 setups, one multi-node HA and the other a single node. I provisioned R3.1 kilo on both and configured keystone v3. I followed the steps as listed in the attached file. I have changed the /etc/contrail/openstackrc file too, to use required v3 environment variables.

       But when I try to execute any service commands like glance image-list, neutron net-list and nova service-list it fails with token authentication failure. On the single node setup it works fine but in case of HA setup I see the failure.
Can you please take a look at the setups.

Single Node:-

root@nodeb2:~# nova --os-project-domain-name default --os-user-domain-name default service-list
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
| 1 | nova-scheduler | nodeb2 | internal | enabled | up | 2016-07-29T16:11:43.000000 | - |
| 2 | nova-console | nodeb2 | internal | enabled | up | 2016-07-29T16:11:37.000000 | - |
| 3 | nova-consoleauth | nodeb2 | internal | enabled | up | 2016-07-29T16:11:35.000000 | - |
| 4 | nova-conductor | nodeb2 | internal | enabled | up | 2016-07-29T16:11:38.000000 | - |
| 5 | nova-compute | nodeb2 | nova | enabled | up | 2016-07-29T16:11:42.000000 | - |
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
root@nodeb2:~# neutron --os-project-domain-name default --os-user-domain-name default net-list
+--------------------------------------+-------------------------+---------+
| id | name | subnets |
+--------------------------------------+-------------------------+---------+
| d68731f3-ceaf-45b0-b901-3e9014475ae3 | default-virtual-network | |
| c3067bea-e3f8-41c7-bf39-9a8bd2fa6ac6 | __link_local__ | |
| e6ba9007-6d80-4914-af6b-5fa632e79375 | ip-fabric | |
+--------------------------------------+-------------------------+---------+
root@nodeb2:~#

HA Setup:-

root@nodeg34:~# nova service-list
+----+------------------+---------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------+----------+---------+-------+----------------------------+-----------------+
| 3 | nova-scheduler | nodeg34 | internal | enabled | up | 2016-07-29T16:10:11.000000 | - |
| 6 | nova-console | nodeg34 | internal | enabled | up | 2016-07-29T16:10:01.000000 | - |
| 9 | nova-consoleauth | nodeg34 | internal | enabled | up | 2016-07-29T16:10:08.000000 | - |
| 12 | nova-conductor | nodeg34 | internal | enabled | up | 2016-07-29T16:10:08.000000 | - |
| 15 | nova-scheduler | nodec48 | internal | enabled | up | 2016-07-29T16:10:10.000000 | - |
| 18 | nova-console | nodec48 | internal | enabled | up | 2016-07-29T16:09:59.000000 | - |
| 21 | nova-consoleauth | nodec48 | internal | enabled | up | 2016-07-29T16:10:07.000000 | - |
| 24 | nova-conductor | nodec48 | internal | enabled | up | 2016-07-29T16:10:06.000000 | - |
| 27 | nova-scheduler | nodec49 | internal | enabled | up | 2016-07-29T16:09:58.000000 | - |
| 30 | nova-console | nodec49 | internal | enabled | up | 2016-07-29T16:10:02.000000 | - |
| 33 | nova-consoleauth | nodec49 | internal | enabled | up | 2016-07-29T16:10:10.000000 | - |
| 36 | nova-conductor | nodec49 | internal | enabled | up | 2016-07-29T16:10:11.000000 | - |
| 42 | nova-compute | nodec51 | nova | enabled | up | 2016-07-29T16:10:03.000000 | - |
| 45 | nova-compute | nodec63 | nova | enabled | up | 2016-07-29T16:10:11.000000 | - |
+----+------------------+---------+----------+---------+-------+----------------------------+-----------------+
root@nodeg34:~# glance --os-project-domain-name default --os-user-domain-name default image-list
404 Not Found: The resource could not be found. (HTTP 404)
root@nodeg34:~# neutron --os-project-domain-name default --os-user-domain-name default net-list
Traceback (most recent call last):
  File "/usr/bin/neutron", line 6, in <module>
    from neutronclient.shell import main
  File "/usr/lib/python2.7/dist-packages/neutronclient/shell.py", line 32, in <module>
    from keystoneclient.openstack.common.apiclient import exceptions as ks_exc
ImportError: No module named openstack.common.apiclient
root@nodeg34:~#

Setup -1 :: HA
env.roledefs = {

    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1, host2, host3],
    'openstack':[host1, host2, host3],
    'control':[host1, host2, host3],
    'compute': [host4, host5],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],

}
env.hostnames ={
    'all': ['nodeg34', 'nodec48', 'nodec49', 'nodec51', 'nodec63']
}
host1 = 'root@10.204.221.24'
host2 = 'root@10.204.221.27'
host3 = 'root@10.204.221.28'
host4 = 'root@10.204.221.25'
host5 = 'root@10.204.221.26'

Setup – 2 :: Single Node

host1 = 'root@10.204.216.33'

env.roledefs = {
    'all': [host1],
    'cfgm': [host1],
    'openstack': [host1],
    'control': [host1],
    'compute': [host1],
    'collector': [host1],
    'webui': [host1],
    'database': [host1],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodeb2']
}

[Deepinder Setia (dsetia)]

Ritam, can you check if this is still broken? Ensure following:

1) admin user has a role in default domain
2) SESSION_ENGINE isn't set to cookies (ok to set to cache)

[Deepinder Setia (dsetia)]

Should be fixed by:

https://review.opencontrail.org/#/c/23013/ and
https://review.opencontrail.org/#/c/23058/

I see the bug is in kilo. Note that we are only claiming keystone V3 support for Mitaka.

[Ritam Gangopadhyay (ritam)]

Fix verified on build 24 mitaka.