Unable to setup context 'users' when launching rally scenario

[Morgan Richomme]

We use Rally to test OPNFV solutions. It has been integrated in CI.
We used to run successfully
- most of the rally scenario (authenticate, cider, glance, keystone, nova, neutron, quotas, requests)
- tempest smoke suite

The OPNFV solution consists in the integration of OpenStack + ODL.
In CI we are testing 2 OPNFV flavours:
- 1 installed from fuel (MOS 6.0.1)
- 1 installed through foreman/puppet installer

We were able to run all the tests automatically (https://build.opnfv.org/ci/view/functest/)
Rally is automatically installed and configured (including tempest part) based on env (fromenv option) using keystone v2.0

Installation looks fine
2015-08-25 17:33:12,297 - config_functest - DEBUG - Creating Rally environment...
2015-08-25 17:33:12,297 - config_functest - DEBUG - Executing command : rally deployment create --fromenv --name=opnfv-arno-rally
2015-08-25 17:33:13,166 - config_functest - DEBUG - 2015-08-25 17:33:12.973 25872 INFO rally.deployment.engine [-] Deployment ee971af8-2369-42d1-96bf-fa6f0768d634 | Starting: OpenStack cloud deployment.
2015-08-25 17:33:13.057 25872 INFO rally.deployment.engine [-] Deployment ee971af8-2369-42d1-96bf-fa6f0768d634 | Completed: OpenStack cloud deployment.
+--------------------------------------+----------------------------+------------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------------+------------------+--------+
| ee971af8-2369-42d1-96bf-fa6f0768d634 | 2015-08-26 00:33:12.918895 | opnfv-arno-rally | deploy->finished | |
+--------------------------------------+----------------------------+------------------+------------------+--------+
...
2015-08-25 17:34:21,107 - config_functest - DEBUG - Executing command : rally deployment check
2015-08-25 17:34:22,028 - config_functest - DEBUG - keystone endpoints are valid and following services are available:
+----------+----------------+-----------+
| services | type | status |
+----------+----------------+-----------+
| cinder | volume | Available |
| cinderv2 | volumev2 | Available |
| cloud | cloudformation | Available |
| ec2 | ec2 | Available |
| glance | image | Available |
| heat | orchestration | Available |
| keystone | identity | Available |
| neutron | network | Available |
| nova | compute | Available |
| novav3 | computev3 | Available |
+----------+----------------+-----------+

rally show flavors OK (so keystone v2.0 authentication ok)

src(https://build.opnfv.org/ci/view/functest/job/functest-foreman-master/2/consoleText)

We still can run Tempest smoke suite but it is now impossible to launch the rally scenario
we got an error (here with the foreman installer but same error with fuel):

2015-08-25 17:37:16.502 28634 INFO rally.plugins.openstack.context.keystone.users [-] Task c2c0352e-f60d-4fdd-8211-8a61eb09d539 | Starting: Enter context: `users`
2015-08-25 17:37:19.892 28634 WARNING rally.common.broker [-] Failed to consume a task from the queue: Unable to establish connection to http://192.168.0.92:35357/v2.0/tenants
....
2015-08-25 17:37:22,292 - run_rally - DEBUG - saving json file2
authenticate
Traceback (most recent call last):
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 217, in <module>
    main()
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 208, in main
    run_task(test_name)
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 189, in run_task
    if task_succeed(json_results):
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 96, in task_succeed
    rally_report = json.loads(json_raw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I am a bit puzzled by the "Unable to establish connection to http://192.168.0.92:35357"
this IP address does not correspond to the IP from en used to build the deployment context

I saw in the IRC log that someone seems to have already faced this change (http://irclog.perlgeek.de/openstack-rally/2015-03-06/text 17:24)
I saw that one cause could be the non declaration of end point
so I tried a deployment using config file instead of env (I added the public IP associated with the port 35357) https://github.com/openstack/rally/blob/master/samples/deployments/existing-with-given-endpoint.json
but I still got an error (test done with MOS installer here) http://pastebin.com/aTfSxFuQ
I tried using the keystone v3 profile (https://github.com/openstack/rally/blob/master/samples/deployments/existing-keystone-v3.json) but still same issue

[Sergey Skripnick (eyerediskin)]

Seems your cloud is configured to allow admin actions only from "admin" network (192.168.0), so if you make this network accessible by rally node, everything will work.

[Morgan Richomme (morgan-richomme)]

you are right, it was a config issue on our side.
Basically, according to our architecture, we need to authorize the access to the management network (which is not ideal) from the jumphost.
So we added a vlan interface and moved the IP to that interface. That is why we got the connection issue. Once properly reconfigured, rally is running well (https://build.opnfv.org/ci/job/functest-fuel-stable-arno/5/console)

I should have focused on this connectivity issue message but I was mis-leaded by the first comments I saw on IRC log that were dealing with a specific configuration of the endpoint

thanks for the answer

[JoseLausuch (jlausuch)]

Sergey,

Is there a way to disable that config (allowing admin actions from public network)?
We would like to get rid of configuring an admin network on the jumphost where we run Rally...

[Sergey Skripnick (eyerediskin)]

There are options admin_bind_host, public_bind_host, admin_port, public_port, etc in keystone.conf. Also you may ask someone on IRC #openstack-keystone or #openstack-rally.