Unable to setup context 'users' when launching rally scenario

Bug #1489746 reported by Morgan Richomme on 2015-08-28
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Rally
Undecided
Unassigned

Bug Description

We use Rally to test OPNFV solutions. It has been integrated in CI.
We used to run successfully
- most of the rally scenario (authenticate, cider, glance, keystone, nova, neutron, quotas, requests)
- tempest smoke suite

The OPNFV solution consists in the integration of OpenStack + ODL.
In CI we are testing 2 OPNFV flavours:
- 1 installed from fuel (MOS 6.0.1)
- 1 installed through foreman/puppet installer

We were able to run all the tests automatically (https://build.opnfv.org/ci/view/functest/)
Rally is automatically installed and configured (including tempest part) based on env (fromenv option) using keystone v2.0

Installation looks fine
2015-08-25 17:33:12,297 - config_functest - DEBUG - Creating Rally environment...
2015-08-25 17:33:12,297 - config_functest - DEBUG - Executing command : rally deployment create --fromenv --name=opnfv-arno-rally
2015-08-25 17:33:13,166 - config_functest - DEBUG - 2015-08-25 17:33:12.973 25872 INFO rally.deployment.engine [-] Deployment ee971af8-2369-42d1-96bf-fa6f0768d634 | Starting: OpenStack cloud deployment.
2015-08-25 17:33:13.057 25872 INFO rally.deployment.engine [-] Deployment ee971af8-2369-42d1-96bf-fa6f0768d634 | Completed: OpenStack cloud deployment.
+--------------------------------------+----------------------------+------------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------------+------------------+--------+
| ee971af8-2369-42d1-96bf-fa6f0768d634 | 2015-08-26 00:33:12.918895 | opnfv-arno-rally | deploy->finished | |
+--------------------------------------+----------------------------+------------------+------------------+--------+
...
2015-08-25 17:34:21,107 - config_functest - DEBUG - Executing command : rally deployment check
2015-08-25 17:34:22,028 - config_functest - DEBUG - keystone endpoints are valid and following services are available:
+----------+----------------+-----------+
| services | type | status |
+----------+----------------+-----------+
| cinder | volume | Available |
| cinderv2 | volumev2 | Available |
| cloud | cloudformation | Available |
| ec2 | ec2 | Available |
| glance | image | Available |
| heat | orchestration | Available |
| keystone | identity | Available |
| neutron | network | Available |
| nova | compute | Available |
| novav3 | computev3 | Available |
+----------+----------------+-----------+

rally show flavors OK (so keystone v2.0 authentication ok)

src(https://build.opnfv.org/ci/view/functest/job/functest-foreman-master/2/consoleText)

We still can run Tempest smoke suite but it is now impossible to launch the rally scenario
we got an error (here with the foreman installer but same error with fuel):

2015-08-25 17:37:16.502 28634 INFO rally.plugins.openstack.context.keystone.users [-] Task c2c0352e-f60d-4fdd-8211-8a61eb09d539 | Starting: Enter context: `users`
2015-08-25 17:37:19.892 28634 WARNING rally.common.broker [-] Failed to consume a task from the queue: Unable to establish connection to http://192.168.0.92:35357/v2.0/tenants
....
2015-08-25 17:37:22,292 - run_rally - DEBUG - saving json file2
authenticate
Traceback (most recent call last):
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 217, in <module>
    main()
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 208, in main
    run_task(test_name)
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 189, in run_task
    if task_succeed(json_results):
  File "/home/jenkins-ci/workspace/functest-foreman-master/testcases/VIM/OpenStack/CI/libraries/run_rally.py", line 96, in task_succeed
    rally_report = json.loads(json_raw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I am a bit puzzled by the "Unable to establish connection to http://192.168.0.92:35357"
this IP address does not correspond to the IP from en used to build the deployment context

I saw in the IRC log that someone seems to have already faced this change (http://irclog.perlgeek.de/openstack-rally/2015-03-06/text 17:24)
I saw that one cause could be the non declaration of end point
so I tried a deployment using config file instead of env (I added the public IP associated with the port 35357) https://github.com/openstack/rally/blob/master/samples/deployments/existing-with-given-endpoint.json
but I still got an error (test done with MOS installer here) http://pastebin.com/aTfSxFuQ
I tried using the keystone v3 profile (https://github.com/openstack/rally/blob/master/samples/deployments/existing-keystone-v3.json) but still same issue

Sergey Skripnick (eyerediskin) wrote :

Seems your cloud is configured to allow admin actions only from "admin" network (192.168.0), so if you make this network accessible by rally node, everything will work.

you are right, it was a config issue on our side.
Basically, according to our architecture, we need to authorize the access to the management network (which is not ideal) from the jumphost.
So we added a vlan interface and moved the IP to that interface. That is why we got the connection issue. Once properly reconfigured, rally is running well (https://build.opnfv.org/ci/job/functest-fuel-stable-arno/5/console)

I should have focused on this connectivity issue message but I was mis-leaded by the first comments I saw on IRC log that were dealing with a specific configuration of the endpoint

thanks for the answer

Changed in rally:
status: New → Invalid
JoseLausuch (jlausuch) wrote :

Sergey,

Is there a way to disable that config (allowing admin actions from public network)?
We would like to get rid of configuring an admin network on the jumphost where we run Rally...

Sergey Skripnick (eyerediskin) wrote :

There are options admin_bind_host, public_bind_host, admin_port, public_port, etc in keystone.conf. Also you may ask someone on IRC #openstack-keystone or #openstack-rally.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers