Rally task hangs in state "Verifying"

[Andrew Kalach]

Rally task hangs in state "Verifying" after finished interaction with Glance subsystem.

Lookup task status on the Fuel master node:

    cd /opt/stack
    source .venv/bin/activate
    rally task list

Rally task YAML file and detailed logs are in attachement

[Alexander Maretskiy (maretskiy)]

I've made some digging on this environment and collect additional info:

Rally process (PID 11852) from `ps aux' output:

develop+ 11852 0.0 0.4 500356 68736 pts/1 S+ Apr13 0:02 /opt/stack/.venv/bin/python /opt/stack/.venv/bin/rally task start --task-args {"controller": 3, "floating_ip_amount": 197, "compute": 44, "vlan_amount": 199, "concurrency": 5, "floating_net": "admin_floating_net", "gre_enabled": false, "current_path": "/opt/stack/rally-scenarios/nova"} /opt/stack/rally-scenarios/nova/boot_and_rebuild_server.yaml --abort-on-sla-failure --tag 11_env_run_rally/114/788afe79

Here we see that process 11852 has opened socket to 172.16.45.102:9292 (Glance service):

(.venv) [root@fuel stack]# netstat -pn --numeric-hosts | grep 11852
tcp 0 0 172.16.44.11:43247 172.16.45.102:9292 ESTABLISHED 11852/python
tcp 0 0 172.16.44.11:35281 172.18.160.54:5432 ESTABLISHED 11852/python

Here is a Glance service endpoint:

(.venv) [root@fuel stack]# openstack catalog show glance
+-----------+----------------------------------------+
| Field | Value |
+-----------+----------------------------------------+
| endpoints | RegionOne |
| | publicURL: http://172.16.45.102:9292 |
| | internalURL: http://192.168.0.2:9292 |
| | adminURL: http://192.168.0.2:9292 |
| | |
| name | glance |
| type | image |
+-----------+----------------------------------------+