Intermittent failure to unarchive nova venv

[Charles Farquhar]

I have an intermittent build failure that seems to happen approximately 1 out of 8 or 10 builds. Re-running the same deployment code on the same hardware generally resolves the issue.

It seems like the "Create nova venv dir" task returns successfully, but when we reach "Unarchive pre-built venv", it fails because the venv directory is not created. If this is unsuccessful on the first pass, it fails indefinitely because we don't retry these tasks unless the SHA for the venv archive in /var/cache does not match the remote venv archive's SHA.

This problem is occurring on the Liberty branch.

First pass failure:
```
TASK: [os_nova | Set nova get_venv fact] **************************************
ok: [infra01-d_nova_api_os_compute_container-10b634d9]
ok: [infra01-d_nova_console_container-19d67f32]
ok: [infra01-d_nova_conductor_container-0a3acd59]
ok: [infra01-d_nova_scheduler_container-65aac9f4]
ok: [infra01-d]
ok: [infra01-d_nova_api_metadata_container-1d93a703]
ok: [infra01-d_nova_cert_container-6e1d9e0a]

TASK: [os_nova | Remove existing venv] ****************************************
changed: [infra01-d_nova_scheduler_container-65aac9f4]
changed: [infra01-d_nova_console_container-19d67f32]
changed: [infra01-d_nova_api_os_compute_container-10b634d9]
changed: [infra01-d_nova_conductor_container-0a3acd59]
changed: [infra01-d_nova_api_metadata_container-1d93a703]
changed: [infra01-d]
changed: [infra01-d_nova_cert_container-6e1d9e0a]

TASK: [os_nova | Create nova venv dir] ****************************************
changed: [infra01-d_nova_console_container-19d67f32]
changed: [infra01-d_nova_cert_container-6e1d9e0a]
changed: [infra01-d]
changed: [infra01-d_nova_conductor_container-0a3acd59]
changed: [infra01-d_nova_scheduler_container-65aac9f4]
changed: [infra01-d_nova_api_metadata_container-1d93a703]
changed: [infra01-d_nova_api_os_compute_container-10b634d9]

TASK: [os_nova | Unarchive pre-built venv] ************************************
failed: [infra01-d_nova_api_os_compute_container-10b634d9] => {"failed": true}
msg: dest '/openstack/venvs/nova-12.0.10' must be an existing dir
changed: [infra01-d]
changed: [infra01-d_nova_console_container-19d67f32]
changed: [infra01-d_nova_cert_container-6e1d9e0a]
changed: [infra01-d_nova_scheduler_container-65aac9f4]
changed: [infra01-d_nova_api_metadata_container-1d93a703]
changed: [infra01-d_nova_conductor_container-0a3acd59]
```

Subsequent pass failure:
```
TASK: [os_nova | Set nova get_venv fact] **************************************
ok: [infra01-d_nova_cert_container-6e1d9e0a]
ok: [infra01-d_nova_api_os_compute_container-10b634d9]
ok: [infra01-d_nova_conductor_container-0a3acd59]
ok: [infra01-d_nova_console_container-19d67f32]
ok: [infra01-d_nova_scheduler_container-65aac9f4]
ok: [infra01-d]
ok: [infra01-d_nova_api_metadata_container-1d93a703]

TASK: [os_nova | Remove existing venv] ****************************************
skipping: [infra01-d_nova_conductor_container-0a3acd59]
skipping: [infra01-d_nova_console_container-19d67f32]
skipping: [infra01-d_nova_cert_container-6e1d9e0a]
skipping: [infra01-d_nova_scheduler_container-65aac9f4]
skipping: [infra01-d_nova_api_os_compute_container-10b634d9]
skipping: [infra01-d]
skipping: [infra01-d_nova_api_metadata_container-1d93a703]

TASK: [os_nova | Create nova venv dir] ****************************************
skipping: [infra01-d_nova_api_os_compute_container-10b634d9]
skipping: [infra01-d_nova_cert_container-6e1d9e0a]
skipping: [infra01-d_nova_scheduler_container-65aac9f4]
skipping: [infra01-d_nova_console_container-19d67f32]
skipping: [infra01-d_nova_api_metadata_container-1d93a703]
skipping: [infra01-d]
skipping: [infra01-d_nova_conductor_container-0a3acd59]

TASK: [os_nova | Unarchive pre-built venv] ************************************
skipping: [infra01-d_nova_cert_container-6e1d9e0a]
skipping: [infra01-d_nova_api_os_compute_container-10b634d9]
skipping: [infra01-d_nova_conductor_container-0a3acd59]
skipping: [infra01-d]
skipping: [infra01-d_nova_console_container-19d67f32]
skipping: [infra01-d_nova_scheduler_container-65aac9f4]
skipping: [infra01-d_nova_api_metadata_container-1d93a703]

TASK: [os_nova | Update virtualenv path] **************************************
failed: [infra01-d_nova_api_os_compute_container-10b634d9] => {"changed": true, "cmd": ["virtualenv-tools", "--update-path=auto", "/openstack/venvs/nova-12.0.10"], "delta": "0:00:00.033160", "end": "2016-05-03 14:03:01.989031", "rc": 1, "start": "2016-05-03 14:03:01.955871", "warnings": []}
stdout: error: /openstack/venvs/nova-12.0.10 does not refer to a python installation
changed: [infra01-d]
changed: [infra01-d_nova_conductor_container-0a3acd59]
changed: [infra01-d_nova_cert_container-6e1d9e0a]
changed: [infra01-d_nova_console_container-19d67f32]
changed: [infra01-d_nova_api_metadata_container-1d93a703]
changed: [infra01-d_nova_scheduler_container-65aac9f4]
```

View of destination directory:
```
root@infra01-d_nova_api_os_compute_container-10b634d9:~# ll -R /openstack/venvs/nova-12.0.10/
/openstack/venvs/nova-12.0.10/:
total 12
drwxr-xr-x 3 root root 4096 May 3 11:48 ./
drwxr-xr-x 3 root root 4096 May 3 10:19 ../
drwxr-xr-x 2 root root 4096 May 3 11:48 bin/

/openstack/venvs/nova-12.0.10/bin:
total 8
drwxr-xr-x 2 root root 4096 May 3 11:48 ./
drwxr-xr-x 3 root root 4096 May 3 11:48 ../
```

There is nothing wrong with the nova venv archive we downloaded:
```
root@infra01-d_nova_api_os_compute_container-10b634d9:/tmp# tar xvzf /var/cache/nova-12.0.10.tgz
./
./local/
./local/lib/
./local/lib/python2.7/
./local/lib/python2.7/stat.py

...

./share/websockify/include/web-socket-js/web_socket.js
./share/websockify/include/web-socket-js/WebSocketMain.swf
./share/websockify/include/websock.js
./share/websockify/include/base64.js
./share/websockify/include/util.js
root@infra01-d_nova_api_os_compute_container-10b634d9:/tmp#
```

[Charles Farquhar (cfarquhar)]

This seems to be related to [1].

[1] https://github.com/ansible/ansible-modules-core/issues/932