Tempest failures after run_ssh = True

[Matt Thompson]

I'm seeing a few smoke tests now failing with:

05:46:37 Traceback (most recent call last):
05:46:37 File "/opt/tempest_d1a391a55482d64b9014e7b41219af195722d990/tempest/api/compute/servers/test_server_actions.py", line 106, in test_reboot_server_hard
05:46:37 self._test_reboot_server('HARD')
05:46:37 File "/opt/tempest_d1a391a55482d64b9014e7b41219af195722d990/tempest/api/compute/servers/test_server_actions.py", line 88, in _test_reboot_server
05:46:37 self.password)
05:46:37 File "/opt/tempest_d1a391a55482d64b9014e7b41219af195722d990/tempest/common/utils/linux/remote_client.py", line 37, in __init__
05:46:37 addresses = server['addresses'][network]
05:46:37 KeyError: 'private'

The issue here is that the tenant which owns the server in question does not have access to the network 'private' (it's marked as not shared and assigned to demo user), and as such the instance doesn't get assigned an IP.

The alternative is to mark the network as shared, but then you run into the issue where nova cannot boot instances due to there being multiple available networks to use.

[Matt Thompson (mattt416)]

One way around this is to set allow_tenant_isolation = False, which results in instances being booted as demo user (which has access to private network). We then run into another issue in that we have network_for_ssh = private, which is not a routable address. We have use_floatingip_for_ssh = True and ssh_connect_method = floating, but tempest does not appear to be creating floating IPs for these instances.

[Matt Thompson (mattt416)]

I got confirmation in #openstack-qa that the affected tests do not spin up floating IPs, which is currently being addressed in https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/ssh-auth-strategy,n,z (I have not had a chance to look in detail). In the mean time, I have set run_ssh = False in the heat multi node jenkins job so that we can continue running the smoke tests until this blueprint is implemented and merged.

[Kevin Carter (kevin-carter)]

is this still a problem with the updated versions of tempest that we're using ?

[Tom Cameron (drdabbles)]

It appears this was already changed in a commit from the April or May timeframe.

[Tom Cameron (drdabbles)]

tempest_compute_run_ssh was set to True in commit 61deb8de8b018fdc2ca8344f1602f9335cdafd9a by Hugh Saunders

[Tom Cameron (drdabbles)]

This bug may not be addressed yet. The issue at hand may still exist upstream, and will take some investigation.

The specific problem is that when executing smoke or API tests, the exception may be triggered. These do not run as a normal part of gating or AIO jobs, so the assumption that those run successfully means this is not an issue was probably wrong.

[Matt Thompson (mattt416)]

I'm not sure if this is still an issue, but closing since it's been opened for so long.