rally boot and delete server failed with "No valid host was found"

[Jason Hobbs]

2020-09-27 18:35:25.194 18677 INFO rally.task.runner [-] Task 7ebb8dae-5747-463b-95de-2ef7c21bd9ef | ITER: 1 END: Error GetResourceErrorStatus: Resource <Server: s_rally_f606ac31_INQohbuY_0-5> has ERROR status.
 Fault: {'code': 500, 'created': '2020-09-27T18:35:24Z', 'message': 'No valid host was found. '}

The same test run also fails creating octavia load balancers:
rally.exceptions.GetResourceErrorStatus: Resource {'id': '2b51e1b5-6d71-47f5-9ed8-1724d203ae11', 'name': 's_rally_ff1c8b0b_lgGbx9hQ', 'description': '', 'provisioning_status': 'ERROR', 'operating_status': 'OFFLINE', 'admin_state_up': True, 'project_id': '9a72ae1523a341d687da85439bfe614d', 'created_at': '2020-09-27T18:20:22', 'updated_at': '2020-09-27T18:20:26', 'vip_address': '10.2.93.114', 'vip_port_id': '056a9b7e-7d08-4dc5-9742-61ac68ded233', 'vip_subnet_id': '98f71b99-b95b-40cc-aae7-802ec1c42070', 'vip_network_id': 'a26b7ceb-da54-47b1-bfe3-e1780767471d', 'listeners': [], 'pools': [], 'provider': 'amphora', 'flavor_id': None, 'vip_qos_policy_id': None, 'tags': [], 'availability_zone': None, 'tenant_id': '9a72ae1523a341d687da85439bfe614d'} has ERROR status.

See https://solutions.qa.canonical.com/openstack/testRun/33d7b6f3-d4a5-4b99-aadd-bb6791136827 for details

[Vladimir Grevtsev (vlgrevtsev)]

FWIW, the sympthoms are very similar to the https://bugs.launchpad.net/charm-placement/+bug/1896227 - I had the same outcome when tried to ran the VM creation without the n-c-c and n-compute service restarts..

[Michael Skalka (mskalka)]

Seen again with this test run: https://solutions.qa.canonical.com/openstack/testRun/fc8807dd-adde-41b3-bbe8-2dadb141d6c0

[Marian Gasparovic (marosg)]

again here
https://solutions.qa.canonical.com/testruns/testRun/e77a6461-ec7b-4a46-a9ba-88eb0ea1f168

[Jason Hobbs (jason-hobbs)]

sub'd to field high; this is blocking tests of stable charms to update our HOV skus.

[Liam Young (gnuoy)]

Having looked at the crashdumps they all show evidence of bug 1826382 ( https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1826382 )

The telltale signs being the messages:

 "Reason: You're speaking plain HTTP to an SSL-enabled server port"

in the nova-scheduler or nova-compute logs.

Run 33d7b6f3 was before the fixes landed, fc8807dd and e77a6461 used the stable charms and the fix hasn't been backported.

Fwiw if the fix is present the nova-cloud-controller units should have the message ""Waiting for catalog cache ttl to expire" in their juju logs.