Upgrades job failing pingtest with "Message: No valid host was found."

[Giulio Fidente]

It looks like the compute nodes are unable to authenticate against keystone after the stack-update and are logging messages like:

  Unable to establish connection to http://[2001:db8:fd00:1000::10]:5000/v3/auth/tokens

After a number of attempts nova will give up with "No valid host".

This can be seen for example in the CI jobs for https://review.openstack.org/#/c/313650

[Sagi (Sergey) Shnaidman (sshnaidm)]

May be related: tempest configuration script failed to configure identity v3 features on upgrades and ha jobs, but succeeds on nonha.

[Derek Higgins (derekh)]

endpoint show for the keystone endpoint show nova compute using the publicurl and it can't reach this IP address, shouldn't it be using the internalurl?

[stack@instack ~]$ openstack endpoint show 0484edd967e34655bf276a8f5bb1dbc5
+--------------+--------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------+
| adminurl | http://192.0.2.6:35357/v2.0 |
| enabled | True |
| id | 0484edd967e34655bf276a8f5bb1dbc5 |
| internalurl | http://[fd00:fd00:fd00:2000::11]:5000/v2.0 |
| publicurl | http://[2001:db8:fd00:1000::10]:5000/v2.0 |
| region | regionOne |
| service_id | cdde9d9091aa4106a0f639213a917876 |
| service_name | keystone |
| service_type | identity |
+--------------+--------------------------------------------+

[Carlos Camacho (ccamacho)]

Hello Giulio, this error happens not only in upgrades jobs, the error is
(resources.server1: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500") in my case when creating/deleting overcloud several times, maybe 7 or 8 times.

In http://logs.openstack.org/50/313650/2/check-tripleo/gate-tripleo-ci-f22-upgrades/c8c0034/console.html (2016-05-09 15:56:18.569 )

I had to remove/reinstall the undercloud/overcloud to overcome this, I will debug the error and put more info about it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/314684

[Dan Prince (dan-prince)]

This patch from Derekh seems to fix it:

https://review.openstack.org/#/c/314510/

That was the only upgrades CI job pass I've seen in the last few days.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314510
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d76f70edcd3bbf0308071638cb4ee8eb2f70503a
Submitter: Jenkins
Branch: master

commit d76f70edcd3bbf0308071638cb4ee8eb2f70503a
Author: Derek Higgins <email address hidden>
Date: Tue May 10 12:21:53 2016 +0100

    Set nova neutron auth back to 'v3password'.

    Puppet-nova recently changed the default neutron auth setting
    in I3416ae594e972e40ff0336779258a887987e46b1 to 'password'.

    This single setting seems to break the tripleo upgrades job.

    Setting it here manually for now and following up in puppet-nova.

    Closes-bug: #1580076

    Change-Id: I3f38a3e1ef3378a272a51ecbc1e8a801c8d3608a

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Giulio Fidente (<email address hidden>) on branch: master
Review: https://review.openstack.org/314684
Reason: has been fixed in I3f38a3e1ef3378a272a51ecbc1e8a801c8d3608a