Basically all of our compute nodes just went down too:
+----+--------------------------------------+-------+----------+ | ID | Hypervisor hostname | State | Status | +----+--------------------------------------+-------+----------+ | 1 | overcloud-novacompute-28.localdomain | down | enabled | | 2 | overcloud-novacompute-29.localdomain | down | enabled | | 3 | overcloud-novacompute-30.localdomain | down | enabled | | 4 | overcloud-novacompute-26.localdomain | down | enabled | | 5 | overcloud-novacompute-6.localdomain | down | enabled | | 6 | overcloud-novacompute-27.localdomain | down | enabled | | 7 | overcloud-novacompute-10.localdomain | down | enabled | | 8 | overcloud-novacompute-31.localdomain | down | enabled | | 9 | overcloud-novacompute-1.localdomain | down | disabled | | 10 | overcloud-novacompute-9.localdomain | down | enabled | | 11 | overcloud-novacompute-4.localdomain | down | enabled | | 12 | overcloud-novacompute-14.localdomain | down | disabled | | 13 | overcloud-novacompute-19.localdomain | down | enabled | | 14 | overcloud-novacompute-5.localdomain | down | enabled | | 15 | overcloud-novacompute-17.localdomain | down | disabled | | 16 | overcloud-novacompute-22.localdomain | down | enabled | | 17 | overcloud-novacompute-7.localdomain | down | enabled | | 18 | overcloud-novacompute-8.localdomain | up | enabled | | 19 | overcloud-novacompute-0.localdomain | down | enabled | | 20 | overcloud-novacompute-3.localdomain | down | enabled | | 21 | overcloud-novacompute-23.localdomain | down | enabled | | 22 | overcloud-novacompute-21.localdomain | down | enabled | | 23 | overcloud-novacompute-25.localdomain | down | enabled | | 24 | overcloud-novacompute-32.localdomain | down | enabled | | 25 | overcloud-novacompute-18.localdomain | down | enabled | | 26 | overcloud-novacompute-20.localdomain | down | enabled | | 27 | overcloud-novacompute-11.localdomain | down | enabled | | 28 | overcloud-novacompute-15.localdomain | down | enabled | | 29 | overcloud-novacompute-2.localdomain | down | disabled | | 30 | overcloud-novacompute-12.localdomain | down | enabled | | 31 | overcloud-novacompute-13.localdomain | down | enabled | | 32 | overcloud-novacompute-16.localdomain | down | enabled | | 33 | overcloud-novacompute-24.localdomain | down | enabled | +----+--------------------------------------+-------+----------+
And I can't get on them anymore:
[bnemec@RedHat ~]$ ssh -i rh1id xxx.xxx.xxx.72 ssh_exchange_identification: Connection closed by remote host
I'm going to start bouncing them in ironic since there's really nothing more I can do to debug this.
Basically all of our compute nodes just went down too:
+----+- ------- ------- ------- ------- ------- --+---- ---+--- ------- + ------- ------- ------- ------- ------- --+---- ---+--- ------- + novacompute- 28.localdomain | down | enabled | novacompute- 29.localdomain | down | enabled | novacompute- 30.localdomain | down | enabled | novacompute- 26.localdomain | down | enabled | novacompute- 6.localdomain | down | enabled | novacompute- 27.localdomain | down | enabled | novacompute- 10.localdomain | down | enabled | novacompute- 31.localdomain | down | enabled | novacompute- 1.localdomain | down | disabled | novacompute- 9.localdomain | down | enabled | novacompute- 4.localdomain | down | enabled | novacompute- 14.localdomain | down | disabled | novacompute- 19.localdomain | down | enabled | novacompute- 5.localdomain | down | enabled | novacompute- 17.localdomain | down | disabled | novacompute- 22.localdomain | down | enabled | novacompute- 7.localdomain | down | enabled | novacompute- 8.localdomain | up | enabled | novacompute- 0.localdomain | down | enabled | novacompute- 3.localdomain | down | enabled | novacompute- 23.localdomain | down | enabled | novacompute- 21.localdomain | down | enabled | novacompute- 25.localdomain | down | enabled | novacompute- 32.localdomain | down | enabled | novacompute- 18.localdomain | down | enabled | novacompute- 20.localdomain | down | enabled | novacompute- 11.localdomain | down | enabled | novacompute- 15.localdomain | down | enabled | novacompute- 2.localdomain | down | disabled | novacompute- 12.localdomain | down | enabled | novacompute- 13.localdomain | down | enabled | novacompute- 16.localdomain | down | enabled | novacompute- 24.localdomain | down | enabled | ------- ------- ------- ------- ------- --+---- ---+--- ------- +
| ID | Hypervisor hostname | State | Status |
+----+-
| 1 | overcloud-
| 2 | overcloud-
| 3 | overcloud-
| 4 | overcloud-
| 5 | overcloud-
| 6 | overcloud-
| 7 | overcloud-
| 8 | overcloud-
| 9 | overcloud-
| 10 | overcloud-
| 11 | overcloud-
| 12 | overcloud-
| 13 | overcloud-
| 14 | overcloud-
| 15 | overcloud-
| 16 | overcloud-
| 17 | overcloud-
| 18 | overcloud-
| 19 | overcloud-
| 20 | overcloud-
| 21 | overcloud-
| 22 | overcloud-
| 23 | overcloud-
| 24 | overcloud-
| 25 | overcloud-
| 26 | overcloud-
| 27 | overcloud-
| 28 | overcloud-
| 29 | overcloud-
| 30 | overcloud-
| 31 | overcloud-
| 32 | overcloud-
| 33 | overcloud-
+----+-
And I can't get on them anymore:
[bnemec@RedHat ~]$ ssh -i rh1id xxx.xxx.xxx.72 identification: Connection closed by remote host
ssh_exchange_
I'm going to start bouncing them in ironic since there's really nothing more I can do to debug this.