I think we need to update the logstash query for this because I'm seeing out of space issues causing ssh to fail but the existing query isn't in the guest console output, e.g.:
https://zuul.opendev.org/t/openstack/build/73e216c0b8d94365a5e5f1c94e6af361/log/job-output.txt#78872
2019-09-29 08:15:56.988235 | controller | b'successful after 1/20 tries: up 16.34. iid=i-0000004a'
2019-09-29 08:15:56.988283 | controller | b'failed to get http://169.254.169.254/2009-04-04/user-data'
2019-09-29 08:15:56.988320 | controller | b'warning: no ec2 metadata for user-data'
2019-09-29 08:15:56.988374 | controller | b'sh: write error: No space left on device'
2019-09-29 08:15:56.988415 | controller | b'Top of dropbear init script'
2019-09-29 08:15:56.988445 | controller | b'Starting dropbear sshd: OK'
2019-09-29 08:15:56.988499 | controller | b'GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown'
Note the http failure to get user-data but the logstash query is looking for meta-data.
We could probably just look for "sh: write error: No space left on device" or "GROWROOT...it cannot be grown".
I think we need to update the logstash query for this because I'm seeing out of space issues causing ssh to fail but the existing query isn't in the guest console output, e.g.:
https:/ /zuul.opendev. org/t/openstack /build/ 73e216c0b8d9436 5a5e5f1c94e6af3 61/log/ job-output. txt#78872
2019-09-29 08:15:56.988235 | controller | b'successful after 1/20 tries: up 16.34. iid=i-0000004a'
2019-09-29 08:15:56.988283 | controller | b'failed to get http:// 169.254. 169.254/ 2009-04- 04/user- data'
2019-09-29 08:15:56.988320 | controller | b'warning: no ec2 metadata for user-data'
2019-09-29 08:15:56.988374 | controller | b'sh: write error: No space left on device'
2019-09-29 08:15:56.988415 | controller | b'Top of dropbear init script'
2019-09-29 08:15:56.988445 | controller | b'Starting dropbear sshd: OK'
2019-09-29 08:15:56.988499 | controller | b'GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown'
Note the http failure to get user-data but the logstash query is looking for meta-data.
We could probably just look for "sh: write error: No space left on device" or "GROWROOT...it cannot be grown".