b/linux: kvm_smoke_test fails on metal/kernel02

[Manuel Diewald]

The test started to fail with bionic/linux 4.15.0-222.233.

16:08:24 ERROR| [stderr] + uvt-kvm wait bjf-test --insecure --ssh-private-key-file /home/ubuntu/.ssh/id_rsa
16:11:22 ERROR| [stderr] uvt-kvm: error: timed out waiting for ssh to open on 192.168.122.148.
16:11:22 ERROR| [stderr] + cleanup
16:11:22 ERROR| [stderr] + uvt-kvm destroy bjf-test
16:11:23 ERROR| [stderr] + '[' s390x = ppc64el ']'
16:11:23 ERROR| Exception escaping from test:
Traceback (most recent call last):
  File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
    return func(*args, **dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
    postprocess_profiled_run, args, dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
    self.run_once(*args, **dargs)
  File "/home/ubuntu/autotest/client/tests/ubuntu_kvm_smoke_test/ubuntu_kvm_smoke_test.py", line 64, in run_once
    self.results = utils.system_output(cmd, retain_output=True)
  File "/home/ubuntu/autotest/client/shared/utils.py", line 1268, in system_output
    verbose=verbose, args=args).stdout
  File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
    "Command returned non-zero exit status")
CmdError: Command <sudo -u ubuntu /home/ubuntu/autotest/client/tests/ubuntu_kvm_smoke_test/the-test s390x> failed, rc=1, Command returned non-zero exit status
* Command:
    sudo -u ubuntu /home/ubuntu/autotest/client/tests/ubuntu_kvm_smoke_test
    /the-test s390x
Exit status: 1
Duration: 189.442286968

This might be related to the changes merged for https://bugs.launchpad.net/uvtool/+bug/2039441

main_wait_remote() in uvtool/wait will now enforce the timeout (default 120 seconds) when waiting for the remote to be ready, while previously before these changes, it did not. The remote-wait.sh script in uvtool does not enforce the timeout.

Looking at the failed test run, uvt-kvm wait time outs after running approximately 3 minutes.

The previous successful run of the test on the same machine (in s2023.10.30) takes about 8 minutes for uvt-kvm wait to return:

07:30:51 ERROR| [stderr] + uvt-kvm wait bjf-test --insecure --ssh-private-key-file /home/ubuntu/.ssh/id_rsa
07:38:53 ERROR| [stderr] ++ uvt-kvm ip bjf-test
07:38:53 ERROR| [stderr] + ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa ubuntu@192.168.122.47 uname -a
07:39:37 DEBUG| [stdout] Linux bjf-test 4.15.0-194-generic #205-Ubuntu SMP Fri Sep 16 19:53:54 UTC 2022 s390x s390x s390x GNU/Linux
07:39:37 ERROR| [stderr] ++ cut '-d ' -f 2
07:39:37 ERROR| [stderr] +++ uvt-kvm ip bjf-test
07:39:38 ERROR| [stderr] ++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -i /home/ubuntu/.ssh/id_rsa ubuntu@192.168.122.47 lsb_release -c
07:40:19 ERROR| [stderr] + RESULT=bionic
07:40:19 ERROR| [stderr] + '[' bionic '!=' bionic ']'
07:40:19 ERROR| [stderr] + cleanup
07:40:19 ERROR| [stderr] + uvt-kvm destroy bjf-test
07:40:20 ERROR| [stderr] + '[' s390x = ppc64el ']'
07:40:21 INFO | GOOD ubuntu_kvm_smoke_test.kvm_smoke_test ubuntu_kvm_smoke_test.kvm_smoke_test timestamp=1706082021 localtime=Jan 24 07:40:21 completed successfully

If my suspicion is correct, the timeout for waiting for the machine to be ready to log in via ssh is reached, but it is not a regression.

[Po-Hsu Lin (cypressyew)]

I give it a try manually and it failed on my first attempt.
On the second attempt I tried to use `virsh console` to see what's going on but test has passed this time.
And so forth for the 3rd and 4th manual attempts.

Perhaps it's the default timeout is too close to what it takes in some occasions.

The test result is now green after retest.