tempest-validation.log missing

[Marian Gasparovic]

When running validation plugin on sunbeam, we encountered this

2024-06-18-15:59:04 root DEBUG [localhost]: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null solqa-lab1-server-08.nosilo.lab1.solutionsqa -- sunbeam validation run quick --output validation_quick.log
2024-06-18-15:59:04 root DEBUG Warning: Permanently added 'solqa-lab1-server-08.nosilo.lab1.solutionsqa' (ED25519) to the list of known hosts.
2024-06-18-16:00:21 root DEBUG Error: {'error': 'Error during test execution.\nFor more information, copy log file from container by running:\n$ juju scp -m openstack --container tempest tempest/0:/var/lib/tempest/workspace/tempest-validation.log validation.log', 'return-code': 0}
2024-06-18-16:00:21 root ERROR [localhost] Command failed: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null solqa-lab1-server-08.nosilo.lab1.solutionsqa -- sunbeam validation run quick --output validation_quick.log
2024-06-18-16:00:21 root ERROR 1[localhost] STDOUT follows:

2024-06-18-16:00:21 root ERROR 2[localhost] STDERR follows:
Warning: Permanently added 'solqa-lab1-server-08.nosilo.lab1.solutionsqa' (ED25519) to the list of known hosts.
Error: {'error': 'Error during test execution.\nFor more information, copy log file from container by running:\n$ juju scp -m openstack --container tempest tempest/0:/var/lib/tempest/workspace/tempest-validation.log validation.log', 'return-code': 0}

logs from tempest pod show

[container-agent] 2024-06-18 16:00:21 ERROR juju-log Exited with code 1. Stderr:
[container-agent] 2024-06-18 16:00:21 ERROR juju-log mv: cannot move '/tmp/tmp.40DYrBvHLP' to '/var/lib/tempest/workspace/tempest-validation.log': No such file or directory
[container-agent] 2024-06-18 16:00:21 ERROR juju-log /usr/local/sbin/tempest-run-wrapper: line 40: /var/lib/tempest/workspace/tempest-validation.log: No such file or directory

Logs and artifacts - https://oil-jenkins.canonical.com/artifacts/6d0543c5-c24f-4359-9ef1-39686d0bd1d4/index.html

We saw it at least twice

[Andrea Ieri (aieri)]

this was observed (rarely) on revision 37 from channel 2024.1/beta

[Marian Gasparovic (marosg)]

After further tests - it happens every single time after machine reboot.