Smoketesting issues started with krillin ubuntu-rtm image #128

[Łukasz Zemczak]

Since around image #128 we have noticed degrading quality of our autopilot image testing. Nothing obvious has landed on that image, but it seems that not all tests are being ran and generally the smoketesting infra seems to be hanging on syncing specific tests. Those tests seem to vary from image to image, sometimes it being reminders, sometimes weather, sometimes system-settings. Even after timing out, the results are still not syncing.

Any ideas what could have happened? Were there any infrastructure changes? Or maybe the apt revert we made could have caused it?

[Łukasz Zemczak (sil2100)]

Francis was kind enough to re-kick the tests that were missing due to devices failure, but it seems those are still getting stuck for different reasons. Could we get anyone looking into that?

[Francis Ginther (fginther)]

133 finally completed testing after kicking it twice. Paul is working on some changes to better detect a failed device at the end of a test run, this fix would eliminate one of the failures. I've also submitted an MP to retry upon ubuntu-device-flash errors which were visible in a number of job failures in the past few days.

One issue that still needs to be addressed are retries around package installation. One of the builds failed because the unity8-autopilot packages could not be installed. The error appeared to be transient as it worked on the next attempt.

[Francis Ginther (fginther)]

Also, it appears that the results are only getting synced now when the individual builds complete. Will dig on this further, but may need to consult with Paul.

[Łukasz Zemczak (sil2100)]

Does anyone know if all the issues we saw here got fixed? The tests certainly look better, but is that because of some fix?