Automatic retries of insert/storage-test/remove jobs occur in wrong order

[Betty Lin]

Description:
The test order of insert/storage-test/remove will wrong when re-testing
The order becomes to:
insert
remove
storage-test

The workaround now is: after finished the test insert and remove then plugged in storage and run storage-test again.

Reproduced steps:
1. Select tests: usb3/insert, usb3/storage-automated, usb3/remove
2. Don't insert USB3 storage when running usb3/insert (let it failed)
3. Re-run usb3/insert, usb3/storage-automated, usb3/remove

Moreover, if we re-run insert/storage-test/remove and after suspend insert/storage-test/remove,
the order becomes to:
suspend/insert
insert
suspend/storage-test
remove
storage-test
suspend/remove

And no suspend/suspend_advanced before running suspend/insert.

Reproduced steps:
1. Select tests: usb3/insert, usb3/storage-automated, usb3/remove, usb3_insert_after_suspend, usb3_storage_automated_after_suspend, usb3_remove_after_suspend
2. Don't insert USB3 storage when running usb3/insert and usb3_storage_automated_after_suspend (let them failed)
3. Re-run usb3/insert, usb3/storage-automated, usb3/remove, usb3_insert_after_suspend, usb3_storage_automated_after_suspend, usb3_remove_after_suspend

The problem affects to:
 mediacard/sd-insert
 mediacard/sd-storage
 mediacard/sd-remove
 mediacard/sdhc-insert
 mediacard/sdhc-storage
 mediacard/sdhc-remove
 usb/insert
 usb/storage-automated
 usb/remove
 usb3/insert
 usb3/storage-automated
 usb3/remove
 usb-c/c-to-a-adapter/insert
 usb-c/c-to-a-adapter/storage-automated
 usb-c/c-to-a-adapter/remove
 usb-c/insert
 usb-c/storage-automated
 usb-c/remove
 thunderbolt/insert
 thunderbolt/storage-test
 thunderbolt/remove
 thunderbolt3/insert
 thunderbolt3/storage-test
 thunderbolt3/remove
 suspend/sd-insert-after-suspend
 suspend/sd-storage-after-suspend
 suspend/sd-remove-after-suspend
 suspend/sdhc-insert-after-suspend
 suspend/sdhc-storage-after-suspend
 suspend/sdhc-remove-after-suspend
 suspend/usb_insert_after_suspend
 suspend/usb_storage_automated_after_suspend
 suspend/usb_remove_after_suspend
 suspend/usb3_insert_after_suspend
 suspend/usb3_storage_automated_after_suspend
 suspend/usb3_remove_after_suspend
 after-suspend-manual-usb-c/c-to-a-adapter/insert
 after-suspend-manual-usb-c/c-to-a-adapter/storage-automated
 after-suspend-manual-usb-c/c-to-a-adapter/remove
 after-suspend-manual-usb-c/insert
 after-suspend-manual-usb-c/storage-automated
 after-suspend-manual-usb-c/remove

u@u-XPS-13-9380:~$ checkbox-cli --version
checkbox-ng: 1.4.0rc1
checkbox-support: 0.41.0rc1
com.canonical.ce:oem: 1.0
certification-client: 0.37.0rc1
plainbox-provider-checkbox: 0.48.0rc3
plainbox-provider-resource-generic: 0.40.0rc1
plainbox-provider-sru: 1.13.0rc1
plainbox-provider-tpm2: 1.10.0rc1

[Jonathan Cave (jocave)]

I tried reproducing this today, but the order seemed to remain the expected insert, storage-test, remove. Checkbox versions:

joc@io:/var/tmp/checkbox-providers$ checkbox-cli --version
checkbox-ng: 1.6.0.dev0
checkbox-support: 0.43.0.dev0

Providers sideloaded from master.

I ran:

$ checkbox-cli

Then selected the usb-cert-manual test plan. (I don't have any USB3 devices but the jobs are identical).