Comment 0 for bug 1892358

Hi,
we had such cases in the past like bug 1817721 for bionic and maybe bug 1892130 is about the same as well. There were more but I didn't want to search for all of them - what I checked is that there are no open ones clearly pointing out the recent further drop in already flaky subtests.

In particular the tests "tests-in-lxd" and "systemd-fsckd" were known to be flaky before, but got even worse.

Here stats of the last 40 runs, it might be a coincidences that this is after 246-2ubuntu1 landed. Could as well be any other change

groovy
  amd64
tests-in-lxd (F 42% S 0% B 10% => P 45%/) FFFFBFFFFFFFB....FF.B.....F.....F...FBF
build-login (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
unit-config (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
networkd-testpy (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
boot-and-services (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
boot-smoke (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
logind (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
storage (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
upstream (F 35% S 0% B 10% => P 52%/) ..FFB.FFF.FFB....FF.B.....F.F..F....FBF
udev (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
systemd-fsckd (F 37% S 0% B 10% => P 50%/) FFFFBFFFFFFFB.FF...FB.....F..........B.
root-unittests (F 0% S 0% B 10% => P 87%/) ....B.......B.......B................B.
  ppc64el
tests-in-lxd (F 25% S 0% B 0% => P 75%/) FFFF....FF............FFF.....F.........
systemd-fsckd (F 35% S 0% B 0% => P 65%/) FFFFFFF...FF........F....FF.F..F........
root-unittests (F 2% S 0% B 0% => P 97%/) ..............................F.........
  s390x
tests-in-lxd (F 52% S 0% B 0% => P 47%/) FFFFFFF.FFFFFFF.FF.........FFFF...F.....
timedated (F 2% S 0% B 0% => P 97%/) ...........F............................
upstream (F 17% S 0% B 0% => P 82%/) .....F......F.F.............FFF...F.....
systemd-fsckd (F 32% S 0% B 0% => P 67%/) FFFFFFF..FF..F.................FF..F....
root-unittests (F 10% S 0% B 0% => P 90%/) ............................FFF...F.....
  arm64
tests-in-lxd (F 40% S 0% B 2% => P 57%/) FFFFF.B...FFF.FF..F..F.........FFF.F....
logind (F 2% S 0% B 2% => P 95%/) ......B...................F.............
upstream (F 22% S 0% B 2% => P 75%/) ...F.FB.....F.F.............F..FFF.F....
root-unittests (F 12% S 0% B 2% => P 85%/) ......B.F...........F.F........F...F....

(I'm sure LP will make this unreadable, but is is nice in monospace)

Whatever the root cause is - the success rate of these has reduced so much that the (even formerly questionable) practice of retry-until-success won't work anymore.

I have run the two tests in a local VM and systemd-fsckd works there while tests-in-lxd seems to trip over the old flaky fellow being "boot-and-services".

We had the discussion in the past, but I think I need to again bring up the suggestion to skip "tests-in-lxd" and "systemd-fsckd" until they are on reasonable success rates.