set-timer-lat from timers in ubuntu_kernel_selftests failed on 4.15/5.4

[Po-Hsu Lin]

Issue found on 5.4.0-1016.16~18.04.1-aws, with instance t3a.2xlarge (Passed on other instances)

 # selftests: timers: set-timer-lat
 # Setting timers for every 1 seconds
 # CLOCK_REALTIME ABSTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_REALTIME ABSTIME PERIODIC max latency: 53219 ns : [OK]
 # CLOCK_REALTIME RELTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_REALTIME RELTIME PERIODIC max latency: 63712 ns : [OK]
 # CLOCK_REALTIME ABSTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_REALTIME ABSTIME ONE-SHOT max latency: 58921 ns : [OK]
 # CLOCK_REALTIME ABSTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_REALTIME RELTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_REALTIME RELTIME ONE-SHOT max latency: 86175 ns : [OK]
 # CLOCK_REALTIME RELTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_MONOTONIC ABSTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_MONOTONIC ABSTIME PERIODIC max latency: 299683 ns : [OK]
 # CLOCK_MONOTONIC RELTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_MONOTONIC RELTIME PERIODIC max latency: 238132 ns : [OK]
 # CLOCK_MONOTONIC ABSTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_MONOTONIC ABSTIME ONE-SHOT max latency: 61408 ns : [OK]
 # CLOCK_MONOTONIC ABSTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_MONOTONIC RELTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_MONOTONIC RELTIME ONE-SHOT max latency: 59345 ns : [OK]
 # CLOCK_MONOTONIC RELTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_BOOTTIME ABSTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME ABSTIME PERIODIC max latency: 63587 ns : [OK]
 # CLOCK_BOOTTIME RELTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME RELTIME PERIODIC max latency: 84887 ns : [OK]
 # CLOCK_BOOTTIME ABSTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME ABSTIME ONE-SHOT max latency: 58429 ns : [OK]
 # CLOCK_BOOTTIME ABSTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_BOOTTIME RELTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME RELTIME ONE-SHOT max latency: 35193 ns : [OK]
 # CLOCK_BOOTTIME RELTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_REALTIME_ALARM ABSTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_REALTIME_ALARM ABSTIME PERIODIC max latency: 67341 ns : [OK]
 # CLOCK_REALTIME_ALARM RELTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_REALTIME_ALARM RELTIME PERIODIC max latency: 75131 ns : [OK]
 # CLOCK_REALTIME_ALARM ABSTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_REALTIME_ALARM ABSTIME ONE-SHOT max latency: 57514 ns : [OK]
 # CLOCK_REALTIME_ALARM ABSTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_REALTIME_ALARM RELTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_REALTIME_ALARM RELTIME ONE-SHOT max latency: 30771 ns : [OK]
 # CLOCK_REALTIME_ALARM RELTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_BOOTTIME_ALARM ABSTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME_ALARM ABSTIME PERIODIC max latency: 64560 ns : [OK]
 # CLOCK_BOOTTIME_ALARM RELTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME_ALARM RELTIME PERIODIC max latency: 72624 ns : [OK]
 # CLOCK_BOOTTIME_ALARM ABSTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME_ALARM ABSTIME ONE-SHOT max latency: 32229 ns : [OK]
 # CLOCK_BOOTTIME_ALARM ABSTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_BOOTTIME_ALARM RELTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_BOOTTIME_ALARM RELTIME ONE-SHOT max latency: 60767 ns : [OK]
 # CLOCK_BOOTTIME_ALARM RELTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_TAI ABSTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_TAI ABSTIME PERIODIC max latency: 36900639844 ns : [FAILED]
 # CLOCK_TAI RELTIME PERIODIC timer fired early: 0 : [OK]
 # CLOCK_TAI RELTIME PERIODIC max latency: 60987 ns : [OK]
 # CLOCK_TAI ABSTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_TAI ABSTIME ONE-SHOT max latency: 42678 ns : [OK]
 # CLOCK_TAI ABSTIME ONE-SHOT count: 1 : [OK]
 # CLOCK_TAI RELTIME ONE-SHOT timer fired early: 0 : [OK]
 # CLOCK_TAI RELTIME ONE-SHOT max latency: 69952 ns : [OK]
 # CLOCK_TAI RELTIME ONE-SHOT count: 1 : [OK]
 # Bail out!
 # # Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
 not ok 4 selftests: timers: set-timer-lat # exit=1

[Po-Hsu Lin (cypressyew)]

Issue found on B-hwe 5.3
5.3.0-65.59

[Po-Hsu Lin (cypressyew)]

Found on B-4.15 (4.15.0-1080.84) AWS with instance m5.xlarge only

[Ian May (ian-may)]

Found on B-gcp-4.15 (4.15.0-1097.110) n1-highcpu-4, n1-standard-8

[Krzysztof Kozlowski (krzk)]

Found on focal/aws-5.8 5.8.0-1032.34~20.04.1

[Krzysztof Kozlowski (krzk)]

focal/aws-5.8 5.8.0-1032.34~20.04.1 in sru-20210412 looks like regression because:
1. on focal/aws from previous cycle (5.8.0-1028.30~20.04.3-aws) this test passed,
2. on groovy/aws from current cycle (5.8.0-1031.33-aws) this test passed.

[Kelsey Steele (kelsey-steele)]

for focal/aws-5.8 5.8.0-1032.34~20.04.1 in sru-20210412 I retriggered this on the two hosts I found the failure on (i3.metal and t3.medium) and set-timer-lat passed. Seems to be a flaky test

[Ian May (ian-may)]

Found on bionic/linux-aws-fips: 4.15.0-2057.59 - i3.metal, r5-metal

[Kleber Sacilotto de Souza (kleber-souza)]

Found with bionic/linux 4.15.0-166.174 node scobee-kernel (arm64).

[Bartlomiej Zolnierkiewicz (bzolnier)]

Also seen in bionic linux-ibm-gt 4.15.0-1117.128 for sru-20220418.

Looking at the past reports the failed clock test is sometimes "CLOCK_TAI ABSTIME PERIODIC max latency" and sometimes "CLOCK_TAI RELTIME PERIODIC max latency" (i.e. in this case or in case of bionic/linux-aws-fips: 4.15.0-2057.59 on i3.metal reported earlier).