rtc01 failed on some instances in kernel_misc from ubuntu_ltp for Bionic/azure-4.15 / 5.4 / 5.8

[Kelsey Steele]

rtc01 has failed on some instances for the prior Bionic/azure-4.15 : 4.15.0-1096.106 : amd64
 version as well.

47. 10/10 05:34:24 DEBUG| utils:0153| [stdout] startup='Sat Oct 10 05:12:49 2020'
48. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC READ TEST:
49. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 1 TPASS : RTC READ TEST Passed
50. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Current RTC date/time is 10-10-2020, 05:12:50.
51. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC ALARM TEST :
52. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Alarm time set to 05:12:55.
53. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Waiting 5 seconds for the alarm...
54. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Alarm rang.
55. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 2 TPASS : RTC ALARM TEST Passed
56. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC UPDATE INTERRUPTS TEST :
57. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Waiting for 5 update interrupts...
58. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Update interrupt 1
59. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Update interrupt 2
60. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Update interrupt 3
61. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Update interrupt 4
62. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 3 TFAIL : rtc01.c:209: Timed out waiting for the update interrupt
63. 10/10 05:34:24 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC Tests Done!
64. 10/10 05:34:24 DEBUG| utils:0153| [stdout] tag=rtc01 stime=1602306769 dur=12 exit=exited stat=1 core=no cu=0 cs=5

[Po-Hsu Lin (cypressyew)]

Spotted on G-5.8 Azure

[Po-Hsu Lin (cypressyew)]

Spotted on G-5.8 Azure with Standard_DS4_v2 (Standard_DS15_v2, Standard_DS5_v2, Standard_F2s_v2, Standard_F32s_v2)

[Po-Hsu Lin (cypressyew)]

Can be found on F-Azure 5.4.0-1032.33 - azure

[Po-Hsu Lin (cypressyew)]

Found on B-azure-5.4.0-1035.36~18.04.1 with instance Standard_DS5_v2

[Po-Hsu Lin (cypressyew)]

Found on F-aws 5.4.0-1037.39 c5.large

[Krzysztof Kozlowski (krzk)]

Focal/azure-5.8 5.8.0-1030.32~20.04.1

48. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC READ TEST:
49. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 1 TPASS : RTC READ TEST Passed
50. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Current RTC date/time is 24-4-2021, 18:22:53.
51. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC ALARM TEST :
52. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Alarm time set to 18:22:58.
53. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Waiting 5 seconds for the alarm...
54. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Alarm rang.
55. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 2 TPASS : RTC ALARM TEST Passed
56. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC UPDATE INTERRUPTS TEST :
57. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Waiting for 5 update interrupts...
58. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : Update interrupt 1
59. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 3 TFAIL : rtc01.c:208: Timed out waiting for the update interrupt
60. 04/24 18:36:00 DEBUG| utils:0153| [stdout] rtc01 0 TINFO : RTC Tests Done!

[Krzysztof Kozlowski (krzk)]

Found similar failure ("Timed out waiting for the alarm") on Azure instances:
 - impish/azure 5.11.0-1007.7+21.10.1-azure (Standard_F32s_v2, Standard_D48_v3)
 - focal/azure 5.8.0-1033.35~20.04.1-azure (Standard_D48_v3)

Reproduced (~33% times) also manually on:
 - hirsue/azure 5.11.0-1009-azure (B1ms, B4ms)
 - focal/azure 5.11.0-1007-azure (B4ms, B1ms)

rtc01 0 TINFO : RTC READ TEST:
rtc01 1 TPASS : RTC READ TEST Passed
rtc01 0 TINFO : Current RTC date/time is 11-6-2021, 09:00:58.
rtc01 0 TINFO : RTC ALARM TEST :
rtc01 0 TINFO : Alarm time set to 09:01:03.
rtc01 0 TINFO : Waiting 5 seconds for the alarm...
rtc01 2 TFAIL : rtc01.c:151: Timed out waiting for the alarm
rtc01 0 TINFO : RTC UPDATE INTERRUPTS TEST :
rtc01 0 TINFO : Waiting for 5 update interrupts...
rtc01 0 TINFO : Update interrupt 1
rtc01 0 TINFO : Update interrupt 2
rtc01 0 TINFO : Update interrupt 3
rtc01 0 TINFO : Update interrupt 4
rtc01 0 TINFO : Update interrupt 5
rtc01 3 TPASS : RTC UPDATE INTERRUPTS TEST Passed
rtc01 0 TINFO : RTC Tests Done!

[Krzysztof Kozlowski (krzk)]

Reproduced also on generic v5.11 kernels on Azure instances (B1ms, B4ms).

[Krzysztof Kozlowski (krzk)]

Found on bionic/linux-azure-5.4/5.4.0-1058.60~18.04.1, instance Standard_D4s_v3

[Krzysztof Kozlowski (krzk)]

The test fails only when alarm minutes are being increased. This will fail:
time: 14:50:57
alarm: 14:51:02

This will work:
time: 14:50:47
alarm: 14:50:52

[Krzysztof Kozlowski (krzk)]

Reproduced with rtcwake on Azyre node: Standard_D8d_v4, iff the time crosses to next minute:
sudo rtcwake -d rtc0 -m on -s 5 -v
with kernels:
H/5.11.0-1013-azure
H/5.11.0-34-generic
B/5.4.0-1056-azure
B/4.15.0-156-generic

If set to 59 or 60 seconds, the alarm succeeds.

AWS seems to work fine (F/5.4.0-1045-aws on t3a.2xlarge instance)

[Krzysztof Kozlowski (krzk)]

rtc01 fails on missed alarm on multiple different Azure instances if the
alarm is set for the next minute.

If alarm is set for now+60 seconds, it works fine. Clearly Microsoft
Hyper-V cloud instances have a broken CMOS RTC which unfortunately
cannot be easily fixed. Adding simple workaround to extend the time to
60 seconds allows to avoid false positives in expense of longer testing
time.

Not a Linux kernel issue. Not a LTP issue, although a workaround is proposed.

Patch sent upstream:
https://lists.linux.it/pipermail/ltp/2021-September/024989.html

[Ian May (ian-may)]

found on impish/linux-aws: 5.13.0-1008.9 a1.metal

[Po-Hsu Lin (cypressyew)]

In cycle 20220711, I can only see this issue with:
  * B-azure-4.15 (4.15.0-1147.162) with Standard_D8d_v4
  * F-azure-5.15 (5.15.0-1016.19~20.04.1) with Standard_F32s_v2