hctosys not reading hardware clock on CM4

[Oliver Grawert]

[Impact]

At boot, the kernel does not synchronize the system clock from an RTC, if the driver that manages the RTC device is built as a module.

[Test Case]

$ dmesg|grep rtc
[ 1.593561] hctosys: unable to open rtc device (rtc0)
[ 10.767256] rtc-pcf85063 10-0051: registered as rtc0

[Where Problems Could Occur]

Loading of the relevant RTC module (which then triggers a clock sync from the RTC) is handled by userspace. This could interfere with systemd (or some other entity) hat handles clock sync'ing which could yield unexpected results.

[Original Description]

There is a timing issue when using the CM4 with the official CM4 IO baseboard.

The CM4 baseboard comes with a built in rtc and a battery holder by default, to enable it a devicetree overlay needs to be enabled and a matching module gets loaded ...

$ tail -2 /run/mnt/ubuntu-seed/config.txt
dtparam=i2c_vc=on
dtoverlay=i2c-rtc,pcf85063a,i2c_csi_dsi,addr=0x51

$ lsmod | grep pcf85063
rtc_pcf85063 20480 0

When booting the kernel runs hctosys about 1.5sec into the boot, the rtc module only gets loaded about 10sec later though:

$ dmesg|grep rtc
[ 1.593561] hctosys: unable to open rtc device (rtc0)
[ 10.767256] rtc-pcf85063 10-0051: registered as rtc0

Looking at the systemd journal the clock does not get set at all, only a network connection actually triggers setting of the clock (note the timestamps in the journal):

---
Apr 26 10:27:53 ubuntu kernel: mmc0: SDHCI controller on fe340000.emmc2 [fe340000.emmc2] using ADMA
Apr 26 10:27:53 ubuntu kernel: hctosys: unable to open rtc device (rtc0)
Apr 26 10:27:53 ubuntu kernel: of_cfs_init
Apr 26 10:27:53 ubuntu kernel: of_cfs_init: OK
...
Apr 26 10:28:02 CM4 systemd[1]: Started Network Name Resolution.
Apr 26 10:28:02 CM4 kernel: rtc-pcf85063 10-0051: registered as rtc0
Apr 26 10:28:02 CM4 systemd-udevd[459]: Using default interface naming scheme 'v245'.
...
Apr 26 10:28:08 CM4 avahi.daemon[593]: Registering new address record for 192.168.2.32 on wlan0.IPv4.
Apr 26 10:28:08 CM4 systemd-timesyncd[495]: Network configuration changed, trying to establish connection.
May 02 17:14:17 CM4 systemd-timesyncd[495]: Initial synchronization to time server 91.189.89.199:123 (ntp.ubu>
May 02 17:14:17 CM4 systemd[1]: Starting Online ext4 Metadata Check for All Filesystems...
---

I think the loading of the rtc_pcf85063 module should trigger an additional hctosys call (preferably from the module itself, but worst case a udev rule calling out to hwclock --hctosys might work too)...

Just for the record, the hwclock works fine otherwise:

$ sudo hwclock
2021-05-02 19:56:11.281975+00:00

[Juerg Haefliger (juergh)]

Which kernel?

[Oliver Grawert (ogra)]

ogra@CM4:~$ snap list pi-kernel
Name Version Rev Tracking Publisher Notes
pi-kernel 5.4.0-1035.38 282 20/edge canonical✓ kernel

[Juerg Haefliger (juergh)]

Seem focal:
$ uname -a
Linux rpi-cm4-rev1d0-e428 5.4.0-1034-raspi #37-Ubuntu SMP PREEMPT Mon Apr 12 23:14:49 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
$ sudo dmesg | grep rtc
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DC:A6:32:D6:C8:D1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 net.ifnames=0 dwc_otg.lpm_enable=0 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc console=tty1 console=ttyS0,115200 loglevel=8 quiet splash
[ 1.974818] hctosys: unable to open rtc device (rtc0)
[ 3.658840] rtc-pcf85063 10-0051: registered as rtc0

[Juerg Haefliger (juergh)]

Hirsute looks fine:
$ uname -a
Linux rpi-cm4-rev1d0-e428 5.11.0-1005-raspi #5 SMP PREEMPT Mon Apr 19 09:04:31 UTC 2021 armv7l armv7l armv7l GNU/Linux

$ sudo dmesg | grep rtc
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DC:A6:32:D6:C8:D1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 net.ifnames=0 dwc_otg.lpm_enable=0 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc splash console=tty1 console=ttyS0,115200 loglevel=8
[ 3.696219] fixrtc
[ 5.599830] rtc-pcf85063 10-0051: registered as rtc0
[ 5.606145] rtc-pcf85063 10-0051: setting system clock to 2021-05-03T09:25:07 UTC (1620033907)

[Oliver Grawert (ogra)]

hirsute does not have Ubuntu Core images ;)

It seems to be solvable via a simple defconfig change according to:

https://github.com/raspberrypi/linux/issues/4205

We have at least one UC customer implementing a CM4 baseboard with TPM for Full disk encryption and secureboot in UC20, for this the clock needs to be correct. The module also needs to move into the UC initrd so it can load before mounting disks, i added an ubuntu-core-initramfs task for this ...

[Oliver Grawert (ogra)]

https://github.com/snapcore/core-initrd/issues/17

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-raspi - 5.4.0-1036.39

---------------
linux-raspi (5.4.0-1036.39) focal; urgency=medium

  * focal/linux-raspi: 5.4.0-1036.39 -proposed tracker (LP: #1927614)

  * Raspberry Pi kernel config doesn't match generic for NFT, breaks LXD
    (LP: #1927108)
    - [Config] raspi: SET NFT_FIB_INET=m

  * hctosys not reading hardware clock on CM4 (LP: #1926911)
    - rtc: class: support hctosys from modular RTC drivers

  * Support Auvidea B101 (TC358743) (LP: #1926755)
    - [Config] raspi: Set VIDEO_TC358743=m

  * No iwilwifi driver in Linux 5.4.0-1034-raspi (LP: #1925209)
    - [Config] raspi: Set IWLWIFI=m

  [ Ubuntu: 5.4.0-74.83 ]

  * focal/linux: 5.4.0-74.83 -proposed tracker (LP: #1927619)
  * Introduce the 465 driver series, fabric-manager, and libnvidia-nscq
    (LP: #1925522)
    - debian/dkms-versions -- add NVIDIA 465 and migrate 450 to 460
  * linux-image-5.0.0-35-generic breaks checkpointing of container
    (LP: #1857257)
    - SAUCE: overlayfs: fix incorrect mnt_id of files opened from map_files
  * Enable CIFS GCM256 (LP: #1921916)
    - smb3: add defines for new crypto algorithms
    - smb3.1.1: add new module load parm require_gcm_256
    - smb3.1.1: add new module load parm enable_gcm_256
    - smb3.1.1: print warning if server does not support requested encryption type
    - smb3.1.1: rename nonces used for GCM and CCM encryption
    - smb3.1.1: set gcm256 when requested
    - cifs: Adjust key sizes and key generation routines for AES256 encryption
  * locking/qrwlock: Fix ordering in queued_write_lock_slowpath() (LP: #1926184)
    - locking/qrwlock: Fix ordering in queued_write_lock_slowpath()
  * [Ubuntu 21.04] net/mlx5: Fix HW spec violation configuring uplink
    (LP: #1925452)
    - net/mlx5: Fix HW spec violation configuring uplink
  * Focal update: v5.4.114 upstream stable release (LP: #1926493)
    - Revert "scsi: qla2xxx: Retry PLOGI on FC-NVMe PRLI failure"
    - Revert "scsi: qla2xxx: Fix stuck login session using prli_pend_timer"
    - scsi: qla2xxx: Dual FCP-NVMe target port support
    - scsi: qla2xxx: Fix device connect issues in P2P configuration
    - scsi: qla2xxx: Retry PLOGI on FC-NVMe PRLI failure
    - scsi: qla2xxx: Add a shadow variable to hold disc_state history of fcport
    - scsi: qla2xxx: Fix stuck login session using prli_pend_timer
    - scsi: qla2xxx: Fix fabric scan hang
    - net/sctp: fix race condition in sctp_destroy_sock
    - Input: nspire-keypad - enable interrupts only when opened
    - gpio: sysfs: Obey valid_mask
    - dmaengine: dw: Make it dependent to HAS_IOMEM
    - ARM: dts: Drop duplicate sha2md5_fck to fix clk_disable race
    - ARM: dts: Fix moving mmc devices with aliases for omap4 & 5
    - lockdep: Add a missing initialization hint to the "INFO: Trying to register
      non-static key" message
    - arc: kernel: Return -EFAULT if copy_to_user() fails
    - ASoC: max98373: Added 30ms turn on/off time delay
    - neighbour: Disregard DEAD dst in neigh_update
    - ARM: keystone: fix integer overflow warning
    - ARM: omap1: fix building with clang IAS
    - drm/msm: Fix a5xx/a6xx timestamps
    - ASoC: fsl_esai: Fix TDM slot setup fo...