improve timesyncd epoch on uc20

Bug #1878422 reported by Ian Johnson on 2020-05-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
snapd
Medium
Ian Johnson
ubuntu-core-initramfs
Undecided
Unassigned

Bug Description

When booting in install mode, and timesyncd clock stamp file is missing we should initialize it to at least as old as the image build.

Ideally from .disk/info file which we don't have at the moment.

When systemd is updated in SRUs we should bump the time-epoch in a reproducible way.

When booting into recover mode, the clock stamp file should be copied up from runmode into recover mode.

Dimitri John Ledkov (xnox) wrote :

No, we do not need fixrtc script.

kernel cmdline must _not_ specify 'fixrtc'

systemd adjusts hardware/kernel clock to be at least when systemd was last released (ie. sometime in 2020). It does that very early during init. And since initrd's init is systemd, the equivalent functionality is already there.

Do you have any symptoms that indicate that above is not working? I.e. do you see system clock coming up as before than 2020? any tls / gpg verification failures (i.e. errors like CA certificate expired, or start date in the future)

Changed in ubuntu-core-initramfs:
status: New → Incomplete
Dimitri John Ledkov (xnox) wrote :

(ian clarified on irc)

So fixrt-mount did two things:

1) set time to a fixed "good enough date" in 2018
2) try to get a timestamp of the snaps folder, and set the time to that if newer

systemd does 1) already by setting time to what it was in 2020.

but it doesn't do 2) in any way at the moment.

Changed in ubuntu-core-initramfs:
status: Incomplete → New
Dimitri John Ledkov (xnox) wrote :

snapsfolder=/root/writable/system-data/var/lib/snapd/snaps is what is checked for a "newer" time.

I wonder what systemd uses to store/restore more dynamic timestamps already. And what we might want to use in UC20 instead.

As if we want to try ubuntu-seed/snaps ubuntu-boot/somthing ubuntu-data/..../snaps

Dimitri John Ledkov (xnox) wrote :

In the running system, timesyncd updates /var/lib/systemd/timesync/clock and that is "more recent" time.

So i think without clock & network. I believe booting, running, rebooting => the second boot should be "ahead of time" and not go back in time.

Maybe we want to copy /var/lib/systemd/timesync/clock from data to recover environment.

Dimitri John Ledkov (xnox) wrote :

        /* Let's try to make sure that the clock is always
         * monotonically increasing, by saving the clock whenever we
         * have a new NTP time, or when we shut down, and restoring it
         * when we start again. This is particularly helpful on
         * systems lacking a battery backed RTC. We also will adjust
         * the time to at least the build time of systemd. */

So i think the "epoch" of when systemd is built + the stored /var/lib/systemd/timesync/clock achieves the equivalent functionality of increasing time between boots. And ensuring the very first boot is "recentish".

Oliver Grawert (ogra) wrote :

fixrtc is originally in ubuntus upstream initramfs-tools, it took us quite some work to get it there and foundations was the team asking us to put it there because "all initrd's in ubuntu have to be based on initramfs-tools" so that broken(non-existing RTCs are covered everywhere ...

Do we not use initramfs-tools as a base anymore in the core20 initrds ?

Also note that fixrtc looks at the filesystem creation time or last mount time in the superblock of the found rootfs, while the additional fixrtc-mount script is a snapd specific add-onn in the initramfs-tools-ubuntu-core package (which is only used in core initrds) ...

I doubt /var/lib/systemd/timesync/clock is of any help with "last mount time of superblock is in the future" rootfs mounting errors (and their fallout, that on some boards ends up in unbootable systems). Unless you plan to replace /init of the initrd with systemd now ...

Where does the var/lib/systemd/timesync/clock file get its initial time from on i.e. boards that have never (and probably will never) been connected to a network (we have such customers) ?

If the clock is too far off any gpg calls will simply fail during first boot
(IIRC snapd nowadays has a hardcoded hack that makes it ignore gpg timestamps during seeding, but application snaps that use gpg internally will not have this) ...
If the clock is too far off in the other direction you easily hit issues with key expiration dates ...

Note I'm not opposed to get rid of fixrtc, it has always been a hack, but it is a reliable and well tested mechanism that is being used on arm devices since nearly a decade and I'd like us to avoid situations like we had in core18 ... where *all* potential hacks blindly got removed without any replacements and it took us 6-12 months to even get the basic regressions fixed again (like keeping a persistent hostname over reboots etc) ...

Michael Vogt (mvo) wrote :

Based on what Dimitri wrote I mark this as invalid, please reopen if I misunderstood.

Changed in ubuntu-core-initramfs:
status: New → Invalid

Sorry Michael, I'm pretty sure that we do in fact need the ubuntu core specific "fixtrc-mount" script in the uc20 initramfs, or rather something like it, as w/o this script I still see my Pi skipping back in time a couple of months to Feb which causes problems with getting a network connection AFAICT.

To be clear, this bug is about the fixrtc-mount script, not the generic fixrtc script which it seems systemd is already implementing the functionality that fixrtc provided.

summary: - ubuntu-core-initramfs is missing the fixrtc script from uc16/uc18
+ ubuntu-core-initramfs is missing the fixrtc-mount script from uc16/uc18
description: updated
Changed in ubuntu-core-initramfs:
status: Invalid → New
Ian Johnson (anonymouse67) wrote :

As an example, on my pi, the initrd logs have the date as something from Nov 2019, then systemd detects this when it runs as /init after the initrd, and sets the time forward to when systemd was built to Feb 2020, which is better but still not good enough.

If I install a snap, then the mtime of /var/lib/snapd/ is updated to when I installed something, so I think that proves that if we had the initramfs change the time of the system to the mtime of /run/mnt/host/var/lib/snapd/ in run mode we would be much much closer.

I think actually we could try doing this in "the-tool" and eventually push this into snapd where it has better visibility for snapd early boot sequence hacks/tricks. I will try this out locally to see if it helps some of the issue I have been seeing with my pi booting and not being able to connect to network.

Ian Johnson (anonymouse67) wrote :

To be clear though, for my first boot pi problems not getting a network connection I have filed https://bugs.launchpad.net/subiquity/+bug/1878640 in case it is not a time issue and is actually a console-conf/general uc20 issue.

Changed in ubuntu-core-initramfs:
assignee: nobody → Ian Johnson (anonymouse67)
Dimitri John Ledkov (xnox) wrote :

@ian updated the bug title and description to match what we have discussed today about all the things we can do better to improve time epoch handling.

@ogra UC20 does not use legacy initramfs-tools-ubuntu-core based initrd, it uses the use ubuntu-core-initramfs which is systemd & snap-bootstrap based.

summary: - ubuntu-core-initramfs is missing the fixrtc-mount script from uc16/uc18
+ improve timesyncd epoch on uc20
description: updated
Changed in ubuntu-core-initramfs:
assignee: Ian Johnson (anonymouse67) → nobody
Dimitri John Ledkov (xnox) wrote :

"it uses the use" => "it uses the new"

Changed in snapd:
assignee: nobody → Ian Johnson (anonymouse67)
Ian Johnson (anonymouse67) wrote :

The snapd task for recover mode is up here https://github.com/snapcore/snapd/pull/8795

Changed in snapd:
status: New → In Progress
importance: Undecided → Medium
Ian Johnson (anonymouse67) wrote :

This will be Fix Released in 2.46

Changed in snapd:
status: In Progress → Fix Committed
Zygmunt Krynicki (zyga) on 2020-06-23
Changed in snapd:
milestone: none → 2.46
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers