improve timesyncd epoch on uc20

[Ian Johnson]

When booting in install mode, and timesyncd clock stamp file is missing we should initialize it to at least as old as the image build.

Ideally from .disk/info file which we don't have at the moment.

When systemd is updated in SRUs we should bump the time-epoch in a reproducible way.

When booting into recover mode, the clock stamp file should be copied up from runmode into recover mode.

[Dimitri John Ledkov (xnox)]

No, we do not need fixrtc script.

kernel cmdline must _not_ specify 'fixrtc'

systemd adjusts hardware/kernel clock to be at least when systemd was last released (ie. sometime in 2020). It does that very early during init. And since initrd's init is systemd, the equivalent functionality is already there.

Do you have any symptoms that indicate that above is not working? I.e. do you see system clock coming up as before than 2020? any tls / gpg verification failures (i.e. errors like CA certificate expired, or start date in the future)

[Dimitri John Ledkov (xnox)]

(ian clarified on irc)

So fixrt-mount did two things:

1) set time to a fixed "good enough date" in 2018
2) try to get a timestamp of the snaps folder, and set the time to that if newer

systemd does 1) already by setting time to what it was in 2020.

but it doesn't do 2) in any way at the moment.

[Dimitri John Ledkov (xnox)]

snapsfolder=/root/writable/system-data/var/lib/snapd/snaps is what is checked for a "newer" time.

I wonder what systemd uses to store/restore more dynamic timestamps already. And what we might want to use in UC20 instead.

As if we want to try ubuntu-seed/snaps ubuntu-boot/somthing ubuntu-data/..../snaps

[Dimitri John Ledkov (xnox)]

In the running system, timesyncd updates /var/lib/systemd/timesync/clock and that is "more recent" time.

So i think without clock & network. I believe booting, running, rebooting => the second boot should be "ahead of time" and not go back in time.

Maybe we want to copy /var/lib/systemd/timesync/clock from data to recover environment.

[Dimitri John Ledkov (xnox)]

        /* Let's try to make sure that the clock is always
         * monotonically increasing, by saving the clock whenever we
         * have a new NTP time, or when we shut down, and restoring it
         * when we start again. This is particularly helpful on
         * systems lacking a battery backed RTC. We also will adjust
         * the time to at least the build time of systemd. */

So i think the "epoch" of when systemd is built + the stored /var/lib/systemd/timesync/clock achieves the equivalent functionality of increasing time between boots. And ensuring the very first boot is "recentish".

[Oliver Grawert (ogra)]

fixrtc is originally in ubuntus upstream initramfs-tools, it took us quite some work to get it there and foundations was the team asking us to put it there because "all initrd's in ubuntu have to be based on initramfs-tools" so that broken(non-existing RTCs are covered everywhere ...

Do we not use initramfs-tools as a base anymore in the core20 initrds ?

Also note that fixrtc looks at the filesystem creation time or last mount time in the superblock of the found rootfs, while the additional fixrtc-mount script is a snapd specific add-onn in the initramfs-tools-ubuntu-core package (which is only used in core initrds) ...

I doubt /var/lib/systemd/timesync/clock is of any help with "last mount time of superblock is in the future" rootfs mounting errors (and their fallout, that on some boards ends up in unbootable systems). Unless you plan to replace /init of the initrd with systemd now ...

Where does the var/lib/systemd/timesync/clock file get its initial time from on i.e. boards that have never (and probably will never) been connected to a network (we have such customers) ?

If the clock is too far off any gpg calls will simply fail during first boot
(IIRC snapd nowadays has a hardcoded hack that makes it ignore gpg timestamps during seeding, but application snaps that use gpg internally will not have this) ...
If the clock is too far off in the other direction you easily hit issues with key expiration dates ...

Note I'm not opposed to get rid of fixrtc, it has always been a hack, but it is a reliable and well tested mechanism that is being used on arm devices since nearly a decade and I'd like us to avoid situations like we had in core18 ... where *all* potential hacks blindly got removed without any replacements and it took us 6-12 months to even get the basic regressions fixed again (like keeping a persistent hostname over reboots etc) ...

[Michael Vogt (mvo)]

Based on what Dimitri wrote I mark this as invalid, please reopen if I misunderstood.

[Ian Johnson (anonymouse67)]

Sorry Michael, I'm pretty sure that we do in fact need the ubuntu core specific "fixtrc-mount" script in the uc20 initramfs, or rather something like it, as w/o this script I still see my Pi skipping back in time a couple of months to Feb which causes problems with getting a network connection AFAICT.

To be clear, this bug is about the fixrtc-mount script, not the generic fixrtc script which it seems systemd is already implementing the functionality that fixrtc provided.

[Ian Johnson (anonymouse67)]

As an example, on my pi, the initrd logs have the date as something from Nov 2019, then systemd detects this when it runs as /init after the initrd, and sets the time forward to when systemd was built to Feb 2020, which is better but still not good enough.

If I install a snap, then the mtime of /var/lib/snapd/ is updated to when I installed something, so I think that proves that if we had the initramfs change the time of the system to the mtime of /run/mnt/host/var/lib/snapd/ in run mode we would be much much closer.

I think actually we could try doing this in "the-tool" and eventually push this into snapd where it has better visibility for snapd early boot sequence hacks/tricks. I will try this out locally to see if it helps some of the issue I have been seeing with my pi booting and not being able to connect to network.

[Ian Johnson (anonymouse67)]

To be clear though, for my first boot pi problems not getting a network connection I have filed https://bugs.launchpad.net/subiquity/+bug/1878640 in case it is not a time issue and is actually a console-conf/general uc20 issue.

[Dimitri John Ledkov (xnox)]

@ian updated the bug title and description to match what we have discussed today about all the things we can do better to improve time epoch handling.

@ogra UC20 does not use legacy initramfs-tools-ubuntu-core based initrd, it uses the use ubuntu-core-initramfs which is systemd & snap-bootstrap based.

[Dimitri John Ledkov (xnox)]

"it uses the use" => "it uses the new"

[Ian Johnson (anonymouse67)]

The snapd task for recover mode is up here https://github.com/snapcore/snapd/pull/8795

[Ian Johnson (anonymouse67)]

This will be Fix Released in 2.46

[Ian Johnson (anonymouse67)]

This has come up again with ARM boards that cannot finish booting out of the initramfs because they have no RTC and so upon booting in the initramfs their time simply gets forwarded to Nov 2019 (the epochd of systemd), but if the model assertion and the account-key used to create the image are both newer than Nov 2019 then the device fails to boot, resulting in something like:

```
[ 15.377377] the-tool[176]: error: cannot load metadata and verify essential bootstrap snaps [base kernel snapd gadget]: cannot accept some assertions:
[ 15.400931] [FAILED] Failed to start the-tool.service.
See 'systemctl status the-tool.service' for details.
the-tool[176]: - assertion is signed with expired public key "PYn7GbIbTMPJk9SyM73DfKfzTXRfH0wPqSe3h7ixlKN3-Lw25rSO2_tZ3WVamWux" from "PcQpbi62D9Fvr3TBxaS5lf3M9650ceGr"
[ 15.436979] systemd[1]: the-tool.service: Main process exited, code=exited, status=1/FAILURE
[ 15.453782] systemd[1]: the-tool.service: Failed with result 'exit-code'.
[ 15.469151] systemd[1]: Failed to start the-tool.service.
```

I think the error message from snap-bootstrap here is a bit confusing it says "expired" when more accurately it is "not yet valid".

I think as a start, we should SRU to focal this bug: https://bugs.launchpad.net/ubuntu-core-initramfs/+bug/1878969 and then also work on adding an equivalent fixrtc-mount script to the initramfs somehow/somewhere that uses additional, trusted information about what time to move forward to using something in ubuntu-seed.

[Dimitri John Ledkov (xnox)]

initrd cannot trust anything that cannot be verified.

snap assertions however are signed and verified... and contain lots of "timestamp: " and "since: " fields.

In the initrd, snap-bootstrap initramfs-mounts should have a mode to set system time, as it is loading and verifying the assertions.

Such that the latest verified time seen, is good enough epoch time for the initrd.

something along the lines of

if ! /usr/lib/snapd/snap-bootstrap initramfs-mounts; then
 . /proc/cmdline
 date -u -D%Y-%m-%dT%H:%M:%S "$(sed -n 's/timestamp: //p;s/since: //p' /run/mnt/ubuntu-seed/systems/"$snapd_recovery_system"/assertions/* | sort | tail -n1)"
 /usr/lib/snapd/snap-bootstrap initramfs-mounts
fi

[Dimitri John Ledkov (xnox)]

Maybe it will need to be a new subcommand / function.

I.e. snap-bootstrap initramfs-set-time-epoch

[Ian Johnson (anonymouse67)]

We do not want to add additional commands to snap-bootstrap, if we need to make snap-bootstrap do additional things, we will just make the initramfs-mounts command do them at an appropriate time.

Dimitri what do you think of the proposal to add a file to the initrd when we build it (and thus before we sign it with the trusted canonical keys), and make something in the initrd read that file and move time forward to that timestamp?

[Ian Johnson (anonymouse67)]

I have a proposal here which moves time forward to at least that of a timestamp file in the initrd (if that timestamp file is not older than what the kernel thinks the time is, so devices with working RTC's don't move backwards in time), and then also moves the time even further ahead to the mtime of the systems dir of ubuntu-seed if we were able to verify the seed and we also moved the time forward using the initrd timestamp file.

We need a bit of coordination here to approve this approach, Dimitri what do you think of the above?

See PR https://github.com/snapcore/snapd/pull/9974

[Dimitri John Ledkov (xnox)]

clock-epoch landed in core-initrd.

I still want snap-bootstrap provide authenticated timestamps out of the assertion stream. But clock-epoch will do for now.