Message-ID: <Pine.GSO.4.63.0601011346350.13415@turing>
Date: Sun, 1 Jan 2006 14:22:25 -0800 (PST)
From: Nate Eldredge <email address hidden>
To: Henrique de Moraes Holschuh <email address hidden>
Cc: <email address hidden>, <email address hidden>,
	Nate Eldredge <email address hidden>, Ken Kennedy <email address hidden>,
	Jan De Luyck <email address hidden>
Subject: Re: hwclock must run after /usr

On Sat, 31 Dec 2005, Henrique de Moraes Holschuh wrote:

> Let's go over the way things work, and see how we can fix them back so that
> they work correctly.  Please bear with me while I go over the entire
> problem, and feel free to correct any mistakes I make.
>
> Reading manpage tzset(3) before you read any further is advised.
>
> AFAIK There are ONLY TWO valid modes of early boot operation:
>
>  1. UTC (hardware clock in UTC, UTC=yes)
>
>  2. Non-UTC with TZ set or /etc/localtime in the root filesystem
>     (hardware clock in local time, UTC=no, system knows timezone)
>
> Anything else is an invalid system configuration, and must be fixed.  By
> invalid configuration, I mean one where the system UTC clock (which has
> nothing to do with the hardware clock) is NOT set to the correct UTC time.

I think this is right.

> I would bet it is common to find people with invalid configurations, since a
> separate /usr is common, TZ is not mentioned at the top of
> /etc/init.d/hwclockfirst.sh, and there isn't a way for d-i to set that up
> either.
>
> Also, do refer to tzset(3), we would need to use the simplest TZ form, since
> it must work without data that is in /usr/share.

Right.  Which is kind of painful in the daylight savings case.  I can't 
just say US/Pacific, or even PST -0800, I have to look up the exact 
definition of when daylight savings changes and figure out the TZ syntax 
to specify that.  I don't know, but there might be places where the time 
zone is even more complicated and can't be fully described in the TZ 
variable.  In which you can't use this approach and be correct always.

> It is good to remember that timezones can be changed at will, by any user
> (just set TZ) at any time, and also that TZ overrides /etc/localtime so we
> don't want to define it system-wide.
>
> What that means?  Below are the two valid configurations and what actions
> hwclock initscripts should do (doing anything else is probably a big bad
> bug ;-) ), and the most common invalid configuration.
>
> (*) are the points where hwclock scripts must take some action.
>
> UTC mode:
>  0. Kernel might boot with correct time (if it reads the RTC by itself)
> * 1. hwclockfirst sets kernel time to the correct UTC time
>  2. system runs with a local timezone (if /etc/localtime is readable)
>     or with UTC as a timezone
>  3. something mounts /usr
>  4. System runs with the proper timezone
>
>  Early boot runs fine, system thinks timezone is UTC if /usr is not
>  mounted, or reads timezone from /etc/localtime if it is mounted.
>  No big deal, as apps use UTC for persitent timekeeping.  e2fsck
>  should not be going bonkers in this case.
>
>  hwclock should try to set the system clock only on early boot.

Yes.

> Non-UTC valid config mode, with /usr as a separate partition
>  0. kernel boots with wrong time (even if it reads the RTC by itself)
> * 1. TZ is set so hwclockfirst sets the kernel time to the correct UTC time
>  2. system runs with a local timezone (if /etc/localtime is readable)
>     or with UTC as a timezone
>  3. Something mounts /usr
>  4. System runs with the proper timezone
>
>  Again, hwclock should try to set the system clock only on early boot.
>  Notice that TZ must be set only for the hwclockfirst initscript, we don't
>  want it in the global environment.  I'd suggest removing UTC from
>  /etc/default/rcS, and adding TZ and UCT to a new file,
>  /etc/default/timezone.

Yes.  Note that setting TZ is rather inconvenient.  First because you have 
to figure out the format, as I mentioned above, and also now if you ever 
want to change your timezone you have to change it in two places 
(/etc/default and the /etc/localtime symlink) which are hard to find.  I 
forget which package has the config script where you select your timezone, 
but I guess it could in principle apply it in both places.  But that's 
kind of ugly, and anyway the Debian philospohy is not to force people into 
using config scripts.

> Non-UTC *valid config* mode, /usr inside the root partition
>  0. kernel boots with wrong time (even if it reads the RTC by itself)
>  1. /etc/localtime is readable, so everything else runs just like it
>     does in UTC mode.

i.e. hwclockfirst sets the time to correct UTC.  Provided UTC=NO.

> With a valid configuration, hwclockfirst does all the job that doesn't need
> a timezone and must run VERY early.  hwclock does the jobs that require a
> timezone (which is actually just displaying the time in the local timezone
> so that the users have a better clue of what's happening, and we don't
> increase d-user traffic with RTC problems again).
>
> This means hwclock.sh *must* run after /usr is mounted.  That's the only
> time we *know* the timezone will be correct, no matter which configuration
> is used (including invalid ones -- hwclock after /usr is a *debugging aid*).
>
>
> What would happen in the typical broken configuration?
>
> Non-UTC, /usr separate, no TZ defined anywhere:
>  0. kernel boots with wrong time (even if it reads the RTC by itself)
> * 1. hwclock sets the kernel UTC time to the local time in the RTC,
>     and thus the system clock could be wrong by several hours in either
>     direction.
>  2. System runs with wrong time. e2fsck can croak, etc.
>  3. something mounts /usr
> * 4. hwclock runs, and tells the user a completely ludricous time
>     (timezone applied twice :P) <=== We could fix the mess here, so
>     if the user doesn't hit a bug (e.g. e2fsck) before, he would not
>     suffer much from his broken config.
>     hwclock MUST be run after /usr is mounted to be able to help fix
>     the mess (well work around it, realy).

I don't have my unstable box handy to check right now, but it gets fixed 
eventually.  I think when hwclock runs after /usr is mounted, it sets the 
time correctly.  So it's only wrong before that point.  I used to have 
this broken config, and until e2fsck was changed to start complaining, I 
never noticed a problem.  (Actually, I think there was something funny 
with computing module dependencies, but I worked around that somehow.)

> Now, there's a bug in that hwclockfirst should bitch to all winds if it
> detects an invalid config, currently it just outputs a meek "System clock
> was not updated at this time" and bangs out with exit status 1.

Agreed.  I always assumed that message was innocuous.

> There is also a bug in that hwclockfirst must be one of the VERY first
> things to be run, AND that hwclock must be run AFTER /etc/localtime is
> working well in a typical Debian system.  I.e. *after* /usr is mounted.
>
> And there is a third bug which is the lack of easy TZ handling and lack of
> documentation.  The funny thing is that I could swear I added TZ and some
> documentation to hwclockfirst, but that script has been extensively
> modified since then.
>
> After hwclock has enough infrastructure for it (i.e. /etc/default/timezone
> or somesuch), d-i should be enhanced to set THAT if it is going to set
> UTC=no.

You know, another solution would be to relocate the whole 
/usr/share/zoneinfo hierarchy to the root filesystem.  I don't really like 
it, but it seems that we really may need all  that information early in 
the boot process.  Setting TZ isn't as good.  OTOH, it's a significant 
change just to benefit those "weird" people who have their hw clocks set 
to local time, which isn't really the Unix way.  (I only did it because I 
was dual-booting Windows, which insisted on having the hw clock in local 
time.)

>
> I might submit a patch soon, it would have been faster than documenting all
> of the above...

Thanks for looking into this!

-- 
Nate Eldredge
<email address hidden>