Let's go over the way things work, and see how we can fix them back so that
they work correctly.  Please bear with me while I go over the entire
problem, and feel free to correct any mistakes I make.

Reading manpage tzset(3) before you read any further is advised.

AFAIK There are ONLY TWO valid modes of early boot operation:

  1. UTC (hardware clock in UTC, UTC=yes)
  
  2. Non-UTC with TZ set or /etc/localtime in the root filesystem
     (hardware clock in local time, UTC=no, system knows timezone)

Anything else is an invalid system configuration, and must be fixed.  By
invalid configuration, I mean one where the system UTC clock (which has
nothing to do with the hardware clock) is NOT set to the correct UTC time.

I would bet it is common to find people with invalid configurations, since a
separate /usr is common, TZ is not mentioned at the top of
/etc/init.d/hwclockfirst.sh, and there isn't a way for d-i to set that up
either.

Also, do refer to tzset(3), we would need to use the simplest TZ form, since
it must work without data that is in /usr/share.

It is good to remember that timezones can be changed at will, by any user
(just set TZ) at any time, and also that TZ overrides /etc/localtime so we
don't want to define it system-wide.

What that means?  Below are the two valid configurations and what actions
hwclock initscripts should do (doing anything else is probably a big bad
bug ;-) ), and the most common invalid configuration.

(*) are the points where hwclock scripts must take some action.

UTC mode:
  0. Kernel might boot with correct time (if it reads the RTC by itself)
* 1. hwclockfirst sets kernel time to the correct UTC time
  2. system runs with a local timezone (if /etc/localtime is readable)
     or with UTC as a timezone
  3. something mounts /usr
  4. System runs with the proper timezone
  
  Early boot runs fine, system thinks timezone is UTC if /usr is not
  mounted, or reads timezone from /etc/localtime if it is mounted.
  No big deal, as apps use UTC for persitent timekeeping.  e2fsck
  should not be going bonkers in this case.

  hwclock should try to set the system clock only on early boot.

Non-UTC valid config mode, with /usr as a separate partition
  0. kernel boots with wrong time (even if it reads the RTC by itself)
* 1. TZ is set so hwclockfirst sets the kernel time to the correct UTC time
  2. system runs with a local timezone (if /etc/localtime is readable)
     or with UTC as a timezone
  3. Something mounts /usr
  4. System runs with the proper timezone

  Again, hwclock should try to set the system clock only on early boot.
  Notice that TZ must be set only for the hwclockfirst initscript, we don't
  want it in the global environment.  I'd suggest removing UTC from
  /etc/default/rcS, and adding TZ and UCT to a new file,
  /etc/default/timezone.
  
Non-UTC *valid config* mode, /usr inside the root partition
  0. kernel boots with wrong time (even if it reads the RTC by itself)
  1. /etc/localtime is readable, so everything else runs just like it
     does in UTC mode.

With a valid configuration, hwclockfirst does all the job that doesn't need
a timezone and must run VERY early.  hwclock does the jobs that require a
timezone (which is actually just displaying the time in the local timezone
so that the users have a better clue of what's happening, and we don't
increase d-user traffic with RTC problems again).

This means hwclock.sh *must* run after /usr is mounted.  That's the only
time we *know* the timezone will be correct, no matter which configuration
is used (including invalid ones -- hwclock after /usr is a *debugging aid*).


What would happen in the typical broken configuration?

Non-UTC, /usr separate, no TZ defined anywhere:
  0. kernel boots with wrong time (even if it reads the RTC by itself)
* 1. hwclock sets the kernel UTC time to the local time in the RTC,
     and thus the system clock could be wrong by several hours in either
     direction.
  2. System runs with wrong time. e2fsck can croak, etc.
  3. something mounts /usr
* 4. hwclock runs, and tells the user a completely ludricous time
     (timezone applied twice :P) <=== We could fix the mess here, so
     if the user doesn't hit a bug (e.g. e2fsck) before, he would not
     suffer much from his broken config.

     hwclock MUST be run after /usr is mounted to be able to help fix
     the mess (well work around it, realy).

Now, there's a bug in that hwclockfirst should bitch to all winds if it
detects an invalid config, currently it just outputs a meek "System clock
was not updated at this time" and bangs out with exit status 1.

There is also a bug in that hwclockfirst must be one of the VERY first
things to be run, AND that hwclock must be run AFTER /etc/localtime is
working well in a typical Debian system.  I.e. *after* /usr is mounted.

And there is a third bug which is the lack of easy TZ handling and lack of
documentation.  The funny thing is that I could swear I added TZ and some
documentation to hwclockfirst, but that script has been extensively
modified since then.

After hwclock has enough infrastructure for it (i.e. /etc/default/timezone
or somesuch), d-i should be enhanced to set THAT if it is going to set
UTC=no.

I might submit a patch soon, it would have been faster than documenting all
of the above...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh