Activity log for bug #978127

Date Who What changed Old value New value Message
2012-04-10 15:36:05 Scott Moser bug added bug
2012-04-10 15:36:05 Scott Moser attachment added screenshot of system https://bugs.launchpad.net/bugs/978127/+attachment/3040935/+files/out.jpg
2012-04-11 00:59:30 Julian Edwards maas: status Confirmed Triaged
2012-04-11 00:59:32 Julian Edwards maas: importance Undecided High
2012-04-11 01:03:59 Julian Edwards bug added subscriber Julian Edwards
2012-04-18 06:40:26 Julian Edwards tags api provisioning
2012-05-04 18:14:33 LaMont Jones bug added subscriber The Canonical Sysadmins
2012-07-27 17:39:11 Khris Richardson bug added subscriber Khris Richardson
2012-08-06 08:21:44 Robie Basak bug added subscriber Robie Basak
2012-08-15 18:40:00 dann frazier bug task added lomond
2012-08-16 16:46:19 dann frazier lomond: status New Triaged
2012-09-17 11:38:42 Robie Basak tags api provisioning api arm provisioning
2012-09-24 21:01:21 Scott Moser bug task added cloud-init
2012-09-24 21:01:30 Scott Moser cloud-init: status New In Progress
2012-09-24 21:01:36 Scott Moser cloud-init: importance Undecided Medium
2012-09-24 21:01:39 Scott Moser cloud-init: assignee Scott Moser (smoser)
2012-09-24 21:14:54 Launchpad Janitor branch linked lp:cloud-init
2012-09-24 21:18:11 Scott Moser bug task added cloud-init (Ubuntu)
2012-09-24 23:55:24 Scott Moser cloud-init (Ubuntu): status New Triaged
2012-09-24 23:55:27 Scott Moser cloud-init (Ubuntu): importance Undecided High
2012-10-01 14:54:12 Launchpad Janitor cloud-init (Ubuntu): status Triaged Fix Released
2012-10-01 18:24:16 Scott Moser cloud-init: status In Progress Fix Released
2012-10-01 18:55:04 mahmoh lomond: status Triaged Fix Released
2012-10-04 20:22:41 Francis J. Lacoste maas: status Triaged Fix Committed
2012-10-05 02:28:33 Julian Edwards maas: status Fix Committed Fix Released
2012-10-05 14:55:01 Scott Moser nominated for series Ubuntu Precise
2012-10-05 14:55:01 Scott Moser bug task added cloud-init (Ubuntu Precise)
2012-10-05 15:07:52 Scott Moser cloud-init (Ubuntu Precise): status New Triaged
2012-10-05 15:07:55 Scott Moser cloud-init (Ubuntu Precise): importance Undecided High
2012-10-05 17:21:17 Launchpad Janitor branch linked lp:ubuntu/cloud-init
2012-10-09 13:21:39 Robie Basak maas: status Fix Released Triaged
2012-10-09 13:21:43 Robie Basak maas: importance High Critical
2012-10-09 13:25:50 Robie Basak maas: assignee Scott Moser (smoser)
2012-10-09 14:29:06 Francis J. Lacoste bug task added maas (Ubuntu)
2012-10-09 14:39:47 Francis J. Lacoste maas: status Triaged In Progress
2012-10-09 14:44:22 Francis J. Lacoste tags api arm provisioning api arm missing-in-quantal provisioning
2012-10-09 14:44:57 Francis J. Lacoste bug task deleted maas (Ubuntu)
2012-10-09 14:45:06 Francis J. Lacoste bug task deleted maas (Ubuntu Precise)
2012-10-09 18:57:55 Launchpad Janitor branch linked lp:~smoser/maas/trunk.maas-signal-clockskew
2012-10-09 19:49:44 MAAS Lander maas: status In Progress Fix Committed
2012-10-10 00:15:12 Francis J. Lacoste maas: status Fix Committed Fix Released
2012-10-12 18:41:52 Launchpad Janitor branch linked lp:~smoser/ubuntu/precise/cloud-init/sru
2012-11-08 16:03:56 Scott Moser tags api arm missing-in-quantal provisioning api arm provisioning
2012-11-08 16:42:38 Scott Moser description In this simple scenario: a. hardware installed b. hardware booted and enlisted c. commissioning d. install to hardware e. cloud-init boot At this point steps 'b' and 'e' do OAUTH to get user-data. If the clock on the system is sufficiently off, then oauth will fail as shown in the attached screenshot. it seems to make sense that 'b' would set the clock. Once the user enlists the systme to MAAS, it seems OK to start changing their hardware settings. There is still a potential for really bad hardware clock that could forget its settings on reboot, or somehow get off between 'b' and 'c'. If we were really interested in fixing that, cloud-init could read a kernel command line parameter pointing to a system that ran an ntp server and just run that very early in boot to set the local date. === Begin SRU Information === [Impact] * For systems that have a broken or incorrectly set hardware clock, enlistment and commissioning into MAAS will fail. This is because ubuntu's system clock is initially seeded by the hardware clock, and OAUTH is used for authentication with the maas server. If the client clock differs by more than 5 minutes from the server clock, authentication will fail, and subsequently enlistment or commisioning will fail. This is also a problem after installation of the system as the same process for authentication is used. * There is a need to backport this change in order to fully utilize 12.04 and MAAS. * The change in cloud-init is essentially this: If a request for access to the MAAS metadata service returns 401 or 403 (unauthorized), then subsequent re-tries will modify the timestamp in the OAUTH request so that it matches the server. [Test Case] * To recreate the bug, you first need to get MAAS set up (http://maas.ubuntu.com/docs), and start a system for enlistment that would have an invalid clock. To force an invalid clock, do one of: * boot to a system bios and change the bios clock * modify the ephemeral image so that the clock is broken during boot. This can be accomplished by appending the following to /etc/init/cloud-init.conf inside an ephemeral image. | pre-start script | offset="10 minutes ago" | past=$(date -R --date "$offset") | date --set "$past" && | echo ===== "set date to $past [$offset]" ===== || | echo ===== "failed to set date to $past [$offset]" ==== | end script This is actually more complex than that, because the ephemeral images already have this fix inside of them. So in order to reproduce, you have to downgrade the version of cloud-init inside the 12.04 ephemeral image to the version available in the ubuntu archive (0.6.3-0ubuntu1.1) * After a sufficiently broken system is obtained, boot the system. If this fix is not present, enlistment or commissioning will fail to do anything as it will not have access to the metadata. * Errors will be written to the MAAS server's /var/log/apache/error.log * When the fix is applied, a single failure will occur, and then cloud-init will modify future requests. [Regression Potential] * Regression is limited to the MAAS datasource, which is not enabled by default for cloud-init. Thus, only a user that is using MAAS or otherwise takes explicit action to enable it will be affected. [Other Info] * This bug has essentially been fixed in maas enlistment and commissioning environments outside of the SRU process. The "ephemeral images" downloaded for MAAS have 12.10's version of cloud-init installed inside them. This all works reliably. We want to properly SRU the change so that installed systems will also be resilient to a bad hardware clock. === End SRU Information === === original bug report === In this simple scenario:  a. hardware installed  b. hardware booted and enlisted  c. commissioning  d. install to hardware  e. cloud-init boot At this point steps 'b' and 'e' do OAUTH to get user-data. If the clock on the system is sufficiently off, then oauth will fail as shown in the attached screenshot. it seems to make sense that 'b' would set the clock. Once the user enlists the systme to MAAS, it seems OK to start changing their hardware settings. There is still a potential for really bad hardware clock that could forget its settings on reboot, or somehow get off between 'b' and 'c'. If we were really interested in fixing that, cloud-init could read a kernel command line parameter pointing to a system that ran an ntp server and just run that very early in boot to set the local date.
2012-12-10 23:05:55 Steve Langasek cloud-init (Ubuntu Precise): status Triaged Fix Committed
2012-12-10 23:05:58 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2012-12-10 23:06:00 Steve Langasek bug added subscriber SRU Verification
2012-12-10 23:06:10 Steve Langasek tags api arm provisioning api arm provisioning verification-needed
2012-12-12 20:48:52 Scott Moser attachment added modify-ephemeral image script https://bugs.launchpad.net/ubuntu/precise/+source/cloud-init/+bug/978127/+attachment/3458662/+files/modify-image
2012-12-12 20:49:11 Scott Moser tags api arm provisioning verification-needed api arm provisioning verification-done
2013-01-08 19:16:39 Clint Byrum tags api arm provisioning verification-done api arm provisioning
2013-01-08 19:16:40 Clint Byrum tags api arm provisioning api arm provisioning verification-needed
2013-01-08 23:48:28 Scott Moser tags api arm provisioning verification-needed api arm provisioning verification-done
2013-01-16 14:20:12 Colin Watson removed subscriber Ubuntu Stable Release Updates Team
2013-01-16 14:20:17 Launchpad Janitor cloud-init (Ubuntu Precise): status Fix Committed Fix Released
2013-08-28 11:32:39 Launchpad Janitor branch linked lp:~ubuntu-branches/ubuntu/precise/cloud-init/precise-proposed
2013-10-25 19:37:51 Nobuto Murata bug added subscriber Nobuto MURATA
2014-01-21 15:29:36 Michael Cunningham bug added subscriber Jonathan Davies
2023-05-09 20:30:08 James Falcon bug watch added https://github.com/canonical/cloud-init/issues/2256