Registering images gives 403 Forbidden

[Soren Hansen]

When trying to register a kernel image, I get this in my
cloud-output.log:

15:09:52 WARN PipelineRegistry | => More than one candidate pipeline. Ignoring offer by: internal-query-pipeline-Eucalyptus of type InternalQueryPipeline
com.eucalyptus.ws.AuthenticationException: Message has expired.
 at com.eucalyptus.ws.handlers.QueryTimestampHandler.incomingMessage(QueryTimestampHandler.java:108)
 at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:115)
 at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:116)
 at com.eucalyptus.ws.server.FilteredPipeline$StageBottomHandler.handleUpstream(FilteredPipeline.java:171)
 at com.eucalyptus.ws.server.NioServerHandler.messageReceived(NioServerHandler.java:111)
 at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:114)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:459)
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:443)
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:381)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
 at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
 at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:636)

And this on the command line:
soren@aalborg2:~$ euca-register kernel/vmlinuz-$(uname -r).manifest.xml --debug
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

This is on a fresh install. It's the first image I'm attempting to add.

 affects ubuntu/eucalyptus
 importance high
 status triaged
 tag eucalyptus

[Soren Hansen (soren)]

This seems to be timezone setting related. I fiddled around with my timezone settings and it suddenly worked. I'm not sure if it's eucalyptus or euca2ools at this point.

[Thierry Carrez (ttx)]

Works with ec2-api-tools:

$ euca-register --debug kernel/vmlinuz-$(uname -r).manifest.xml
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden
$ ec2-register --debug kernel/vmlinuz-$(uname -r).manifest.xml
IMAGE eki-448E1260

[Thierry Carrez (ttx)]

It is indeed TZ-related. Any timezone <= UTC will work. UTC+n will fail.

Current default time zone: 'Europe/Paris'
Local time is now: Mon Sep 21 14:27:25 CEST 2009.
Universal Time is now: Mon Sep 21 12:27:25 UTC 2009.
$ euca-describe-images
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden
$ sudo dpkg-reconfigure tzdata
Current default time zone: 'Europe/London'
Local time is now: Mon Sep 21 13:19:30 BST 2009.
Universal Time is now: Mon Sep 21 12:19:30 UTC 2009.
ubuntu@ubuntu:~$ sudo service eucalyptus-cloud restart
 * Stopping Eucalyptus Cloud Controller [ OK ]
 * Starting Eucalyptus Cloud Controller eucalyptus-cloud [ OK ]
ubuntu@ubuntu:~$ euca-describe-images
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden
$ sudo dpkg-reconfigure tzdata
Current default time zone: 'Etc/UTC'
Local time is now: Mon Sep 21 12:16:37 UTC 2009.
Universal Time is now: Mon Sep 21 12:16:37 UTC 2009.
$ sudo service eucalyptus-cloud restart
 * Stopping Eucalyptus Cloud Controller [ OK ]
 * Starting Eucalyptus Cloud Controller eucalyptus-cloud [ OK ]
$ euca-describe-images
IMAGE eri-96A8139C ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eri-96A71399 ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eki-448E1260 kernel/vmlinuz-2.6.31-10-generic.manifest.xml admin available public x86_64 kernel
$ sudo dpkg-reconfigure tzdata
Current default time zone: 'US/Pacific'
Local time is now: Mon Sep 21 05:18:26 PDT 2009.
Universal Time is now: Mon Sep 21 12:18:26 UTC 2009.
$ sudo service eucalyptus-cloud restart
 * Stopping Eucalyptus Cloud Controller [ OK ]
 * Starting Eucalyptus Cloud Controller eucalyptus-cloud [ OK ]
$ euca-describe-images
IMAGE eri-96A8139C ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eri-96A71399 ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eki-448E1260 kernel/vmlinuz-2.6.31-10-generic.manifest.xml admin available public x86_64 kernel

[Thierry Carrez (ttx)]

I think I nailed it... At UTC+2 and 14:12
Timestamp received is of the form "2009-09-23T12:12:34", missing the final Z.
In clc/modules/wsstack/src/main/java/com/eucalyptus/ws/util/HmacUtils.java if there is no timezone indication in the ISO8601 timestamp, it uses server local time. So the message is expired by 2 hours.
Would work for all timezones <= UTC.

Two options to fix:
* Set TZ to UTC before making the local time conversion in HmacUtils.java (eucalyptus)
* Fix whatever is producing the timestamp to make it properly add Z for Zulu time (?)

[Thierry Carrez (ttx)]

Upon further analysis this needs to be fixed in eucalyptus itself.
clc/modules/wsstack/src/main/java/com/eucalyptus/ws/util/HmacUtils.java evaluates all ISO8601 timestamps as local time, whether the final Z is present or not. Forcing UTC TZ is that evaluation fixes it.

[Thierry Carrez (ttx)]

Not a bug in euca2ools

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6~bzr854-0ubuntu2

---------------
eucalyptus (1.6~bzr854-0ubuntu2) karmic; urgency=low

  * Fix query timeout issues in timezones > UTC (LP: #431847)
  * Add --local-sync to allow euca_conf --register-* to sync keys locally
    if an external (but local) IP address is used to register (LP: #434651)

 -- Thierry Carrez <email address hidden> Wed, 23 Sep 2009 18:27:06 +0200

[Thierry Carrez (ttx)]

Fix breaks euca_conf (for TZ < UTC), since euca_conf is sending local time instead of UTC.
See more complete upstream fix at:
http://bazaar.launchpad.net/~eucalyptus-maintainers/eucalyptus/1.6/revision/895

[Thierry Carrez (ttx)]

Was applied by Dustin in
http://bazaar.launchpad.net/~ubuntu-core-dev/eucalyptus/ubuntu/revision/615