Registering images gives 403 Forbidden

Bug #431847 reported by Soren Hansen on 2009-09-17
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
euca2ools (Ubuntu)
High
Thierry Carrez
Karmic
High
Thierry Carrez
eucalyptus (Ubuntu)
High
Thierry Carrez
Karmic
High
Thierry Carrez

Bug Description

When trying to register a kernel image, I get this in my
cloud-output.log:

15:09:52 WARN PipelineRegistry | => More than one candidate pipeline. Ignoring offer by: internal-query-pipeline-Eucalyptus of type InternalQueryPipeline
com.eucalyptus.ws.AuthenticationException: Message has expired.
 at com.eucalyptus.ws.handlers.QueryTimestampHandler.incomingMessage(QueryTimestampHandler.java:108)
 at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:115)
 at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:116)
 at com.eucalyptus.ws.server.FilteredPipeline$StageBottomHandler.handleUpstream(FilteredPipeline.java:171)
 at com.eucalyptus.ws.server.NioServerHandler.messageReceived(NioServerHandler.java:111)
 at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:114)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:459)
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:443)
 at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:381)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
 at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
 at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:636)

And this on the command line:
soren@aalborg2:~$ euca-register kernel/vmlinuz-$(uname -r).manifest.xml --debug
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

This is on a fresh install. It's the first image I'm attempting to add.

 affects ubuntu/eucalyptus
 importance high
 status triaged
 tag eucalyptus

Soren Hansen (soren) wrote :

This seems to be timezone setting related. I fiddled around with my timezone settings and it suddenly worked. I'm not sure if it's eucalyptus or euca2ools at this point.

Soren Hansen (soren) on 2009-09-17
Changed in eucalyptus (Ubuntu):
milestone: none → ubuntu-9.10-beta
Thierry Carrez (ttx) wrote :

Works with ec2-api-tools:

$ euca-register --debug kernel/vmlinuz-$(uname -r).manifest.xml
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden
$ ec2-register --debug kernel/vmlinuz-$(uname -r).manifest.xml
IMAGE eki-448E1260

Changed in euca2ools (Ubuntu Karmic):
assignee: nobody → Soren Hansen (soren)
importance: Undecided → High
milestone: none → ubuntu-9.10-beta
status: New → Triaged
Changed in eucalyptus (Ubuntu Karmic):
status: Triaged → Invalid
Thierry Carrez (ttx) wrote :

It is indeed TZ-related. Any timezone <= UTC will work. UTC+n will fail.

Current default time zone: 'Europe/Paris'
Local time is now: Mon Sep 21 14:27:25 CEST 2009.
Universal Time is now: Mon Sep 21 12:27:25 UTC 2009.
$ euca-describe-images
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden
$ sudo dpkg-reconfigure tzdata
Current default time zone: 'Europe/London'
Local time is now: Mon Sep 21 13:19:30 BST 2009.
Universal Time is now: Mon Sep 21 12:19:30 UTC 2009.
ubuntu@ubuntu:~$ sudo service eucalyptus-cloud restart
 * Stopping Eucalyptus Cloud Controller [ OK ]
 * Starting Eucalyptus Cloud Controller eucalyptus-cloud [ OK ]
ubuntu@ubuntu:~$ euca-describe-images
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden
$ sudo dpkg-reconfigure tzdata
Current default time zone: 'Etc/UTC'
Local time is now: Mon Sep 21 12:16:37 UTC 2009.
Universal Time is now: Mon Sep 21 12:16:37 UTC 2009.
$ sudo service eucalyptus-cloud restart
 * Stopping Eucalyptus Cloud Controller [ OK ]
 * Starting Eucalyptus Cloud Controller eucalyptus-cloud [ OK ]
$ euca-describe-images
IMAGE eri-96A8139C ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eri-96A71399 ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eki-448E1260 kernel/vmlinuz-2.6.31-10-generic.manifest.xml admin available public x86_64 kernel
$ sudo dpkg-reconfigure tzdata
Current default time zone: 'US/Pacific'
Local time is now: Mon Sep 21 05:18:26 PDT 2009.
Universal Time is now: Mon Sep 21 12:18:26 UTC 2009.
$ sudo service eucalyptus-cloud restart
 * Stopping Eucalyptus Cloud Controller [ OK ]
 * Starting Eucalyptus Cloud Controller eucalyptus-cloud [ OK ]
$ euca-describe-images
IMAGE eri-96A8139C ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eri-96A71399 ramdisk/initrd.img-2.6.31-10-generic.manifest.xml admin available public x86_64 ramdisk
IMAGE eki-448E1260 kernel/vmlinuz-2.6.31-10-generic.manifest.xml admin available public x86_64 kernel

Thierry Carrez (ttx) on 2009-09-23
Changed in euca2ools (Ubuntu Karmic):
assignee: Soren Hansen (soren) → Thierry Carrez (ttx)
Changed in eucalyptus (Ubuntu Karmic):
milestone: ubuntu-9.10-beta → none
Thierry Carrez (ttx) wrote :

I think I nailed it... At UTC+2 and 14:12
Timestamp received is of the form "2009-09-23T12:12:34", missing the final Z.
In clc/modules/wsstack/src/main/java/com/eucalyptus/ws/util/HmacUtils.java if there is no timezone indication in the ISO8601 timestamp, it uses server local time. So the message is expired by 2 hours.
Would work for all timezones <= UTC.

Two options to fix:
* Set TZ to UTC before making the local time conversion in HmacUtils.java (eucalyptus)
* Fix whatever is producing the timestamp to make it properly add Z for Zulu time (?)

Changed in euca2ools (Ubuntu Karmic):
status: Triaged → In Progress
Thierry Carrez (ttx) wrote :

Upon further analysis this needs to be fixed in eucalyptus itself.
clc/modules/wsstack/src/main/java/com/eucalyptus/ws/util/HmacUtils.java evaluates all ISO8601 timestamps as local time, whether the final Z is present or not. Forcing UTC TZ is that evaluation fixes it.

Changed in eucalyptus (Ubuntu Karmic):
assignee: nobody → Thierry Carrez (ttx)
milestone: none → ubuntu-9.10-beta
status: Invalid → In Progress
Thierry Carrez (ttx) wrote :

Not a bug in euca2ools

Changed in euca2ools (Ubuntu Karmic):
milestone: ubuntu-9.10-beta → none
status: In Progress → Invalid
Thierry Carrez (ttx) on 2009-09-23
Changed in eucalyptus (Ubuntu Karmic):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6~bzr854-0ubuntu2

---------------
eucalyptus (1.6~bzr854-0ubuntu2) karmic; urgency=low

  * Fix query timeout issues in timezones > UTC (LP: #431847)
  * Add --local-sync to allow euca_conf --register-* to sync keys locally
    if an external (but local) IP address is used to register (LP: #434651)

 -- Thierry Carrez <email address hidden> Wed, 23 Sep 2009 18:27:06 +0200

Changed in eucalyptus (Ubuntu Karmic):
status: Fix Committed → Fix Released
Thierry Carrez (ttx) wrote :

Fix breaks euca_conf (for TZ < UTC), since euca_conf is sending local time instead of UTC.
See more complete upstream fix at:
http://bazaar.launchpad.net/~eucalyptus-maintainers/eucalyptus/1.6/revision/895

Changed in eucalyptus (Ubuntu Karmic):
status: Fix Released → Triaged
Thierry Carrez (ttx) wrote :
Changed in eucalyptus (Ubuntu Karmic):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers