juju bootstrap fails due to wrong Date header (openstack_s3)

[Thomas Leonard]

After "juju bootstrap", machine 0 fails to start correctly. /var/log/cloud-init-output.log shows:

usage: juju-admin [-h] [--verbose] [--version] [--log-file LOG_FILE] ...

juju cloud orchestration internal tools

positional arguments:

    initialize Initialize Zookeeper hierarchy

optional arguments:
  -h, --help show this help message and exit
  --verbose, -v Enable verbose logging
  --version show program's version number and exit
  --log-file LOG_FILE, -l LOG_FILE
                        Log output to file
juju-admin: error: unrecognized arguments: version="1.0" encoding="UTF-8"?>^M <Error>^M <Code>AccessDenied</Code>

This is because the curl command failed.

The AccessDenied error comes from /usr/share/pyshared/swift3/middleware.py, which does:

  date = email.utils.parsedate(req.headers['Date'])

However, printing out req.headers['Date'] shows that it is an integer (which time.ctime() shows is exactly 10 years in the future).

I don't know what the correct fix is, but I changed it to ignore unparsable dates and that allowed Juju to work.

I have done two OpenStack installs on two different machines, one starting with Essex and upgrading to Folsom and another starting with Folsom, and I hit this issue both times.

[Kapil Thangavelu (hazmat)]

does it work with native openstack provider ie not openstack_s3

[Thomas Leonard (tal-it-innovation)]

Hi Kapil,

The "openstack" provider does not appear to work at all, due to the permissions issues you mentioned previously:

https://lists.ubuntu.com/archives/juju/2012-December/002085.html

[Kapil Thangavelu (hazmat)]

it sounds like a compatibility issue between swift's s3 middleware and the actual s3 behavior, hence a bug in swift's s3 middleware compatibility. we could try to address in txaws s3 implementation as a workaround, but that's not quite right..

looking over the s3 swift middleware history this is the commit that causes the issue

https://github.com/fujita/swift3/commit/c3341b8de8befc7ed6dfca94de70904baa853c61

its basically disallowing a time delta of more 1m from the server current time, which is broken for this case, in addition to parsing it with the wrong format.

[Kapil Thangavelu (hazmat)]

err.. disallowing a time delta of more 10m