sync-tools requires aws credentials be set in the (shell) environment

Reported by Michael Hudson-Doyle on 2013-04-25
46
This bug affects 8 people
Affects Status Importance Assigned to Milestone
juju-core
Critical
Frank Mueller

Bug Description

I was setting up a environment that had MAAS as its provider. bootstrap said this:

$ juju -v bootstrap
2013/04/25 21:32:12 INFO environs: reading tools with major version 1
2013/04/25 21:32:12 INFO environs: falling back to public bucket
2013/04/25 21:32:12 ERROR command failed: no tools available
error: no tools available

A hint on IRC got me to try sync-tools:

$ juju -v sync-tools
2013/04/25 21:46:20 INFO environs/ec2: opening environment "juju-public"
2013/04/25 21:46:20 ERROR failed to initialize the official bucket environment
2013/04/25 21:46:20 ERROR command failed: environment has no access-key or secret-key
error: environment has no access-key or secret-key

Setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the shell made sync-tools succeed and I could make progress. But it's pretty obscure.

Related branches

Michael Hudson-Doyle (mwhudson) wrote :

Chasing through the code a little (and I'm not a go programmer at all, but this seems fairly clear):

cmd/juju/synctools.go:officialBucketAttrs does not have an access-key or secret-key set
cmd/juju/synctools.go:Run calls environs.NewFromAttrs(officialBucketAttrs)
environs/open.go:NewFromAttrs: calls cNew
environs/config.go:New calls Open on an ec2.environProvider
environs/ec2/ec2.go:Open calls SetConfig on an environ
SetConfig calls newConfig
newConfig calls Validate
Validate complains if the passed in config _and_ the environment do not have an access key and a secret key -- and as the passed in config does not (it is made from officialBucketAttrs), the only way this can pass is if the environment has the keys.

I'd guess there needs to be some way to tell Validate that not having the keys is ok, but I'll leave that sort of thing to someone who understands the architecture :)

Martin Packman (gz) on 2013-05-01
Changed in juju-core:
importance: Undecided → High
status: New → Confirmed
Haw Loeung (hloeung) on 2013-05-10
tags: added: canonical-webops-juju
John A Meinel (jameinel) wrote :

Note that this is actually because of bug #1161383 in goaws.

Having written sync-tools and been unhappy about this fact, here is the background:
1) creating an environment in juju-core always validates the full config. And the config code says you must have credentials set (either in the config or in the environment).
2) goaws is used for listing the S3 bucket. It currently requires you to pass credentials.
3) Amazon will always validate the credentials you supply, even if the bucket itself is public. (It checks passed credentials before it checks if the bucket can be read.)

If we could pass empty auth for (2) then (3) would be happy. But you can't set empty credentials due to (1). And even if you could, we have to be careful with how goaws handles empty credentials. (naively it might try to sign the request with an key of "" and Amazon might reject that as invalid, rather than actually not signing the request at all.)

sync-tools is rather unique in juju-core, as it is the only code that wants to talk to 2 providers. Also, it is unique in being the only code that will never want to write to one of the providers. (arguably 'juju status' is read code, but at some point in the future, you could write to that provider. sync-tools is talking to an environment where you will never have write access.)

I don't have a good feeling for whether we should make it more apparent that you have to source your AWS credentials, or whether we can actually fix it. There are enough layers and specific differences for sync-tools that I'm hesitant to make a clear cut decision.

Michael Hudson-Doyle (mwhudson) wrote :

IMHO, you have to fix it. If all you're doing is deploying to MAAS or openstack, you might not even *have* AWS credentials.

Kapil Thangavelu (hazmat) wrote :

A public bucket resource is simply a url that can be obtained without resorting to goamz at all.

Kapil Thangavelu (hazmat) wrote :

A log of my session, fwiw

ubuntu@maas-dev:~/.juju$ juju status
error: file 'provider-state' not found not found
ubuntu@maas-dev:~/.juju$ juju bootstrap
2013-06-06 00:15:37 DEBUG juju environprovider.go:24 environs/maas: opening environment "maas".
2013-06-06 00:15:38 DEBUG juju environ.go:94 environs/maas: bootstrapping environment "maas"
2013-06-06 00:15:38 INFO juju tools.go:25 environs: reading tools with major version 1
2013-06-06 00:15:38 DEBUG juju storage.go:49 environs/tools: reading v1.* tools
2013-06-06 00:15:38 INFO juju tools.go:29 environs: falling back to public bucket
2013-06-06 00:15:38 DEBUG juju storage.go:49 environs/tools: reading v1.* tools
2013-06-06 00:15:38 ERROR juju supercommand.go:234 command failed: no tools available
error: no tools available
ubuntu@maas-dev:~/.juju$ env | grep AWS
ubuntu@maas-dev:~/.juju$ juju sync-tools --debug
error: environment has no access-key or secret-key
ubuntu@maas-dev:~/.juju$ juju sync-tools --debug
2013-06-06 00:16:57 INFO juju ec2.go:132 environs/ec2: opening environment "juju-public"
2013-06-06 00:16:57 ERROR juju synctools.go:115 failed to initialize the official bucket environment
2013-06-06 00:16:57 ERROR juju supercommand.go:234 command failed: environment has no access-key or secret-key
error: environment has no access-key or secret-key
ubuntu@maas-dev:~/.juju$ cat environments.yaml
default: maas
environments:
  maas:
    type: maas
    # Change this to where your MAAS server lives. It must specify the base path.
    maas-server: 'http://10.0.3.83/MAAS/'
    maas-oauth: 'kmtLZj6UmgN3TLPpgJ:79r2Hjts2L42ntdJwE:5FK4FQB2J9yF5ZcWSpCR9hMyedPVLtUQ'
    admin-secret: d0e51b71cebd919c36f4f5f3bad1432d
    default-series: precise
    authorized-keys-path: ~/.ssh/authorized_keys # or any file you want.

Jason Sievert (jsievert) wrote :

This might be related but by setting the AWS information I was able to get a bit farther now I am getting the below error.

2013/06/07 13:20:35 DEBUG environs/tools: reading v1.* tools
found 0 tools in target; 6 tools to be copied
2013/06/07 13:20:35 INFO copying 1.10.0-precise-amd64 from https://s3.amazonaws.com/juju-dist/tools/juju-1.10.0-precise-amd64.tgz?AWSAccessKeyId=AKIAI6DEQNFHGINORYIA&Expires=1686162035&Signature=E8RH9lvtpcvMvjliEXq5tLcRiAE%3D
copying tools/juju-1.10.0-precise-amd64.tgz2013/06/07 13:20:36 ERROR command failed: use of closed network connection
error: use of closed network connection

Copying the listed URL into a browser I am able to download the files but I am unsure how to proceed.

William Reade (fwereade) wrote :

Frank, I understand you've already started working on this?

Changed in juju-core:
status: Confirmed → Triaged
importance: High → Critical
assignee: nobody → Frank Mueller (themue)
Frank Mueller (themue) wrote :

Yes, change is in progress. Looks good as discussed so far, rough testing on the live location works. Next step will be the change of the tests to use a simulated storage.

Frank Mueller (themue) on 2013-06-17
Changed in juju-core:
status: Triaged → In Progress
Frank Mueller (themue) on 2013-06-19
Changed in juju-core:
status: In Progress → Fix Committed
Tim Penhey (thumper) on 2013-06-20
Changed in juju-core:
milestone: none → 1.11.1
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers