[jaas] add-model doesn't respect region

[james beedy]

$ juju add-model bdx-k8s aws/us-west-2 --credential james-pdl
Using credential 'james-pdl' cached in controller
Added 'bdx-k8s' model on aws/us-east-1 with credential 'james-pdl' for user 'jamesbeedy'

I try to add a model in aws/us-west-2, but I get aws/us-east-1.

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
bdx-k8s jaas aws/us-east-1 2.6.8 unsupported 03:52:03Z

Model "jamesbeedy@external/bdx-k8s" is empty.

[Ian Booth (wallyworld)]

I tried this and it seemed to work using juju 2.7-beta1 client.

$ juju login jaas

$ juju add-model foo aws/us-west-1 --credential wallyworld
Uploading credential 'aws/wallyworld@external/wallyworld' to controller
Added 'foo' model on aws/us-west-1 with credential 'wallyworld' for user 'wallyworld'

$ juju models
Controller: jaas

Model Cloud/Region Type Status Machines Cores Access Last connection
foo* aws/us-west-1 ec2 available 0 0 admin never connected

$ juju add-model bar aws/us-east-2
Added 'bar' model on aws/us-east-2 with credential 'wallyworld' for user 'wallyworld'

$ juju models
Controller: jaas

Model Cloud/Region Type Status Machines Cores Access Last connection
bar* aws/us-east-2 ec2 available 0 0 admin never connected
foo aws/us-west-1 ec2 available 0 0 admin never connected

[Ian Booth (wallyworld)]

I also tried with the CLI as the stable 2.6.9 snap

$ /snap/bin/juju add-model foobar aws/us-west-2
Added 'foobar' model on aws/us-west-2 with credential 'wallyworld' for user 'wallyworld'

Any further info on how to reproduce would be great.

[james beedy (jamesbeedy)]

I can reproduce this as follows:

$ juju --version
2.7-rc1-bionic-amd64

$ juju add-model ceph-k8s-demo-00 aws/us-west-2 --credential james-pdl
Using credential 'james-pdl' cached in controller
Added 'ceph-k8s-demo-00' model on aws/eu-west-1 with credential 'james-pdl' for user 'jamesbeedy'

$ juju add-space nat 172.31.102.0/24 172.31.103.0/24 172.31.105.0/24
juju add-ERROR cannot add space "nat": adding space "nat": subnet "172.31.102.0/24" not found

$ juju add-space nat 172.31.102.0/24 172.31.103.0/24 172.31.104.0/24
ERROR cannot add space "nat": adding space "nat": subnet "172.31.102.0/24" not found

$ juju add-space nat 172.31.103.0/24 172.31.104.0/24 172.31.105.0/24
ERROR cannot add space "nat": adding space "nat": subnet "172.31.103.0/24" not found

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
ceph-k8s-demo-00 jaas aws/eu-west-1 2.6.8 unsupported 16:16:44Z

Model "jamesbeedy@external/ceph-k8s-demo-00" is empty.

I should have noticed from the output of the add-model command that the model was created in eu-west-1. As we would expect, I cannot add the subnets because those subnets only exist in us-west-2.

[james beedy (jamesbeedy)]

A comparison of using JAAS vs my own controller https://paste.ubuntu.com/p/VBkRQxz6pK/

[Anastasia (anastasia-macmood)]

I can confirm that the problem is not on Juju side but JAAS. I am now investigating its implementation to pinpoint and fix the problem.

[james beedy (jamesbeedy)]

Thanks @anastasia-macmood!

[Martin Hilton (martin-hilton)]

Tracing the add-model request, it looks like the juju client isn't including the cloud-region in the CreateModel request so JAAS is using the default.

% juju add-model test-us-west-2-002 aws/us-west-2 --debug --logging-config trace --credential mhilton
15:29:12 INFO juju.cmd supercommand.go:80 running juju [2.7-rc4 gc go1.10.4]
15:29:12 DEBUG juju.cmd supercommand.go:81 args: []string{"/snap/juju/9701/bin/juju", "add-model", "test-us-west-2-002", "aws/us-west-2", "--debug", "--logging-config", "trace", "--credential", "mhilton"}
15:29:12 INFO juju.juju api.go:67 connecting to API addresses: [jimm.jujucharms.com:443]
15:29:12 TRACE juju.api apiclient.go:1089 dialing: "wss://jimm.jujucharms.com:443/api" 162.213.33.88:443
15:29:12 TRACE juju.api apiclient.go:1089 dialing: "wss://jimm.jujucharms.com:443/api" 162.213.33.244:443
15:29:12 DEBUG juju.api apiclient.go:745 looked up jimm.jujucharms.com -> [162.213.33.244 162.213.33.88]
15:29:12 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://jimm.jujucharms.com:443/api"
15:29:12 INFO juju.api apiclient.go:624 connection established to "wss://jimm.jujucharms.com:443/api"
15:29:12 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://jimm.jujucharms.com:443/api"
15:29:12 TRACE juju.rpc.jsoncodec codec.go:228 -> {"request-id":1,"type":"Admin","version":3,"request":"Login","params":{"auth-tag":"","credentials":"","nonce":"","macaroons":["***redacted***"],"cli-args":"/snap/juju/9701/bin/juju add-model test-us-west-2-002 aws/us-west-2 --debug --logging-config trace --credential mhilton","user-data":""}}
15:29:12 TRACE juju.rpc.jsoncodec codec.go:122 <- {"request-id":1,"response":{"controller-tag":"controller-a030379a-940f-4760-8fcf-3062b41a04e7","user-info":{"display-name":"martin-hilton","identity":"user-martin-hilton@external","controller-access":"","model-access":""},"facades":[{"name":"Bundle","versions":[1]},{"name":"Cloud","versions":[1,2,3,4,5]},{"name":"Controller","versions":[3]},{"name":"JIMM","versions":[1]},{"name":"ModelManager","versions":[2,3,4,5]},{"name":"Pinger","versions":[1]},{"name":"UserManager","versions":[1]}],"server-version":"2.6.10"}}
15:29:12 INFO cmd authkeys.go:114 Adding contents of "/home/mhilton/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
15:29:12 INFO cmd authkeys.go:114 Adding contents of "/home/mhilton/.ssh/id_rsa.pub" to authorized-keys
15:29:12 TRACE juju.rpc.jsoncodec codec.go:228 -> {"request-id":2,"type":"Cloud","version":5,"request":"Cloud","params":{"entities":[{"tag":"cloud-aws"}]}}
15:29:12 TRACE juju.rpc.jsoncodec codec.go:122 <- {"request-id":2,"response":{"results":[{"cloud":{"type":"ec2","auth-types":["access-key"],"regions":[{"name":"ap-northeast-1","endpoint":"https://ec2.ap-northeast-1.amazonaws.com"},{"name":"ap-northeast-2","endpoint":"https://ec2.ap-northeast-2.amazonaws.com"},{"name":"ap-south-1","endpoint":"https://ec2.ap-south-1.amazonaws.com"},{"name":"ap-southeast-1","endpoint":"https://ec2.ap-southeast-1.amazonaws.com"},{"name":"ap-southeast-2","endpoint":"https://ec2.ap-southeast-2.amazonaws.com"},{"name":"eu-central-1","endpoint":"https://ec2.eu-central-1.amazonaws.com"},{"name":"eu-west-1","endpoint":"h...

[Anastasia (anastasia-macmood)]

So I ran the same command as you @Martin Hilton (martin-hilton), and am getting the region in the request [2]. In fact, at the end of my run, I do get 2 models on a the default region created by bootstrap and the model I added manually on the request region [1]...

[1]
https://pastebin.ubuntu.com/p/fG6YZqRgSB/

[2]
https://pastebin.ubuntu.com/p/zbnBCy3SsK/

I'll try adding '--credential' option to the command but am not convinced that this is the difference. I am a bit puzzled at the moment ;D

[Anastasia (anastasia-macmood)]

No, specifying the credential does not affect region selection. I still create a model on the region I request successfully: https://pastebin.ubuntu.com/p/4s4dN2RdKq/

[Anastasia (anastasia-macmood)]

The same works for me on develop (heading into 2.8), 2.7 and 2.6.. So it's a standard behavior in Juju going back a while.

[james beedy (jamesbeedy)]

hello,

What is the status of this bug? This has been a massive thorn for me, and is still blocking myself and others from creating models on jaas.

@anastasis-macmood are you not able to reproduce?

What can I do to help here?

Thanks

[Ian Booth (wallyworld)]

We've tried again today to reproduce this with no luck :-(

Just to see what difference there is with your set up, does your credential in ~/.local/share/juju/credentials.yaml have the default-region attribute set?

Are you able to run juju show-credential compare what the controller has stored vs what's in the local credentials.yaml file and pastebin the results with the access key/secrets redacted?

[Ian Booth (wallyworld)]

Digging a bit, I think I found the issue. If the named credential is not cached locally, it is retrieved from what has been uploaded previously to the controller. And in that case, whatever region was supplied on to the command as the cloud/region arg is overwritten with "". This then causes Juju to choose the default region when adding the model.

The reason we didn't see it trying to reproduce is that we all had a local copy of the named credential cause that's what was needed to add a model in the first place. But if you then add a model from another, different client, there will be no local credential and it will behave incorrectly.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/11399

[Ian Booth (wallyworld)]

It's a small client fix so when the PR hits the 2.8 edge snap, it can be easily tested to confirm it works for you.