juju 1.21-alpha1, unable to bootstrap to maas with --constraints arch=armhf / arm64

[Sean Feole]

Hello, I'm testing out juju 1.21-alpha1 in attempt to talk to our maas server that hosts all of our armhf / arm64 nodes. I am unable to bootstrap to them nor start a machine with --constraints arch=arm64

Here is an example below: I had a similar problem in juju 1.20.X ref bug: https://bugs.launchpad.net/juju-core/+bug/1362856

ii juju-core 1.21-alpha1-0ubuntu1~14.04.1~juju1 amd64 Juju is devops distilled - client

$ juju bootstrap --debug --constraints arch=armhf
2014-10-17 03:28:02 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha1-trusty-amd64 gc]
2014-10-17 03:28:03 DEBUG juju.provider.maas environprovider.go:30 opening environment "maas".
2014-10-17 03:28:03 DEBUG juju.environs.configstore disk.go:330 writing jenv file to /home/sfeole/.juju/environments/maas.jenv
2014-10-17 03:28:03 DEBUG juju.environs.simplestreams simplestreams.go:389 fetchData failed for "images/streams/v1/index.sjson": file 'cf0e49a9-d45d-4bbe-8d3c-61f2082a5ddf-images/streams/v1/index.sjson' not found not found
2014-10-17 03:28:03 DEBUG juju.environs.simplestreams simplestreams.go:363 cannot load index "streams/v1/index.sjson": invalid URL "images/streams/v1/index.sjson" not found
2014-10-17 03:28:03 DEBUG juju.environs.simplestreams simplestreams.go:389 fetchData failed for "images/streams/v1/index.json": file 'cf0e49a9-d45d-4bbe-8d3c-61f2082a5ddf-images/streams/v1/index.json' not found not found
2014-10-17 03:28:03 DEBUG juju.environs.simplestreams simplestreams.go:363 cannot load index "streams/v1/index.json": invalid URL "images/streams/v1/index.json" not found
2014-10-17 03:28:03 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-17 03:28:03 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-17 03:28:04 DEBUG juju.environs.simplestreams simplestreams.go:389 fetchData failed for "http://cloud-images.ubuntu.com/releases/streams/v1/mirrors.sjson": cannot find URL "http://cloud-images.ubuntu.com/releases/streams/v1/mirrors.sjson" not found
2014-10-17 03:28:04 DEBUG juju.environs.simplestreams simplestreams.go:466 no mirror index file found
2014-10-17 03:28:04 DEBUG juju.environs.simplestreams simplestreams.go:367 read metadata index at "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
2014-10-17 03:28:04 DEBUG juju.environs.simplestreams simplestreams.go:534 candidate matches for products ["com.ubuntu.cloud:server:12.10:amd64" "com.ubuntu.cloud:server:12.10:i386" "com.ubuntu.cloud:server:12.10:armhf" "com.ubuntu.cloud:server:12.10:arm64" "com.ubuntu.cloud:server:12.10:ppc64el" "com.ubuntu.cloud:server:13.10:amd64" "com.ubuntu.cloud:server:13.10:i386" "com.ubuntu.cloud:server:13.10:armhf" "com.ubuntu.cloud:server:13.10:arm64" "com.ubuntu.cloud:server:13.10:ppc64el" "com.ubuntu.cloud:server:14.10:amd64" "com.ubuntu.cloud:server:14.10:i386" "com.ubuntu.cloud:server:14.10:armhf" "com.ubuntu.cloud:server:14.10:arm64" "com.ubuntu.cloud:server:14.10:ppc64el" "com.ubuntu.cloud:server:win2012hvr2:amd64" "com.ubuntu.cloud:server:win2012hvr2:i386" "com.ubuntu.cloud:server:win2012hvr2:armhf" "com.ubuntu.cloud:server:win2012hvr2:arm64" "com.ubuntu.cloud:server:win2012hvr2:ppc64el" "com.ubuntu.cloud:server:win2012hv:amd64" "com.ubuntu.cloud:server:win2012hv:i386" "com.ubuntu.cloud:server:win2012hv:armhf" "com.ubuntu.cloud:server:win2012hv:arm64" "com.ubuntu.cloud:server:win2012hv:ppc64el" "com.ubuntu.cloud:server:win2012r2:amd64" "com.ubuntu.cloud:server:win2012r2:i386" "com.ubuntu.cloud:server:win2012r2:armhf" "com.ubuntu.cloud:server:win2012r2:arm64" "com.ubuntu.cloud:server:win2012r2:ppc64el" "com.ubuntu.cloud:server:win81:amd64" "com.ubuntu.cloud:server:win81:i386" "com.ubuntu.cloud:server:win81:armhf" "com.ubuntu.cloud:server:win81:arm64" "com.ubuntu.cloud:server:win81:ppc64el" "com.ubuntu.cloud:server:12.04:amd64" "com.ubuntu.cloud:server:12.04:i386" "com.ubuntu.cloud:server:12.04:armhf" "com.ubuntu.cloud:server:12.04:arm64" "com.ubuntu.cloud:server:12.04:ppc64el" "com.ubuntu.cloud:server:13.04:amd64" "com.ubuntu.cloud:server:13.04:i386" "com.ubuntu.cloud:server:13.04:armhf" "com.ubuntu.cloud:server:13.04:arm64" "com.ubuntu.cloud:server:13.04:ppc64el" "com.ubuntu.cloud:server:14.04:amd64" "com.ubuntu.cloud:server:14.04:i386" "com.ubuntu.cloud:server:14.04:armhf" "com.ubuntu.cloud:server:14.04:arm64" "com.ubuntu.cloud:server:14.04:ppc64el" "com.ubuntu.cloud:server:win2012:amd64" "com.ubuntu.cloud:server:win2012:i386" "com.ubuntu.cloud:server:win2012:armhf" "com.ubuntu.cloud:server:win2012:arm64" "com.ubuntu.cloud:server:win2012:ppc64el" "com.ubuntu.cloud:server:win7:amd64" "com.ubuntu.cloud:server:win7:i386" "com.ubuntu.cloud:server:win7:armhf" "com.ubuntu.cloud:server:win7:arm64" "com.ubuntu.cloud:server:win7:ppc64el" "com.ubuntu.cloud:server:win8:amd64" "com.ubuntu.cloud:server:win8:i386" "com.ubuntu.cloud:server:win8:armhf" "com.ubuntu.cloud:server:win8:arm64" "com.ubuntu.cloud:server:win8:ppc64el"] are [{Mon, 13 Oct 2014 14:14:07 +0000 products:1.0 image-ids joyent [{eu-ams-1 https://eu-ams-1.api.joyentcloud.com} {us-sw-1 https://us-sw-1.api.joyentcloud.com} {us-east-3 https://us-east-3.api.joyentcloud.com} {us-east-2 https://us-east-2.api.joyentcloud.com} {us-east-1 https://us-east-1.api.joyentcloud.com} {us-west-1 https://us-west-1.api.joyentcloud.com}] streams/v1/com.ubuntu.cloud:released:joyent.sjson [com.ubuntu.cloud:server:12.04:amd64 com.ubuntu.cloud:server:14.04:amd64]} {Fri, 17 Oct 2014 03:04:45 +0000 products:1.0 image-ids aws [{us-east-1 https://ec2.us-east-1.amazonaws.com} {ap-northeast-1 https://ec2.ap-northeast-1.amazonaws.com} {eu-west-1 https://ec2.eu-west-1.amazonaws.com} {ap-southeast-1 https://ec2.ap-southeast-1.amazonaws.com} {ap-southeast-2 https://ec2.ap-southeast-2.amazonaws.com} {us-west-2 https://ec2.us-west-2.amazonaws.com} {us-west-1 https://ec2.us-west-1.amazonaws.com} {sa-east-1 https://ec2.sa-east-1.amazonaws.com}] streams/v1/com.ubuntu.cloud:released:aws.sjson [com.ubuntu.cloud:server:13.10:amd64 com.ubuntu.cloud:server:13.04:amd64 com.ubuntu.cloud:server:12.10:i386 com.ubuntu.cloud:server:12.04:i386 com.ubuntu.cloud:server:14.04:i386 com.ubuntu.cloud:server:13.10:i386 com.ubuntu.cloud:server:10.04:amd64 com.ubuntu.cloud:server:10.04:i386 com.ubuntu.cloud:server:14.04:amd64 com.ubuntu.cloud:server:12.10:amd64 com.ubuntu.cloud:server:13.04:i386 com.ubuntu.cloud:server:11.10:amd64 com.ubuntu.cloud:server:12.04:amd64 com.ubuntu.cloud:server:11.10:i386 com.ubuntu.cloud:server:14.10:amd64 com.ubuntu.cloud:server:14.10:i386]} {Thu, 09 Oct 2014 06:58:34 +0000 products:1.0 image-ids brightbox [{gb1 https://api.gb1.brightbox.com}] streams/v1/com.brightbox:official.sjson [com.ubuntu.cloud:server:12.04:amd64 com.ubuntu.cloud:server:14.04:amd64 com.ubuntu.cloud:server:12.04:i386 com.ubuntu.cloud:server:14.04:i386]} {Mon, 29 Sep 2014 19:59:58 +0000 products:1.0 image-ids hpcloud [{region-b.geo-1 https://region-b.geo-1.identity.hpcloudsvc.com:35357/v2.0/} {region-a.geo-1 https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/}] streams/v1/com.ubuntu.cloud:released:hpcloud.sjson [com.ubuntu.cloud:server:12.04:amd64 com.ubuntu.cloud:server:13.10:amd64 com.ubuntu.cloud:server:14.04:amd64 com.ubuntu.cloud:server:12.10:amd64 com.ubuntu.cloud:server:14.10:amd64]} {Tue, 30 Sep 2014 16:27:06 +0000 products:1.0 image-ids azure [{Japan West https://management.core.windows.net/} {West Europe https://management.core.windows.net/} {East US 2 https://management.core.windows.net/} {Southeast Asia https://management.core.windows.net/} {South Central US https://management.core.windows.net/} {Australia Southeast https://management.core.windows.net/} {Australia East https://management.core.windows.net/} {East Asia https://management.core.windows.net/} {East US https://management.core.windows.net/} {Central US https://management.core.windows.net/} {West US https://management.core.windows.net/} {North Europe https://management.core.windows.net/} {North Central US https://management.core.windows.net/} {Japan East https://management.core.windows.net/} {China East https://management.core.chinacloudapi.cn/} {China North https://management.core.chinacloudapi.cn/} {Brazil South https://management.core.windows.net/}] streams/v1/com.ubuntu.cloud:released:azure.sjson [com.ubuntu.cloud:server:12.04:amd64 com.ubuntu.cloud:server:14.04:amd64 com.ubuntu.cloud:server:12.10:amd64 com.ubuntu.cloud:server:14.10:amd64]} {Tue, 14 Oct 2014 18:23:36 +0000 products:1.0 image-ids aws-govcloud [{us-gov-west-1 https://ec2.us-gov-west-1.amazonaws-govcloud.com}] streams/v1/com.ubuntu.cloud:released:aws-govcloud.sjson [com.ubuntu.cloud:server:13.10:amd64 com.ubuntu.cloud:server:12.04:i386 com.ubuntu.cloud:server:14.04:i386 com.ubuntu.cloud:server:13.10:i386 com.ubuntu.cloud:server:10.04:amd64 com.ubuntu.cloud:server:14.04:amd64 com.ubuntu.cloud:server:10.04:i386 com.ubuntu.cloud:server:12.04:amd64]} {Tue, 30 Sep 2014 05:56:56 +0000 products:1.0 image-ids aws-cn [{cn-north-1 https://ec2.cn-north-1.amazonaws-cn.com.cn}] streams/v1/com.ubuntu.cloud:released:aws-cn.sjson [com.ubuntu.cloud:server:13.10:amd64 com.ubuntu.cloud:server:12.04:i386 com.ubuntu.cloud:server:14.04:i386 com.ubuntu.cloud:server:13.10:i386 com.ubuntu.cloud:server:10.04:amd64 com.ubuntu.cloud:server:14.04:amd64 com.ubuntu.cloud:server:10.04:i386 com.ubuntu.cloud:server:12.04:amd64]}]
2014-10-17 03:28:04 DEBUG juju.environs.simplestreams simplestreams.go:847 finding products at path "streams/v1/com.ubuntu.cloud:released:joyent.sjson"
2014-10-17 03:28:04 INFO juju.utils http.go:59 hostname SSL verification enabled
2014-10-17 03:28:04 DEBUG juju.environs.simplestreams simplestreams.go:885 metadata: &{map[com.ubuntu.cloud:server:12.04:amd64:{precise 12.04 amd64 map[20140519:0xc21014c600 20140806:0xc21014c6c0 20140408:0xc21014c780 20140227:0xc21014c840 20140929:0xc21014c900 20140608:0xc21014c9c0 20140926:0xc21014ca80]} com.ubuntu.cloud:server:14.04:amd64:{trusty 14.04 amd64 map[20140519:0xc21014cba0 20140416:0xc21014cc60 20140729:0xc21014cd20 20140929:0xc21014cde0 20140608:0xc21014cea0 20140926:0xc21014cf60]}] map[crsn:map[euww1:map[region:eu-ams-1 endpoint:https://eu-ams-1.api.joyentcloud.com] ussw1:map[region:us-sw-1 endpoint:https://us-sw-1.api.joyentcloud.com] usee3:map[region:us-east-3 endpoint:https://us-east-3.api.joyentcloud.com] usee2:map[region:us-east-2 endpoint:https://us-east-2.api.joyentcloud.com] usee1:map[region:us-east-1 endpoint:https://us-east-1.api.joyentcloud.com] usww1:map[region:us-west-1 endpoint:https://us-west-1.api.joyentcloud.com]]] Mon, 13 Oct 2014 14:14:07 +0000 products:1.0 com.ubuntu.cloud:released:joyent }
2014-10-17 03:28:04 INFO juju.cmd cmd.go:113 Bootstrap failed, destroying environment
2014-10-17 03:28:04 INFO juju.provider.common destroy.go:15 destroying environment "maas"
2014-10-17 03:28:04 ERROR juju.cmd supercommand.go:323 invalid constraint value: arch=armhf
valid values are: [amd64]

[Ian Booth (wallyworld)]

What version of MAAS are you using? Juju uses MAAS APIs to query the architectures of the nodes available to the cluster in order to validate that the requested arch can be satisfied. Those APIs are different for MAAS 1.5 vs 1.6 vs 1.7 etc.

[Sean Feole (sfeole)]

 I apologize Ian, I thought I had already added that info. I'm using maas 1.7 beta 6, the full build listed below:

ii maas 1.7.0~beta6+bzr3260-0ubuntu1~trusty1 all MAAS server all-in-one metapackage

[Ian Booth (wallyworld)]

For MAAS 1.7, Juju uses the nodegroups boot-images API call to gather a list of availabl boot images from which the arches are recorded. If this API calls returns 0 boot images, Juju will get confused.

[Julian Edwards (julian-edwards)]

MAAS 1.7 changed the way in which it imports images. They are now imported into the region controller's database and then synced across all the clusters later.

The API call above that Ian mentions queries what's on the clusters, not what is in the region. Therefore it's possible that the clusters didn't finish syncing, or the region didn't finish importing yet. 1.7 requires users to re-import images even though they may have had them already imported in a previous release before upgrading.

[Ian Booth (wallyworld)]

Sean, are you able to try Julian's suggestion above and see if that helps?

[Sean Feole (sfeole)]

I'll try to reimport my images, the maas ui does say that my latest set of images are (out-of-sync)

[Sean Feole (sfeole)]

Ok i was misled from my maas-ui , i'm hitting bug https://bugs.launchpad.net/maas/+bug/1378527 .. Could that be causing an issue? although the maas - cli says everything is synced.

ubuntu@ubuntu:/var/log/maas$ maas my-maas boot-resources read
[
    {
        "name": "ubuntu/trusty",
        "kflavor": "generic",
        "architecture": "amd64/hwe-t",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t",
        "type": "Synced",
        "id": 14,
        "resource_uri": "/MAAS/api/1.0/boot-resources/14/"
    },
    {
        "name": "ubuntu/trusty",
        "kflavor": "generic",
        "architecture": "arm64/hwe-t",
        "subarches": "generic,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t",
        "type": "Synced",
        "id": 16,
        "resource_uri": "/MAAS/api/1.0/boot-resources/16/"
    },
    {
        "name": "ubuntu/trusty",
        "kflavor": "generic",
        "architecture": "arm64/xgene-uboot",
        "subarches": "xgene-uboot",
        "type": "Synced",
        "id": 18,
        "resource_uri": "/MAAS/api/1.0/boot-resources/18/"
    },
    {
        "name": "ubuntu/trusty",
        "kflavor": "generic-lpae",
        "architecture": "armhf/generic-lpae",
        "subarches": "generic-lpae",
        "type": "Synced",
        "id": 19,
        "resource_uri": "/MAAS/api/1.0/boot-resources/19/"
    },
    {
        "name": "ubuntu/trusty",
        "kflavor": "generic",
        "architecture": "armhf/hwe-t",
        "subarches": "generic,highbank,hwe-p,hwe-q,hwe-r,hwe-s,hwe-t",
        "type": "Synced",
        "id": 17,
        "resource_uri": "/MAAS/api/1.0/boot-resources/17/"
    },
    {
        "name": "ubuntu/trusty",
        "kflavor": "keystone",
        "architecture": "armhf/keystone",
        "subarches": "keystone",
        "type": "Synced",
        "id": 15,
        "resource_uri": "/MAAS/api/1.0/boot-resources/15/"
    }
]

[Ian Booth (wallyworld)]

Is there a MAAS CLI tool that can be used to invoke the nodegroups boot-images API call to see what that returns? You'd need to made in the node group UUID I think. I think this would be a different internal API to what was used in the previous comment. Juju uses the boot-images API so if we can see what that returns, we can see if Juju is getting the information it needs.

[Curtis Hovey (sinzui)]

I think this bug is a duplicate of bug 1376906 which is about the arm& constraint gets machines/agents that cannot call home.