Invalid binary version, version "1.23.3--amd64" or "1.23.3--armhf"

[Jorge Niedbalski]

[Environment]

Vivid
Juju-core 1.23.3 ( client )
Juju-core 1.23.2 ( Bootstrap node with 1.23.3-trusty-amd64 )
Maas 1.5.4 on Arm ( raspberry pi 2)

[Description]

niedbalski@theos-mobile:~$ juju get-env | grep series
default-series: ""

I executed the following sequence.

$ juju bootstrap (successful)
$ juju sync-tools --debug --verbose --series=1.23

The following log entry is generated

machine-0.log
434572:2015-05-15 18:17:37 DEBUG juju.apiserver tools.go:119 sending error: 400 cannot store tools metadata: invalid binary version "1.23.3--amd64"

From this point if I try to execute add-machine or sync-tools again, i got

2015-05-26 22:09:19 ERROR juju.cmd supercommand.go:430 invalid binary version "1.23.3--armhf"

environment: maas-hq
machines:
  "0":
    agent-state: started
    agent-version: 1.23.2.1
    dns-name: neb4g.maas
    instance-id: /MAAS/api/1.0/nodes/node-5fa197d0-f812-11e4-9eb1-b827eb833655/
    series: trusty
    containers:
      0/lxc/7:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.8
        instance-id: juju-machine-0-lxc-7
        series: trusty
        hardware: arch=amd64
      0/lxc/8:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.14
        instance-id: juju-machine-0-lxc-8
        series: trusty
        hardware: arch=amd64
      0/lxc/9:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.15
        instance-id: juju-machine-0-lxc-9
        series: trusty
        hardware: arch=amd64
      0/lxc/10:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.17
        instance-id: juju-machine-0-lxc-10
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=16384M tags=use-fastpath-installer
    state-server-member-status: has-vote
  "7":
    agent-state: started
    agent-version: 1.23.2.1
    dns-name: j3gbx.maas
    instance-id: /MAAS/api/1.0/nodes/node-31909a92-f8e6-11e4-9eb1-b827eb833655/
    series: trusty
    containers:
      7/lxc/4:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.18
        instance-id: juju-machine-7-lxc-4
        series: trusty
        hardware: arch=amd64
      7/lxc/5:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.19
        instance-id: juju-machine-7-lxc-5
        series: trusty
        hardware: arch=amd64
      7/lxc/9:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.22
        instance-id: juju-machine-7-lxc-9
        series: trusty
        hardware: arch=amd64
      7/lxc/10:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.23
        instance-id: juju-machine-7-lxc-10
        series: trusty
        hardware: arch=amd64
      7/lxc/11:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.24
        instance-id: juju-machine-7-lxc-11
        series: trusty
        hardware: arch=amd64
      7/lxc/12:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.13
        instance-id: juju-machine-7-lxc-12
        series: trusty
        hardware: arch=amd64
      7/lxc/13:
        agent-state: started
        agent-version: 1.23.2.1
        dns-name: 10.1.1.25
        instance-id: juju-machine-7-lxc-13
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=16384M tags=use-fastpath-installer
  "21":
    agent-state-info: invalid binary version "1.23.3--armhf"
    instance-id: pending
    series: trusty

[Observations]

- The MAAS server is running a different architecture (arm) to the allocated machines (amd64)
- THe add-machine command started failing after performing sync-tools.

[Curtis Hovey (sinzui)]

This issue looks familiar. We know from other maas related bugs that machines and agents are not properly matched by maas unless the arch and series is explicitly passed to deploy or add-machine.

[Curtis Hovey (sinzui)]

This issue relates to and may be a duplicate of bug 1358219

[Jorge Niedbalski (niedbalski)]

Curtis,

I tried passing specific series via --series=trusty, it doesn't works anyways.

[Jorge Niedbalski (niedbalski)]

niedbalski@theos-mobile:~$ juju add-machine --series=trusty --constraints="arch=amd64"

  "22":
    agent-state-info: invalid binary version "1.23.3--armhf"
    instance-id: pending
    series: trusty

[Jorge Niedbalski (niedbalski)]

Also, on the database i can find these entries, so probably a constraint is not being considered on the upload
tools phase.

{
    "_id": "1.23.3--armhf",
    "version": "1.23.3--armhf",
    "size": NumberLong(11059752),
    "sha256": "b0cc9a4f0e0a4efd40785105552e535ae7cbed1bfd66fe9cb33d4894fe1eed4f",
    "path": "tools/1.23.3--armhf-b0cc9a4f0e0a4efd40785105552e535ae7cbed1bfd66fe9cb33d4894fe1eed4f",
    "txn-revno": NumberLong(2),
    "txn-queue": [
        "5556376d56027769f40241bf_2ed3b636"
    ]
}{
    "_id": "1.23.3-precise-i386",
    "version": "1.23.3-precise-i386",
    "size": NumberLong(11196031),
    "sha256": "bb959442b4d5eccea01a84307d3d4f2115ebee2ef8bfc29d7d3188615b530104",
    "path": "tools/1.23.3-precise-i386-bb959442b4d5eccea01a84307d3d4f2115ebee2ef8bfc29d7d3188615b530104",
    "txn-revno": NumberLong(2),
    "txn-queue": [
        "555637d756027769f4024268_86c84d66"
    ]
}{
    "_id": "1.23.3--i386",
    "version": "1.23.3--i386",
    "size": NumberLong(11196031),
    "sha256": "bb959442b4d5eccea01a84307d3d4f2115ebee2ef8bfc29d7d3188615b530104",
    "path": "tools/1.23.3--i386-bb959442b4d5eccea01a84307d3d4f2115ebee2ef8bfc29d7d3188615b530104",
    "txn-revno": NumberLong(2),
    "txn-queue": [
        "555637d856027769f402426d_1122fe3c"
    ]
}{
    "_id": "1.23.3-trusty-amd64",
    "version": "1.23.3-trusty-amd64",
    "size": NumberLong(11566458),
    "sha256": "007c62a742c974c3f082964f37b04c28d46345e4816a926c31f8bdef53000552",
    "path": "tools/1.23.3-trusty-amd64-007c62a742c974c3f082964f37b04c28d46345e4816a926c31f8bdef53000552",
    "txn-revno": NumberLong(2),
    "txn-queue": [
        "5556383f56027769f4024311_c070c0eb"
    ]
}{
    "_id": "1.23.3--amd64",
    "path": "tools/1.23.3--amd64-ca9452252708abc4538c09741ba00e7eaece36cf90d0afb53933a535fe2e3ef0",
    "sha256": "ca9452252708abc4538c09741ba00e7eaece36cf90d0afb53933a535fe2e3ef0",
    "size": NumberLong(11566328),
    "txn-queue": [

    ],
    "txn-revno": NumberLong(2),
    "version": "1.23.3--amd64"
}

As a workaround I removed this entries manually by running:

juju:PRIMARY> db.toolsmetadata.remove({'_id': {$regex: /1\.23\.3\-\-\.*/i}})

This solved the issue, then i re-ran the sync-tools

[Jorge Niedbalski (niedbalski)]

OK running the following command:

$ juju --debug -v sync-tools --version=1.23

Triggers this issue again:

copying 1.23.3-trusty-amd64 from https://streams.canonical.com/juju/tools/releases/juju-1.23.3-trusty-amd64.tgz
2015-05-27 20:41:40 INFO juju.environs.sync sync.go:175 copying 1.23.3-trusty-amd64 from https://streams.canonical.com/juju/tools/releases/juju-1.23.3-trusty-amd64.tgz
2015-05-27 20:41:40 INFO juju.environs.sync sync.go:186 downloading "testing" tools/releases/juju-1.23.3-trusty-amd64.tgz (https://streams.canonical.com/juju/tools/releases/juju-1.23.3-trusty-amd64.tgz)
downloading "testing" tools/releases/juju-1.23.3-trusty-amd64.tgz (https://streams.canonical.com/juju/tools/releases/juju-1.23.3-trusty-amd64.tgz)
2015-05-27 20:42:47 INFO juju.environs.sync sync.go:204 uploading tools/releases/juju-1.23.3-trusty-amd64.tgz (11295kB) to environment
uploading tools/releases/juju-1.23.3-trusty-amd64.tgz (11295kB) to environment
UPLOAD TOOLS %s https://10.1.1.3:17070/tools?binaryVersion=1.23.3-trusty-amd64&series=
2015-05-27 20:42:54 ERROR juju.cmd supercommand.go:430 tools upload failed: 400 ({"Tools":null,"DisableSSLHostnameVerification":false,"Error":{"Message":"cannot store tools metadata: invalid binary version \"1.23.3--amd64\"","Code":""}})

And the database got this broken entry again:

{
    "_id": "1.23.3-trusty-amd64",
    "path": "tools/1.23.3-trusty-amd64-007c62a742c974c3f082964f37b04c28d46345e4816a926c31f8bdef53000552",
    "sha256": "007c62a742c974c3f082964f37b04c28d46345e4816a926c31f8bdef53000552",
    "size": NumberLong(11566458),
    "txn-queue": [
        "55662c53560277053e092f77_f9c4df44"
    ],
    "txn-revno": NumberLong(2),
    "version": "1.23.3-trusty-amd64"
}{
    "_id": "1.23.3--amd64",
    "path": "tools/1.23.3--amd64-ca9452252708abc4538c09741ba00e7eaece36cf90d0afb53933a535fe2e3ef0",
    "sha256": "ca9452252708abc4538c09741ba00e7eaece36cf90d0afb53933a535fe2e3ef0",
    "size": NumberLong(11566328),
    "txn-queue": [

    ],
    "txn-revno": NumberLong(2),
    "version": "1.23.3--amd64"
}

[Jorge Niedbalski (niedbalski)]

After manually removing the entry as described on #5, I can add new machines in the environment:

  "23":
    agent-state: pending
    dns-name: hbpkx.maas
    instance-id: /MAAS/api/1.0/nodes/node-e8a0b09c-03e9-11e5-bff9-b827eb833655/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=use-fastpath-installer

[Tim Kuhlman (timkuhlman)]

I encountered this error on a machine running 1.22.8 while trying to run 'juju sync-tools' and fixed in a similar way

juju:PRIMARY> db.toolsmetadata.find({'_id': {$regex: /1\.24\.7\-\-\.*/i}})
{ "_id" : "1.24.7--amd64", "version" : "1.24.7--amd64", "size" : NumberLong(16431961), "sha256" : "e4c51217fc1d4793b58edc0cd3eca4d5d716e2c44ba91f4a880e44faff2b04c8", "path" : "tools/1.24.7--amd64-e4c51217fc1d4793b58edc0cd3eca4d5d716e2c44ba91f4a880e44faff2b04c8", "txn-revno" : NumberLong(2), "txn-queue" : [ "566076e2662068282c000114_72542e1a" ] }
juju:PRIMARY> db.toolsmetadata.remove({'_id': {$regex: /1\.24\.7\-\-\.*/i}})
juju:PRIMARY> db.toolsmetadata.find({'_id': {$regex: /1\.24\.7\-\-\.*/i}})

[Andrew Wilkins (axwalk)]

There was a bug in the tools-upload HTTP handler, which is fixed in 1.25 and master; we'll no longer add the invalid metadata, but we'll also fail earlier when attempting to add metadata into Mongo. Additionally, I've added an upgrade step to remove any invalid tools metadata for 1.25.

[Andrew Wilkins (axwalk)]

There is no upgrade step for 2.0, as we're requiring that the user upgrade from 1.25.2, which *will* have the upgrade step.

[Cheryl Jennings (cherylj)]

Looks like this was fixed in PR: https://github.com/juju/juju/pull/3933 for 1.25