Upgrade fails because product json is renamed

[Martin Packman]

Upgrade ci jobs are broken on master at present:

<http://reports.vapour.ws/releases/2120>

machine-0: 2014-11-26 23:33:14 ERROR juju.worker.upgrader upgrader.go:157 failed to fetch tools from "http://juju-dist.s3.amazonaws.com/testing/tools/releases/juju-1.22-alpha1-precise-amd64.tgz": cannot unpack tools: tarball sha256 mismatch, expected 1d45ea821ab50e719df68d7a887449ff5f63f7d3347b665f2e522a99703a2577, got 00c0b630d88383bf0866a5a9846b1b945cac1936b88242a163d2d84a6eed83aa

<http://data.vapour.ws/juju-ci/products/version-2120/aws-upgrade-precise-amd64/build-2103/all-machines.log.gz>

This is somewhere after revsion f110dd9dc7a172fa07d1fb1897d1832aef1dfcc3 and before 9fbb70d8bc83212b75c320aee008360deb7254d1 which is from pr #1211 and pr #1218 landing.

This is caused by a change to product file names in the metadata. The old index.json is pointing to the old product file which doesn't match the information in the new product file. The shasum issue will only be seen in volatile streams like testing and weekly. The shasum issue a manifestation that another problem, that we don't have a strategy to preserve upgrade from 1.20 to 1.22. User and enterprises with private streams cannot upgrade with rewriting metadata by hand.

[Martin Packman (gz)]

Caused by the changes for windows filename compatibility:

https://github.com/juju/juju/pull/908

     // ProductMetadataPath returns the tools product metadata path for the given stream.
     func ProductMetadataPath(stream string) string {
    - return fmt.Sprintf("streams/v1/com.ubuntu.juju:%s:tools.json", stream)
    + return fmt.Sprintf("streams/v1/com.ubuntu.juju-%s-tools.json", stream)
     }

In the upgrade testing, we end up with both the colon version and the dash version.

[Ian Booth (wallyworld)]

Having both files shouldn't matter as the old juju will read the ":" version and new juju will read the "-" version. We'd really want to see the json files to understand what's going on. It's also unclear why there are being generated two tools tarballs with the same name but different checksums. That normally indicates a deeper systemic issue.

[Ian Booth (wallyworld)]

I can't reproduce this. The steps I took:

1. bootstrap a Juju 1.20.13 environment with config
tools-metadata-url=http://juju-dist.s3.amazonaws.com/testing/tools

From looking at the build logs, it seemed that the above tools-metata-url was what was used. And indeed, there was metadata at that location for 1.22-alpha1 tools.

2. Attempt to upgrade
juju upgrade-juju --version=1.22-alpha1

This fails with "no matching tools available".

The above message makes sense. The tools are searched for by the current 1.20 agent. The simplestreams metadata that's at the tools url location doesn't have metadata for 1.22 tools. There's new metadata there which does - this is in the index2.json file. But Juju 1.20 doesn't know about this file.

So I'm not sure what steps were followed to get a 1.20 -> 1.22 upgrade going without the correct tools metadata being in place.

I also checked with Tim - he's been testing 1.20 -> 1.22 upgrades all week, albeit using --upload-tools, and there's been no problem.

I still can't see how the new metadata file names are to blame here, and without clarity on how to reproduce the issue, I'm marking as Incomplete.

[Aaron Bentley (abentley)]

We ran publish_revision.py, from lp:juju-ci-tools. This in turn runs various scripts from lp:juju-release-tools, notably assemble-streams.bash and publish-public-tools.bash.

When used with the latest juju, this script uploads
com.ubuntu.juju-released-tools.json
index.json
index2.json
But NOT com.ubuntu.juju:released:tools.json, presumably because juju did not create it.

Example output from http://juju-ci.vapour.ws:8080/job/publish-revision/1244/consoleFull

Phase 1: Publishing testing to AWS.
File '/var/lib/jenkins/streams/juju-dist/testing/tools/releases/juju-1.22-alpha1-precise-amd64.tgz' stored as 's3://juju-dist/testing/tools/releases/juju-1.22-alpha1-precise-amd64.tgz' (9183095 bytes in 0.4 seconds, 19.91 MB/s) [1 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-amd64.tgz' stored as 's3://juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-amd64.tgz' (9183261 bytes in 0.5 seconds, 17.30 MB/s) [2 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-i386.tgz' stored as 's3://juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-i386.tgz' (8872568 bytes in 0.6 seconds, 14.85 MB/s) [3 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-ppc64.tgz' stored as 's3://juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-ppc64.tgz' (10251430 bytes in 0.5 seconds, 20.55 MB/s) [4 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-ppc64el.tgz' stored as 's3://juju-dist/testing/tools/releases/juju-1.22-alpha1-trusty-ppc64el.tgz' (10251430 bytes in 0.5 seconds, 18.65 MB/s) [5 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/releases/juju-1.22-alpha1-utopic-amd64.tgz' stored as 's3://juju-dist/testing/tools/releases/juju-1.22-alpha1-utopic-amd64.tgz' (9183261 bytes in 0.6 seconds, 15.64 MB/s) [6 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/streams/v1/com.ubuntu.juju-released-tools.json' stored as 's3://juju-dist/testing/tools/streams/v1/com.ubuntu.juju-released-tools.json' (336988 bytes in 0.1 seconds, 4.58 MB/s) [7 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/streams/v1/index.json' stored as 's3://juju-dist/testing/tools/streams/v1/index.json' (1802 bytes in 0.0 seconds, 43.57 kB/s) [8 of 9]
File '/var/lib/jenkins/streams/juju-dist/testing/tools/streams/v1/index2.json' stored as 's3://juju-dist/testing/tools/streams/v1/index2.json' (2111 bytes in 0.1 seconds, 29.49 kB/s) [9 of 9]
Done. Uploaded 57265946 bytes in 3.3 seconds, 16.70 MB/s

[Ian Booth (wallyworld)]

If index.json is uploaded, then the ":" product files must also be uploaded. The older index.json file will have the product file references with ":", since the index2.json file was in place prior to the change to use "-". If just the index file is uploaded, then any attempt to use that index file will fail, because the product files it references will be missing.
Note that Juju versions which know about "-" product files also look at index2.json files, and only if no metadata is found will fall back to index.json files. However, older versions of juju will fail to locate any metadata.

[Curtis Hovey (sinzui)]

We need to discuss who is affected by this issue to decide which team makes a fix where.

We froze index.json to point to the coloned released product file because juju 1.21 writes index2.json. Both index files would point to the *same* released product file. This ensure old juju found the current products and only saw the released versions.

As of 1.22 the two indexes no longer point to the same file, so we see an upgrade error. If this change was in production, users will be told that 1.22 doesn't exist (volatile streams, like testing and weekly see a shasum error because of rewites)

If this is just a Canonical problem, the qa team can change the index.json to point to the dashed product file. This is not trivial because multiples jujus are creating streams in testing and *production*, a fix requires that us to force old juju product file to take the name of the new juju files. Maybe we just want to ensure anything creating a dashed product file will also create coloned version too for upgrade purposes.

This might be a problem for enterprises with private streams. If an enterprise is on 1.20 and chooses to skip 1.21, jumping to 1.22, they cannot upgrade without changes to their scripts or manual intervention. 1.20 still sees the coloned released product file which was not updated with the new agents. Someone must rewite index.json or copy the dashed file over the coloned file.

But this is a different problem for an enterprise migrating from --upload-tools to private streams. they never had an index.json. In this case, the fix is to create an index.json. If index2.json has one stream, listed, the enterprise just copies index.json to index2.json. If there are more streams in index2.json, surgery is needed on the json to remove the extra entries.

Also consider that the inverse is also an issue, when old juju regenerates streams *after* new juju created streams, the new juju will have a stale streams. New juju wont see the new product file :(

So, generate-tool will need to create a backward compatible index.json to point to the new product file will allow new juju to remain compatible with juju 1.18 and 1.20. A compatible index.json only contains the "released" stanza and product file. No other stanzas.

Or revert this change. Solve the file name on windows could be accommodate be escaping.

[Ian Booth (wallyworld)]

After discussing with Curtis, the generate tools metadata utility will be modified to make both index.json and index2.json. The index.json will point to the "-" files, but will just contain metadata for the "released" stream tools.

[Curtis Hovey (sinzui)]

sigh :( We need this backported to 1.21-beta4. All jujus need to make index.json. If not, we see the inverse case from the initial report. testing with 1.22 we get a dashed file, then downgrade to 1.21, and the index still points to a dashed file, but the data in in the coloned file.