bootstrap on maas with --metadata-source fails

[Curtis Hovey]

The maas lab cannot bootstrap juju. To debug the issue and try extending the timeout, Diogo pulled tip, to run juju locally. Using a created by CI, Diogo generated simple streams metadata to use with his cloud. Bootstrap fails after monodb-server is installed. Diogo attached gdp to the process to prevent the machine from being torndown so that he could read what was happening. It looks like the tools cannot be found.
    http://pastebin.ubuntu.com/6799799/

Though when tries:
    juju bootstrap --debug --metadata-source=/root/juju-simplestreams
or
   juju bootstrap --debug --upload-tools --metadata-source=/root/juju-simplestreams

This is the content of juju-simplestreams/tools/streams/v1/com.ubuntu.juju:released:tools.json
    http://pastebin.ubuntu.com/6799401/

[Diogo Matsubara (matsubara)]

I got more info about this bug:

Today I actually read the documentation in juju-core/doc/simplestreams-metadata.txt and there it explains that the --metadata-source option needs a local directory that's accessible by the nodes. The current help for --metadata-source says:
--metadata-source (= "")
    local path to use as tools and/or metadata source
Private clouds may need to specify their own custom image metadata, and possibly upload
Juju tools to cloud storage if no outgoing Internet access is available. In this case,
use the --metadata-source paramater to tell bootstrap a local directory from which to
upload tools and/or image metadata.

So I just assume that juju would make some kind of magic to make the local directory file accessible to the bootstrapped node. So at a minimum this bug is about improve that documentation, but this is not the extra info I want to add.
Following the instruction in simplestreams-metadata.txt, I set up a local webserver to serve the metadata files to juju.

Created the metadata directory under /var/www/juju-metadata
Copied the 1.17.1-precise-amd64.tgz tarball from http://juju-dist.s3.amazonaws.com
Generated the metadata using metadata generate-tools option

$ wget -c http://juju-dist.s3.amazonaws.com/testing/tools/releases/juju-1.17.1-precise-amd64.tgz
$ sha256sum tools/releases/juju-1.17.1-precise-amd64.tgz
3c3c04ec40c2f52d1e5f9fa356825c91ce9bbfce92b4142e44827f32d8289753 tools/releases/juju-1.17.1-precise-amd64.tgz
I used sudo so juju would have permission to write to /var/www/juju-metadata
$ sudo juju --debug metadata generate-tools -d juju-metadata
$ juju --version
1.16.5-trusty-amd64
$ sudo cat juju-metadata/tools/streams/v1/com.ubuntu.juju\:released\:tools.json
{
    "products": {
        "com.ubuntu.juju:12.04:amd64": {
            "version": "1.17.1",
            "arch": "amd64",
            "versions": {
                "20142301": {
                    "items": {
                        "1.17.1-precise-amd64": {
                            "release": "precise",
                            "version": "1.17.1",
                            "arch": "amd64",
                            "size": 19,
                            "path": "releases/juju-1.17.1-precise-amd64.tgz",
                            "ftype": "tar.gz",
                            "sha256": "b16e15764b8bc06c5c3f9f19bc8b99fa48e7894aa5a6ccdad65da49bbf564793"
                        }
                    }
                }
            }
        }
    },
    "updated": "Thu, 23 Jan 2014 13:01:51 -0500",
    "format": "products:1.0"

Notice how the sha256 differ but I continued anyway.
Changed the permission of the generated index.json and com.ubuntu.juju:released:tools.json so they will be accessible through the webserver
$ sudo chmod 644 juju-metadata/tools/streams/v1/*

I logged into the Trusty VM where I tried to bootstrap MAAS:
# juju bootstrap --debug --metadata-source=http://10.98.3.20/juju-metadata/tools

and got this failure in the node's /var/log/cloud-init-output.log
...
Adding new user `mongodb' (UID 107) with group `nogroup' ...
Not creating home directory `/home/mongodb'.
Adding group `mongodb' (GID 114) ...
Done.
Adding...

[Curtis Hovey (sinzui)]

Diogo, your first run generate metadata from the initial report was successful. I did suspect the metadata was wrong (the file size and sha256 are wrong when juju fails to read the files) but the example you provides are correct. I have learned to pass absolute paths to juju since it does not believe it is running on a computer run by humans.

Note that since several sets of tools have been created since the tar.gz was pulled down, so the file size you used in the followup test will not match subsequent replays of the test.

[Diogo Matsubara (matsubara)]

Hi Curtis,

I understand that I might be doing something wrong while reproducing the environment and that's ok. I'm still learning to debug juju failures. I did download the tools again every time I tried to reproduce the environment and verified the checksum of the downloaded tarball and the checksum generated by the `metadata generate-tools` command and they were always different.
The downloaded tarball had this hash: 3c3c04ec40c2f52d1e5f9fa356825c91ce9bbfce92b4142e44827f32d8289753 while the `metadata generate-tools` command had b16e15764b8bc06c5c3f9f19bc8b99fa48e7894aa5a6ccdad65da49bbf564793. Is it possible that the `metadata generate-tools` is actually hashing some other tarball than the one in the directory passed via -d option? Ok, I just confirmed your suggestion of using the absolute paths, and got the 3c3c04ec40c2f52d1e5f9fa356825c91ce9bbfce92b4142e44827f32d8289753 hash for the tool in the juju-metadata dir.

In any case, I think there are a couple of bugs here:

1. Juju bootstrap --debug doesn't output the whole cloud-init-output.log from the bootstrapped node, which makes debugging a lot harder, since juju stops the instance on failure.
2. I think would be nice to have an option to tell juju to not stop the instance in case of failure so one can do some analysis of what went wrong during its initialization
3. The `metadata generate-tools` -d option doesn't work correctly with relative paths.

Hope this helps.

[Eric Snow (ericsnowcurrently)]

This has impacted another user, though unrelated to maas:

  http://irclogs.ubuntu.com/2015/02/09/%23juju-dev.html#t09:43

I'll take a look to see if anything jumps out at me as the problem. I'll probably also update the bug summary.

[Anastasia (anastasia-macmood)]

Bug #1591488 is dealing with general failure to bootstrap with --metadata-source.

I am marking this as a duplicate as failure to bootstrap maas specifically with this flag is just another symptom.