api-endpoints fails if run just after bootstrap

[Free Ekanayaka]

To reproduce:

juju bootstrap -e maas --to my.node && juju api-endpoints; echo $?

and the output looks like:

Launching instance
<... rest of bootstrap log ...>
Starting Juju machine agent (jujud-machine-0)
ERROR EOF
1

This is a regression introduced in 1.20.0, it was working fine in 1.19.4. It seems to happen only when using the MAAS provider, not when using the Amazon one.

The diff between 1.19.4 and 1.20.0 is very small:

http://paste.ubuntu.com/7747675/

so I'm not really sure what could be the cause.

The workaround is to put a little sleep before running "juju api-endpoints".

[Frank Mueller (themue)]

Same error with local provider, but only once.So maybe a race condition due to loading of dependent software packages.

[Free Ekanayaka (free.ekanayaka)]

Not sure if it matters, but the MAAS server I used to trigger this bug was in a VPN. Didn't try with a local provider, but with that MAAS server I could reproduce the bug fairly consistently (it failed at every run I attempted).

[Andreas Hasenack (ahasenack)]

I reproduced it immediately with a local maas:

andreas@nsn7:~$ export JUJU_ENV=kvm-maas
andreas@nsn7:~$ juju bootstrap --to 43nhg.maaslocal && juju api-endpoints; echo $?
Launching instance
WARNING picked arbitrary tools &{1.20.0-trusty-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.0-trusty-amd64.tgz 69ad204e52ed88c2bcbb60ebcde9bc1137f6db4ceed4b9ae3807ed8392627524 8054312}
 - /MAAS/api/1.0/nodes/node-a24cb932-02ba-11e4-81f2-525400d76d7e/
Waiting for address
Attempting to connect to 43nhg.maaslocal:22
Attempting to connect to 10.0.5.70:22
Warning: Permanently added '10.0.5.70' (ECDSA) to the list of known hosts.
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: git
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' -o $bin/tools.tar.gz 'https://streams.canonical.com/juju/tools/releases/juju-1.20.0-trusty-amd64.tgz'
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
ERROR EOF
1
andreas@nsn7:~$

After a while, api-endpoints works:

andreas@nsn7:~$ juju api-endpoints
10.0.5.70:17070
43nhg.maaslocal:17070

[Frank Mueller (themue)]

Found how to simulate the same problem for the local provider, seems to be a more general race condition when the initialization of MongoDB and the opening of the state inside the machine agent have a too long interval.

[Andrew Wilkins (axwalk)]

Turned out that there was a bug that caused mongo to be restarted briefly after startup. This has been rectified on trunk and in the 1.20 branch for a pending 1.20.1 release.