jujud won't start if apt-get of juju-mongodb package fails

[James Troup]

If apt-get fails for any reason, jujud on machine 0 will go into an
infinite retry loop and never recover or even abort, e.g.:

  http://paste.ubuntu.com/10777463/

To reproduce:

 * Bootstrap an older (e.g. 1.18) environment
 * 'juju ssh 0' and ensure apt-get will fail; e.g. make it a symlink to /bin/false
 * 'juju upgrade-juju'

[Menno Finlay-Smits (menno.smits)]

This problem isn't upgrade specific. Juju runs "apt-get install juju-mongodb" every time it starts as part of ensuring that mongodb is ready to start. Another way to trigger it is:

juju bootstrap # with say 1.23
juju ssh 0
make apt-get point to /bin/false
sudo restart jujud-machine-0

The problem also isn't new. juju has had this functionality since 1.19. I can trigger the problem with any version since then.

One possible way to deal with this is to make juju retry the apt-get command a number of times (already in place) and if that fails continue on anyway, the logic being that in most cases a valid juju-mongodb will already be in place even if the apt-get command fails. We shouldn't let the apt-get failure prevent juju from at least trying to start up.

I'm not sure there's much more we can do beyond that given that mongodb is required in order for the machine agent to do anything.

I'll track down people at the sprint today to discuss further.

Updating the bug title to reflect what I've found.

[Menno Finlay-Smits (menno.smits)]

Fix for 1.23 here: https://github.com/juju/juju/pull/2182

[Menno Finlay-Smits (menno.smits)]

Fix for 1.24 here: https://github.com/juju/juju/pull/2183

[Menno Finlay-Smits (menno.smits)]

Fix for master here: https://github.com/juju/juju/pull/2184