juju needs to correctly report failure to clean up manual machine

[james beedy]

Using the manual provider, and juju-2.0 -> http://paste.ubuntu.com/23549833/

I'm getting this error after adding and removing a machine multiple times -> http://paste.ubuntu.com/23549809/

I've tried ssh'ing in and 'rm -rf /var/{log,lib}/juju`, and `sudo deluser --remove-all-files ubuntu` to no avail.

[Richard Harding (rharding)]

Looks like this is mongodb state.

Per
https://github.com/juju/juju/blob/6cf1bc9d917a4c56d0034fd6e0d6f394d6eddb6e/environs/manual/provisioner.go#L167

It's looking if it's provisioned and that's using the hostname as the nonce for the document below:

https://github.com/juju/juju/blob/ef17f71281d245540a9e5ed54a00095610ac0797/state/machine.go#L1450

Which means that document didn't get cleaned with removing of the machine at some point.

Not sure what failed on the clean up end.

Do you have any notes on how the machine was added/removed leading up to this?

[james beedy (jamesbeedy)]

Controller logs

[james beedy (jamesbeedy)]

Update

Whilst troubleshooting this issue, I killed my controller and rebootstrapped to the same machine (these machines are not ephemeral), to my dismay, I still get this error even after bootstrapping a new controller :-(

[james beedy (jamesbeedy)]

debug output from add-machine -> http://paste.ubuntu.com/23549954/

[Richard Harding (rharding)]

Update, I was wrong. It was hitting that there were left over upstart init scripts for

jujud-machine-17
jujud-unit-elasticsearch-5

Those were left over causing it to not be able to add-machine again.

I think we should make sure to make the logging clear on what we found as well as checking the code that removes those for how we might have had them left over.

[james beedy (jamesbeedy)]

manually removing /etc/init/juju* on the target machine fixes

[Curtis Hovey (sinzui)]

See http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/trunk/view/head:/remove-manual-juju.bash for an example of how to properly check and clean if juju left something behind after a manual provisioning.

This issue might relate to bug 1642295 which CI sees from time to time.

[Curtis Hovey (sinzui)]

SO to be clear, Juju *correctly* reported the host was provisioned. It neglected to state that this jight be because Juju failed to clean up.

[Anastasia (anastasia-macmood)]

Triaged to medium as there a manual workaround.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.