Juju agent in a "failed" state after machine reboot on some charms

[Pen Gale]

We ran into this while building and testing Matrix, but it appears to be reproducible outside of it.

To reproduce:

1) Deploy wiki-simple (or just mysql or the wiki charm).
2) juju run 'sudo reboot' on one of the units.
3) Note that the controller marks the unit as being in a "failed" state.

This happens even in the latest Juju 2.1 beta.

The interesting thing is that it doesn't happen for all charms. The latest 'zookeeper' charm from the store reboots without incident, for example. There are also no big obvious errors in the machine logs for the unit that the controller has marked as failed -- the agent appears to be in good shape, despite the controller thinking that it has failed.

[Pen Gale (pengale)]

Controller logs from a repro of this bug, for reference: http://paste.ubuntu.com/23624621/

[Anastasia (anastasia-macmood)]

If it does not happen for all the charms, only for latest 'zookeeper' one, maybe the problem is not with juju but with the charm?

[Cory Johns (johnsca)]

Anastasia,

That's backwards; it's fine on ZK but fails for both mysql and mediawiki. It's noteworthy that both mediawiki and mysql are trusty, while ZK is xenial. To test that further, I deployed the same charm, cs:ubuntu-8, on both xenial and trusty, and ran `juju run $unit 'sudo reboot'` on both, and the trusty version failed while the xenial version did not.

ubuntu-trusty/0* active failed 4 10.171.42.173 ready
ubuntu-xenial/0* active idle 3 10.171.42.78 ready

[Anastasia (anastasia-macmood)]

Re-targeted to "juju" project that tracks Juju 2.x issues.

[Anastasia (anastasia-macmood)]

Could you please supply output of juju status? Is it agent or charm status reporting failure?

[Cory Johns (johnsca)]

Full output of `juju status --format=yaml`: http://pastebin.ubuntu.com/23626030/

It's the juju-status field, and the message is "resolver loop error"

[Tim Penhey (thumper)]

Removing the milestone to stop this just getting punted.

How often does it happen? I'm trying to get an understanding of frequency.

[Tim Penhey (thumper)]

Possibly related to bug 1694734

[Tim Penhey (thumper)]

Actually I'm pretty sure this is related, almost certainly timing related which is why it sometimes appears. Making this bug as the dupe as the other has more info.

[Cory Johns (johnsca)]

I can reproduce it with 100% consistency (or near enough that I can't tell) on trusty units with the lxd provider, but it does only happen with `juju run` and not `juju ssh`, so it being related to action status transactions seems reasonable.