juju stop responding after juju-upgrade

[lithium]

After running:

juju upgrade-juju --version 1.23-beta2 --upload-tools

All unit start saying in debug-log

....
unit-pgsql-medium-0[9015]: 2015-03-31 00:24:01 ERROR juju.worker runner.go:219 exited "uniter": failed to initialize uniter for "unit-pgsql-medium-0": cannot read "/var/lib/juju/agents/unit-pgsql-medium-0/state/uniter": invalid operation state: unexpected hook info with Kind Continue
....

After a while juju stop responding. Ex: juju status

I already try to restart juju-machine0 service and juju-mongodb

If i keep running "juju status" in a console and restart juju-machine0 service from machine0 it show statuses and then hang again. As if there were something in the queue.

Attached extra big large /var/log/juju/machine-0.log

[lithium (rudicba)]

I forget to upload all-machine.log

[John Weldon (johnweldon4)]

It seems the upgrade step for adding the Stopped field to the uniter state isn't working. The logs indicate that the upgrade steps ran, but many (most / all ?) of the units all report the error that the uniter state has a hook value in ModeContinue, which the upgrade step should have taken care of.

[John Weldon (johnweldon4)]

Once the uniter operation state file is manually fixed (by removing the hook key and subkeys) another error that seems upgrade related appears too:

unit-jw-charm-test-0[10017]: 2015-03-31 17:09:18 ERROR juju.worker.uniter.filter filter.go:137 cannot retrieve meter status for unit jw-charm-test/0: not found
unit-jw-charm-test-0[10017]: 2015-03-31 17:09:19 ERROR juju.worker runner.go:219 exited "uniter": cannot retrieve meter status for unit jw-charm-test/0: not found

[John Weldon (johnweldon4)]

so the issue is that the upgrade steps do not run against units, but only against machines. The upgrade steps need to explicitly iterate the units and apply the upgrade to the units.

I'm working on implementing this, but until it's in this upgrade won't work.

[lithium (rudicba)]

So, when the new update is available, juju environment with wrong uniter state will be fixed? or i have to start from scratch?

Thanks for the quick response

[Liam Young (gnuoy)]

I've just upgraded from 1.22.1 to 1.23.0 (from proposed ppa) and after the upgrade I seem to have hit this bug.

Last entry in debug-log is:

unit-swift-proxy-0[8375]: 2015-04-17 10:20:02 ERROR juju.cmd supercommand.go:430 must restart: an agent upgrade is available
unit-swift-proxy-0[4189]: 2015-04-17 10:20:03 ERROR juju.worker.uniter.operation state.go:137 unexpected hook info with Kind Continue

juju set no longer fires hooks on the units.

I did not use upload-tools for the upgrade. I did:

juju set-env agent-metadata-url=https://streams.canonical.com/juju/tools
juju set-env agent-stream=proposed
juju upgrade-juju --version 1.23.0

[John Weldon (johnweldon4)]

Even after the committed fix, hook operations don't fire any more after the upgrade. Investigating further.

[Martin Packman (gz)]

I can reproduce this with our standard upgrade job by adding a step at the end that sets a new value on our testing charm and expects it to propagate across a relation. It seems any juju deployment upgraded from 1.22 to 1.23 will basically be in an unusable state afterwards.

[Curtis Hovey (sinzui)]

As there is bot a fix and a release with this issue. I opened bug 1447846 to resolve the hooks issue.

[Menno Finlay-Smits (menno.smits)]

John, can you confirm that the fix is made it into the 1.24 branch as well as master? (and update this bug accordingly)

[John Weldon (johnweldon4)]

Yes, this fix was in revision [2fb4df33](https://github.com/johnweldon/juju/commit/2fb4df3367c061d408a6ec8325f13f5e2a74f8aa)

running `git branch -r --contains 2fb4df33` will confirm that the revision is in both master and 1.24