1.22 cannot upgrade to 1.26-alpha1: run.socket: no such file or directory

[Curtis Hovey]

Upgrades from 1.22.6 to 1.26-alpha1 fail:
    http://reports.vapour.ws/releases/3039/job/aws-upgrade-22-trusty-amd64/attempt/123
    http://reports.vapour.ws/releases/3039/job/joyent-upgrade-22-trusty-amd64/attempt/127

1.22.6: dummy-source/0
2015-09-09 05:54:29 ERROR juju run on machine 1 returned 1: error: dial unix /var/lib/juju/agents/unit-dummy-source-0/run.socket: no such file or directory

[Curtis Hovey (sinzui)]

The suspect commits are:
    https://github.com/juju/juju/commit/7455bb02b02d23c8413fd650124ae72a239581b6
    https://github.com/juju/juju/commit/b1efe7aaf4a8afaeb958ff04d8383089317e8f64
    https://github.com/juju/juju/commit/c93d9528c072384bfe30fa25732b2a79a06c88fc
    https://github.com/juju/juju/commit/370a5f1c2a395e5a28476860859ef9db0ec14501

[Martin Packman (gz)]

This also causes the OS-deployer job to fail:

<http://reports.vapour.ws/releases/3039/job/OS-deployer/attempt/247>

  swift-storage-z3:
    charm: local:trusty/swift-storage-92
    exposed: false
    service-status:
      current: error
      message: 'hook failed: "update-status"'

From the unit-swift-storage-z3-0.log:

2015-09-09 09:44:47 ERROR juju.juju.sockets sockets_nix.go:22 failed to listen on unix:@/var/lib/juju/agents/unit-swift-storage-z3-0/agent.socket: listen unix @/var/lib/juju/agents/unit-swift-storage-z3-0/agent.socket: bind: address already in use
2015-09-09 09:44:48 ERROR juju.worker.uniter.operation runhook.go:108 hook "update-status" failed: listen unix @/var/lib/juju/agents/unit-swift-storage-z3-0/agent.socket: bind: address already in use

[Casey Marshall (cmars)]

http://reviews.vapour.ws/r/2617/

[Casey Marshall (cmars)]

Update: http://reviews.vapour.ws/r/2620/

So it turns out that unit agent upgrades aren't actually getting applied, possibly since ever. To confirm this, try running a 1.22 -> 1.26 upgrade (pre-maltese-falcon landing, if you like) with some logging to updates/steps123, after the unit tag check. It never gets executed.

Landed a workaround that handles the potential for inconsistent unit local state, as a stop-gap until we can fix unit agent upgrades.

[Ian Booth (wallyworld)]

New CI test run now passes.
http://juju-ci.vapour.ws:8080/job/aws-upgrade-22-trusty-amd64/127/

Marking as fix committed.

[Martin Packman (gz)]

Have we got a follow up bug for having upgrade steps that never run?