Cannot deploy charms in jes envs

[Curtis Hovey]

This bug is like bug 1513552, but is limited to deployments with the jes feature flag set. Li

As seen in
    http://reports.vapour.ws/releases/3307/job/functional-jes/attempt/262

Charms are in error.
       cannot assign unit "dummy-sink/0" to machine: cannot assign unit "dummy-sink/0"
        to new machine or container: cannot assign unit "dummy-sink/0" to new machine:
        unit is already assigned to a machine

This job tests juju by bootstrapping with the jes feature flag, then deploys two trivial stacks to two different environments. The error happens during the deployment of unctional-jes-env1. The test never get to deploy unctional-jes-env2.

[Nate Finch (natefinch)]

The real error appears to be

2015-11-13 19:01:10 INFO juju.worker runner.go:275 stopped "unitassigner", err: "4873074d-7936-4e66-89dd-4ffb6e4d7f3b:dummy-source/0" is not a valid unit id

I think this is a difference with how tags and unit ids are converted with jes enabled.

[Tim Penhey (thumper)]

The ids are not changed at all with the flag on or off.

[Menno Finlay-Smits (menno.smits)]

The problem is that the collectionwatcher is used without filtering by environment. The unitassigners for each environment are seeing unit ids for the other environments.

[Menno Finlay-Smits (menno.smits)]

Fix is here: https://github.com/juju/juju/pull/3737

[Nate Finch (natefinch)]

Trying to reproduce the problem, but struggling a lot with getting CI happy so it can run locally. Manual attempts to reproduce the bug were unsuccessful.

[Andrew Wilkins (axwalk)]

I'm not sure if this is *the* problem, but I can see *a* problem with the current unit assigner code in state.

Unit.assignToMachineOps calls the removeStagedAssignmentOp method to get an op that removes the unit-assignment doc. Unit.assignToNewMachine does not do that. The unit assigner changes have changed the behaviour of unit-to-machine assignment, which has uncovered another issue with the assignToNewMachine method -- it doesn't add storage attachments to the new machine.

I'm not entirely sure how this would have triggered, but if the machine agent restarts it's going to see old unit assignment docs in the collection still, and try to assign them again.

[Andrew Wilkins (axwalk)]

https://github.com/juju/juju/pull/3768

If someone could review and land (if it's good), that'd be good. I have to go out now.

[Dimiter Naydenov (dimitern)]

I've reviewed axw's PR and since it looks reasonable, I've set it to land.

[Canonical Juju QA Bot (juju-qa-bot)]

Juju-CI verified that this issue is Fix Released in juju-core master:
    http://reports.vapour.ws/releases/3325

[Nate Finch (natefinch)]

I reviewed Andrew's commit post-submit, and I also think it looks good. Thanks for finding that!