juju ssh to wily/xenial broken

[Curtis Hovey]

As seen in
    http://reports.vapour.ws/releases/issue/563c4dc1749a5669a5aed8a7

CI's dummy charms exchange a random token when hooks fire. This allows the test know hooks fire and that charms can respond to them. Ci occasionally sees the exchange fail, but a recent change in master is making a single test fail, aws-deploy-trusty-amd64-on-xenial-amd64. For example:
    http://reports.vapour.ws/releases/3579/job/aws-deploy-trusty-amd64-on-xenial-amd64/attempt/186

Using an updated test script, we can see that "juju ssh" to the unit is failing. When the script falls back to ssh, it can read the token.

Changing the services to trusty works. changing the services to wily or xenial fails.

[Cheryl Jennings (cherylj)]

This is really weird. I see the token being set on the relation in the unit log, and the whole operation from the unit agent starting to the relation-set -r sink:0 token=1HTTTEGLXU8C60D9JT5M log entry is like 15 seconds, so I don't think it's the case that we just didn't wait long enough.

[Curtis Hovey (sinzui)]

I see 3 1.25 failures too. I think the host might be a factor.

[Curtis Hovey (sinzui)]

Logs with ---debug and JUJU_LOGGING_CONFIG='<root>=DEBUG' are at
    -ci.vapour.ws:8080/job/aws-deploy-trusty-amd64-on-xenial-amd64/220

[Curtis Hovey (sinzui)]

Logs with ---debug and JUJU_LOGGING_CONFIG='<root>=DEBUG' are at
    http://juju-ci.vapour.ws:8080/job/aws-deploy-trusty-amd64-on-xenial-amd64/220/

[Curtis Hovey (sinzui)]

The xenial image deployed on AWS is bad, the issue is fixed when bootstrapping with daily image steams. CI is currently working around the issue by the addition of a sleep 2 at the end of the script that gets the token.