maas upgrades do not complete

[Curtis Hovey]

We see the old maas (1.5) upgrade test and the new maas-devel (1.7) jobs consistently time out:
    http://juju-ci.vapour.ws:8080/job/maas-upgrade-trusty-amd64/
    http://juju-ci.vapour.ws:8080/job/maas-devel-upgrade-trusty-amd64/

We ran this locally can confirmed that after 30 minutes machines still said they were waiting for the api server:
    :26:51 DEBUG juju.apiserver apiserver.go:156 <- [601] <unknown> {"RequestId":1,"Type":"Admin","Request":"Login","Params":{"AuthTag":"unit-dummy-source-0","Password":"0ettuQAGyoQkdHobHcmqzi0u","Nonce":"unused"}}
2014-12-16 20:26:51 DEBUG juju.apiserver apiserver.go:163 -> [601] <unknown> 137.986us {"RequestId":1,"Error":"login for \"unit-dummy-source-0\" blocked because upgrade is in progress","Response":{}} Admin[""].Login
2014-12-16 20:26:51 INFO juju.apiserver apiserver.go:171 [601] <unknown> API connection terminated after 1.225763ms

[Curtis Hovey (sinzui)]

We can see that 1.21 last passed with commit 5d9c8dcb and the next commit 5d9c8dcb failed.

[John George (jog)]

Here are the steps that were taken to setup the environment.
The dummy-source and dummy-sink charms are charms we use with our tests.
This was done using KVM guests running a MaaS.

juju --show-log bootstrap -e my_maas
export JUJU_REPOSITORY=/home/jog/src/juju-ci-tools_repository/
======= Machine-1 not seen in the logs because a precise charm was tried and removed ========
juju --show-log deploy -e my_maas local:precise/dummy-source
juju --show-log remove-service -e my_maas dummy-source
juju --show-log remove-machine -e my_maas --force 1
==============================================================================
juju --show-log deploy -e my_maas local:trusty/dummy-source
juju --show-log deploy -e my_maas local:trusty/dummy-sink
juju --show-log status -e my_maas
juju --show-log add-relation -e my_maas dummy-source dummy-sink
juju --show-log expose -e my_maas dummy-sink
mkdir -p /home/jog/tmp/2178/extracted
cd /home/jog/tmp/2178
wget http://juju-ci.vapour.ws:8080/view/Release,%20package,%20publish/job/publish-revision/1296/artifact/juju-core_1.21-beta4-0ubuntu1%7E14.04.1%7Ejuju1_amd64.deb
dpkg -x juju-core_1.21-beta4-0ubuntu1~14.04.1~juju1_amd64.deb extracted
/home/jog/tmp/2178/extracted/usr/lib/juju-1.21-beta4/bin/juju --show-log upgrade-juju -e my_maas --upload-tools --version 1.21-beta4

[Menno Finlay-Smits (menno.smits)]

Fixed for master in a7dd525571bf720da355d2fb2396a9f3ae42d9d9

[Menno Finlay-Smits (menno.smits)]

Fixed for 1.21 in c515fb4c607e3f1b644175a3185515fef3ec8ee8

[Dimiter Naydenov (dimitern)]

Marking as Fix Released to unblock CI, as both upgrade jobs linked in the description have passed for the revisions the fix got merged in.