Unable to connect to environment after local upgrade on precise

[John George]

Unable to connect to the environment after upgrading from Juju version 1.20.6-precise-amd64 to 1.21-alpha2-precise-amd64.
gitbranch:master:github.com/juju/juju r34b9ec5c

The steps to reproduce the error are as follows:

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ juju --version
1.20.6-precise-amd64

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ juju status
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.20.6.1
    dns-name: localhost
    instance-id: localhost
    series: precise
    state-server-member-status: has-vote
services: {}

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ export PATH=./extracted-bin/usr/lib/juju-1.21-alpha2/bin:$PATH

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ juju --version
1.21-alpha2-precise-amd64

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ juju status
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.20.6.1
    dns-name: localhost
    instance-id: localhost
    series: precise
    state-server-member-status: has-vote
services: {}

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ juju --show-log upgrade-juju -e local --version 1.21-alpha2 --upload-tools
2014-09-17 06:58:06 INFO juju.cmd supercommand.go:37 running juju [1.21-alpha2-precise-amd64 gc]
2014-09-17 06:58:06 INFO juju.api api.go:234 connecting to API addresses: [localhost:17070 10.0.1.1:17070 10.0.0.191:17070 192.168.122.1:17070 [fe80::f816:3eff:fe1f:67e1]:17070 [fe80::ac79:d9ff:fe11:83d5]:17070]
2014-09-17 06:58:06 INFO juju.api apiclient.go:252 dialing "wss://localhost:17070/environment/8ceb3492-6a1b-4706-8eb5-4ef3953f5ba6/api"
2014-09-17 06:58:06 INFO juju.api apiclient.go:175 connection established to "wss://localhost:17070/environment/8ceb3492-6a1b-4706-8eb5-4ef3953f5ba6/api"
2014-09-17 06:58:07 INFO juju.environs.tools build.go:168 found existing jujud
2014-09-17 06:58:07 INFO juju.environs.tools build.go:178 target: /tmp/juju-tools217094332/jujud
2014-09-17 06:58:18 INFO juju.environs.sync sync.go:304 built tools 1.21-alpha2.1-precise-amd64 (8388kB)
2014-09-17 06:58:18 INFO juju.cmd.juju upgradejuju.go:325 uploading tools 1.21-alpha2.1-precise-amd64 (8388kB) to Juju state server
2014-09-17 06:58:18 INFO juju.utils http.go:66 hostname SSL verification disabled
available tools:
    1.21-alpha2.1-precise-amd64
best version:
    1.21-alpha2.1
2014-09-17 06:58:22 INFO juju.cmd.juju upgradejuju.go:213 started upgrade to 1.21-alpha2.1
2014-09-17 06:58:22 INFO juju.cmd supercommand.go:329 command finished

jenkins@precise-slave:~/workspace/local-upgrade-precise-amd64$ juju status -e local
WARNING discarding API open error: environment not found
ERROR Unable to connect to environment "local".
Please check your credentials or use 'juju bootstrap' to create a new environment.

Error details:
invalid entity name or password

Messages from from machine-0.log (full log attached):
2014-09-16 07:07:43 DEBUG juju.apiserver apiserver.go:151 <- [103] <unknown> {"RequestId":1,"Type":"Admin","Request":"Login","Params":{"AuthTag":"user-admin","Password":"<removed to protect the innocent>","Nonce":""}}
2014-09-16 07:07:43 DEBUG juju.apiserver apiserver.go:158 -> [103] <unknown> 88.624491ms {"RequestId":1,"Error":"invalid entity name or password","Response":{}} Admin[""].Login
2014-09-16 07:07:43 INFO juju.apiserver apiserver.go:166 [103] <unknown> API connection terminated after 89.65875ms

[Nate Finch (natefinch)]

Can you add the contents of your ~/.juju/environments/local.jenv ? That may help find the error.

[Menno Finlay-Smits (menno.smits)]

The problem can be reliably reproduced and the revision at fault has been found. I will attach the script I used with git bisect.

The problem rev is 69ee37f.

[Menno Finlay-Smits (menno.smits)]

This script was used to track down the problem using git bisect. It exits with 0 if the upgrade worked and with 1 otherwise.

[Menno Finlay-Smits (menno.smits)]

This is a chicken-and-egg situation.

As of rev 69ee37f, an upgrade step needs to run before environments can be looked up when API connections are opened, but the upgrade infrastructure needs to open an API connection to give to upgrade steps so the upgrade steps fail to run.

Exploring solutions now... Backing out this revision probably isn't an option because there's been a lot built on top of it.

[Menno Finlay-Smits (menno.smits)]

A fix is merging now:
https://github.com/juju/juju/pull/787

[Curtis Hovey (sinzui)]

Seen to be fixed in http://juju-ci.vapour.ws:8080/job/local-upgrade-precise-amd64/1804/