wily 1.24 cannot bootstrap local-provider: 127.0.0.1:37017: getsockopt: connection refused

[Curtis Hovey]

Juju 1.24 but NOT 1.25 and master
    ...strap failed, cleaning up the environment.
2015-08-19 02:02:42 ERROR juju.cmd supercommand.go:430 there was an issue examining the environment: dial tcp 127.0.0.1:37017: getsockopt: connection refused

This issue looked at first to be a substrate issue. We can see the machine is clean, and 1.25 and 1.26 clients can boostrap. Just 1.24 cannot bootstrap. This issue was also obscured by the systemd regression that broke all version of juju.

The first occurrence is seen at
    http://reports.vapour.ws/releases/2986/job/local-deploy-wily-amd64/attempt/237

^ We are using --debug, but don't get any rich information.

The suspect commits are
    https://github.com/juju/juju/commit/2743244680959269186d23e14f957b5a0395b310
    https://github.com/juju/juju/commit/5e0b33208e3f946c240b366e5197170038acf690

[Curtis Hovey (sinzui)]

We can see nothing is holding the port on the wily machine.
sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:6514 0.0.0.0:* LISTEN 751/rsyslogd
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 1072/dnsmasq
tcp 0 0 10.0.1.1:53 0.0.0.0:* LISTEN 866/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1157/sshd
tcp6 0 0 :::6514 :::* LISTEN 751/rsyslogd
tcp6 0 0 fe80::c0d9:c0ff:fe31:53 :::* LISTEN 866/dnsmasq
tcp6 0 0 :::22 :::* LISTEN 1157/sshd

We know mongod is not running and the machine has been started many times ver the last few weeks. Again. only 1.24.6 is affected.

[Eric Snow (ericsnowcurrently)]

I've been able to verify that the issue affects the 1.24 head (1.24.6) as well as the "juju-1.24.5" and "juju-1.24.3" tags. I've also verified that it does *not* impact 1.25 nor 1.24 before 1.24.4 (e.g. 1.24.3).

[Eric Snow (ericsnowcurrently)]

That should have been the "juju-1.24.4" tag (affected), not "juju-1.24.3".

[Eric Snow (ericsnowcurrently)]

Strike that. I haven't been able to get any juju pre-1.25 to work. I've tried all of 1.24 and the 1.22 branch. Perhaps it's just some state that my Wily install got into. However, I've certainly verified that it's not a problem on 1.25.

[Eric Snow (ericsnowcurrently)]

This appears to be a Go 1.5-related issue, fixed in 1.25/master in db5080a3313051dd83f24b4c7e6199b618e1d1dd (https://github.com/juju/juju/pull/2304), but not in 1.24. I'll put up a backport of that patch.

[Eric Snow (ericsnowcurrently)]

BTW, bisect FTW.

[Eric Snow (ericsnowcurrently)]

http://reviews.vapour.ws/r/2624/

[Canonical Juju QA Bot (juju-qa-bot)]

Juju-CI verified that this issue is Fix Released in juju-core 1.24:
    http://reports.vapour.ws/releases/3045