Juju2 cannot deploy centos workloads on maas 1.9

[Curtis Hovey]

Juju 2 in master has not been able to deploy centos workloads since commit c5a433b
    http://reports.vapour.ws/releases/3934

As of commit 2354e12
    http://reports.vapour.ws/releases/3935
we see centos workloads timeout
    http://reports.vapour.ws/releases/3935/job/maas-1_9-deploy-centos-amd64/attempt/364

The suspect commits are
    https://github.com/juju/juju/commit/1db39037bcc844b2ebeb2b3c612e2faecf848315
    https://github.com/juju/juju/commit/f2d229f8b539b649b351a0285f59f68506433d53
    https://github.com/juju/juju/commit/c2e6513f24002cd5d013a988ae6535ba73fcaea2
    https://github.com/juju/juju/commit/2354e12ac6d8149394735de860c45d7387c4c60c

[Martin Packman (gz)]

From cloud-init-output.log from a centos deploy with maas 2.0:

+ curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --noproxy '*' --insecure -o /var/lib/juju/tools/2.0-beta7-centos7-amd64/tools.tar.gz https://10.0.210.121:17070/model/8d1e7eb2-04d6-4283-8215-a8049ad753f0/tools/2.0-beta7-centos7-amd64
curl: (35) Peer reports incompatible or unsupported protocol version.

Suggests issue is from natefinch/useSecureTLS:

<http://reviews.vapour.ws/r/4735/>

[Nate Finch (natefinch)]

Looking at this now

[Nate Finch (natefinch)]

I can't repro this problem using a generic CentOS7 VM deployed on GCE, downloading tools from a juju controller deployed using master.

$ curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --noproxy '*' --insecure -o tools.tar.gz
 https://104.197.80.154:17070/model/da4c12d1-7993-4b45-845f-36a18ee1c9bd/tools/2.0-beta6-xenial-amd64
tools from https://104.197.80.154:17070/model/da4c12d1-7993-4b45-845f-36a18ee1c9bd/tools/2.0-beta6-xenial-amd64 downloaded: HTTP 200; time 43.077s; size 19445680 bytes; speed 451415.000 bytes/s
$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC
zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

[Nate Finch (natefinch)]

Looks like the difference is that the image that we're getting from maas uses NSS 3.15, which defaults to disabling TLS 1.1 and 1.2. you can fix the command to work correctly by adding --tlsv1 --ciphers ecdhe_rsa_aes_256_sha to the curl command. The version on GCE's CentOS7 uses NSS 3.19. (in 3.18 they enabled tls 1.1 and 1.2 by default).

[Nate Finch (natefinch)]

see also https://bugs.launchpad.net/juju-core/+bug/1578286

[Nate Finch (natefinch)]

Curtis is pinging the MAAS team to see why the image we're providing appears to be out of date.

[Katherine Cox-Buday (cox-katherine-e)]

We want to fix the problem upstream, not make this work by being less secure.

[Martin Packman (gz)]

Updates cloud-images in CI and verified this is now fixed:

<http://juju-ci.vapour.ws/job/maas-1_9-deploy-centos-amd64/415/>