Client connects to wrong host when proxy is configured

[Paul Gear]

After upgrading from juju 2.1.3 to juju 2.2.0, I can no longer connect to my controllers, nor upgrade them to 2.2:

[master*]paulgear@peleg:~$ juju status
ERROR unable to connect to API: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

It takes 10 minutes before this error message is seen.

This appears to be related to proxy support in the juju client (I run a local squid cache), but even if I unset all of my {ftp,http,https,no}_proxy environment variables, the behaviour does not change.

See https://pastebin.canonical.com/191338/ for full debug messages and a comparison with 2.1.3 behaviour.

[John A Meinel (jameinel)]

My immediate thought is that something is treating the connection as plain
HTTP rather than HTTPS. The 10min sounds like a retry that times out.

John
=:->

On Jun 20, 2017 05:50, "Paul Gear" <email address hidden> wrote:

> Public bug reported:
>
> After upgrading from juju 2.1.3 to juju 2.2.0, I can no longer connect
> to my controllers, nor upgrade them to 2.2:
>
> [master*]paulgear@peleg:~$ juju status
> ERROR unable to connect to API: malformed HTTP response
> "\x15\x03\x01\x00\x02\x02\x16"
>
> It takes 10 minutes before this error message is seen.
>
> This appears to be related to proxy support in the juju client (I run a
> local squid cache), but even if I unset all of my
> {ftp,http,https,no}_proxy environment variables, the behaviour does not
> change.
>
> See https://pastebin.canonical.com/191338/ for full debug messages and a
> comparison with 2.1.3 behaviour.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
>
> ** Tags: canonical-is
>
> ** Tags added: canonical-is
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1698989
>
> Title:
> Can't connect to controllers, juju status hangs for 10 minutes
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1698989/+subscriptions
>

[Joel Sing (jsing)]

I would concur with that, you're almost certainly seeing a TLS fatal alert in the response:

0x15 == Record Type Alert
0x0301 == TLS Version 1.0
0x0002 == Length
0x02 == Alert Type - Fatal
0x16 == Alert Description - Record Overflow

Which would likely caused by talking HTTP to a HTTPS server.

[Menno Finlay-Smits (menno.smits)]

I can reproduce locally and can trigger the problem by setting
$https_proxy. I've been instrumenting the gorilla/websocket client code to
figure out what's happening.

Initial packet captures indicate the client might be connecting to the API
server but talking as if it's a HTTP proxy (not confirmed for sure yet)

On 20 Jun 2017 4:41 pm, "Joel Sing" <email address hidden> wrote:

> I would concur with that, you're almost certainly seeing a TLS fatal
> alert in the response:
>
> 0x15 == Record Type Alert
> 0x0301 == TLS Version 1.0
> 0x0002 == Length
> 0x02 == Alert Type - Fatal
> 0x16 == Alert Description - Record Overflow
>
> Which would likely caused by talking HTTP to a HTTPS server.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1698989
>
> Title:
> Can't connect to controllers, juju status hangs for 10 minutes
>
> Status in juju:
> In Progress
>
> Bug description:
> After upgrading from juju 2.1.3 to juju 2.2.0, I can no longer connect
> to my controllers, nor upgrade them to 2.2:
>
> [master*]paulgear@peleg:~$ juju status
> ERROR unable to connect to API: malformed HTTP response
> "\x15\x03\x01\x00\x02\x02\x16"
>
> It takes 10 minutes before this error message is seen.
>
> This appears to be related to proxy support in the juju client (I run
> a local squid cache), but even if I unset all of my
> {ftp,http,https,no}_proxy environment variables, the behaviour does
> not change.
>
> See https://pastebin.canonical.com/191338/ for full debug messages and
> a comparison with 2.1.3 behaviour.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1698989/+subscriptions
>

[Menno Finlay-Smits (menno.smits)]

I have verified that when $https_proxy is set, the client is connects to the Juju API server and but talks to it as if it is a proxy.

The root of the problem is https://github.com/juju/juju/blob/juju-2.2.0/api/apiclient.go#L614

As part of the DNS caching change added in 874fbd53dd898c325edc36ec37d0518f03bfd987 the dial func we pass to gorilla/websocket ignores the address it's given and uses a pre-resolved one. This breaks proxy functionality causing the client to connect to the API server instead of the proxy.

The correct fix needs some further thought. I'll reach out to some people.

[Menno Finlay-Smits (menno.smits)]

The PRs that relate to the cause of the issue are:

https://github.com/juju/juju/pull/7407
https://github.com/juju/juju/pull/7438

[Andrew Wilkins (axwalk)]

Short-term solution: https://github.com/juju/juju/pull/7529