[2.4-beta3] wildcard syntax is not supported by no-proxy/juju-no-proxy

[Dmitrii Shcherbakov]

Looks like *.domain.com[:port] syntax is no supported by no-proxy (CIDR syntax is supported judging by other tests I performed):

cat model-config.yaml
logging-config: '<root>=ERROR;unit=TRACE;juju.worker.proxyupdater=TRACE'
#no-proxy: 'localhost,127.0.0.1,10.10.101.0/24,10.10.10.0/24'
#http-proxy: 'http://10.10.10.88:3128'
#https-proxy: 'http://10.10.10.88:3128'
no-proxy: ''
http-proxy: ''
https-proxy: ''
juju-no-proxy: 'localhost,127.0.0.1,10.10.101.0/24,10.10.10.0/24,*.canonical.com,*.ubuntu.com'
juju-http-proxy: 'http://10.10.10.88:3128'
juju-https-proxy: 'http://10.10.10.88:3128'
apt-http-proxy: 'http://10.10.10.88:3128'
apt-https-proxy: 'http://10.10.10.88:3128'

juju model-config -m controller model-config.yaml

dd6ca7e2-1304-456b-8d0a-d2eaa3993f53: machine-0 2018-05-25 22:17:54 DEBUG juju.worker.proxyupdater proxyupdater.go:165 new legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.10.101.3", AutoNoProxy:""}
dd6ca7e2-1304-456b-8d0a-d2eaa3993f53: machine-0 2018-05-25 22:17:54 DEBUG juju.worker.proxyupdater proxyupdater.go:186 new apt proxy settings proxy.Settings{Http:"http://10.10.10.88:3128", Https:"http://10.10.10.88:3128", Ftp:"", NoProxy:"*.canonical.com,*.ubuntu.com,10.10.10.0/24,10.10.101.0/24,127.0.0.1,localhost", AutoNoProxy:""}
32041e72-121e-4e4c-89f7-7a35918f4a4c: unit-ubuntu-0 2018-05-25 22:18:13 INFO unit.ubuntu/0.juju-log server.go:284 Reactive main running for hook update-status

ubuntu@proxytest:~$ juju model-config -m controller | grep proxy
apt-ftp-proxy default ""
apt-http-proxy model http://10.10.10.88:3128
apt-https-proxy model http://10.10.10.88:3128
apt-no-proxy default ""
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
juju-ftp-proxy default ""
juju-http-proxy model http://10.10.10.88:3128
juju-https-proxy model http://10.10.10.88:3128
juju-no-proxy model localhost,127.0.0.1,10.10.101.0/24,10.10.10.0/24,*.canonical.com,*.ubuntu.com
logging-config model <root>=ERROR;unit=TRACE;juju.worker.proxyupdater=TRACE
no-proxy model ""
proxy-ssh default false

With this model-config I still get requests sent out to a proxy.

1527286866.141 285 10.10.101.3 TCP_TUNNEL/200 3529 CONNECT streams.canonical.com:443 - HIER_DIRECT/91.189.88.141 -
1527286866.431 289 10.10.101.3 TCP_TUNNEL/200 6692 CONNECT streams.canonical.com:443 - HIER_DIRECT/91.189.88.141 -
1527286866.929 150 10.10.101.3 TCP_MISS/404 507 GET http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson - HIER_DIRECT/91.189.92.141 text/html
1527286866.994 64 10.10.101.3 TCP_MISS/200 2894 GET http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson - HIER_DIRECT/91.189.92.141 -
1527286867.061 65 10.10.101.3 TCP_MISS/404 507 GET http://cloud-images.ubuntu.com/releases/streams/v1/mirrors.sjson - HIER_DIRECT/91.189.92.141 text/html
1527286867.200 137 10.10.101.3 TCP_MISS/200 21719 GET http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:gce.sjson - HIER_DIRECT/91.189.92.141 -
1527286869.411 5170 10.10.101.3 TCP_TUNNEL/200 52275 CONNECT api.jujucharms.com:443 - HIER_DIRECT/162.213.33.121 -

Example implementation of wildcard support:
https://go-review.googlesource.com/c/go/+/75730/3/src/net/http/no_proxy.go#162
https://go-review.googlesource.com/c/go/+/75730/3/src/net/http/no_proxy.go#60

[John A Meinel (jameinel)]

* is not supported (as it wasn't supported by the underlying Go proxy code).
However
"bar.com" matches foo.bar.com, as does ".bar.com"
So while we don't allow "*.bar.com" it can be expressed as just "bar.com"
and ".bar.com".

On Sat, May 26, 2018 at 2:43 AM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Public bug reported:
>
> Looks like *.domain.com[:port] syntax is no supported by no-proxy (CIDR
> syntax is supported judging by other tests I performed):
>
> cat model-config.yaml
> logging-config: '<root>=ERROR;unit=TRACE;juju.worker.proxyupdater=TRACE'
> #no-proxy: 'localhost,127.0.0.1,10.10.101.0/24,10.10.10.0/24'
> #http-proxy: 'http://10.10.10.88:3128'
> #https-proxy: 'http://10.10.10.88:3128'
> no-proxy: ''
> http-proxy: ''
> https-proxy: ''
> juju-no-proxy: 'localhost,127.0.0.1,10.10.101.0/24,10.10.10.0/24,*.
> canonical.com,*.ubuntu.com'
> juju-http-proxy: 'http://10.10.10.88:3128'
> juju-https-proxy: 'http://10.10.10.88:3128'
> apt-http-proxy: 'http://10.10.10.88:3128'
> apt-https-proxy: 'http://10.10.10.88:3128'
>
> juju model-config -m controller model-config.yaml
>
> dd6ca7e2-1304-456b-8d0a-d2eaa3993f53: machine-0 2018-05-25 22:17:54 DEBUG
> juju.worker.proxyupdater proxyupdater.go:165 new legacy proxy settings
> proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.10.101.3",
> AutoNoProxy:""}
> dd6ca7e2-1304-456b-8d0a-d2eaa3993f53: machine-0 2018-05-25 22:17:54 DEBUG
> juju.worker.proxyupdater proxyupdater.go:186 new apt proxy settings
> proxy.Settings{Http:"http://10.10.10.88:3128", Https:"
> http://10.10.10.88:3128", Ftp:"", NoProxy:"*.canonical.com,*.ubuntu.com,
> 10.10.10.0/24,10.10.101.0/24,127.0.0.1,localhost", AutoNoProxy:""}
> 32041e72-121e-4e4c-89f7-7a35918f4a4c: unit-ubuntu-0 2018-05-25 22:18:13
> INFO unit.ubuntu/0.juju-log server.go:284 Reactive main running for hook
> update-status
>
> ubuntu@proxytest:~$ juju model-config -m controller | grep proxy
> apt-ftp-proxy default ""
> apt-http-proxy model http://10.10.10.88:3128
> apt-https-proxy model http://10.10.10.88:3128
> apt-no-proxy default ""
> ftp-proxy default ""
> http-proxy default ""
> https-proxy default ""
> juju-ftp-proxy default ""
> juju-http-proxy model http://10.10.10.88:3128
> juju-https-proxy model http://10.10.10.88:3128
> juju-no-proxy model localhost,127.0.0.1,10.10.101.
> 0/24,10.10.10.0/24,*.canonical.com,*.ubuntu.com
> logging-config model <root>=ERROR;unit=TRACE;juju.
> worker.proxyupdater=TRACE
> no-proxy model ""
> proxy-ssh default false
>
> With this model-config I still get requests sent out to a proxy.
>
> 1527286866.141 285 10.10.101.3 TCP_TUNNEL/200 3529 CONNECT
> streams.canonical.com:443 - HIER_DIRECT/91.189.88.141 -
> 1527286866.431 289 10.10.101.3 TCP_TUNNEL/200 6692 CONNECT
> streams.canonical.com:443 - HIER_DIRECT/91.189.88.141 -
> 1527286866.929 150 10.10.101.3 TCP_MISS/404 507 GET
> http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson -...

[John A Meinel (jameinel)]

In code:
                p = strings.ToLower(strings.TrimSpace(p))
                if len(p) == 0 {
                        continue
                }
                if hasPort(p) {
                        p = p[:strings.LastIndex(p, ":")]
                }
                if addr == p {
                        return false
                }
                if p[0] == '.' && (strings.HasSuffix(addr, p) || addr ==
p[1:]) {
                        // no_proxy ".foo.com" matches "bar.foo.com" or "
foo.com"
                        return false
                }
                if p[0] != '.' && strings.HasSuffix(addr, p) &&
addr[len(addr)-len(p)-1] == '.' {
                        // no_proxy "foo.com" matches "bar.foo.com"
                        return false
                }
                if _, net, err := net.ParseCIDR(p); ip != nil && err == nil
&& net.Contains(ip) {
                        return false
                }

If we wanted to support "*.bar.com" we could add it to the first if with
something like:

if len(p) > 2 && p[0] == '*' && p[1] == '.' {
  p = p[1:]
}
(eg, treat *.com as just .com)

On Mon, May 28, 2018 at 8:20 AM, John Meinel <email address hidden> wrote:

> * is not supported (as it wasn't supported by the underlying Go proxy
> code).
> However
> "bar.com" matches foo.bar.com, as does ".bar.com"
> So while we don't allow "*.bar.com" it can be expressed as just "bar.com"
> and ".bar.com".
>
>
> On Sat, May 26, 2018 at 2:43 AM, Dmitrii Shcherbakov <
> <email address hidden>> wrote:
>
>> Public bug reported:
>>
>> Looks like *.domain.com[:port] syntax is no supported by no-proxy (CIDR
>> syntax is supported judging by other tests I performed):
>>
>> cat model-config.yaml
>> logging-config: '<root>=ERROR;unit=TRACE;juju.worker.proxyupdater=TRACE'
>> #no-proxy: 'localhost,127.0.0.1,10.10.101.0/24,10.10.10.0/24'
>> #http-proxy: 'http://10.10.10.88:3128'
>> #https-proxy: 'http://10.10.10.88:3128'
>> no-proxy: ''
>> http-proxy: ''
>> https-proxy: ''
>> juju-no-proxy: 'localhost,127.0.0.1,10.10.101
>> .0/24,10.10.10.0/24,*.canonical.com,*.ubuntu.com'
>> juju-http-proxy: 'http://10.10.10.88:3128'
>> juju-https-proxy: 'http://10.10.10.88:3128'
>> apt-http-proxy: 'http://10.10.10.88:3128'
>> apt-https-proxy: 'http://10.10.10.88:3128'
>>
>> juju model-config -m controller model-config.yaml
>>
>> dd6ca7e2-1304-456b-8d0a-d2eaa3993f53: machine-0 2018-05-25 22:17:54
>> DEBUG juju.worker.proxyupdater proxyupdater.go:165 new legacy proxy
>> settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.10.101.3",
>> AutoNoProxy:""}
>> dd6ca7e2-1304-456b-8d0a-d2eaa3993f53: machine-0 2018-05-25 22:17:54
>> DEBUG juju.worker.proxyupdater proxyupdater.go:186 new apt proxy settings
>> proxy.Settings{Http:"http://10.10.10.88:3128", Https:"
>> http://10.10.10.88:3128", Ftp:"", NoProxy:"*.canonical.com,*.ubuntu.com,
>> 10.10.10.0/24,10.10.101.0/24,127.0.0.1,localhost", AutoNoProxy:""}
>> 32041e72-121e-4e4c-89f7-7a35918f4a4c: unit-ubuntu-0 2018-05-25 22:18:13
>> INFO unit.ubuntu/0.juju-log server.go:284 Reactive main running for hook
>> update-status
>>
>> ubuntu@proxytest:~$ juju model-config -m controller | grep proxy
>> apt-ftp-proxy ...

[Dmitrii Shcherbakov (dmitriis)]

I'd say the wildcard syntax is more common as it is described in a relevant RFC: https://tools.ietf.org/html/rfc4592#section-2.1.3
"
A wildcard "blocks itself" in the sense that a wildcard does not match its own subdomains. That is, "*.example." does not match all names in the "example." zone; it fails to match the names
   below "*.example."
"

~~

no-proxy=.domain.com
or no-proxy=*.domain.com
or no-proxy=.domain.com.
or no-proxy=*.domain.com.

will match:

domainhost1.domain.com
domainhost1.domain.com.
domainhost2.domain.com
domainhost2.domain.com.

but not

subhost.subdomain.domain.com
subhost.subdomain.domain.com.

[John A Meinel (jameinel)]

I'm fine with supporting *.domain.com, just wanted to point out that Go
doesn't, by default, support it, though it does support .domain.com, etc.
(So the same limitations are part of lxd and probably snaps.)

John
=:->

On Thu, May 31, 2018 at 11:48 AM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> I'd say the wildcard syntax is more common as it is described in a
> relevant RFC: https://tools.ietf.org/html/rfc4592#section-2.1.3
> "
> A wildcard "blocks itself" in the sense that a wildcard does not match its
> own subdomains. That is, "*.example." does not match all names in the
> "example." zone; it fails to match the names
> below "*.example."
> "
>
> ~~
>
> no-proxy=.domain.com
> or no-proxy=*.domain.com
> or no-proxy=.domain.com.
> or no-proxy=*.domain.com.
>
> will match:
>
> domainhost1.domain.com
> domainhost1.domain.com.
> domainhost2.domain.com
> domainhost2.domain.com.
>
> but not
>
> subhost.subdomain.domain.com
> subhost.subdomain.domain.com.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1773463
>
> Title:
> [2.4-beta3] wildcard syntax is not supported by no-proxy/juju-no-proxy
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1773463/+subscriptions
>

[Dmitrii Shcherbakov (dmitriis)]

I think we could wait for upstream to fix this as we have a working version already with a slightly different syntax than the one used in the RFC due to the fact that LXD and snapd are also affected.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.