proxy tries ipv6 and gets 503 when no ipv6 routes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
squid3 (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Precise |
Fix Released
|
High
|
Unassigned | ||
Trusty |
Fix Released
|
High
|
Unassigned | ||
Wily |
Fix Released
|
Medium
|
Unassigned | ||
Xenial |
Fix Released
|
High
|
Unassigned |
Bug Description
== Begin SRU Information ==
[Impact]
Users of squid3 as a proxy on a host without ipv6 connectivity will see http '503' errors if they attempt to access a url through that proxy that has greater than 9 ipv6 addresses associated with it.
The failure case is that affected ubuntu users specifically was:
a.) user uses squid from Ubuntu as a proxy
b.) security.ubuntu.com and archive.ubuntu.com had additional IPV6 addresses added to their dns, such that there were 10 ipv6 addresses for each.
c.) the squid system does not have access to the ipv6 addresses. Most likely that woudl be a result of having no routable ipv6 traffic.
The change as described in the upstream commit is:
| Update forward_max_tries to permit 25 server paths
|
| With cloud sites becoming more popular more CDN servers are producing
| long lists of IPv6 and IPv4 addresses. If there are not enough paths
| selected the IPv4 ones may never be reached.
[Test Case]
The attached 'lp-1547640.sh' can be run with:
./lp-1547640.sh setup
./lp-1547640.sh test
It installs squid3 and sets up dnsmasq to know about 10 ipv6 addresses for a host, and then attempts to use that squid proxy.
[Regression Potential]
Likely scenarios to cause regression would be for hosts that have several ipv6 addresses. The change has been in squid3 upstream in trunk since 2013-08-21 and for quite a while though. It is released in squids 3.5 branch.
[Other Info]
After we saw and diagnosed this failure, Canonical's IS team removed one of the ipv6 addresses from security.ubuntu.com and archive.ubuntu.com, so that there are only 9 present now.
$ host archive.ubuntu.com | grep 'has IPv6'
archive.
archive.
archive.
archive.
archive.
archive.
archive.
archive.
archive.
There *were* 10 on the day this caused a problem. Canonical will hold off on adding more ipv6 until this change is rolled out widely.
The fix for this bug will come to xenial through a merge with debian under bug 1473691.
== End SRU Information ==
Many people run squid (squid-deb-proxy, or maas-proxy) to provide ubuntu archive mirror caching and proxying. MAAS sets this up by default for users with the 'maas-proxy' package.
On or about Friday February 19, this setup began to fail for many people.
Users would see 'apt-get update' returning 503 errors. For me, I saw 503 on security.ubuntu.com addresses.
The reason for the failure was that the DNS records for Ubuntu reacheda threshold of 10 IPv6 entries. The squid proxy host did not have ipv6 connectivity and with a limit of 10 retries the failover does not reach any IPv4 addresses - thus would fail.
The fix/workaround is to add the following to your squid config:
# http://
forward_max_tries 25
The appropriate squid config file depends on what is running squid.
maas-proxy: /usr/share/
squid-deb-proxy: /etc/init/
I'm not sure how this previously worked, nor what change was made.
One change that was made in this time frame was a glibc update (2.19-0ubuntu6.6 to 2.19-0ubuntu6.7) for security (CVE-2013-7423 CVE-2014-9402 CVE-2015-1472 CVE-2015-1473). But it doesn't seem to make sense that that would change squid3 to start looking for AAAA records when it did not previously.
i can verify that as late as
Thu Feb 18 06:36:07 EST 2016
i was seeing entries in my squid logs with
1455713142.896 335 10.7.2.103 TCP_REFRESH_
but now i get
1455879482.210 1 10.7.2.103 TCP_REFRESH_
Related Bugs:
description: | updated |
Changed in maas: | |
status: | New → Triaged |
importance: | Undecided → High |
milestone: | none → 2.0.0 |
Changed in maas: | |
assignee: | nobody → Andres Rodriguez (andreserl) |
tags: | added: cloud-installer |
affects: | squid (Ubuntu) → squid3 (Ubuntu) |
no longer affects: | maas |
no longer affects: | maas/1.10 |
no longer affects: | maas/1.9 |
description: | updated |
description: | updated |
Changed in squid3 (Ubuntu Precise): | |
status: | New → Triaged |
importance: | Undecided → High |
Adding dns_v4_first on to my 14.04 LTS /etc/squid- deb-proxy/ squid-deb- proxy.conf solved this for me.
My personal best guess is that something happened during machine reboots in the Canonical datacenter to address the glibc updates.
My failures were to both security.ubuntu.com and archive.ubuntu.com, e.g.:
W: Failed to fetch http:// security. ubuntu. com/ubuntu/ dists/trusty- security/ restricted/ binary- amd64/Packages 503 Service Unavailable
W: Failed to fetch http:// archive. ubuntu. com/ubuntu/ dists/trusty- proposed/ restricted/ binary- amd64/Packages 503 Service Unavailable
(there were dozens more like this, these two were just side-by-side in scrollback.)
Thanks