[2.x] Make maas-proxy more configurable

[Peter Sabaini]

MAAS built-in squid will try to connect to ipv6 upstreams first, and hangs for quite some time if upstream ipv6 is not reachable

Squid has a knob dns_v4_first to change this behaviour [0] but there's no way to configure that persistently in maas 2.2

[0] http://www.squid-cache.org/Doc/config/dns_v4_first/

[Peter Sabaini (peter-sabaini)]

I wonder about the wishlist classification. The impact if the missing dns_v4_first is imho quite severe, as lots of juju charms like to do 'apt-get update' during hook runs. With a malfunctioning maas-proxy this will hang juju units. Similarly, eg. security updates could fail to roll out.

[Andres Rodriguez (andreserl)]

Wishlist are feature request. These end up in the roadmap for discussion in order to get prioritized.

You can work around this issue by modifying the squid3 template config in:

/usr/lib/python3/dist-packages/provisioningserver/templates/proxy/maas-proxy.conf.template

While this wont survive an upgrade, it will give you the means to work around the issue for now.

That said, I have a question for you? How many IPv6 addresses does the apt mirror or archive have ? The issue i believe you experiencing is [0], which was to fix the number of retries squid3 does. Before, it would just do 10, which meant that if the mirror had 10 ipv6 address, it would never retry on IPv4. The fix was to increase such retries to 25, so if there were to be 10 ipv6 addresses, it would retry ipv4 addresses after ipv6. If this is not working, then the bug could potentially be an issue with squid.

[0]: https://bugs.launchpad.net/ubuntu/+source/squid3/+bug/1547640

[Peter Sabaini (peter-sabaini)]

IMHO a failing maas-proxy qualifies as a bug, not a feature request. But thanks for the pointer to the workaround and squid bug.

To answer your question, none of the archives that are queried have 10+ ipv6 ipaddr (can see 0-6 ipv6). Also, we do have the fix in bug #1547640 so that limit should even be higher. I wonder if it just slows down so much that the (numerous) requests from juju agents start failing?

[Björn Tillenius (bjornt)]

Could you check how long it takes to wget/curl each of the ipv6 ips? If you don't have ipv6 configured, it usually fails immediately.

But if it takes longer, that's certainly a problem for squid. That would make squid requests take a long time probably.

[Peter Sabaini (peter-sabaini)]

It indeed seems to take some time to fail:

$ dig +short aaaa archive.ubuntu.com
2001:67c:1360:8001::21
2001:67c:1560:8001::11
2001:67c:1360:8001::17
2001:67c:1562::19
2001:67c:1560:8001::14

$ curl -v --connect-timeout 10 http://[2001:67c:1360:8001::21]/
* Trying 2001:67c:1360:8001::21...
* Connection timed out after 10001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 10001 milliseconds

[Andres Rodriguez (andreserl)]

So based on the last two comments, it seems that this is an issue with squid itself, rather than MAAS.

That said, we can make this option default and always try ipv4 over ipv6 first. However, if we are to do so, I would like to understand what the implications of that would be. Any thoughts ?

[Peter Sabaini (peter-sabaini)]

Not sure about the implications of making ipv4 default - might just create the same effect for ppl running ipv6?

IMHO it'd be safest to give admins control over this and provide means to pass through config to the underlying squid.

Alternatively, maybe it would be possible to probe the network and try to autoconfig squid based on avail. of ipv6?

[Andres Rodriguez (andreserl)]

Ok, so y default we will send the option as 'on' for ipv4, otherwise, users will have to disable such option to set it 'off'.

For now I think we will only have API for this thoough.

[Peter Sabaini (peter-sabaini)]

We would need this for Maas 2.3.5 as distributed by xenial too please.

[Andres Rodriguez (andreserl)]

I'm switching this back as fix released, because this was fixed as per the milestone on 2.4. This is a new feature, and new features are never usually backported to far advanced point releases, specially when these involve database changes such as this one.

THis is to allow upgrade path to work. My recommendation would be for you to upgrade your MAAS server and use 2.4.

On the other hand, I'll open a task for 2.3 and evaluate the likelihood of this being backported, and what risks it carries, as again, changes that involve database updates are not likely to be backported.