MAAS does not configure squid-deb-proxy for node local networks

[Paul]

Commissioning nodes fail due to not being able to connect with the squid-deb-proxy server running on the MAAS server:

"1 1395953765.664 0 XXX.XXX.XXX.XXX TCP_DENIED/403 3848 GET http://archive.ubuntu.com/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages - HIER_NONE/- text/html

It seems there are 3 private networks and the local loopback hardcoded in /etc/squid-deb-proxy/allowed-networks-src.acl :

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
127.0.0.1

Since MAAS relies on the functionality of squid-deb-proxy server, we should have a way of configuring the allowed networks within the MAAS web interface or at the very least, determine the current ip/subnet of the server and add in the network to the above file during installation time.

=============== previous description =====================

MAAS with latest 14.04 Trusty failing on Console output: failed [3/5] (00-maas-03-install-lldpd, 99-maas-01-wait-for-lldpd, 99-maas-02-capture-lldp)

>> contents of /var/log/maas/*

Please see attachment: var_log_maas.zip

Also attached with backdoor into the ephemeral image: ephemeral-var_log.zip

>> Output from dpkg -l '*maas*'|cat

root@ubuntu-virtual-machine:/var/log/maas# dpkg -l '*maas*' | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-====================== =============================-============-===================================== ==========================================
ii maas 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii maas-cli 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Client Tool
ii maas-cluster-controller 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Cluster Controller
ii maas-common 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii maas-dhcp 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server - DHCP Configurati on (meta-package)
ii maas-dns 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server - DNS configuratio n (metapackage)
ii maas-region-controller 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii maas-region-controller-min 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server
ii python-django-maas 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server - (django files)
ii python-maas-client 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS API Client - (python file s)
ii python-maas-provisioningserver 1.5+bzr1977-0ubuntu5 all Ubuntu MAAS Server

>> Instructions on how we can re-create your bug and what the expected result should be.
Please do not use words like "should" and "needs to" - report the actual problem, not how you think it should be solved.

1. Follow the MAAS instruction under: http://maas.ubuntu.com/docs/install.html
2. When trying to commission the node failed commissioning and was marked "Failed tests" failed [3/5] (00-maas-03-install-lldpd, 99-maas-01-wait-for-lldpd, 99-maas-02-capture-lldp)

[Paul (pachen2)]

attached: log_var_maas.zip

[Paul (pachen2)]

attached: ephemeral-var_log.zip

[Julian Edwards (julian-edwards)]

Thanks for attaching all the logs, very useful!

Your cloud-init-output.log shows a lot of errors when it tries to download packages:
Err http://archive.ubuntu.com trusty/main amd64 Packages
  403 Forbidden

I expect this is the root cause of your problem. Can you please make sure the node can see a valid archive, and try again.

[Mike Rushton (leftyfb)]

The problem was with the squid-deb-proxy server denying traffic from the local network that the nodes and maas server are on. It seems there are 3 private networks and the local loopback hardcoded in /etc/squid-deb-proxy/allowed-networks-src.acl :

# private networks
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
127.0.0.1

I guess we are making assumptions on which networks people are likely to choose for their local net. After adding in the subnet for the network Paul was using, the node was able to commission and subsequently run an installation of Ubuntu 14.04.

[Mike Rushton (leftyfb)]

It should be noted that in this case, MAAS was not controlling DNS or DHCP.

[Rod Smith (rodsmith)]

One more twist on this tale: If your network is within the defined range but doesn't follow traditional class A/B/C numbering (for instance, 10.20.30.0/24), you also hit this bug. Adding your network to the list without deleting or adjusting the existing network that covers your range causes squid-deb-proxy to ignore your network (with an error at startup time).

[Rod Smith (rodsmith)]

See also https://bugs.launchpad.net/maas/+bug/1300266 for a similar (but distinct) bug. Note that two different configuration files must be edited to work around both bugs; this bug affects the squid-deb-proxy's clients, whereas 1300266 limits the servers to which it can connect.

[Rajasekar Karthik (karthik085)]

I am having the problem outlined in Bug Description - getting
failed [3/6] (00-maas-03-install-lldpd, 99-maas-01-wait-for-lldpd, 99-maas-02-capture-lldp)

I am running my own ISC DHCP Server on 10.0.0.1 and restored original squid3 settings, which is
# allowed-networks-src.conf
#
# network sources that you want to allow access to the cache

# private networks
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
127.0.0.1

# IPv6 private addresses
fe80::/64
::1/128

# additional non-private networks can be added to the directory:
# /etc/squid-deb-proxy/allowed-networks-src.acl.d

Any suggestions on how to resolve the bug are appreciated!

[Chris Glass (tribaal)]

A couple of comments:

- You don't need to edit the configuration file, you can simply print the network to a /etc/squid-deb-proxy/allowed-networks-src.acl.d/ file.

- Do we really need the restrictive acls for MAAS at all? What is realistically stopping us from doing: "echo '0.0.0.0/0' > /etc/squid-deb-proxy/allowed-networks-src.acl.d/99-maas" at install time and let whoever use port 8000 as a deb proxy? It seems like the common case would benefit from one less potential hurdle, and the security impact would be quite small: I assume the cases where MAAS is on a publicly-routable non-firewalled interface to be quite rare?

[mahmoh (mahmoh)]

I've seen this symptom at a customer site, the problem was the router that they specified was not the correct one for the Dynamic DHCP range that MAAS was configured to assign the enlisting nodes; once we fixed them to point to the MAAS server (a known quantity and configured as a router) then everything started working. We'll fix the routing information now to match what it should have been initially to keep things working and no longer need the MAAS server as a router.

[mahmoh (mahmoh)]

^All behind a proxy.

[Andres Rodriguez (andreserl)]

Hi!

We believe this is no longer an issue in the latest MAAS releases. As such, we are marking this bug report as invalid. If you believe this is still an issue, please re-open this bug report.

Thanks