[2.1.1] regiond requests to 127.0.0.1:5240 going via external proxy and failing
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Invalid
|
High
|
Unassigned | ||
2.1 |
Won't Fix
|
High
|
Unassigned |
Bug Description
I upgraded my 2.0 installation to 2.1.1. I noticed requests failing in the regiond log, and that these requests to 127.0.0.1 were being sent to my external APT proxy.
/etc/maas/
maas_url: http://
/etc/maas/
maas_url: http://
Error from /var/log/
2016-11-21 10:41:10 stderr: [error] request to http://
From proxy log:
1479695886.079 1 100.64.0.253 TCP_MISS/503 4653 POST http://
A few interesting notes:
- The localhost config is stored in the rackd config file
- The errors were being generated by regiond, seemingly connecting to itself based on the rackd config
- If I change the rackd.conf URL from localhost to 100.64.0.253 then not only do the errors go away, but the requests are no longer sent via the proxy
- Disabling the proxy also 'fixed' the problem
Seems there is a bug in the logic for when to use the proxy, seems like it has an 'exception' for it's own IPs but does not include localhost. If fixing this by whitelist, should probably whitelist ::1 also.
Changed in maas: | |
status: | New → Triaged |
importance: | Undecided → Critical |
milestone: | none → 2.2.0 |
importance: | Critical → High |
tags: | added: docteam |
tags: | added: landscape |
Changed in maas: | |
milestone: | 2.2.0 → 2.2.x |
Changed in maas: | |
milestone: | 2.2.x → 2.3.x |
I am not sure why the region seems to be making HTTP calls to itself in this example. That's strange to me and I think it requires explanation. (The only thing I can think of is, I think we run a subset of the commissioning scripts on the region in order to collect data to populate the database; perhaps one of those scripts is doing it.)
I think the "TCP_MISS/503" in the proxy log simply means that the proxy tried to access the server, but it returned an HTTP 503 (unavailable) error. That seems like it should be temporary. Is this consistent?
The other thing I would note here is (at least in trunk), we have a rule as follows in the maas-proxy. conf.template:
acl localnet src 127.0.0.0/8
So the requests are indeed being allowed by the proxy. I think the issue, then, is with the HTTP server answering the request on that port, not the proxy. But it's still weird that the requests are going through the proxy.
This is a long shot, but the other thing I would check in this situation are the iptables rules. I have seen situations where broad NAT rules cause the system to communicate to itself in weird ways (such as SNAT-ing traffic toward localhost).