MAAS 1.8 requires arbitrary high-numbered port connections between cluster and region controllers

[Nick Moffitt]

MAAS's separation of regions and clusters allows network admins to centralise policy and interface while erecting management presence in isolated networks. This feature is most useful in situations where the firewall policy walls off clusters more strictly.

MAAS 1.5 seems to require that the cluster controllers call back to the region controller via arbitrary high-numbered ports. I checked our installation and could not find any specification of whcih ports in the code or our local configuration, and only spotted them because our firewall denied this traffic and caused node commissioning to fail.

I'm not familiar enough with twisted or the MAAS source enough to work out why this is, but I suspected there were no defined ports when I found the comment here:

    http://bazaar.launchpad.net/~maas-maintainers/maas/1.5/view/head:/src/maasserver/rpc/regionservice.py#L177

            """Start listening on an ephemeral port."""

Unfortunately the exact situation that makes splitting out your cluster controllers attractive is going to make this behaviour a deal-breaker. Getting all unprivileged ports opened on a firewall is not something to be done lightly.

I do not believe that MAAS 1.4 exhibited this behaviour.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: maas (not installed)
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Aug 5 15:24:02 2014
InstallationDate: Installed on 2014-05-29 (67 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: maas
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gavin Panella (allenap)]

A few thoughts:

- We may be able to use SO_REUSEPORT to have all event-loops in the
  region listening on the same port, but then it's unclear how we would
  ensure that each cluster has a connection to each and every
  event-loop, which is the model we rely upon.

- We /could/ change the logic so that the region initiates connections
  with clusters. The clusters could listen on a fixed port, and because
  there's only one of them (...so far), it would make firewalls simpler
  to configure. But, if we do ever run multiple cluster processes then
  we're back to the same problem.

- We could restrict the ports used to a narrow range.

Before we spend more time on this, can you explain why opening up a
large port range is problematic? The ports need only to be open between
the cluster controllers and the region controllers, not to the whole
cluster network.

My hand-wavy thoughts on using a narrow range, which is probably the
easiest thing to implement:

- A narrow port range isn't a significant security improvement over
  using a wide range as far as I can tell, unless the range is beneath
  1024, when we can assume that only a root or root-authorised user
  could attempt to impersonate an event-loop in the region.

- The clusters will only connect to ports on the region controllers that
  have been advertised (the clusters request the list of ports via the
  web API). This also narrows the chance of impersonation.

- The RPC connection between region and cluster is not yet over TLS, but
  will be for 1.7. Each end will authenticate the other via certificate.
  (The code is in place, we just need to figure out the upgrade workflow
  to introduce certificates.)

- On the region machines, other processes could be listening on high
  ports, and these would be exposed to the cluster controllers. These
  could be selectively blocked, or reconfigured to only listen on a
  local-only interface (e.g. 127.0.0.1). Granted, this isn't as nice as
  poking small holes in the firewall, but it's doable.

[Nick Moffitt (nick-moffitt)]

Gavin Panella:
> Before we spend more time on this, can you explain why opening up a
> large port range is problematic? The ports need only to be open between
> the cluster controllers and the region controllers, not to the whole
> cluster network.

I don't have a unified or coherent way of explaining this, but I'll try
to outline what I see as some of the reasoning involved here and hope
that others more articulate on the subject can chime in.

1. Principle of Least Access

    The reason we have firewalls is to allow desired traffic through
    while keeping out undesired traffic. The costs for disallowing
    desired traffic are felt immediately, and (depending on
    administration of the firewalls) easily remedied. The costs for
    allowing undesired traffic through are subtle and hidden, and
    magnify with time.

    As a result, the most sensible policy is to DENY by default and then
    allow through specific host/port combinations. Occasionally it is
    necessary to use entire network ranges instead of hosts, but that is
    a considered thing and has no alternatives (a central monitoring
    system, for example, needs to be able to probe all machines on a
    given network).

    Fortunately we never have to open large ranges of ports. Protocols
    such as the old NFS portmap used to require arbitrary UDP
    connections, and that's one of the many reasons nobody uses NFS any
    more. Protocols and connections are identified by port numbers,
    even if they're narrow ranges or de facto standards (such as 6667
    for IRC and 6697 for IRC over SSL).

    It is reasonable that the burden of proof be on the request to open
    access, as that is the operation that comes with risk.

2. Bound Ports

    We can be confident that a process has laid claim to a port
    (privileged or no) and that unprivileged processes won't be able to
    set up connections without first disabling the official one and any
    monitoring we perform.

    You mentioned privileged ports (sub-1024) which add a handy extra
    layer of constraints on all this, but the fact that the ports are
    unprivileged doesn't mean that there's no benefit to limiting them.
    If the apache2 process holds port 8080 on example.canonical.com,
    someone who exploits ntpd and gains access as the ntp user won't be
    able to kick apache off of that port without further effort.

3. Pervasiveness of the Approach

    Network firewalls are something we've had to defend less and less as
    Canonical has gained experience on customer sites. Even if there
    were no rational basis for the setup, its pervasiveness means that
    it's something MAAS needs to support.

    The role of the cluster controller is such that it is MAAS's
    representative in a variety of walled-off networks, and it cannot
    assume that it has unrestricted access to the region controller.
    The firewalls between MAAS controllers may not be under the control
    of the people running the MAAS infrastructure, and changes to
    filtering rules on such a site may be vastly more difficult than it
    is for us internally.

    What happens if an ISP filters bittorrent ports, and a ...

[Gavin Panella (allenap)]

Thanks Nick, that's good stuff.

[Christian Reis (kiko)]

We should put serious thought into making a single port whitelist the only thing which is required to communicate between cluster and region. I'm not sure whether making that 80 or 443 is feasible or significantly better than requiring a random unprivileged port.

[ubuntudotcom1 (ubuntudotcom1)]

Hmm maybe there is a way we can leverage multicast to help out here.