upgrade from MAAS 2.3.5 -> 2.4.2 (or later) does not account for 80 -> 5240 port change

[Dmitrii Shcherbakov]

Upgraded (do-release-upgrade) a long-lived MAAS from 2.3.5/xenial to 2.4.2/bionic. After an upgrade I started getting:

Nov 19 15:56:37 maas sh[31121]: 2018-11-19 15:56:37 provisioningserver.rpc.clusterservice: [critical] Failed to contact region. (While requesting RPC info at b'http://
Nov 19 15:56:37 maas sh[31121]: Traceback (most recent call last):
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
Nov 19 15:56:37 maas sh[31121]: current.result = callback(current.result, *args, **kw)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
Nov 19 15:56:37 maas sh[31121]: _inlineCallbacks(r, g, deferred)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
Nov 19 15:56:37 maas sh[31121]: result = result.throwExceptionIntoGenerator(g)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
Nov 19 15:56:37 maas sh[31121]: return g.throw(self.type, self.value, self.tb)
Nov 19 15:56:37 maas sh[31121]: --- <exception caught here> ---
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1060, in _doUpdate
Nov 19 15:56:37 maas sh[31121]: info = yield self._fetch_rpc_info(info_url)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
Nov 19 15:56:37 maas sh[31121]: current.result = callback(current.result, *args, **kw)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1107, in catch_503_error
Nov 19 15:56:37 maas sh[31121]: failure.trap(web.error.Error)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap
Nov 19 15:56:37 maas sh[31121]: self.raiseException()
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException
Nov 19 15:56:37 maas sh[31121]: raise self.value.with_traceback(self.tb)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
Nov 19 15:56:37 maas sh[31121]: current.result = callback(current.result, *args, **kw)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1118, in <lambda>
Nov 19 15:56:37 maas sh[31121]: d.addCallback(lambda data: json.loads(data.decode('utf-8')))
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
Nov 19 15:56:37 maas sh[31121]: return _default_decoder.decode(s)
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
Nov 19 15:56:37 maas sh[31121]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov 19 15:56:37 maas sh[31121]: File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
Nov 19 15:56:37 maas sh[31121]: raise JSONDecodeError("Expecting value", s, err.value) from None
Nov 19 15:56:37 maas sh[31121]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Turns out that the reason was simple and the maas_url used on a region still contained an implicit :80 port usage.

/etc/maas/rackd.conf
cluster_uuid: 731d7023-d486-453e-bce4-3ca6dd45eabe
maas_url:
- http://192.0.2.2/MAAS

Adding the port http://192.0.2.2:5240/MAAS fixed the problem.

Might be worthwhile to handle this during upgrade automatically.

[Andres Rodriguez (andreserl)]

Hi Dimitrii,

MAAS does not bind to port 80, and it hasn't been doing so since 2.0. In fact, this was made public in the transition from Trusty to Xenial, and users were asked to stop using port 80 as their MAAS endpoint (as it was no longer supported). Due to Juju, we decided to keep apache2 as a dependency to continue to provide port 80 as a reverse proxy via apache2, so that Juju users were not impacted, but it was communicated that port 80 was no longer supported.

That said, MAAS upgrades do not change the endpoint configured in rackd.conf:maas_url, provided that you could be putting the communication between rack and region behind a proxy (such as HAProxy), as such, MAAS doesn't enforce having a port in the config. (e.g. FCB uses port 80 to connect racks to regions).

Lastly, as of 2.4.0, we completely dropped the dependency on apache2.

[Dmitrii Shcherbakov (dmitriis)]

Makes sense, sorry about the noise.