[2.5] Attempting to configure RBAC while MAAS was unable to restart maas-proxy caused internal server error

[Andres Rodriguez]

maas-proxy was failing to start because of this:

Nov 28 12:44:26 rivals.local squid[23937]: Ipc::Mem::Segment::create failed to shm_open(/squid-cf__queues.shm): (17) File exists
Nov 28 12:44:26 rivals.local sh[23929]: FATAL: Ipc::Mem::Segment::create failed to shm_open(/squid-cf__queues.shm): (17) File exists
Nov 28 12:44:26 rivals.local systemd[1]: maas-proxy.service: Control process exited, code=exited status=1
Nov 28 12:44:26 rivals.local systemd[1]: maas-proxy.service: Failed with result 'exit-code'.

Which caused the following:

2018-11-28 12:42:29 maasserver.region_controller: [critical] Failed configuring proxy.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/service_monitor.py", line 404, in reloadService
            state = yield self.ensureService(name)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/service_monitor.py", line 645, in _ensureService
            yield self._performServiceAction(service, action)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/service_monitor.py", line 498, in _performServiceAction
            raise ServiceActionError(error_msg)
        provisioningserver.utils.service_monitor.ServiceActionError: Service 'proxy' failed to start: Job for maas-proxy.service failed because the control process exited with error code.
        See "systemctl status maas-proxy.service" and "journalctl -xe" for details.

When trying to make operations agains the API to register RBAC while this was happening, I kept getting the following (with nothing on regiond.log other than the traceback above):

2 roaksoax@rivals:~/project/crbs$ sudo maas configauth --rbac-url http://localhost:8000/ --rbac-service-name maas-2
usage: maas [-h] COMMAND ...

optional arguments:
  -h, --help show this help message and exit

drill down:
  COMMAND
    login Log in to a remote API, and remember its description and
                  credentials.
    logout Log out of a remote API, purging any stored credentials.
    list List remote APIs that have been logged-in to.
    refresh Refresh the API descriptions of all profiles.
    init Initialize controller.
    apikey Used to manage a user's API keys. Shows existing keys unless
                  --generate or --delete is passed.
    configauth Configure external authentication.
    createadmin Create a MAAS administrator account.
    changepassword
                  Change a MAAS user's password.
    admin Interact with http://localhost:5240/MAAS/api/2.0/
    test Interact with http://192.168.1.13:5240/MAAS/api/2.0/

http://maas.io/

Internal Server Error

Once I manually fixed the start-up of maas-proxy, attempting to run the command above worked successfully.

[Björn Tillenius (bjornt)]

So there are several bugs here.

  1) maas-proxy fails to start
  2) RBAC was using the same port, 8000, as the proxy
  3) The configauth command shows help when something
     goes wrong.

I think the real bug that we need to focus on is 1)

2) has already been fixed, RBAC now uses port 5000 by default

3) is not RBAC specific and nothing new. You can reproduce
the same behavior by running the createadmin command twice,
using the same parameters both times.

[Björn Tillenius (bjornt)]

Andres, you seem to indicate that the real bug was that configauth wasn't working during the failing maas-proxy restart. That should be fixed now, since i think RBAC and the proxy were fighting for the ports.

You said that you fixed the maas-proxy problem manually, though. How did you fix it? Is that something that we need to fix?

[Björn Tillenius (bjornt)]

BTW, I did reproduce the maas-proxy failure, and I was able to use configauth to configure MAAS to use RBAC without any problems.

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]