Capomastro needs a command to safely lock users out and stop builds for service maintenance

[Caio Begotti]

With the new scheduling features of Capomastro it can get a bit complicated to stop the service to perform any type of maintenance as it will be talking to Jenkins constantly with multiple users logged in. We'd need a way to clear all current sessions, log these users out, make them inactive temporarily and (ideally, also) pause the scheduler beat. This command would have an opposite counterpart so it can all be re-enabled after the service maintenance.

Of course we can't stop Jenkins in any way, but Django seems to support what we need to do here. We'd just need to investigate how to pause the Celery beat but this seems doable, otherwise we may break things while trying to deploy fixes to production.

[Daniel Manrique (roadmr)]

Celery Beat periodic tasks can be "paused" by setting their "enabled" attribute to false. Then they will be skipped by the scheduler.

[Daniel Manrique (roadmr)]

We can use the horrid inline way of fiddling with the models as a workaround:

echo "from django.contrib.auth.models import User; User.objects.all().update(is_active=False)" | capomastro shell >/dev/null 2>&1

echo "from djcelery.models import PeriodicTask; PeriodicTask.objects.all().update(enable=False)" | capomastro shell >/dev/null 2>&1

Then just set them to True again to re-enable them.

This solution is a bit horrible because it just re-enables everything (i.e. if we had any disabled users or tasks they'd be enabled after this). We'd need something that is able to retrieve the existing state and then restore it after finishing. That would also require fetching the ID for each record (who knows, it may change as part of the process).

Also, the way to violently and without warning log all users out:

from django.contrib.sessions.models import Session; Session.objects.all().delete()

Maybe we could put this all into a "maintenance-mode" command for ease of use. An elegant solution would also have a table to store the current status of PeriodicTasks and Users for later restoring.

Anyway, it sounds like a feature that needs thought and planning so perhaps for the upcoming deployment we can use the one-liners, and work on a proper solution for the next milestone, since it looks like it will be an important component of the deployment process going forward.