Rotation for the maas shared secret key does not work

[Nishant Kumar]

when rotating the 'rpc_shared_secret' I see the following error:

Oct 24 19:26:38 maas-region-1 maas.start-up[92]: [warn] Error during start-up; pausing for 3 seconds.
Oct 24 19:26:38 maas-region-1 sh[88]: 2019-10-24 19:26:38 maasserver.start_up: [error] Error during start-up.
Oct 24 19:26:38 maas-region-1 sh[88]: Traceback (most recent call last):
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/maasserver/start_up.py", line 68, in start_up
Oct 24 19:26:38 maas-region-1 sh[88]: yield security.get_shared_secret()
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
Oct 24 19:26:38 maas-region-1 sh[88]: result = inContext.theWork()
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
Oct 24 19:26:38 maas-region-1 sh[88]: inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
Oct 24 19:26:38 maas-region-1 sh[88]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
Oct 24 19:26:38 maas-region-1 sh[88]: return func(args,*kw)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
Oct 24 19:26:38 maas-region-1 sh[88]: return func(*args, **kwargs)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
Oct 24 19:26:38 maas-region-1 sh[88]: result = func(*args, **kwargs)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 710, in call_with_connection
Oct 24 19:26:38 maas-region-1 sh[88]: return func(*args, **kwargs)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/maasserver/utils/_init_.py", line 192, in call_with_lock
Oct 24 19:26:38 maas-region-1 sh[88]: return func(*args, **kwargs)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 754, in call_within_transaction
Oct 24 19:26:38 maas-region-1 sh[88]: return func_outside_txn(*args, **kwargs)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 561, in retrier
Oct 24 19:26:38 maas-region-1 sh[88]: return func(*args, **kwargs)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3.5/contextlib.py", line 30, in inner
Oct 24 19:26:38 maas-region-1 sh[88]: return func(*args, **kwds)
Oct 24 19:26:38 maas-region-1 sh[88]: File "/usr/lib/python3/dist-packages/maasserver/security.py", line 111, in get_shared_secret_txn
Oct 24 19:26:38 maas-region-1 sh[88]: get_shared_secret_filesystem_path())
Oct 24 19:26:38 maas-region-1 sh[88]: AssertionError: The secret stored in the database does not match the secret stored on the filesystem at /var/lib/maas/secret. Please investigate.

I am using the maas chart here - https://opendev.org/airship/maas/src/branch/master/charts/maas which contains the 'rpc_shared_secret' as k8s secret - https://opendev.org/airship/maas/src/branch/master/charts/maas/templates/secret-region.yaml

By looking at the maas source code it appears that the rotation is not supported i.e. the setting of 'rpc_shared_secret' to database is only done if the value is None (first time).

Problematic code (according to me) - https://github.com/maas/maas/blob/master/src/maasserver/security.py#L90

For supporting the rotation i would expect the secret to be set to database even in the case when it is not none while the secret in filesystem is already set.

[Jerzy Husakowski (jhusakowski)]

The helm chart is 3rd party, and may contain issues - there is a way to rotate the secret by stopping MAAS, updating the database and the files that contain it (all region secret files), and the chart may not be doing that.
Having said that - MAAS today does not offer an easy way to rotate secrets. We added secret rotation feature to our roadmap, no timeline yet.