invalid cmr macaroon when getting cmr secret

[Guillaume Boutry]

This happened on a sunbeam deployed after a few days:
Command '('/var/lib/juju/tools/unit-openstack-hypervisor-0/secret-get', 'secret://29fcc45b-0a2a-4d46-81c2-cf9b508daa3c/cou5laofd6dn9f3mocv0', '--format=json')' returned non-zero exit status 1.

ops.model.ModelError: ERROR invalid cmr macaroon

juju: 3.4.2
cloud: maas provider

All 3 units are failing to read the secrets and are in error state. Rebooting the controller fixed it.

[Ian Booth (wallyworld)]

To help diagnose this, we really need a bit more information:
- logs from controller and affected models
- possibly a db dump of the application collection from the consuming model

Can we start by getting the logs and we can take a look and see if anything relevant reveals itself?

[Guillaume Boutry (gboutry)]

Here's the controller logs

[Guillaume Boutry (gboutry)]

Here's the affected model

[Guillaume Boutry (gboutry)]

Here's the model offering the CMR.

The secret is created by Keystone and sent over the CMR.

[Ian Booth (wallyworld)]

Unfortunately there's not enough in the logs to pin point the problem. If the problem were ongoing, or reproducible, we could increase the logging and get some extra diagnostics to look at. But as it's now fixed after a reboot, it's hard to say exactly what happened. One guess is clock skew, but it's just a guess.