Juju controller fails to remove offers from model

[Steven Parker]

performed service restarts on all juju controllers just in case that worked but it did not help

---

juju offers
Offer User Relation id Status Endpoint Interface Role Ingress subnets
filebeat-maas-infra -
nrpe-maas-infra -
telegraf-maas-infra -

--------------------------------------

juju status --relations

...

Offer Application Charm Rev Connected Endpoint Interface Role
filebeat-maas-infra filebeat-maas-infra filebeat 24 0/0

...

------------------------------------

 juju remove-offer admin/maas-infra.filebeat-maas-infra --debug
01:07:03 INFO juju.cmd supercommand.go:56 running juju [2.9.19 0 bae1644c23fd86e58dbf7249af55a236d0159099 gc go1.14.15]
01:07:03 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/17725/bin/juju", "remove-offer", "admin/maas-infra.filebeat-maas-infra", "--debug"}
01:07:03 INFO juju.juju api.go:78 connecting to API addresses: [10.145.252.41:17070 10.145.252.40:17070 10.145.252.42:17070]
01:07:03 DEBUG juju.api apiclient.go:1149 successfully dialed "wss://10.145.252.40:17070/api"
01:07:03 INFO juju.api apiclient.go:680 connection established to "wss://10.145.252.40:17070/api"
01:07:04 DEBUG juju.api monitor.go:35 RPC connection died
ERROR cannot delete application offer "filebeat-maas-infra": state changing too quickly; try again soon
01:07:04 DEBUG cmd supercommand.go:537 error stack:
/build/snapcraft-juju-35d6cf/parts/juju/src/apiserver/params/params.go:103: cannot delete application offer "filebeat-maas-infra": state changing too quickly; try again soon

----------------------------

Tailing the logsink.log on one of the controllers did not show any logs for this issue

[Steven Parker (sbparke)]

This bug occurred on the same juju controller as bug https://bugs.launchpad.net/juju/+bug/1952796

Relevant artifacts for this bug...

Juju controller SOS reports:
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-1-Juju-CMR-Controller-0-2021-11-30-npxesvi.tar.xz
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-3-Juju-CMR-Controller-1-2021-11-30-olgynuc.tar.xz
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-2-Juju-CMR-Controller-2-2021-11-30-lahomhg.tar.xz

Crash Dumps for LMA models:
https://private-fileshare.canonical.com/~sbparke/juju-crashdump-59a7cfdd-f754-4641-aebd-26c072ca29eb-lmaModel.tar.xz

Export of maas-infra model failed due to bug https://bugs.launchpad.net/juju/+bug/1952799

Controller database:
https://private-fileshare.canonical.com/~sbparke/sanitizedMongoDB.tgz

[Ian Booth (wallyworld)]

It would be good to understand how we got to this state. Regardless, there should not be errors like "state changing too quickly; try again soon" - that indicates slightly broken logic. So there's 2 parts here - how did the model end up in an inconsistent state; and juju should be able to deal with broken data better.
For now, we may need a little db surgery to fix it.

[Ian Booth (wallyworld)]

The error message 'cannot delete application offer "blah"' is printed when remove-offer is used without '--force'.

The use of '--force' might unblock the removal as it tries to avoid errors related bad data etc. What happens if you try this?

[Ian Booth (wallyworld)]

Looking at the db dump, it seems that:

relation count on the filebeat-maas-infra app is 0
yet there's a relation in the model between filebeat-maas-infra and infra-node
(this has id 4)

And I think this consistency check is what might be stopping the offer deletion.

You could try to fix it

db.applications.update(
{"_id": "d06afd52-89a7-41bb-8441-81a80782bf48:filebeat-maas-infra"},
{$set: {"relationcount": 1}}
)

[Steven Parker (sbparke)]

Thanks Ian,

We ended up redeploying. But I will put this in our notes if it happens again.

[Launchpad Janitor (janitor)]

[Expired for Canonical Juju because there has been no activity for 60 days.]