After further investigation, I can see that we definitely can end up in a twist.
It looks like we may not release a connection to a model in a timely manner after the model has been destroyed (pool item flagged as removed).
I have seen similar errors from other API calls and in the workers' code. It can actually be fairly frequent, depending on the model and how saturated it was - deployed charms, users, etc.
This is just a small sample of places that I have extracted both from the log output above as well as from my own testing:
* "model-upgrader" [DEBUG juju.worker.dependency engine.go:553 stack trace:
model XXX has been removed
github.com/juju/juju/worker/modelupgrader/worker.go:86:
github.com/juju/juju/worker/modelupgrader/manifold.go:66:]
* Controller[""].AllModels
* ModelManager[""].ModelStatus
* ModelManager[""].ListModels
I will work on tracing where we are not releasing model reference.
I will also investigate all callers of the pool.Get() and pool.GetModel() to consider how they should handle NotFound error.
After further investigation, I can see that we definitely can end up in a twist.
It looks like we may not release a connection to a model in a timely manner after the model has been destroyed (pool item flagged as removed).
I have seen similar errors from other API calls and in the workers' code. It can actually be fairly frequent, depending on the model and how saturated it was - deployed charms, users, etc.
This is just a small sample of places that I have extracted both from the log output above as well as from my own testing:
* "model-upgrader" [DEBUG juju.worker. dependency engine.go:553 stack trace: com/juju/ juju/worker/ modelupgrader/ worker. go:86: com/juju/ juju/worker/ modelupgrader/ manifold. go:66:]
model XXX has been removed
github.
github.
* Controller[ ""].AllModels
* ModelManager[ ""].ModelStatus
* ModelManager[ ""].ListModels
I will work on tracing where we are not releasing model reference.
I will also investigate all callers of the pool.Get() and pool.GetModel() to consider how they should handle NotFound error.