"model-upgrader" manifold worker returned unexpected error: model has been removed

[Paul Gear]

We have a 2.3.8 controller reporting constant logspam of the form:

{{CONTROLLER_MODEL_UUID}}: machine-0 2018-07-11 01:03:27 ERROR juju.worker.dependency engine.go:551 "model-upgrader" manifold worker returned unexpected error: model {{OTHER_MODEL_UUID}} has been removed

This controller is also experiencing occasional loss of connectivity from the outside world.

It is not confirmed yet whether these two facts are related, but regardless, the model upgrader should not continue to attempt upgrades on models which have been removed.

[Paul Gear (paulgear)]

Log extract: https://pastebin.canonical.com/p/NNGTzMKpbb/

[Paul Gear (paulgear)]

Logs at https://private-fileshare.canonical.com/~paulgear/lp1781096/

[Anastasia (anastasia-macmood)]

I will fix the model-upgrader worker to uninstall instead of restarting when the model it operates on is removed.

However, this is a very infrequent corner case that will only be observed on a system with problematic mongo (which usually indicates disk issues). Indeed, other log messages in the attached extract point to a problem on mongo. I'd expect you to experience other issues on this environment.

[Anastasia (anastasia-macmood)]

PR against 2.3: https://github.com/juju/juju/pull/8914

[Anastasia (anastasia-macmood)]

PR for 2.4: https://github.com/juju/juju/pull/8915

[Anastasia (anastasia-macmood)]

PR for 2.5 (develop): https://github.com/juju/juju/pull/8916

[John A Meinel (jameinel)]

Note that we still see this happening in juju 2.5.0:
2019-01-23 07:41:28 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
...
2019-01-23 07:41:37 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
79c03f03-eff2-41dc-831c-b7296c971b95 has been removed
...
2019-01-23 07:41:37 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed

...
2019-01-23 07:41:47 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
9b6a9a6a-e464-4f13-8339-322b8441962f has been removed
2019-01-23 07:41:47 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed

Grepping for a single model shows:
2019-01-23 07:41:01 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:05 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:09 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:14 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:21 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:28 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:37 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed
2019-01-23 07:41:47 ERROR juju.worker.dependency engine.go:632
"model-upgrader" manifold worker returned unexpected error: model
235f7c2f-b531-46e1-8438-1a663a614a68 has been removed

Which is bouncing approx every 3-10s. (The slower part might be due to
controller load as the controller isn't very happy.)

On Tue, Sep 18, 2018 at 7:25 PM Canonical Juju QA Bot <
<email address hidden>> wrote:

> ** Changed in: juju/2.3
> Status: Fix Committed => Fix Released
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1781096
>
> Title:
> "model-upgrader" manifold worker returned unexpected error: model has
> been removed
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1781096/+subscriptions
>

[Anastasia (anastasia-macmood)]

HA, I've seen this recently too and was going to address it. Glad I came across this bug!

[Anastasia (anastasia-macmood)]

PR for 2.5: https://github.com/juju/juju/pull/10265

[Anastasia (anastasia-macmood)]

PR for 2.6: https://github.com/juju/juju/pull/10267

[Anastasia (anastasia-macmood)]

PR for develop (heading into 2.7): https://github.com/juju/juju/pull/10273