Juju status fails due to timeout getting MAAS API version

[Michael Skalka]

As seen during this test run: https://solutions.qa.canonical.com/testruns/testRun/25c338a2-43e6-447e-9e1a-c3c31fd0a682
Full artifcats here: https://oil-jenkins.canonical.com/artifacts/25c338a2-43e6-447e-9e1a-c3c31fd0a682/index.html
Controller crashdump: https://oil-jenkins.canonical.com/artifacts/25c338a2-43e6-447e-9e1a-c3c31fd0a682/generated/generated/juju_maas_controller/juju-crashdump-controller-2020-12-13-17.48.03.tar.gz

A call to `juju status` during execution of juju-wait (a service that polls status checking for certain condition) resulted in this error:

...
ERROR:root:ERROR Get "http://10.X.X.X:80/MAAS/api/2.0/version/": proxyconnect tcp: dial tcp 91.X.X.X:3128: i/o timeout

ERROR:root:juju status --format=json failed: 1
...

This status call was being made continually (every 5-10 seconds) for the past 30-40 minutes, indicating that Juju could indeed talk to the MAAS server at some point seconds before the failed call. Juju should be falling back to cached information for these status calls whenever possible and only surfacing an error when it truly cannot give status about the model. The information from MAAS in a deployed model will likely not be changing over time.

[Pen Gale (pengale)]

We probably should be using cached information here, rather than hitting the provider API for each status.

[Pen Gale (pengale)]

Note that it looks like you are routing to MAAS through a proxy here. You might be able to work around this by talking directly to MAAS, avoiding soaking whatever box Juju is jumping through.

[John A Meinel (jameinel)]

As a general design issue, 'juju status' shouldn't be connecting to the underlying provider to return the status results.

[John A Meinel (jameinel)]

We also got a field issue on an external cloud:

lihuiguo
21:41

Hi folks, I hit this issue on a customer cloud that juju status is working but juju status --format always returned 'timed out'. any thoughts of how to fix it? thanks

$ juju status --format=yaml
ERROR health ping timed out after 30s
{}
ERROR Get "http://172.18.247.21/MAAS/api/2.0/version/": EOF

$ juju export-bundle --filename bundle-$(date +%F).yaml
ERROR getting provider registry: Get "http://172.18.247.21/MAAS/api/2.0/version/": EOF

[John A Meinel (jameinel)]

Both of those are signs of poor communication with the underlying provider, but that shouldn't impact status.

[Ian Booth (wallyworld)]

The server side facade was eagerly creating a storage registry and pool manager which hit the maas api, even though status was just reading data from mongo and never using those apis. We now lazily create the storage pool manager etc

https://github.com/juju/juju/pull/13027