test_audit_custom_policy takes longer than 15 minutes to run config changed hooks

[Alexander Balderson]

Test audit custom policy took longer than 30 mins to run the config changed hook.

The kubernetes-master/1 unit running the hook says it is running an action via juju-run.

There is nothing major in the logging, except the syslog shows some requests to the metrics failing, but has no return code, possibly because the request was too large.

This is running on commit hash c7b59b7

testrun can be found at https://solutions.qa.canonical.com/testruns/testRun/fcf8979e-cb7b-4fd0-9990-fed45d4fd675

and crashdump at https://oil-jenkins.canonical.com/artifacts/fcf8979e-cb7b-4fd0-9990-fed45d4fd675/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-02-27-03.12.21.tar.gz

[George Kraft (cynerva)]

It's hard to say exactly where the test failed since there's no stacktrace. But I don't think it's stuck waiting for a config-changed hook.

02:39:06 test_audit_custom_policy started
02:39:54 kubernetes-master/1 completes config-changed hook
02:42:08 kubernetes-master/1 acquires machine lock for juju-run (action 208), never releases it
03:09:06 test times out

Seems like it got stuck later in the test, maybe with one of the calls to juju run kubectl.

I'm marking this as incomplete since there's not much we can do with this report. Please fix:
https://github.com/juju/juju-crashdump/issues/81
https://github.com/juju/juju-crashdump/issues/82

so we can get some visibility into what these blocking juju-run calls are.

[Konstantinos Kaskavelis (kaskavel)]

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)