recovering from syntax errors in custom-rules needs manual intervention

[Andrea Ieri]

If you pass malformed rules via the custom-rules charm option, `promtool check config` will fail, as well as the config-changed hook (as expected).

If you then fix your custom rules and update the charm option, the config-changed hook will however keep failing, because /var/snap/prometheus/current/custom.rules will not be overwritten.

If you `juju config prometheus --reset custom-rules` the config-changed hook will succeed, but only because the rule_files import will be disabled. The custom.rules file will not be deleted.

If you then reconfigure the charm with properly formatted rules, `promtool check config` will kick in before the custom.rules has a chance to be overwritten, so you'll still end up with a hook error.

Workaround: manually truncate /var/snap/prometheus/current/custom.rules via ssh. If you rm the file you'll only get a different hook error ("custom.rules does not point to an existing file") due to LP#1845329