instance_faults entries are created on InstanceInvalidState exceptions

[Pavlo Shchelokovskyy]

This might somewhat be related to https://bugs.launchpad.net/nova/+bug/1800755 and discussion there.

Recently the following problem was reported in one of our clouds:

- a homegrown self-written monitoring that polls servers diagnostics
- the monitoring script is naive and does not check the server state before requesting server diagnostics
- several servers in shutdown state
- instance_faults table is growing and ballooning database size on disk

During handling of GET /servers/<uuid>/diagnostics call for anything but RUNNING instance nova raises InstanceInvalidState exception which is then:
- stored in instance_faults table;
- returns as HTTP409 Conflict to the user.

https://opendev.org/openstack/nova/src/commit/03d2715ed492350fa11908aea0fdd0265993e284/nova/compute/manager.py#L6550-L6558

Effectively benign 'read-only' GET requests are recorded in the DB. Also, these instance_faults entries can not purged by standard means since the instance is not deleted yet. What's more, they won't be shown in any API at all, since the server is also not in ERROR state.

This got me thinking - should the InvalidInstanceState be saved as instance_faults at all?
After all, usually this exception indicates not the problem (fault) with the instance, but the mismatch between instance state and requested action upon instance, which might not warrant storing it.

There's also a slight DoS potential here, but since default policy for get diagnostics call is admin-only, this is probably not worth worrying.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/860702