Monitor expiration of OVN certs

[Giuseppe Petralia]

There is currently no monitoring for the expiry of certificates used by ovn-chassis, ovn-central and neutron-api-plugin-ovn:

* /etc/ovn/cert_host
* /etc/neutron/plugins/ml2/cert_host

If these certificates are left to expire, neutron-server can't talk to ovn NB and neutron API become unreachable. The NRPE relation could be extended to issue a warning or a critical based on configurable threshold that may default respectively to 30 and 15 days.

[Frode Nordahl (fnordahl)]

Hello, Giuseppe, and thank you for your bug report.

Providing a way to easily monitor certificate lifetime in a system that make use of PKI for authentication and authorization is indeed important.

The OVN charms make use of Juju to integrate with an application providing the Certificate Authority and automation for Certificate Issuance, for example Vault.

Would it perhaps make sense to implement monitoring of lifetime of issued certificates in the charm/application responsible for managing the certificates instead?

[Giuseppe Petralia (peppepetra)]

Hi Frode, thanks for your reply.

If I read correctly you are suggesting to implement the check on the application providing the certs, which in our cases for OVN is 100% of times Vault.

I see some pros and cons of this choice:

Pros:
* we get out of the box monitoring for all certs provided by Vault, i.e. OpenStack API certs, Octavia certs, OVN certs etc.

Cons:
* We are only monitoring the certs that are in Vault. If the distribution of these certs fails, see LP#1940549, our monitoring will miss that as it will think the certs are all renewed but in reality the certs in the ovn-chassis or ovn-central units are still the old ones as Vault failed to provide the updated ones.

To be on the safe side, I think we should always monitor what is actually being used by the applications so I would like to see these checks in the OVN charms.

[Edward Hope-Morley (hopem)]

https://review.opendev.org/c/x/charm-ovn-central/+/887411

[Edward Hope-Morley (hopem)]

Having looked into this a bit further I think it might actually not be possible. Nrpe checks run as the nagios user and the ovn certs have the following permissions:

# ll /etc/ovn/
total 24
dr-xr-xr-x 2 root root 4096 Jun 23 13:07 ./
drwxr-xr-x 101 root root 4096 Jul 3 12:44 ../
-rw-r----- 1 root root 1532 Jun 23 13:07 cert_host
-rw-r----- 1 root root 1674 Jun 23 13:07 key_host
-rw-r--r-- 1 root root 1244 Jul 3 10:50 ovn-central.crt
-rw-r----- 1 root root 211 Jun 23 13:07 ovn-northd-db-params.conf

So an nrpe check cannot read the cert file unless its permissions are opened up which I assume is not what anyone wants to do.

[Edward Hope-Morley (hopem)]

It has been brought to my attention that we do have some charms with nrpe checks for root owned resources and they are implemented using a cron job that runs as root and performs the check itself while the nrpe check checks the output of the cron check e.g. rabbitmq-server:

https://github.com/openstack/charm-rabbitmq-server/blob/3c155e2bdaeaec5090111749a2cf366b55875575/hooks/rabbit_utils.py#L1630

Another option is to use the update-status hook, which runs by default every 5 minutes, to perform the check then store the output in a location that the nrpe check can read.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/887411
Committed: https://opendev.org/x/charm-ovn-central/commit/e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1
Submitter: "Zuul (22348)"
Branch: master

commit e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1
Author: Edward Hope-Morley <email address hidden>
Date: Fri Jun 30 17:27:22 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: I7c7cd238ddf3fd9f92bfa5879d19d78c091cf2ac

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/22.09
Review: https://review.opendev.org/c/x/charm-ovn-central/+/895021

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/23.03
Review: https://review.opendev.org/c/x/charm-ovn-central/+/895022

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/896409

[Eric Chen (eric-chen)]

decide not to implement it in charm-openstack-service-checks , so mark it as invalid

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-chassis/+/896409
Committed: https://opendev.org/x/charm-ovn-chassis/commit/fec04c22a84e5be8867712f3c925efef69d64d70
Submitter: "Zuul (22348)"
Branch: master

commit fec04c22a84e5be8867712f3c925efef69d64d70
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 25 16:27:48 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: Ia6df3fc2e3bc0d64ef2128ed65959a5d2d9b3d62

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/23.09
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/907065

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/895022
Committed: https://opendev.org/x/charm-ovn-central/commit/6b971674072b08f8ff64882990ae5961269ede38
Submitter: "Zuul (22348)"
Branch: stable/23.03

commit 6b971674072b08f8ff64882990ae5961269ede38
Author: Edward Hope-Morley <email address hidden>
Date: Fri Jun 30 17:27:22 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: I7c7cd238ddf3fd9f92bfa5879d19d78c091cf2ac
    (cherry picked from commit e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/895021
Committed: https://opendev.org/x/charm-ovn-central/commit/3b100ab6a18a0835c6e96dc3e34cd0f3b1e6d61e
Submitter: "Zuul (22348)"
Branch: stable/22.09

commit 3b100ab6a18a0835c6e96dc3e34cd0f3b1e6d61e
Author: Edward Hope-Morley <email address hidden>
Date: Fri Jun 30 17:27:22 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: I7c7cd238ddf3fd9f92bfa5879d19d78c091cf2ac
    (cherry picked from commit e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-central/+/890216
Committed: https://opendev.org/x/charm-ovn-central/commit/2a5157b50fb4860ba48a4082200d9387819cf68d
Submitter: "Zuul (22348)"
Branch: stable/22.03

commit 2a5157b50fb4860ba48a4082200d9387819cf68d
Author: Edward Hope-Morley <email address hidden>
Date: Fri Jun 30 17:27:22 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: I7c7cd238ddf3fd9f92bfa5879d19d78c091cf2ac
    (cherry picked from commit e3502b2c9c5f65ee8c0f3ed1aff8ef66c94de2f1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/23.03
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/909404

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/22.09
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/909405

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/22.03
Review: https://review.opendev.org/c/x/charm-ovn-chassis/+/909406

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-chassis/+/909405
Committed: https://opendev.org/x/charm-ovn-chassis/commit/b682697da04c03009a144846c9fb98c432cc3d79
Submitter: "Zuul (22348)"
Branch: stable/22.09

commit b682697da04c03009a144846c9fb98c432cc3d79
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 25 16:27:48 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: Ia6df3fc2e3bc0d64ef2128ed65959a5d2d9b3d62
    (cherry picked from commit fec04c22a84e5be8867712f3c925efef69d64d70)
    (cherry picked from commit 9c670651d81c50ede859f26ca0547497be0cba3b)
    (cherry picked from commit f64ca7dac00db70a091a36c0a1e45d51a8e01d17)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-chassis/+/909404
Committed: https://opendev.org/x/charm-ovn-chassis/commit/f64ca7dac00db70a091a36c0a1e45d51a8e01d17
Submitter: "Zuul (22348)"
Branch: stable/23.03

commit f64ca7dac00db70a091a36c0a1e45d51a8e01d17
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 25 16:27:48 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: Ia6df3fc2e3bc0d64ef2128ed65959a5d2d9b3d62
    (cherry picked from commit fec04c22a84e5be8867712f3c925efef69d64d70)
    (cherry picked from commit 9c670651d81c50ede859f26ca0547497be0cba3b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-chassis/+/907065
Committed: https://opendev.org/x/charm-ovn-chassis/commit/9c670651d81c50ede859f26ca0547497be0cba3b
Submitter: "Zuul (22348)"
Branch: stable/23.09

commit 9c670651d81c50ede859f26ca0547497be0cba3b
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 25 16:27:48 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: Ia6df3fc2e3bc0d64ef2128ed65959a5d2d9b3d62
    (cherry picked from commit fec04c22a84e5be8867712f3c925efef69d64d70)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/x/charm-ovn-chassis/+/909406
Committed: https://opendev.org/x/charm-ovn-chassis/commit/9d2a99ed99d7be68d299cb19e4f01c37cf62aad0
Submitter: "Zuul (22348)"
Branch: stable/22.03

commit 9d2a99ed99d7be68d299cb19e4f01c37cf62aad0
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 25 16:27:48 2023 +0100

    Add ovn cert nrpe check

    Certs are root readable so we use a cron job to perform
    the check and save state for an nrpe check to read and
    send back to nagios.

    Closes-Bug: #1979539
    Change-Id: Ia6df3fc2e3bc0d64ef2128ed65959a5d2d9b3d62
    (cherry picked from commit fec04c22a84e5be8867712f3c925efef69d64d70)
    (cherry picked from commit 9c670651d81c50ede859f26ca0547497be0cba3b)
    (cherry picked from commit f64ca7dac00db70a091a36c0a1e45d51a8e01d17)
    (cherry picked from commit b682697da04c03009a144846c9fb98c432cc3d79)