Restore with expired certificates fails

[Virginia Martins Perozim]

Brief Description
-----------------
If the backup is taken and the certs are expired or expiring before the restore the restore fails and cannot be recovered. The step that fails is 'system certificate-install.." step of the bootstrap.

Severity
--------
Major

Steps to Reproduce
------------------
1) Create an CA that will expire in a short timeframe
2) Wait for the CA to expire
3) Perform a backup
4) Follow the procedures to restore the system.

Expected Behavior
------------------
If certificates are expired the restore should not fail and if the certificates expire a restart of restore would be possible.

Actual Behavior
----------------
Restore is failing and there is no way to recover.

Reproducibility
---------------
100% reproducible.

System Configuration
--------------------
One node system.

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
-

Timestamp/Logs
--------------
TASK [bootstrap/persist-config : Add ssl_ca certificate] **********************************************************************
Saturday 06 August 2022 13:28:42 +0000 (0:00:00.175) 0:10:28.546 *******
fatal: [localhost]: FAILED! => changed=true
  cmd: source /etc/platform/openrc; system certificate-install -m ssl_ca /tmp/ca-cert.pem
  delta: '0:00:02.533496'
  end: '2022-08-06 13:28:45.665924'
  msg: non-zero return code
  rc: 1
  start: '2022-08-06 13:28:43.132428'
  stderr: 'Certificate /tmp/ca-cert.pem not installed: certificate is not valid before 2022-08-04 21:13:50 nor after 2022-08-05 21:13:50'
  stderr_lines:
  - 'Certificate /tmp/ca-cert.pem not installed: certificate is not valid before 2022-08-04 21:13:50 nor after 2022-08-05 21:13:50'
  stdout: |-
    WARNING: For security reasons, the original certificate,
    containing the private key, will be removed,
    once the private key is processed.
  stdout_lines: <omitted>

Test Activity
-------------
B&R.

Workaround
----------
-

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/852785

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/852785
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/f3bb3a64fc5064c16fd0a203a717872462d53ae7
Submitter: "Zuul (22348)"
Branch: master

commit f3bb3a64fc5064c16fd0a203a717872462d53ae7
Author: Virginia Martins Perozim <email address hidden>
Date: Wed Aug 10 14:12:05 2022 -0400

    Restore with ssl_ca certificate as command option

    ssl_ca certificate was included as an optional parameter of restore
    playbook.
    If it is defined in the command line, it will replace the certificate
    defined in backup file.
    If it is not defined in the command line, the certificate defined in
    the backup file will be used.

    Test Plan:

    PASSED: expired certificate in backup file during restore
            ssl_ca certificate not defined
            bug reproduced

    PASSED: expired certificate in backup file during restore
            valid ssl_ca certificate defined
            restore pass with no errors

    PASSED: valid certificate in backup file during restore
            valid ssl_ca certificate not defined
            restore pass with no errors

    PASSED: test on remote play

    Closes-Bug: 1984116
    Signed-off-by: Virginia Martins Perozim <email address hidden>
    Change-Id: I6c1ddca1097fbbdb0acde7df3097287ab59b4c1c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/871853

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/871853
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/a03a5af13c511dcb6c52b8b63256a55d73c8f404
Submitter: "Zuul (22348)"
Branch: master

commit a03a5af13c511dcb6c52b8b63256a55d73c8f404
Author: Gustavo Pereira <email address hidden>
Date: Thu Jan 26 17:35:19 2023 -0300

    Fix backup-restore with expired certificate

    This commit updates the certificates database table according to
    the ssl_ca certificates installed during the restore procedure.
    The installed certificates can be obtained either from the
    backup file or by the file specified by the parameter
    'ssl_ca_certificate_file'.
    This parameter was implemented on the following reviews:

    https://review.opendev.org/c/starlingx/ansible-playbooks/+/852785
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/855362

    In the end of the original restore procedure, the certificates
    database table is restored according to its state from the
    backup file, instead of considering the ssl_ca certificates
    installed earlier in the restore playbook from the file
    specified in the ssl_ca_certificate_file parameter.
    This can cause a mismatch between the certificate database table
    and the certificates actually installed in the filesystem,
    making the database system commands such as
    'system certificate-uninstall', not work properly.

    The solution was to backup the certificate database table to
    a tmp file before postgres DB restore and, once postgres DB is
    restored, replace the certificate table in postgres with the
    data from the tmp file. To guarantee that the certificates
    are inserted correctly to the database table, a sequence
    reset step was created, this step rearrenge the table rows
    avoiding a duplicate row id error. That will ensure the
    database table matches with certificates in the filesystem.

    PASS: Tested on a SX VM with an expired certificate,
    using the ssl_ca_certificate flag and without using it.

    PASS: Tested the commands to install
    (system certificate-install -m ssl_ca <cert.pem>)
    and uninstall certificates
    (system certificate-uninstall -m ssl_ca <ssl_uuid>).

    PASS: Tested scenario to restore backup with multiple
    expired certificates.

    PASS: Tested scenario adding a valid certificate
    before running backup process.

    Closes-Bug: 1984116

    Signed-off-by: Gustavo Pereira <email address hidden>
    Change-Id: Iaac2cfe9f456e4ae4a5ba21a6cf8ca96ca1e1eb8