Mismatch in external_ids:system-id and certificate CN leads to ovn-chassis node not registering itself

[Edward Hope-Morley]

I have an Openstack Train deployment on Bionic with one compute node. When I try to launch an instance it fails and i see the following in networking-ovn-metadata-agent.log

2020-01-06 11:12:17.548 3276 CRITICAL neutron [-] Unhandled error: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=82b33248-b723-4c9e-9731-d7ffeadeaffa
2020-01-06 11:12:17.548 3276 ERROR neutron Traceback (most recent call last):
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 111, in transaction
2020-01-06 11:12:17.548 3276 ERROR neutron yield self._nested_txns_map[cur_thread_id]
2020-01-06 11:12:17.548 3276 ERROR neutron KeyError: 139686119317728
2020-01-06 11:12:17.548 3276 ERROR neutron
2020-01-06 11:12:17.548 3276 ERROR neutron During handling of the above exception, another exception occurred:
2020-01-06 11:12:17.548 3276 ERROR neutron
2020-01-06 11:12:17.548 3276 ERROR neutron Traceback (most recent call last):
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/bin/networking-ovn-metadata-agent", line 10, in <module>
2020-01-06 11:12:17.548 3276 ERROR neutron sys.exit(main())
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/networking_ovn/cmd/eventlet/agents/metadata.py", line 17, in main
2020-01-06 11:12:17.548 3276 ERROR neutron metadata_agent.main()
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/networking_ovn/agent/metadata_agent.py", line 38, in main
2020-01-06 11:12:17.548 3276 ERROR neutron agt.start()
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/networking_ovn/agent/metadata/agent.py", line 193, in start
2020-01-06 11:12:17.548 3276 ERROR neutron self.register_metadata_agent()
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/networking_ovn/agent/metadata/agent.py", line 203, in register_metadata_agent
2020-01-06 11:12:17.548 3276 ERROR neutron ext_ids).execute(check_error=True)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2020-01-06 11:12:17.548 3276 ERROR neutron t.add(self)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-01-06 11:12:17.548 3276 ERROR neutron next(self.gen)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction
2020-01-06 11:12:17.548 3276 ERROR neutron del self._nested_txns_map[cur_thread_id]
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__
2020-01-06 11:12:17.548 3276 ERROR neutron self.result = self.commit()
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2020-01-06 11:12:17.548 3276 ERROR neutron raise result.ex
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2020-01-06 11:12:17.548 3276 ERROR neutron txn.results.put(txn.do_commit())
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
2020-01-06 11:12:17.548 3276 ERROR neutron command.run_idl(txn)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 149, in run_idl
2020-01-06 11:12:17.548 3276 ERROR neutron record = self.api.lookup(self.table, self.record)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 107, in lookup
2020-01-06 11:12:17.548 3276 ERROR neutron return self._lookup(table, record)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 147, in _lookup
2020-01-06 11:12:17.548 3276 ERROR neutron row = idlutils.row_by_value(self, rl.table, rl.column, record)
2020-01-06 11:12:17.548 3276 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 65, in row_by_value
2020-01-06 11:12:17.548 3276 ERROR neutron raise RowNotFound(table=table, col=column, match=match)
2020-01-06 11:12:17.548 3276 ERROR neutron ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=82b33248-b723-4c9e-9731-d7ffeadeaffa

and ovn-controller.log is full of:

2020-01-06T11:16:08.207Z|25079|chassis|WARN|Could not find Chassis : stored (82b33248-b723-4c9e-9731-d7ffeadeaffa) ovs (82b33248-b723-4c9e-9731-d7ffeadeaffa)

[Edward Hope-Morley (hopem)]

Adding another compute unit changed the behaviour slightly:

2020-01-06T11:40:10.160Z|170421|chassis|WARN|Could not find Chassis : stored (82b33248-b723-4c9e-9731-d7ffeadeaffa) ovs (juju-5cc013-ovn-testing-6.cloud.sts)
2020-01-06T11:40:10.161Z|170422|chassis|WARN|Could not find Chassis : stored (juju-5cc013-ovn-testing-6.cloud.sts) ovs (juju-5cc013-ovn-testing-6.cloud.sts)

And the new unit using is also using fqdn for chassis name.

sbdb shows:

Chassis "juju-5cc013-ovn-testing-6.cloud.sts"
    hostname: "juju-5cc013-ovn-testing-6.cloud.sts"
    Encap geneve
        ip: "10.5.0.51"
        options: {csum="true"}
    Port_Binding "cr-lrp-e9f37398-2d49-45b5-b044-87c30823029c"
Chassis "juju-5cc013-ovn-testing-13.cloud.sts"
    hostname: "juju-5cc013-ovn-testing-13.cloud.sts"
    Encap geneve
        ip: "10.5.0.42"
        options: {csum="true"}

[Andrew McLeod (admcleod)]

I am having a problem which may be related (focal ussuri on s390x):

https://pastebin.canonical.com/p/8TQP2qq9MK/

Permission denied in inserting items into the Chassis table

[Frode Nordahl (fnordahl)]

The charms deploy OVN with RBAC enabled, this in turn makes the database check client identity (CN of certificate) against which chassis it wants to makes writes against, if there is a mismatch it will be denied.

Could you provide the contents of the Open_vSwitch table which includes the external_id:hostname and external_id:system-id fields? (``ovs-vsctl list open_vswitch``)

Knowing the CN of the certificate in /etc/ovn and the host systems interpretation of its FQDN (``hostname -f``) would also be useful.

I guess the most complete way to get information on certificates would be:

    juju run --application ovn-chassis 'relation-ids certificates'
    juju run --application vault 'relation-get -r certificates:N - ovn-chassis/0'

And look at the 'common_name' and 'sans' keys

[Andrew McLeod (admcleod)]

cert CN:

ubuntu@s4lpa:~$ sudo openssl x509 -in /etc/ovn/cert_host -text -noout|grep internal
        Subject: CN = 10-13-3-10.internal
                DNS:10-13-3-10.internal, IP Address:10.13.3.10

ubuntu@s4lpa:~$ hostname -f
s4lpa

ovs-vsctl list open_vswitch: https://pastebin.ubuntu.com/p/FtpzQZHNtc/

~$ juju run --application vault 'relation-get -r certificates:64 - ovn-chassis/0'
certificate_name: 4eb1eeeb-001e-4ee4-b1ce-990faa4d11a6
common_name: 10-13-3-10.internal
egress-subnets: 10.13.3.10/32
ingress-address: 10.13.3.10
private-address: 10.13.3.10
sans: '["10.13.3.10"]'
unit_name: ovn-chassis_0

[Florian Guitton (f-guitton)]

Hello everybody,

Has anyone got any news about this issue ? We are encountering the same problem.
It would seem that the root cause is SSL validation and the charm might not be appropriately seting the Root CA from Vault on all nodes. We use Self-signed Root CA and no connections could be verified between any of the ovn-chassis/ovn-central and the system stalls.

Adding manually the certificate seem to have allowed us to get the northd to start running. But we are still experiencing large numbers of "connection dropped (Protocol error)" all over the place.

This is all the more strange considering that the cert place in /etc/ovn/ seem to check out:

root@od-13: ~# openssl verify -CAfile <(cat /etc/ovn/ovn-chassis.crt) /etc/ovn/cert_host
/etc/ovn/cert_host: OK
root@od-13: ~# openssl verify -CAfile <(cat /etc/ssl/certs/Imperial_DSI_Root_CA.pem) /etc/ovn/ovn-chassis.crt
/etc/ovn/ovn-chassis.crt: OK

[Frode Nordahl (fnordahl)]

Hello Florian,

Thank you for reaching out.

I would suspect a mismatch between the CN in the certificates and the name the chassis use to identify themselves to the database. Please take a look at comment #3 (https://bugs.launchpad.net/charm-ovn-chassis/+bug/1858416/comments/3).

Could you share some detail on your setup and provide the information requested in #3?

If there is a mismatch I would be very interested in figuring out how we get into the situation so we can try to avoid it.

[Florian Guitton (f-guitton)]

Hello Frode !

It would appear there might be a mismatch indeed.
For our setup we use MAAS to PXE boot and commission our environment. This happens on a 10.80.0.0/16 interface. We expect OpenStack to be deployed on interface bond0.30 with a range 172.30.0.0/16.
In Juju all spaces are bind to the 172.30.0.0/16 subnet.

Thank you for you support on this.
Feel free to ask for any other details.

You can find the information as follows:

root@od-13:~# hostname -f
od-13.metal.dsi.ic.ac.uk

root@od-13:~# ovs-vsctl list open_vswitch
_uuid : 72d5ae76-1cb1-4731-8179-32c6999ee2f9
bridges : [6db7c672-9e40-4a7f-9de3-e31438dc95ed]
cur_cfg : 7
datapath_types : [netdev, system]
datapaths : {}
db_version : "8.2.0"
dpdk_initialized : false
dpdk_version : none
external_ids : {hostname=od-13.metal.dsi.ic.ac.uk, ovn-encap-ip="172.30.200.51", ovn-encap-type=geneve, ovn-remote="ssl:172.30.200.46:6642,ssl:172.30.200.45:6642,ssl:172.30.200.39:6642", rundir="/var/run/openvswitch", system-id=od-13.metal.dsi.ic.ac.uk}
iface_types : [erspan, geneve, gre, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options : [2c189015-c836-4a38-a217-deb6f8de2d12]
next_cfg : 7
other_config : {}
ovs_version : "2.13.0"
ssl : c8c2e84e-6084-4b51-906b-893913d2b08c
statistics : {}
system_type : ubuntu
system_version : "20.04"

root@od-13:~# openssl x509 -text -in /etc/ovn/cert_host | grep CN
        Issuer: CN = Vault Intermediate Certificate Authority (charm-pki-local)
        Subject: CN = bond0.30.od-13.metal.dsi.ic.ac.uk

root@juju-playground:~# juju run --application ovn-chassis 'relation-ids certificates'
- Stdout: |
    certificates:68
  UnitId: ovn-chassis/0
- Stdout: |
    certificates:68
  UnitId: ovn-chassis/1

root@juju-playground:~# juju run --application vault 'relation-get -r certificates:68 - ovn-chassis/0'
cert_requests: '{"bond0.30.od-13.metal.dsi.ic.ac.uk": {"sans": ["172.30.200.51"]}}'
certificate_name: a46a8550-ccbe-4b0f-b2ea-25fbe1909ac7
common_name: od-13.metal.dsi.ic.ac.uk
egress-subnets: 10.80.0.11/32
ingress-address: 10.80.0.11
private-address: 10.80.0.11
sans: '["10.80.0.11"]'
unit_name: ovn-chassis_0

[Frode Nordahl (fnordahl)]

Excellent, thank you for your swift response Florian. Now we have the why, and need to figure out how.

For completeness could you provide the charm versions involved and information about any space bindings?

The charm makes an attempt at avoiding the many pitfalls of determining the correct FQDN to use by using the one the Open vSwitch init script populates the external_ids:hostname field in the database with on initial startup.

This should be used in the certificate request and it should be used to populate the external_ids:system-id field, which is what the `ovn-controller` process uses when identifying to the database.

The charm should also explicitly not request sans [0], but from your output it appears to have done so anyway.

0: https://github.com/openstack-charmers/charm-layer-ovn/blob/244d88779f10eed85d796d98ddfc678a64333ece/lib/charms/ovn_charm.py#L234-L242

[Florian Guitton (f-guitton)]

Certainly,

Below all of the information about the charm versions and spaces for them.
Please let me know if you need any other information.

root@juju-playground:~# juju status ovn*
Model Controller Cloud/Region Version SLA Timestamp
dsi-r1-openstack dsi-juju-controller dsi-maas/default 2.7.6 unsupported 11:07:37Z

App Version Status Scale Charm Store Rev OS Notes
nova-compute 21.0.0 active 2 nova-compute jujucharms 316 ubuntu
ovn-central 20.03.0 active 3 ovn-central jujucharms 0 ubuntu
ovn-chassis 20.03.0 active 2 ovn-chassis jujucharms 0 ubuntu

Unit Workload Agent Machine Public address Ports Message
nova-compute/0* active idle 7 10.80.0.11 Unit is ready
  ovn-chassis/0* active idle 10.80.0.11 Unit is ready
nova-compute/1 active idle 8 10.80.0.12 Unit is ready
  ovn-chassis/1 active idle 10.80.0.12 Unit is ready
ovn-central/0 active idle 0/lxd/4 172.30.200.46 6641/tcp,6642/tcp Unit is ready
ovn-central/1 active idle 1/lxd/5 172.30.200.45 6641/tcp,6642/tcp Unit is ready (northd: active)
ovn-central/2* active idle 2/lxd/3 172.30.200.39 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db)

Machine State DNS Inst id Series AZ Message
0 started 10.80.0.10 rh-09 focal SFO-02 Deployed
0/lxd/4 started 172.30.200.46 juju-92b05d-0-lxd-4 focal SFO-02 Container started
1 started 10.80.0.1 rh-08 focal SFO-02 Deployed
1/lxd/5 started 172.30.200.45 juju-92b05d-1-lxd-5 focal SFO-02 Container started
2 started 10.80.0.2 rh-07 focal SFO-02 Deployed
2/lxd/3 started 172.30.200.39 juju-92b05d-2-lxd-3 focal SFO-02 Container started
7 started 10.80.0.11 od-13 focal SFO-02 Deployed
8 started 10.80.0.12 od-14 focal SFO-02 Deployed

root@juju-playground:~# juju show-application ovn-chassis
ovn-chassis:
  charm: ovn-chassis
  series: focal
  principal: false
  exposed: false
  remote: false
  endpoint-bindings:
    "": dsi
    amqp: dsi
    certificates: dsi
    data: dsi
    juju-info: dsi
    nova-compute: dsi
    ovsdb: dsi
    ovsdb-subordinate: dsi

root@juju-playground:~# juju show-application ovn-central
ovn-central:
  charm: ovn-central
  series: focal
  principal: true
  exposed: false
  remote: false
  endpoint-bindings:
    "": dsi
    certificates: dsi
    ovsdb: dsi
    ovsdb-cms: dsi
    ovsdb-peer: dsi
    ovsdb-server: dsi

root@juju-playground:~# juju spaces
Name Space ID Subnets
alpha 0
dsi 1 172.30.0.0/16
                      fd51:0:d51:30::/64
pxe 3 10.80.0.0/16
r1-os-data 4 172.17.0.0/16
storage 2 172.18.0.0/16
undefined 6 10.1.0.0/16
           ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/730555

[Frode Nordahl (fnordahl)]

We have a built charm containing the proposed fix [0][1] up in cs:~fnordahl/ovn-chassis-bug1858416

To help confirm it fixes the issue you can run:

    juju upgrade-charm ovn-chassis --path cs:~fnordahl/ovn-chassis-bug1858416

To get back to the promulgated version:

    juju upgrade-charm ovn-chassis --path cs:ovn-chassis

0: https://github.com/openstack-charmers/charm-layer-ovn/pull/15
1: https://review.opendev.org/730555

[Florian Guitton (f-guitton)]

Good morning Frode !

Awesome ! This new charm seems to have done the trick.
Although I did have to explicitly remove the certificate relationship and re-create it for this to finally reissue. The chassis now seem to be registering in the service catalog.

I am now seeing SSL issues popping up on the ovn-central charm side.
Would it be that something is related (see below) ?
I can open a separate issue for this, that would make tracking easier.

Thank you for the support !

root@juju-92b05d-0-lxd-7:~# tail -f /var/log/ovn/ov*log
==> /var/log/ovn/ovn-northd.log <==
2020-05-25T09:07:34.988Z|29257|stream_ssl|ERR|Private key must be configured to use SSL
2020-05-25T09:07:34.988Z|29258|stream_ssl|ERR|Certificate must be configured to use SSL
2020-05-25T09:07:42.996Z|29259|stream_ssl|ERR|Private key must be configured to use SSL
2020-05-25T09:07:42.996Z|29260|stream_ssl|ERR|Certificate must be configured to use SSL

==> /var/log/ovn/ovsdb-server-nb.log <==
2020-05-25T09:07:08.151Z|09909|reconnect|WARN|ssl:172.30.200.45:56434: connection dropped (Protocol error)
2020-05-25T09:07:10.966Z|09910|stream_ssl|WARN|SSL_accept: system error (Success)
2020-05-25T09:07:10.966Z|09911|jsonrpc|WARN|ssl:172.30.200.39:51838: receive error: Protocol error
2020-05-25T09:07:10.966Z|09912|reconnect|WARN|ssl:172.30.200.39:51838: connection dropped (Protocol error)
2020-05-25T09:07:34.988Z|09913|stream_ssl|WARN|SSL_accept: system error (Success)
2020-05-25T09:07:34.988Z|09914|jsonrpc|WARN|ssl:172.30.200.39:52312: receive error: Protocol error

==> /var/log/ovn/ovsdb-server-sb.log <==
2020-05-25T09:06:30.925Z|12212|reconnect|WARN|ssl:172.30.200.39:40060: connection dropped (Protocol error)
2020-05-25T09:06:54.949Z|12213|stream_ssl|WARN|SSL_accept: system error (Success)
2020-05-25T09:06:54.949Z|12214|jsonrpc|WARN|ssl:172.30.200.39:40526: receive error: Protocol error
2020-05-25T09:06:54.950Z|12215|reconnect|WARN|ssl:172.30.200.39:40526: connection dropped (Protocol error)
2020-05-25T09:07:18.975Z|12216|stream_ssl|WARN|SSL_accept: system error (Success)
2020-05-25T09:07:18.975Z|12217|jsonrpc|WARN|ssl:172.30.200.39:40974: receive error: Protocol error

==> /var/log/ovn/ovn-northd.log <==
2020-05-25T09:07:59.004Z|29267|stream_ssl|ERR|Private key must be configured to use SSL
2020-05-25T09:07:59.004Z|29268|stream_ssl|ERR|Certificate must be configured to use SSL
2020-05-25T09:07:59.004Z|29269|stream_ssl|ERR|Private key must be configured to use SSL
2020-05-25T09:07:59.004Z|29270|stream_ssl|ERR|Certificate must be configured to use SSL

==> /var/log/ovn/ovsdb-server-nb.log <==
2020-05-25T09:07:59.004Z|09919|stream_ssl|WARN|SSL_accept: system error (Success)
2020-05-25T09:07:59.004Z|09920|jsonrpc|WARN|ssl:172.30.200.39:52768: receive error: Protocol error
2020-05-25T09:07:59.004Z|09921|reconnect|WARN|ssl:172.30.200.39:52768: connection dropped (Protocol error)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/730603

[Frode Nordahl (fnordahl)]

Thank you for confirming, we will start the process of landing the fix to the chassis charms and you can track the status in this bug.

If you have certificate related issues on the ovn-central units too, I would love to hear about them and would appreciate a separate bug report. Make sure the messages you refer to are recent and are not from the time runining up to when the charm actually writes certificates to disk and tells the ovn-northd and ovsdb-server processes to use them.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730555
Committed: https://git.openstack.org/cgit/x/charm-ovn-chassis/commit/?id=2eae38a8eff3208e28e0d2a444a40b9e7a263680
Submitter: Zuul
Branch: master

commit 2eae38a8eff3208e28e0d2a444a40b9e7a263680
Author: Frode Nordahl <email address hidden>
Date: Mon May 25 07:22:43 2020 +0200

    Fix certificate request and response handling

    Rebuild to pull in fixes [0] from layer-ovn.

    0: https://github.com/openstack-charmers/charm-layer-ovn/pull/15

    Change-Id: Id24d17ed3807e712479fb994ec1dd813510d1fc8
    Closes-Bug: #1858416

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730603
Committed: https://git.openstack.org/cgit/x/charm-ovn-dedicated-chassis/commit/?id=434ccef5b41b720a2e76b5030f9cbf051c0642fe
Submitter: Zuul
Branch: master

commit 434ccef5b41b720a2e76b5030f9cbf051c0642fe
Author: Frode Nordahl <email address hidden>
Date: Mon May 25 09:47:45 2020 +0200

    Fix certificate request and response handling

    Rebuild to pull in fixes [0] from layer-ovn.

    0: https://github.com/openstack-charmers/charm-layer-ovn/pull/15

    Change-Id: I9df1f5523426358b2c88bf96c7559dc119828ce1
    Closes-Bug: #1858416

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.05
Review: https://review.opendev.org/730610

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.05
Review: https://review.opendev.org/730612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730612
Committed: https://git.openstack.org/cgit/x/charm-ovn-dedicated-chassis/commit/?id=d0e0c89f19d587fe1b55a5275081817cfe89d32b
Submitter: Zuul
Branch: stable/20.05

commit d0e0c89f19d587fe1b55a5275081817cfe89d32b
Author: Frode Nordahl <email address hidden>
Date: Mon May 25 09:47:45 2020 +0200

    Fix certificate request and response handling

    Rebuild to pull in fixes [0] from layer-ovn.

    0: https://github.com/openstack-charmers/charm-layer-ovn/pull/15

    Change-Id: I9df1f5523426358b2c88bf96c7559dc119828ce1
    Closes-Bug: #1858416
    (cherry picked from commit 434ccef5b41b720a2e76b5030f9cbf051c0642fe)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730610
Committed: https://git.openstack.org/cgit/x/charm-ovn-chassis/commit/?id=2c1c24bace1b68fb50740046579d6997a7624571
Submitter: Zuul
Branch: stable/20.05

commit 2c1c24bace1b68fb50740046579d6997a7624571
Author: Frode Nordahl <email address hidden>
Date: Mon May 25 07:22:43 2020 +0200

    Fix certificate request and response handling

    Rebuild to pull in fixes [0] from layer-ovn.

    0: https://github.com/openstack-charmers/charm-layer-ovn/pull/15

    Change-Id: Id24d17ed3807e712479fb994ec1dd813510d1fc8
    Closes-Bug: #1858416
    (cherry picked from commit 2eae38a8eff3208e28e0d2a444a40b9e7a263680)