Designate DNS and SSLError 524297

[Francesco Di Nucci]

High level description
Neutron can't delete a floating IP with a DNS entry in Designate when Designate endpoint is over HTTPS

Pre-conditions/Environment
Controller node (my-controller.example.com) with relevant packages for Ceilometer, Designate, Gnocchi, Heat, Nova, Placement. Designate is configured with WSGI and HTTPS, so API is shutdown.
Network node (my-controller.example.com) with relevant packages for Neutron and OVS. Neutron also uses WSGI.

Step-by-step reproduction steps
Usage of dashboard or CLI is not relevant, result is the same.
* Create a floating IP and with a DNS entry in OpenStack
* Try to delete it, get an error

Additional information
- Created a token with 'openstack token issue', used it to manually interrogate Designate API with curl, got a valid answer
- Downloaded certificate served by Designate (https://my-controller:9001) and manually verified it against Neutron's CA file with OpenSSL utilities. Checks are passed
- The same certificate is used for multiple services such as Keystone (it is a SAN certificate) and Neutron can interrogate them successfully
- Tried to set [designate]/insecure=true in neutron.conf on network node, restarted the services, seems to have no effect on the issue

Expected output
Floating IP is deleted, DNS entry is removed

Actual output
Neutron fails to delete the IP, reports this error. Full log attached.
delete failed: No details.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://my-controller.example.com:9001/v2/zones?name=my-zone.cloud.example.com.: HTTPSConnectionPool(host='my-controller.example.com', port=9001): Max retries exceeded with url: /v2/zones?name=my-zone.cloud.example.com. (Caused by SSLError(SSLError(524297, '[SSL] PEM lib (_ssl.c:4065)')))

Version:
  * OpenStack version - 2024.1 Caracal, RDO distribution
  * Linux distro, kernel - AlmaLinux 9.4, Linux 5.14.0-427.16.1.el9_4.x86_64;
  * Deployment mechanism - Puppet Openstack modules;

Attachments
Relevant neutron log at IP deletion attempt, sanitized from sensitive info

[Oleg Bondarev (obondarev)]

Looks like a setup issue to me, not sure how it could be fixed from neutron side, ideas?

[Francesco Di Nucci (d1nuc0m)]

I don't know, as the setup seems to be correct, as the API responds when interrogated otherwise and openssl is able to validate the certificates...

I think that there also is an issue with the insecure option, as it seems to have no effect - I assume that setting it to true should disable certificate verification, but I still get the same error

[MicheleDV (micheledelliveneri)]

We have fixed our SSL certificates setup on both the controller node (my-controller.example.com) hosting the designate service and the networking node hosting the neutron services.

On the network node, we can do the following:
- openstack token issue
- TOKEN="idofthetoken"
- curl -v -H "X-Auth-Token: $TOKEN" https://my-controller.example.com:9001/v2/zones?name=myzone.example.com

obtaining the following reply:

Trying 10.138.0.7:9001...
* Connected to my-controller.example.com (10.138.0.7) port 9001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=IT; ST=Roma; O=Istituto Nazionale di Fisica Nucleare; CN=my-controller.example.com
* start date: Jul 23 00:00:00 2024 GMT
* expire date: Jul 23 23:59:59 2025 GMT
* subjectAltName: host "my-controller.example.com" matched cert's "my-controller.example.com"
* issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV RSA CA 4
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /v2/zones?name=myzone.example.com HTTP/1.1
> Host: my-controller.example.com:9001
> User-Agent: curl/7.76.1
> Accept: */*
> X-Auth-Token: gAAAAABmn73qEXjhYYq-OE5RYZOjcXBk5W0W7r-W4s0ap6RHqTntuUsRmAp1p34dXNEzPWr0ZGtU-41JAGobqrxnT3Eia8RiWq0bz27knGo_g5jHLPc6L3CpP9LcfT_xB7WW6Y_mdouLOKI90G0QWc0BOBYnxP6GJ2udOi48n4EboKffmQVlunA
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 23 Jul 2024 14:28:20 GMT
< Server: Apache
< Content-Length: 142
< x-openstack-request-id: req-13a3f65c-b823-4a46-866e-97b30a0669cf
< Vary: Accept-Encoding
< Content-Type: application/json
<
* Connection #0 to host my-controller.example.com left intact
{"zones": [], "links": {"self": "https://my-controller.example.com:9001/v2/zones?name=myzone.example.com"}, "metadata": {"total_count": 0}}

while if we try to create a floating ip filling the dns domain and name info, or delete it later, we get the following error in /var/log/httpd/neutron_wsgi_error_ssl.log

2024-07-23 16:32:58.6...

[MicheleDV (micheledelliveneri)]

Also by enabling trace7 in designate wsgi, we see the following error in /etc/httpd/designate_wsgi_error_ssl.log

[Tue Jul 23 17:09:25.037724 2024] [ssl:info] [pid 18308:tid 18308] [client 10.138.0.10:34276] AH01964: Connection to child 9 established (server my-controller.example.com:443)
[Tue Jul 23 17:09:25.038070 2024] [ssl:trace2] [pid 18308:tid 18308] ssl_engine_rand.c(125): Server: Seeding PRNG with 656 bytes of entropy
[Tue Jul 23 17:09:25.038269 2024] [ssl:trace3] [pid 18308:tid 18308] ssl_engine_kernel.c(2202): [client 10.138.0.10:34276] OpenSSL: Handshake: start
[Tue Jul 23 17:09:25.038371 2024] [ssl:trace3] [pid 18308:tid 18308] ssl_engine_kernel.c(2210): [client 10.138.0.10:34276] OpenSSL: Loop: before SSL initialization
[Tue Jul 23 17:09:25.038391 2024] [ssl:trace1] [pid 18308:tid 18308] ssl_engine_io.c(588): [client 10.138.0.10:34276] BUG: bio_filter_in_ctrl() should not be called with cmd=76
[Tue Jul 23 17:09:25.038444 2024] [ssl:trace4] [pid 18308:tid 18308] ssl_engine_io.c(2415): [client 10.138.0.10:34276] OpenSSL: I/O error, 5 bytes expected to read on BIO#563612a4fb00 [mem: 563612cc9393]
[Tue Jul 23 17:09:25.038497 2024] [ssl:trace6] [pid 18308:tid 18308] ssl_engine_io.c(220): [client 10.138.0.10:34276] bio_filter_out_write: 7 bytes
[Tue Jul 23 17:09:25.038552 2024] [ssl:trace4] [pid 18308:tid 18308] ssl_engine_io.c(2405): [client 10.138.0.10:34276] OpenSSL: write 7/7 bytes to BIO#563612bc3460 [mem: 563612cd1550] (BIO dump follows)
[Tue Jul 23 17:09:25.038568 2024] [ssl:trace7] [pid 18308:tid 18308] ssl_engine_io.c(2335): [client 10.138.0.10:34276] +-------------------------------------------------------------------------+
[Tue Jul 23 17:09:25.038587 2024] [ssl:trace7] [pid 18308:tid 18308] ssl_engine_io.c(2372): [client 10.138.0.10:34276] | 0000: 15 03 03 00 02 02 32 ......2 |
[Tue Jul 23 17:09:25.038597 2024] [ssl:trace7] [pid 18308:tid 18308] ssl_engine_io.c(2377): [client 10.138.0.10:34276] +-------------------------------------------------------------------------+
[Tue Jul 23 17:09:25.038608 2024] [ssl:trace6] [pid 18308:tid 18308] ssl_engine_io.c(155): [client 10.138.0.10:34276] bio_filter_out_write: flush
[Tue Jul 23 17:09:25.038678 2024] [core:trace6] [pid 18308:tid 18308] core_filters.c(828): [client 10.138.0.10:34276] writev_nonblocking: 7/7
[Tue Jul 23 17:09:25.038705 2024] [ssl:trace3] [pid 18308:tid 18308] ssl_engine_kernel.c(2220): [client 10.138.0.10:34276] OpenSSL: Write: error
[Tue Jul 23 17:09:25.038717 2024] [ssl:trace1] [pid 18308:tid 18308] ssl_engine_io.c(588): [client 10.138.0.10:34276] BUG: bio_filter_in_ctrl() should not be called with cmd=76
[Tue Jul 23 17:09:25.038729 2024] [ssl:trace3] [pid 18308:tid 18308] ssl_engine_kernel.c(2239): [client 10.138.0.10:34276] OpenSSL: Exit: error in error
[Tue Jul 23 17:09:25.038748 2024] [ssl:info] [pid 18308:tid 18308] (70014)End of file found: [client 10.138.0.10:34276] AH02008: SSL library error 1 in handshake (server my-controller.example.com:443)
[Tue Jul 23 17:09:25.038787 2024] [ssl:info] [pid 18308:tid 18308] SSL Library Error: error:0A000126:SSL routines::unexpected eof while reading
[Tue Jul 23 17:09:25.038864 2024...