certificate verify failed when curl requests succeed to same address

Bug #1859092 reported by Alexander Balderson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Barbican-Vault Charm
Fix Committed
High
Frode Nordahl

Bug Description

During a deploy of Openstack with Octavia, Barbican, Vault. This deploy is using a generated SSL certificate and setting hostnames for public address and ip addresses for internal addresses.

Every service is working as expected after unsealing vault EXCEPT barbican-vault, which is erroring with:
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.33.15', port=8200): Max retries exceeded with url: /v1/sys/wrapping/unwrap (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

192.168.33.15 is the vault VIP.

from the same machine, however, i can verify that the certificate is connecting and valid:
$ curl -v https://192.168.33.15:8200/v1
* Trying 192.168.33.15...
* TCP_NODELAY set
* Connected to 192.168.33.15 (192.168.33.15) port 8200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.solutionsqa
* start date: Jan 6 18:23:37 2020 GMT
* expire date: Jan 3 18:23:37 2030 GMT
* subjectAltName: host "192.168.33.15" matched cert's IP address!
* issuer: CN=solutionsqa
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55a717334580)
> GET /v1 HTTP/2
> Host: 192.168.33.15:8200
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< cache-control: no-store
< content-type: application/json
< content-length: 14
< date: Thu, 09 Jan 2020 21:42:23 GMT
<
{"errors":[]}
* Connection #0 to host 192.168.33.15 left intact

Crashdump and bundles + ssl keys to follow

Revision history for this message
Alexander Balderson (asbalderson) wrote :
Revision history for this message
Alexander Balderson (asbalderson) wrote :
Revision history for this message
Frode Nordahl (fnordahl) wrote :

This is most likely a bug in the charm or one of its layers/interfaces. The charm distributes all its dependencies and does not make use of system python packages.

A side effect of that is that the charm code does not get advantage of the Ubuntu patch for the ``certifi`` package which automatically includes system wide TLS certificates for verification of outgoing connections.

For connections to glance this is handled [0], but I do not see this referenced for the code talking to Vault.

So it is most likely easily fixable by passing on the system certificate bundle to the vault client.

0: https://github.com/openstack/charm-octavia-diskimage-retrofit/blob/ea828d58e0196265974be1e1d316148bd2885453/src/lib/charm/openstack/glance_retrofitter.py#L21

Frode Nordahl (fnordahl)
Changed in charm-barbican-vault:
status: New → Confirmed
importance: Undecided → Critical
importance: Critical → High
Revision history for this message
Frode Nordahl (fnordahl) wrote :
Changed in charm-barbican-vault:
status: Confirmed → In Progress
Revision history for this message
Frode Nordahl (fnordahl) wrote :

It would be helpful if you could test ``cs:~fnordahl/barbican-vault-bug1859092`` and report back if it fixes the issue.

Revision history for this message
Alexander Balderson (asbalderson) wrote :

I cant seem to find this charm when i build:
cannot resolve URL "cs:~fnordahl/barbican-vault-bug1859092"

I dont see it under you on the charm store either.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

Ah, sorry about that, I have re-applied the permissions and confirmed that a juju deploy is able to find the charm.

Revision history for this message
Alexander Balderson (asbalderson) wrote :

Thanks Frode
the fix seems to work, barbican-vault came up as expected and is using the cert.

thanks for the fix

Frode Nordahl (fnordahl)
Changed in charm-barbican-vault:
assignee: nobody → Frode Nordahl (fnordahl)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-barbican-vault (master)

Reviewed: https://review.opendev.org/702289
Committed: https://git.openstack.org/cgit/openstack/charm-barbican-vault/commit/?id=e23f232a6883d9629c8ed6cf1a103f1cff455a44
Submitter: Zuul
Branch: master

commit e23f232a6883d9629c8ed6cf1a103f1cff455a44
Author: Frode Nordahl <email address hidden>
Date: Mon Jan 13 21:15:06 2020 +0100

    Use system CA bundle when verifying Vault certificate

    Change-Id: I39d761dfbe1500f06abd617dd97eced671971b7d
    Closes-Bug: #1859092

Changed in charm-barbican-vault:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-barbican-vault (stable/19.10)

Fix proposed to branch: stable/19.10
Review: https://review.opendev.org/703387

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-barbican-vault (stable/19.10)

Change abandoned by Frode Nordahl (<email address hidden>) on branch: stable/19.10
Review: https://review.opendev.org/703387

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.