s3 backend fails with invalid certificate when using s3 compatible storage

[Antony Messerli]

When using the Glance s3 backend, if you are using an s3 compatible store, image operations fail with:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129).

The current implementation uses boto3 and assumes you are only using Amazon's implementation as there are not currently any settings for overriding the CA. In my case, we are using an s3 compatible on-prem device which has internal corporate certs. If I override using an environment variable of AWS_CA_BUNDLE to my CA bundle, the s3 backend then works great.

Can we see about adding an option to the configuration file for the s3_backend so that we can specify the location of a CA bundle so that the default CA can be overridden? It appears a few of the other options have this functionality already, so we would need to add the support for boto3.

This was tested in Antelope and validated to work once the environment variable was added.

[Cyril Roelandt (cyril-roelandt)]

> It appears a few of the other options have this functionality already, so we would need to add the support for boto3.

What options are you talking about? 's3_store_host' for instance?

What do you mean by "adding support for boto3"? Do you think this feature would require changes to boto3 itself?

I think we could add a "verify=path/to/cert/bundle.pem" (see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client) argument to the "session.client()" call here https://github.com/openstack/glance_store/blob/0c60291637d1c941dcd8d2e022acb22ba0bed440/glance_store/_drivers/s3.py#L503 .

[Antony Messerli (antonym)]

No I don’t think we’d need to make changes to boto itself. I think we probably just need to add what you suggested so that we have an option to overide the path to the cert bundle.

I was testing adding support to Kolla for s3 backend and it didn’t seem like anything worked other than setting the ENV var during the container build process so figured the best place to add support for overriding the cert was upstream as a glance configuration.

[Cyril Roelandt (cyril-roelandt)]

So I put together https://review.opendev.org/c/openstack/glance_store/+/893980 that we may end up discussing during the next PTG if this solves your issue. Could you take a look at it and try it?