Bionic: Luminous radosgw incompatible with libssl1.1
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ceph (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Bionic |
Fix Released
|
High
|
Eric Desrochers |
Bug Description
[Impact]
Since the introduction of OpenSSL 1.1.1 in 18.04 LTS:
https:/
This is breaking Ceph cluster https service.
# logs:
2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run
[Test Case]
1) Generate a self-signed certificate or use whatever existing SSL certificate already in place.
If one want to create a PEM file for civetweb, instructions can be found here :
https:/
** Note: "CivetWeb requires one certificate file in PEM format" **
2) Enable logging and debugging in "/etc/ceph/
Example:
------
log to syslog = true
err to syslog = true
clog to syslog = true
debug rgw = 10/5
debug civetweb = 1/10
------
http://
3) From the radosgw node, modify "/etc/ceph/
rgw_frontends = civetweb port=443s ssl_certificate
4) Restart the daemon:
systemctl restart ceph-radosgw@
5) Look logs:
2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
6) Look radosgw which should FAILED to start.
systemctl status ceph-radosgw@
What we are looking for here is radosgw to be 'Active' and to have a LISTEN port on 443 as follow :
$ netstat -anputa | grep LISTEN | grep 443 # or any port mentioned in the configuration above.
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 10153/radosgw
[Potential Regression]
* Same downgrade approach has been made for 'nodejs' via LP: #1798367
* The proposed packages has been tested on at least 2 different Ceph clusters impacted by the issue, and have been tested at various level (no package update problem, radosgw is now working fine when civetweb is configure over ssl, ...)
* Nothing can be worst than current situation, considering that civetweb is non-functional when SSL is in used due to the incompatibility with 1.1 and make radosgw daemon to fail.
* libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a problem here.
* See discussion IRC discussion (xnox/jamespage) on comment #11
* All autopkgtest 'passed'
http://
[Other Information]
* Adding the OpenSSL 1.1 support has been explored and revealed to be non-trivial :
https:/
https:/
https:/
http://
See discussion IRC discussion on comment #11
[Original Description]
Bionic's radosgw package (Version 12.2.11-
This has been known about upstream for a while now, and as noted in the bug-tracker (https:/
Changed in ceph (Ubuntu Bionic): | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in ceph (Ubuntu): | |
status: | New → Fix Released |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in ceph (Ubuntu Bionic): | |
status: | Confirmed → In Progress |
assignee: | nobody → Eric Desrochers (slashd) |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in ceph (Ubuntu Bionic): | |
importance: | Medium → High |
description: | updated |
Changed in ceph (Ubuntu): | |
importance: | Undecided → High |
description: | updated |
description: | updated |
Here's what has been brought to my attention by someone impacted by the problem:
"
This is breaking our test cluster's https service, and blocks upgrading our production cluster to 18.04.
2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run
"