Regression: ceph dashboard upgrade crashes totally w/SSL

[Harry Coin]

After the latest upgrade to ceph, the previously normal and well working dashboard crashes on load. There were no configuration changes, just the update. The dashboard is protected by an SSL cert that is valid. The only change was the upgrade. Was this tested by anyone before it was distributed? The crash was immediate. 100% reproducible.

Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:04] ENGINE Bus STARTING
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: CherryPy Checker:
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: The Application mounted at '' has an empty config.
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:04] ENGINE Started monitor thread '_TimeoutMonitor'.
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:04] ENGINE Serving on https://:::8443
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:04] ENGINE Bus STARTED
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:04] ENGINE Error in HTTPServer.tick
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: Traceback (most recent call last):
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: self.tick()
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: s, ssl_env = self.ssl_adapter.wrap(s)
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: server_side=True)
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: session=session
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3.7/ssl.py", line 870, in _create
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: self.do_handshake()
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: self._sslobj.do_handshake()
Mar 19 00:05:04 nocsupport2 ceph-mgr[19871]: OSError: [Errno 0] Error
Mar 19 00:05:26 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:26] ENGINE Bus STOPPING
Mar 19 00:05:26 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 8443)) shut down
Mar 19 00:05:26 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:26] ENGINE Stopped thread '_TimeoutMonitor'.
Mar 19 00:05:26 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:26] ENGINE Bus STOPPED
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: 2020-03-19 00:05:43.041 7f671ed1d700 -1 client.0 error registering admin socket command: (17) File exists
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: message repeated 4 times: [ 2020-03-19 00:05:43.041 7f671ed1d700 -1 client.0 error registering admin socket command: (17) File exists]
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:43] ENGINE Bus STARTING
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:43] ENGINE Started monitor thread '_TimeoutMonitor'.
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:43] ENGINE Serving on https://:::8443
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:43] ENGINE Bus STARTED
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: [19/Mar/2020:00:05:43] ENGINE Error in HTTPServer.tick
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: Traceback (most recent call last):
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: self.tick()
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: s, ssl_env = self.ssl_adapter.wrap(s)
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: server_side=True)
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: session=session
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: File "/usr/lib/python3.7/ssl.py", line 870, in _create
Mar 19 00:05:43 nocsupport2 ceph-mgr[19871]: self.do_handshake()

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: ceph 14.2.4-0ubuntu0.19.10.2
ProcVersionSignature: Ubuntu 5.3.0-42.34-generic 5.3.18
Uname: Linux 5.3.0-42-generic x86_64
ApportVersion: 2.20.11-0ubuntu8.6
Architecture: amd64
CurrentDesktop: MATE
Date: Thu Mar 19 13:47:34 2020
InstallationDate: Installed on 2018-11-12 (492 days ago)
InstallationMedia: Lubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.2)
SourcePackage: ceph
UpgradeStatus: Upgraded to eoan on 2019-07-11 (251 days ago)

[James Page (james-page)]

Hi Harry

The 14.2.4-0ubuntu0.19.10.2 was a security fix in a different area of Ceph (RGW). What was the prior version you had installed? and did your upgrade include updates to any other packages?

[Harry Coin (hcoin)]

Just vanilla eoan on a vm with only the ceph subsystem loaded. No osds on that vm. Are you saying you have a working dashboard that uses ssl?

[James Page (james-page)]

No I'm just saying its unlikely that the ceph update is the root cause of this issue as it was so specifically focused on a security update in an unrelated area.

Back to my question - what was the prior version of the package that did work? and did the package update updated any other (non-ceph) packages?

[Harry Coin (hcoin)]

James, As you'll note by the upgrade log below, your thought above that the dashboard was not changed needs a second look. Notice 2020-03-17 16:54:39, the dashboard module was in fact changed, and it broke SSL.

Does Canonical have a working ceph cluster anywhere that uses ssl on the dashboard? It has got to be broken as well.

root@nocsupport2:~# cd /var/log
root@nocsupport2:/var/log# more apt/history.log

Start-Date: 2020-03-03 06:06:34
Commandline: /usr/bin/unattended-upgrade
Upgrade: libarchive13:amd64 (3.4.0-1, 3.4.0-1ubuntu0.1)
End-Date: 2020-03-03 06:07:01

Start-Date: 2020-03-04 02:01:38
Commandline: apt-get -qy upgrade
Upgrade: libegl-mesa0:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), libglapi-mesa:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), libxatracker2:amd64 (19.2.8-
0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), libegl1-mesa:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), libgbm1:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3),
 apport:amd64 (2.20.11-0ubuntu8.4, 2.20.11-0ubuntu8.5), python3-apport:amd64 (2.20.11-0ubuntu8.4, 2.20.11-0ubuntu8.5), libgl1-mesa-dri:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubunt
u0~19.10.3), libgl1-mesa-glx:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), apport-gtk:amd64 (2.20.11-0ubuntu8.4, 2.20.11-0ubuntu8.5), mesa-vdpau-drivers:amd64 (19.2.8-0
ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), mesa-vulkan-drivers:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), python3-problem-report:amd64 (2.20.11-0ubuntu8.4, 2.20.11-0
ubuntu8.5), mesa-va-drivers:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3), libglx-mesa0:amd64 (19.2.8-0ubuntu0~19.10.2, 19.2.8-0ubuntu0~19.10.3)
End-Date: 2020-03-04 02:04:41

Start-Date: 2020-03-10 02:05:06
Commandline: apt-get -qy upgrade
Upgrade: fdisk:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), libreoffice-style-breeze:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libreoffice-math:amd64 (1:6.3.4-0u
buntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), uuid-runtime:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), libfdisk1:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), libreoffice-gtk3:amd64
 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libreoffice-core:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libmount1:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubun
tu2.3), util-linux:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), python3-uno:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libreoffice-qt5:amd64 (1:6.3.4-0ubuntu0.19.
10.1, 1:6.3.5-0ubuntu0.19.10.1), mount:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), libreoffice-base-core:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libreoffice-o
gltrans:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libblkid1:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), libreoffice-impress:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6
.3.5-0ubuntu0.19.10.1), libreoffice-style-elementary:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0ubuntu0.19.10.1), libreoffice-help-common:amd64 (1:6.3.4-0ubuntu0.19.10.1, 1:6.3.5-0u
buntu0.19.10.1), libuuid1:amd64 (2.34-0.1ubuntu2.2, 2.34-0.1ubuntu2.3), libreoffice-style-colibre:amd64 (1:6.3.4-0ubuntu0.19.10.1...

[Harry Coin (hcoin)]

This looks to be relevant, upstream:
https://tracker.ceph.com/issues/38378

[James Page (james-page)]

That upstream bug reference does look relevant - the ceph update was not the cause of any sort of regression as the bug is related to a race in something accessing the HTTPS interface of the mgr before SSL init has been completed.

14.2.8 has the referenced fixes and will be managed under bug 1861789

[James Page (james-page)]

14.2.8 packages for eoan in:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3535/+packages

Will upload for SRU team review today.

[James Page (james-page)]

Please reconfirm whether this is still and issue with the 14.2.8 updates that where released yesterday.

Thanks!

[Harry Coin (hcoin)]

Yes, the upstream patches appear to have resolved this.