disco: ceph-mgr unable to load crash module under py3

[James Page]

[Impact]
ceph-mgr daemon is unable to load the crash module due.

[Test Case]
Deploy ceph
Check ceph-mgr log file
Syntax error will be seen

[Regression Potential]
Minimal; python 3 syntax changes are well understood and this fix is already in the newer ceph release in eoan.

[Original Bug Report]
On startup the ceph-mgr daemon fails to load the crash module due to invalid Python 3 syntax:

2019-07-04 04:11:54.533 7f1af75ab300 1 mgr[py] Loading python module 'crash'
2019-07-04 04:11:54.605 7f1af75ab300 -1 mgr[py] Module not found: 'crash'
2019-07-04 04:11:54.605 7f1af75ab300 -1 mgr[py] Traceback (most recent call last):

  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/module.py", line 48
    def inner((_, meta)):
              ^
SyntaxError: invalid syntax

Looking at the history for this module there are a number of syntax/compat fixes for Python3:

https://github.com/ceph/ceph/commit/9aa1d49583ae8fb658967c054247cac6ec2125ab#diff-9433ab33e8bbf1048734205a3a1141ed
https://github.com/ceph/ceph/commit/bcf0f4df2db8b51c58e5ca4d411d05e2a4d081ab#diff-9433ab33e8bbf1048734205a3a1141ed
https://github.com/ceph/ceph/commit/6dc95544512362c202535139c323717ec867e20b#diff-9433ab33e8bbf1048734205a3a1141ed

[James Page (james-page)]

Nautilus release includes these commits so marked Fix Released in Eoan development.

[Przemyslaw Hausman (phausman)]

I can see the same error on bionic-stein.

2019-07-12 07:21:26.665 7f5c1d348300 0 set uid:gid to 64045:64045 (ceph:ceph)
2019-07-12 07:21:26.665 7f5c1d348300 0 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable), process ceph-mgr, pid 64002
2019-07-12 07:21:26.669 7f5c1d348300 0 pidfile_write: ignore empty --pid-file
2019-07-12 07:21:26.701 7f5c1d348300 1 mgr[py] Loading python module 'balancer'
2019-07-12 07:21:26.769 7f5c1d348300 1 mgr[py] Loading python module 'crash'
2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Module not found: 'crash'
2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/module.py", line 48
    def inner((_, meta)):
              ^
SyntaxError: invalid syntax

2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Class not found in module 'crash'
2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Error loading module 'crash': (2) No such file or directory
2019-07-12 07:21:26.813 7f5c1d348300 1 mgr[py] Loading python module 'dashboard'
2019-07-12 07:21:27.257 7f5c1d348300 1 mgr[py] Loading python module 'hello'
2019-07-12 07:21:27.301 7f5c1d348300 1 mgr[py] Loading python module 'influx'
2019-07-12 07:21:27.349 7f5c1d348300 1 mgr[py] Loading python module 'iostat'
2019-07-12 07:21:27.389 7f5c1d348300 1 mgr[py] Loading python module 'localpool'
2019-07-12 07:21:27.429 7f5c1d348300 1 mgr[py] Loading python module 'prometheus'
2019-07-12 07:21:27.793 7f5c1d348300 1 mgr[py] Loading python module 'restful'
2019-07-12 07:21:28.121 7f5c1d348300 1 mgr[py] Loading python module 'selftest'
2019-07-12 07:21:28.233 7f5c1d348300 1 mgr[py] Loading python module 'smart'
2019-07-12 07:21:28.273 7f5c1d348300 1 mgr[py] Loading python module 'status'
2019-07-12 07:21:28.329 7f5c1d348300 1 mgr[py] Loading python module 'telegraf'
2019-07-12 07:21:28.381 7f5c1d348300 1 mgr[py] Loading python module 'telemetry'
2019-07-12 07:21:28.637 7f5c1d348300 1 mgr[py] Loading python module 'zabbix'
2019-07-12 07:21:28.789 7f5c1d348300 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: crash
2019-07-12 07:21:28.789 7f5c1d348300 1 mgr send_beacon standby
2019-07-12 07:21:28.817 7f5c09be3700 0 ms_deliver_dispatch: unhandled message 0x55de1e1a7d40 mon_map magic: 0 v1 from mon.2 10.246.68.52:6789/0
2019-07-12 07:21:28.817 7f5c09be3700 0 client.0 ms_handle_reset on 10.246.68.40:6800/112543
2019-07-12 07:21:30.813 7f5c063dc700 1 mgr send_beacon standby
2019-07-12 07:21:32.837 7f5c063dc700 1 mgr send_beacon standby
2019-07-12 07:21:34.861 7f5c063dc700 1 mgr send_beacon standby
2019-07-12 07:21:36.885 7f5c063dc700 1 mgr send_beacon standby

[David Coronel (davecore)]

subscribed ~field-medium

[David Coronel (davecore)]

We're seeing this with ceph 13.2.6 on bionic with the stein cloud archive

[Michael Skalka (mskalka)]

Confirming this is present in Bionic-stein as well:

root@juju-1101fd-18-lxd-0:~# dpkg -l|grep ceph

ii ceph 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 distributed storage and file system
ii ceph-base 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 common ceph daemon libraries and management tools
ii ceph-common 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 manager for the ceph distributed file system
ii ceph-mon 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 monitor server for the ceph storage system
ii ceph-osd 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 OSD server for the ceph storage system
ii libcephfs2 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 Ceph distributed file system client library
ii python3-cephfs 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 Python 3 libraries for the Ceph libcephfs library

root@juju-1101fd-18-lxd-0:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
root@juju-1101fd-18-lxd-0:~# ceph status
  cluster:
    id: 832cb0a6-a3e7-11e9-bedc-00163e7eca36
    health: HEALTH_WARN
            Module 'crash' has failed dependency: invalid syntax (module.py, line 48)

  services:
    mon: 3 daemons, quorum juju-1101fd-19-lxd-0,juju-1101fd-18-lxd-0,juju-1101fd-20-lxd-0
    mgr: juju-1101fd-18-lxd-0(active), standbys: juju-1101fd-19-lxd-0, juju-1101fd-20-lxd-0
    osd: 72 osds: 72 up, 72 in
    rgw: 3 daemons active

  data:
    pools: 18 pools, 2040 pgs
    objects: 578 objects, 577 MiB
    usage: 74 GiB used, 251 TiB / 252 TiB avail
    pgs: 2040 active+clean

  io:
    client: 3.6 KiB/s rd, 3 op/s rd, 0 op/s wr

root@juju-1101fd-18-lxd-0:~# cat /var/log/ceph/ceph-mgr.juju-1101fd-18-lxd-0.log |grep -C10 syntax
2019-07-11 14:34:03.118 7fa37dd89300 0 pidfile_write: ignore empty --pid-file
2019-07-11 14:34:03.146 7fa37dd89300 1 mgr[py] Loading python module 'balancer'
2019-07-11 14:34:03.202 7fa37dd89300 1 mgr[py] Loading python module 'crash'
2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Module not found: 'crash'
2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/module.py", line 48
    def inner((_, meta)):
              ^
SyntaxError: invalid syntax

2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Class not found in module 'crash'
2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Error loading module 'crash': (2) No such file or directory

[Sage Weil (sage-newdream)]

https://github.com/ceph/ceph/pull/29029

[Brian Murray (brian-murray)]

Hello James, or anyone else affected,

Accepted ceph into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/13.2.6-0ubuntu0.19.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

Hello James, or anyone else affected,

Accepted ceph into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[David Coronel (davecore)]

Tested packages from stein-proposed on ceph-mon units:

BEFORE:

root@juju-1101fd-18-lxd-0:~# ceph -s
  cluster:
    id: 832cb0a6-a3e7-11e9-bedc-00163e7eca36
    health: HEALTH_WARN
            Module 'crash' has failed dependency: invalid syntax (module.py, line 48)
[...]

root@juju-1101fd-18-lxd-0:/var/log/ceph# grep crash *
[...]
ceph.log:2019-07-16 15:00:00.000113 mon.juju-1101fd-19-lxd-0 mon.0 10.191.4.29:6789/0 20888 : cluster [WRN] overall HEALTH_WARN Module 'crash' has failed dependency: invalid syntax (module.py, line 48)
ceph.log:2019-07-16 16:00:00.000185 mon.juju-1101fd-19-lxd-0 mon.0 10.191.4.29:6789/0 21040 : cluster [WRN] overall HEALTH_WARN Module 'crash' has failed dependency: invalid syntax (module.py, line 48)
[...]

AFTER:

root@juju-1101fd-18-lxd-0:/var/log/ceph# ceph -s
  cluster:
    id: 832cb0a6-a3e7-11e9-bedc-00163e7eca36
    health: HEALTH_OK

root@juju-1101fd-18-lxd-0:/var/log/ceph# dpkg -l | grep ceph
ii ceph 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 distributed storage and file system
ii ceph-base 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 common ceph daemon libraries and management tools
ii ceph-common 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 manager for the ceph distributed file system
ii ceph-mon 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 monitor server for the ceph storage system
ii ceph-osd 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 OSD server for the ceph storage system
ii libcephfs2 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 Ceph distributed file system client library
ii python3-cephfs 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 Python 3 libraries for the Ceph libcephfs library

No more crash invalid syntax errors in /var/log/ceph/* either.

[James Page (james-page)]

ubuntu@juju-05ad98-disco-proposed-3:~$ sudo ceph -s
  cluster:
    id: 57558fde-a86a-11e9-acd6-fa163e0779e9
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum juju-05ad98-disco-proposed-5,juju-05ad98-disco-proposed-3,juju-05ad98-disco-proposed-4
    mgr: juju-05ad98-disco-proposed-3(active), standbys: juju-05ad98-disco-proposed-4, juju-05ad98-disco-proposed-5
    osd: 3 osds: 3 up, 3 in

  data:
    pools: 3 pools, 44 pgs
    objects: 1 objects, 14 B
    usage: 3.0 GiB used, 27 GiB / 30 GiB avail
    pgs: 44 active+clean

ubuntu@juju-05ad98-disco-proposed-3:~$ apt-cache policy ceph-mon
ceph-mon:
  Installed: 13.2.6-0ubuntu0.19.04.2
  Candidate: 13.2.6-0ubuntu0.19.04.2
  Version table:
 *** 13.2.6-0ubuntu0.19.04.2 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     13.2.6-0ubuntu0.19.04.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco-updates/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2.1 500
        500 http://security.ubuntu.com/ubuntu disco-security/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco/main amd64 Packages
ubuntu@juju-05ad98-disco-proposed-3:~$ apt-cache policy ceph-mgr
ceph-mgr:
  Installed: 13.2.6-0ubuntu0.19.04.2
  Candidate: 13.2.6-0ubuntu0.19.04.2
  Version table:
 *** 13.2.6-0ubuntu0.19.04.2 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     13.2.6-0ubuntu0.19.04.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco-updates/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2.1 500
        500 http://security.ubuntu.com/ubuntu disco-security/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco/main amd64 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.2

---------------
ceph (13.2.6-0ubuntu0.19.04.2) disco; urgency=medium

  * d/p/bug1832105.patch: Cherry pick fix to avoid cython interpreter
    check raising import error when loading ceph mgr modules
    (LP: #1832105).
  * d/p/mgr-*.patch: Misc fixes to resolve Python 3 syntax issues
    (LP: #1835354).

 -- James Page <email address hidden> Fri, 12 Jul 2019 12:03:05 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.2~cloud0
---------------

 ceph (13.2.6-0ubuntu0.19.04.2~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (13.2.6-0ubuntu0.19.04.2) disco; urgency=medium
 .
   * d/p/bug1832105.patch: Cherry pick fix to avoid cython interpreter
     check raising import error when loading ceph mgr modules
     (LP: #1832105).
   * d/p/mgr-*.patch: Misc fixes to resolve Python 3 syntax issues
     (LP: #1835354).