disco: ceph-mgr unable to load crash module under py3

Bug #1835354 reported by James Page on 2019-07-04
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
High
Unassigned
Stein
High
Unassigned
Train
High
Unassigned
ceph (Ubuntu)
High
Unassigned
Disco
High
James Page
Eoan
High
Unassigned

Bug Description

[Impact]
ceph-mgr daemon is unable to load the crash module due.

[Test Case]
Deploy ceph
Check ceph-mgr log file
Syntax error will be seen

[Regression Potential]
Minimal; python 3 syntax changes are well understood and this fix is already in the newer ceph release in eoan.

[Original Bug Report]
On startup the ceph-mgr daemon fails to load the crash module due to invalid Python 3 syntax:

2019-07-04 04:11:54.533 7f1af75ab300 1 mgr[py] Loading python module 'crash'
2019-07-04 04:11:54.605 7f1af75ab300 -1 mgr[py] Module not found: 'crash'
2019-07-04 04:11:54.605 7f1af75ab300 -1 mgr[py] Traceback (most recent call last):

  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/module.py", line 48
    def inner((_, meta)):
              ^
SyntaxError: invalid syntax

Looking at the history for this module there are a number of syntax/compat fixes for Python3:

https://github.com/ceph/ceph/commit/9aa1d49583ae8fb658967c054247cac6ec2125ab#diff-9433ab33e8bbf1048734205a3a1141ed
https://github.com/ceph/ceph/commit/bcf0f4df2db8b51c58e5ca4d411d05e2a4d081ab#diff-9433ab33e8bbf1048734205a3a1141ed
https://github.com/ceph/ceph/commit/6dc95544512362c202535139c323717ec867e20b#diff-9433ab33e8bbf1048734205a3a1141ed

James Page (james-page) wrote :

Nautilus release includes these commits so marked Fix Released in Eoan development.

Changed in ceph (Ubuntu Eoan):
status: New → Fix Released
Changed in ceph (Ubuntu Disco):
status: New → Triaged
importance: Undecided → High
Changed in ceph (Ubuntu Eoan):
importance: Undecided → Critical
importance: Critical → High
Changed in ceph (Ubuntu Disco):
assignee: nobody → James Page (james-page)
Przemyslaw Hausman (phausman) wrote :

I can see the same error on bionic-stein.

2019-07-12 07:21:26.665 7f5c1d348300 0 set uid:gid to 64045:64045 (ceph:ceph)
2019-07-12 07:21:26.665 7f5c1d348300 0 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable), process ceph-mgr, pid 64002
2019-07-12 07:21:26.669 7f5c1d348300 0 pidfile_write: ignore empty --pid-file
2019-07-12 07:21:26.701 7f5c1d348300 1 mgr[py] Loading python module 'balancer'
2019-07-12 07:21:26.769 7f5c1d348300 1 mgr[py] Loading python module 'crash'
2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Module not found: 'crash'
2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/module.py", line 48
    def inner((_, meta)):
              ^
SyntaxError: invalid syntax

2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Class not found in module 'crash'
2019-07-12 07:21:26.813 7f5c1d348300 -1 mgr[py] Error loading module 'crash': (2) No such file or directory
2019-07-12 07:21:26.813 7f5c1d348300 1 mgr[py] Loading python module 'dashboard'
2019-07-12 07:21:27.257 7f5c1d348300 1 mgr[py] Loading python module 'hello'
2019-07-12 07:21:27.301 7f5c1d348300 1 mgr[py] Loading python module 'influx'
2019-07-12 07:21:27.349 7f5c1d348300 1 mgr[py] Loading python module 'iostat'
2019-07-12 07:21:27.389 7f5c1d348300 1 mgr[py] Loading python module 'localpool'
2019-07-12 07:21:27.429 7f5c1d348300 1 mgr[py] Loading python module 'prometheus'
2019-07-12 07:21:27.793 7f5c1d348300 1 mgr[py] Loading python module 'restful'
2019-07-12 07:21:28.121 7f5c1d348300 1 mgr[py] Loading python module 'selftest'
2019-07-12 07:21:28.233 7f5c1d348300 1 mgr[py] Loading python module 'smart'
2019-07-12 07:21:28.273 7f5c1d348300 1 mgr[py] Loading python module 'status'
2019-07-12 07:21:28.329 7f5c1d348300 1 mgr[py] Loading python module 'telegraf'
2019-07-12 07:21:28.381 7f5c1d348300 1 mgr[py] Loading python module 'telemetry'
2019-07-12 07:21:28.637 7f5c1d348300 1 mgr[py] Loading python module 'zabbix'
2019-07-12 07:21:28.789 7f5c1d348300 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: crash
2019-07-12 07:21:28.789 7f5c1d348300 1 mgr send_beacon standby
2019-07-12 07:21:28.817 7f5c09be3700 0 ms_deliver_dispatch: unhandled message 0x55de1e1a7d40 mon_map magic: 0 v1 from mon.2 10.246.68.52:6789/0
2019-07-12 07:21:28.817 7f5c09be3700 0 client.0 ms_handle_reset on 10.246.68.40:6800/112543
2019-07-12 07:21:30.813 7f5c063dc700 1 mgr send_beacon standby
2019-07-12 07:21:32.837 7f5c063dc700 1 mgr send_beacon standby
2019-07-12 07:21:34.861 7f5c063dc700 1 mgr send_beacon standby
2019-07-12 07:21:36.885 7f5c063dc700 1 mgr send_beacon standby

James Page (james-page) on 2019-07-12
description: updated
David Coronel (davecore) wrote :

subscribed ~field-medium

David Coronel (davecore) wrote :

We're seeing this with ceph 13.2.6 on bionic with the stein cloud archive

Michael Skalka (mskalka) wrote :

Confirming this is present in Bionic-stein as well:

root@juju-1101fd-18-lxd-0:~# dpkg -l|grep ceph

ii ceph 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 distributed storage and file system
ii ceph-base 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 common ceph daemon libraries and management tools
ii ceph-common 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 manager for the ceph distributed file system
ii ceph-mon 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 monitor server for the ceph storage system
ii ceph-osd 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 OSD server for the ceph storage system
ii libcephfs2 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 Ceph distributed file system client library
ii python3-cephfs 13.2.6-0ubuntu0.19.04.1~cloud0 amd64 Python 3 libraries for the Ceph libcephfs library

root@juju-1101fd-18-lxd-0:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
root@juju-1101fd-18-lxd-0:~# ceph status
  cluster:
    id: 832cb0a6-a3e7-11e9-bedc-00163e7eca36
    health: HEALTH_WARN
            Module 'crash' has failed dependency: invalid syntax (module.py, line 48)

  services:
    mon: 3 daemons, quorum juju-1101fd-19-lxd-0,juju-1101fd-18-lxd-0,juju-1101fd-20-lxd-0
    mgr: juju-1101fd-18-lxd-0(active), standbys: juju-1101fd-19-lxd-0, juju-1101fd-20-lxd-0
    osd: 72 osds: 72 up, 72 in
    rgw: 3 daemons active

  data:
    pools: 18 pools, 2040 pgs
    objects: 578 objects, 577 MiB
    usage: 74 GiB used, 251 TiB / 252 TiB avail
    pgs: 2040 active+clean

  io:
    client: 3.6 KiB/s rd, 3 op/s rd, 0 op/s wr

root@juju-1101fd-18-lxd-0:~# cat /var/log/ceph/ceph-mgr.juju-1101fd-18-lxd-0.log |grep -C10 syntax
2019-07-11 14:34:03.118 7fa37dd89300 0 pidfile_write: ignore empty --pid-file
2019-07-11 14:34:03.146 7fa37dd89300 1 mgr[py] Loading python module 'balancer'
2019-07-11 14:34:03.202 7fa37dd89300 1 mgr[py] Loading python module 'crash'
2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Module not found: 'crash'
2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Traceback (most recent call last):
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/lib/x86_64-linux-gnu/ceph/mgr/crash/module.py", line 48
    def inner((_, meta)):
              ^
SyntaxError: invalid syntax

2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Class not found in module 'crash'
2019-07-11 14:34:03.242 7fa37dd89300 -1 mgr[py] Error loading module 'crash': (2) No such file or directory

Hello James, or anyone else affected,

Accepted ceph into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/13.2.6-0ubuntu0.19.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Disco):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-disco
James Page (james-page) wrote :

Hello James, or anyone else affected,

Accepted ceph into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
David Coronel (davecore) wrote :

Tested packages from stein-proposed on ceph-mon units:

BEFORE:

root@juju-1101fd-18-lxd-0:~# ceph -s
  cluster:
    id: 832cb0a6-a3e7-11e9-bedc-00163e7eca36
    health: HEALTH_WARN
            Module 'crash' has failed dependency: invalid syntax (module.py, line 48)
[...]

root@juju-1101fd-18-lxd-0:/var/log/ceph# grep crash *
[...]
ceph.log:2019-07-16 15:00:00.000113 mon.juju-1101fd-19-lxd-0 mon.0 10.191.4.29:6789/0 20888 : cluster [WRN] overall HEALTH_WARN Module 'crash' has failed dependency: invalid syntax (module.py, line 48)
ceph.log:2019-07-16 16:00:00.000185 mon.juju-1101fd-19-lxd-0 mon.0 10.191.4.29:6789/0 21040 : cluster [WRN] overall HEALTH_WARN Module 'crash' has failed dependency: invalid syntax (module.py, line 48)
[...]

AFTER:

root@juju-1101fd-18-lxd-0:/var/log/ceph# ceph -s
  cluster:
    id: 832cb0a6-a3e7-11e9-bedc-00163e7eca36
    health: HEALTH_OK

root@juju-1101fd-18-lxd-0:/var/log/ceph# dpkg -l | grep ceph
ii ceph 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 distributed storage and file system
ii ceph-base 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 common ceph daemon libraries and management tools
ii ceph-common 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 manager for the ceph distributed file system
ii ceph-mon 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 monitor server for the ceph storage system
ii ceph-osd 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 OSD server for the ceph storage system
ii libcephfs2 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 Ceph distributed file system client library
ii python3-cephfs 13.2.6-0ubuntu0.19.04.2~cloud0 amd64 Python 3 libraries for the Ceph libcephfs library

No more crash invalid syntax errors in /var/log/ceph/* either.

tags: added: verification-stein-done
removed: verification-stein-needed
James Page (james-page) wrote :

ubuntu@juju-05ad98-disco-proposed-3:~$ sudo ceph -s
  cluster:
    id: 57558fde-a86a-11e9-acd6-fa163e0779e9
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum juju-05ad98-disco-proposed-5,juju-05ad98-disco-proposed-3,juju-05ad98-disco-proposed-4
    mgr: juju-05ad98-disco-proposed-3(active), standbys: juju-05ad98-disco-proposed-4, juju-05ad98-disco-proposed-5
    osd: 3 osds: 3 up, 3 in

  data:
    pools: 3 pools, 44 pgs
    objects: 1 objects, 14 B
    usage: 3.0 GiB used, 27 GiB / 30 GiB avail
    pgs: 44 active+clean

ubuntu@juju-05ad98-disco-proposed-3:~$ apt-cache policy ceph-mon
ceph-mon:
  Installed: 13.2.6-0ubuntu0.19.04.2
  Candidate: 13.2.6-0ubuntu0.19.04.2
  Version table:
 *** 13.2.6-0ubuntu0.19.04.2 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     13.2.6-0ubuntu0.19.04.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco-updates/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2.1 500
        500 http://security.ubuntu.com/ubuntu disco-security/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco/main amd64 Packages
ubuntu@juju-05ad98-disco-proposed-3:~$ apt-cache policy ceph-mgr
ceph-mgr:
  Installed: 13.2.6-0ubuntu0.19.04.2
  Candidate: 13.2.6-0ubuntu0.19.04.2
  Version table:
 *** 13.2.6-0ubuntu0.19.04.2 500
        500 http://archive.ubuntu.com/ubuntu disco-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     13.2.6-0ubuntu0.19.04.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco-updates/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2.1 500
        500 http://security.ubuntu.com/ubuntu disco-security/main amd64 Packages
     13.2.4+dfsg1-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu disco/main amd64 Packages

tags: added: verification-done verification-done-disco
removed: verification-needed verification-needed-disco
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.2

---------------
ceph (13.2.6-0ubuntu0.19.04.2) disco; urgency=medium

  * d/p/bug1832105.patch: Cherry pick fix to avoid cython interpreter
    check raising import error when loading ceph mgr modules
    (LP: #1832105).
  * d/p/mgr-*.patch: Misc fixes to resolve Python 3 syntax issues
    (LP: #1835354).

 -- James Page <email address hidden> Fri, 12 Jul 2019 12:03:05 +0100

Changed in ceph (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.2~cloud0
---------------

 ceph (13.2.6-0ubuntu0.19.04.2~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (13.2.6-0ubuntu0.19.04.2) disco; urgency=medium
 .
   * d/p/bug1832105.patch: Cherry pick fix to avoid cython interpreter
     check raising import error when loading ceph mgr modules
     (LP: #1832105).
   * d/p/mgr-*.patch: Misc fixes to resolve Python 3 syntax issues
     (LP: #1835354).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers