missing erasure code plugin library libec_isa.so

[Marco Savoca]

[Impact]
An upstream change to build requirements for the ISA-L EC plugin caused it not to be included in the Ceph package for Hirsute and Impish.

Users upgrading from previous versions of Ubuntu making use of this feature would as such face a regression in Ubuntu.

[Test Plan]
The original bug description section below contains a clear reproducible way to test if the issue has been fixed.

[Regression Potential]
We are adding a build dependency which adds back a shared library that was previously included in the package, so I would consider the regression potential to be minimal.

[Original Bug Description]
After a fresh deployment of ceph-mon on maas/ubuntu-focal (source: cloud:focal-wallaby) the required library libec_isa.so for the isa erasure code plugin is missing.

The command

juju run-action ceph-mon/leader create-erasure-profile --params ./create_erasure_profile.yml --verbose --wait

with this parameters in create_erasure_profile.yml

name: ectest
#m
coding-chunks: 2
#k
data-chunks: 4
device-class: hdd
failure-domain: host
plugin: isa

fails with the error message

unit-ceph-mon-0:
  UnitId: ceph-mon/0
  id: "14"
  message: 'Create erasure profile failed with message: Command ''[''ceph'', ''--id'',
    ''admin'', ''osd'', ''erasure-code-profile'', ''set'', ''ectest'', ''plugin=isa'',
    ''k=4'', ''m=2'', ''crush-failure-domain=host'', ''crush-device-class=hdd'']''
    returned non-zero exit status 5.'
  results:
    Stderr: |
      Error ENOENT: unknown erasure code profile 'ectest'
      Error EIO: load dlopen(/usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_isa.so): /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_isa.so: cannot open shared object file: No such file or directory
  status: failed
  timing:
    completed: 2021-08-20 16:39:02 +0000 UTC
    enqueued: 2021-08-20 16:38:58 +0000 UTC
    started: 2021-08-20 16:39:00 +0000 UTC

[Frode Nordahl (fnordahl)]

Marco, thank you for your bug report.

I am able to reproduce the issue and it does appear the ISA plugin is not included in the Ceph packages from Hirsute and onward:

# dpkg-query -W ceph-base
ceph-base 15.2.13-0ubuntu0.20.04.1
# dpkg -S libec
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_lrc.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_isa.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_clay.so

# dpkg-query -W ceph-base
ceph-base 16.2.4-0ubuntu0.21.04.1
# dpkg -S libec
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_lrc.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_clay.so

# dpkg-query -W ceph-base
ceph-base 16.2.5-0ubuntu2
# dpkg -S libec
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_lrc.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_clay.so

[Frode Nordahl (fnordahl)]

Moving the ceph-mon charm task to Invalid as this will be fixed by a update to the ceph package itself.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 16.2.5-0ubuntu4

---------------
ceph (16.2.5-0ubuntu4) impish; urgency=medium

  * d/control: The ISA-L EC plugin changed its build dependency from yasm to
    nasm 2.13.03 or newer (LP: #1940697).

 -- Frode Nordahl <email address hidden> Fri, 10 Sep 2021 14:56:29 +0100

[Brian Murray (brian-murray)]

Hello Marco, or anyone else affected,

Accepted ceph into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/16.2.6-0ubuntu0.21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

 juju run-action ceph-mon/leader create-erasure-profile --params ./create_erasure_profile.yaml --verbose --wait
unit-ceph-mon-1:
  UnitId: ceph-mon/1
  id: "154"
  results:
    Stderr: |
      Error ENOENT: unknown erasure code profile 'ectest'
  status: completed
  timing:
    completed: 2021-10-27 11:51:17 +0000 UTC
    enqueued: 2021-10-27 11:51:15 +0000 UTC
    started: 2021-10-27 11:51:15 +0000 UTC

confirmed erasure-code profile was created correctly and validated that ec_isa was part of the packaging.

[James Page (james-page)]

$ dpkg -S libec
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_lrc.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_generic.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_isa.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse3.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_shec_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_jerasure_sse4.so
ceph-base: /usr/lib/x86_64-linux-gnu/ceph/erasure-code/libec_clay.so

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 16.2.6-0ubuntu0.21.04.1

---------------
ceph (16.2.6-0ubuntu0.21.04.1) hirsute; urgency=medium

  [ Frode Nordahl ]
  * d/control: The ISA-L EC plugin changed its build dependency from yasm to
    nasm 2.13.03 or newer (LP: #1940697)

  [ James Page ]
  * New upstream point release (LP: #1943932).

 -- James Page <email address hidden> Sun, 19 Sep 2021 09:29:10 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.