ceph: Does not respect compiler flags for Python extensions

[Graham Inggs]

[Impact]

Ceph packaging does not respect compiler flags when building Python extensions, notably hardening flags like the stack protector, as well as frame-pointer.

They were enabled and present in focal, but lost somewhere after impish. Jammy is missing them, and noble is too.

If we look at the buildlog for 19.2.1-0ubuntu0.24.04.2 on noble amd64, we see CFLAGS="".

[ 76%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-312-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc" CFLAGS="" CPPFLAGS="-iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.12 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

This should be fixed to include the standard Ubuntu defined hardening CFLAGS.

[Testcase]

Check the buildlog for CFLAGS on "cython_modules" binaries.

There are test packages available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/lp2119024-updates

If you check the build log for noble amd64, you should see:

[ 77%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-312-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc" CFLAGS=""\ -g\ -O2\ -fno-omit-frame-pointer\ -mno-omit-leaf-frame-pointer\ -ffile-prefix-map=/<<PKGBUILDDIR>>=.\ -fstack-protector-strong\ -fstack-clash-protection\ -Wformat\ -Werror=format-security\ -fcf-protection\ -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/ceph-19.2.3-0ubuntu0.24.04.1~lp2119024v20251106b1\ -Wdate-time\ -D_FORTIFY_SOURCE=3 CPPFLAGS="-iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib\ \ -Wl,-Bsymbolic-functions\ -Wl,-z,relro\ -Wl,-z,now\ -Wl,--as-needed\ -latomic CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.12 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

Where all CFLAGS are present.

[Where problems can occur]

We are enabling compiler hardening flags which tighten up execution and security requirements as expected of any package in the Ubuntu archive.

This could potentially lead to these compiled python libraries segmentation faulting if they happen to rely on faulty behaviour of exceeding the stack boundary and overwriting the newly placed stack canaries.

Ideally, all python modules should be exercised before release, but in general, a certification pass of the ceph point release verification should be enough to smoke test these modules.

[Other info]

Gentoo commit:
https://github.com/gentoo/gentoo/commit/c588d0c55a2facb7b98331abf3ccb2439cc34286

Gentoo bugs:
https://bugs.gentoo.org/866159
https://bugs.gentoo.org/866161

[Graham Inggs (ginggs)]

I am currently looking at this, and hopefully can provide a patch soon.

I'd like your opinion on whether this would be worthwhile.

[Graham Inggs (ginggs)]

It turns out CFLAGS was being cleared by ceph's build system in cmake/modules/Distutils.cmake, note CFLAGS="" below.

From a recent build log:
https://launchpad.net/ubuntu/+source/ceph/19.2.0~is.really.19.1.0-0ubuntu2/+build/28692419

[ 78%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-312-x86_64-linux-gnu.so
cd "/<<PKGBUILDDIR>>/src/pybind/cephfs" && env CC="/usr/bin/cc" CFLAGS="" CPPFLAGS="-iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.12 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

[James Page (james-page)]

Hi Graham

I think this is worth fixing however I'm not sure the proposed patch is the right approach - the CMake module it touches actively manages PY_CFLAGS:

  # Note: no quotes, otherwise distutils will execute "/usr/bin/ccache gcc"
  # CMake's implicit conversion between strings and lists is wonderful, isn't it?
  set(PY_CFLAGS ${COMPILE_OPTIONS})
  cmake_parse_arguments(DU "DISABLE_VTA" "" "" ${ARGN})
  if(DU_DISABLE_VTA AND HAS_VTA)
    list(APPEND PY_CFLAGS -fno-var-tracking-assignments)
  endif()

so although this does result in passing the flags through for the Python module, it inhibits the function of the CMake module (and I'd like to get whatever fix we use upstream).

It feels like PY_CFLAGS should be populated with the environmental CFLAGS but that's not getting passed through correctly? How does COMPILE_OPTIONS get generated by cmake?

[Graham Inggs (ginggs)]

Evidently, COMPILE_OPTIONS does not get generated by cmake. ;-)

In one of my my attempts, I tried changing:

set(PY_CFLAGS ${COMPILE_OPTIONS})

to:

set(PY_CFLAGS ${CMAKE_C_FLAGS})

but every flag ended up with a trailing \ and gcc choked, and my cmake-fu is not strong.

For *our* builds, nothing gets appended to PY_CFLAGS, so dropping CFLAGS="${PY_CFLAGS}" does not miss anything. However, I agree this is not upstreamable.

I will try to take a another look at this.

[Graham Inggs (ginggs)]

I found this upstream commit touching Distutils.cmake and included in release 17.1.0:
https://github.com/ceph/ceph/commit/1f6cf5e4031358721624f4d2e79162b2b1bbf0a0

Comparing build logs of 15.2.17-0ubuntu0.20.04.6 in Focal:
https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+build/27668885

[ 95%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-38-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -rdynamic -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fdiagnostics-color=auto -iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.8 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --verbose --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

...and 17.1.0-0ubuntu3 in Jammy:
https://launchpad.net/ubuntu/+source/ceph/17.1.0-0ubuntu3/+build/23490357

[ 84%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-310-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc" CFLAGS="" CPPFLAGS="-iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.10 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

We can see the build in Focal includes the hardening flags, and the build in Jammy does not.

[Graham Inggs (ginggs)]

However, the change in that commit from:

set(cflags ${COMPILE_OPTIONS})

to:

set(PY_CFLAGS ${COMPILE_OPTIONS})

Doesn't seem a likely cause of the error.

Looking at the build of 16.2.9-0ubuntu0.21.10.1 in Impish:

[ 93%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-39-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc -iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.9 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --verbose --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

We can see the hardening flags are already missing, even though it still has the older Distutils.cmake.

[Graham Inggs (ginggs)]

Gentoo ran into this same issue some time ago. I've confirmed that build flags are correctly passed when building the Python extensions with the attached patch.

[James Page (james-page)]

Approach looks sound to me +1

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.0-0ubuntu2

---------------
ceph (19.2.0-0ubuntu2) oracular; urgency=medium

  * d/p/pass-build-flags.patch: Pass buildflags to Python extensions
    (LP: #2078244).

 -- Graham Inggs <email address hidden> Wed, 02 Oct 2024 15:16:31 +0000

[Chris Halse Rogers (raof)]

Hello Graham, or anyone else affected,

Accepted ceph into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/19.2.3-0ubuntu0.24.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Wesley Hershberger (whershberger)]

Test plan cephfs cython_modules build command for reference:

[ 77%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-312-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc" CFLAGS=""\ -g\ -O2\ -fno-omit-frame-pointer\ -mno-omit-leaf-frame-pointer\ -ffile-prefix-map=/<<PKGBUILDDIR>>=.\ -fstack-protector-strong\ -fstack-clash-protection\ -Wformat\ -Werror=format-security\ -fcf-protection\ -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/ceph-19.2.3-0ubuntu0.24.04.1~lp2119024v20251106b1\ -Wdate-time\ -D_FORTIFY_SOURCE=3 CPPFLAGS="-iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib\ \ -Wl,-Bsymbolic-functions\ -Wl,-z,relro\ -Wl,-z,now\ -Wl,--as-needed\ -latomic CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.12 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

### Verification Done Noble ###

From the amd64 build: https://launchpad.net/ubuntu/+source/ceph/19.2.3-0ubuntu0.24.04.1/+build/31507095
Log: https://launchpadlibrarian.net/832449066/buildlog_ubuntu-noble-amd64.ceph_19.2.3-0ubuntu0.24.04.1_BUILDING.txt.gz

[ 86%] Generating ../../../lib/cython_modules/lib.3/cephfs.cpython-312-x86_64-linux-gnu.so
cd /<<PKGBUILDDIR>>/src/pybind/cephfs && env CC="/usr/bin/cc" CFLAGS=""\ -g\ -O2\ -fno-omit-frame-pointer\ -mno-omit-leaf-frame-pointer\ -ffile-prefix-map=/<<PKGBUILDDIR>>=.\ -fstack-protector-strong\ -fstack-clash-protection\ -Wformat\ -Werror=format-security\ -fcf-protection\ -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/ceph-19.2.3-0ubuntu0.24.04.1\ -Wdate-time\ -D_FORTIFY_SOURCE=3 CPPFLAGS="-iquote/<<PKGBUILDDIR>>/src/include -w -D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG##0'" CXX="/usr/bin/c++" LDSHARED="/usr/bin/cc -shared" OPT="-DNDEBUG -g -fwrapv -O2 -w" LDFLAGS=-L/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib\ \ -Wl,-Bsymbolic-functions\ -Wl,-z,relro\ -Wl,-z,now\ -Wl,--as-needed\ -latomic CYTHON_BUILD_DIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/pybind/cephfs CEPH_LIBDIR=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib /usr/bin/python3.12 /<<PKGBUILDDIR>>/src/pybind/cephfs/setup.py build --build-base /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules --build-platlib /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib/cython_modules/lib.3

### Verification Done Noble ###

[Nick Rosbrook (enr0n)]

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 19.2.3-0ubuntu0.24.04.1

---------------
ceph (19.2.3-0ubuntu0.24.04.1) noble; urgency=medium

  [ Luciano Lo Giudice ]
  * New upstream stable release (LP: #2119024)
  * d/control: Update dependencies.
  * d/rules: Enable LTTNG.
  * d/p/pyo3-fix.patch: Fix issues with PyO3.
  * d/p/CVE-2024-48916.patch: Removed, no longer needed.
  * d/p/patch-out-libnbd.patch: build without libnbd.
  * d/clean: Add btrfs and dashboard removals.
  * d/source/lintian-overrides: Fixup changed directory.
  * Backported from resolute to noble. Differences:
  * d/p/questing-boost-fix.patch: Removed, not needed on noble.
  * d/p/py313-compat/*.patch: Removed, not needed on noble.

  [ Graham Inggs ]
  * d/p/pass-build-flags.patch: Pass buildflags to Python extensions
    (LP: #2078244).

 -- Luciano Lo Giudice <email address hidden> Wed, 29 Oct 2025 22:09:45 +0000