python3-dbg-config syntax error

[Andreas Hasenack]

$ python3-dbg-config --cflags --libs
/usr/bin/python3-dbg-config: 117: Syntax error: Unterminated quoted string

This is caused by an unterminated string on line 38:
CFLAGS=$(echo "-g -Og -fstack-protector-strong -Wformat -Werror=format-security | sed "s#$prefix#$prefix_real#")

That should be:
CFLAGS=$(echo "-g -Og -fstack-protector-strong -Wformat -Werror=format-security" | sed "s#$prefix#$prefix_real#")

[Nick Rosbrook (enr0n)]

I have also been looking at this bug because it shows up in the foundations list on the devel-proposed report: https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses_by_team.html#foundations-bugs.

The script is generated from a template at build time. What's odd to me is that the similar python3-config does not have this syntax error, so maybe it has to do with the CFLAGS used in debug vs non-debug builds.

[Andreas Hasenack (ahasenack)]

The Misc/python-config.sh script is generated after a ./configure run. I patched debian/rules to show me Misc/python-config.sh after each such run, to see if I could figure out which one broke it, but turns out none do.

$ grep ^CFLAGS build-*/Misc/python-config.sh
build-debug/Misc/python-config.sh:CFLAGS=$(echo "-g -Og -ffile-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5" | sed "s#$prefix#$prefix_real#")

build-nopie/Misc/python-config.sh:CFLAGS=$(echo "-g -ffile-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5 " | sed "s#$prefix#$prefix_real#")

build-shared/Misc/python-config.sh:CFLAGS=$(echo "-g -ffile-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5 " | sed "s#$prefix#$prefix_real#")

build-shdebug/Misc/python-config.sh:CFLAGS=$(echo "-g -Og -ffile-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5" | sed "s#$prefix#$prefix_real#")

build-static/Misc/python-config.sh:CFLAGS=$(echo "-g -ffile-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5 " | sed "s#$prefix#$prefix_real#")

And the python-config.sh file we have at that stage is not yet the final one, because it still has stuff like -fdebug-prefix-map and others that get removed somewhere else further down the road. Maybe that's the point when the script gets broken.

[Andreas Hasenack (ahasenack)]

It must be this step:

    sed -i $(sysconfig_substflags) \
        $(d)/usr/bin/$(PVER)-config
    cp $(d)/usr/bin/$(PVER)-config $(d)/usr/bin/$(DEB_HOST_MULTIARCH)-$(PVER)-config

[Andreas Hasenack (ahasenack)]

Yeah, that step breaks these builds:
build-shdebug
build-debug

$ cat > testfile
CFLAGS=$(echo "-g -Og -ffile-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5" | sed "s#$prefix#$prefix_real#")

$ cat testfile | sed -e '/^OPT/s,-O3,-O2,' \
    -e 's/-O3/-O2/g' \
    -e 's/-fprofile-use *-fprofile-correction//g' \
    -e 's/-fstack-protector /-fstack-protector-strong /g' \
    -e "s/-specs=[^ ']*//g" \
    -e "s/-fdebug-prefix-map=[^ ']*//g" \
    -e "s/-ffile-prefix-map=[^ ']*//g" \
    -e 's/-Wl,-z,relro//g' \
    -e 's,^RUNSHARED *=.*,RUNSHARED=,' \
    -e '/BLDLIBRARY/s/-L\. //'
CFLAGS=$(echo "-g -Og -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security | sed "s#$prefix#$prefix_real#")

broken^

[Andreas Hasenack (ahasenack)]

The crucial difference seems to be this extra space at the end of the last CFLAGS option in the non-debug builds:

build-shared: -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5 "

build-shdebug: -fdebug-prefix-map=/home/ubuntu/git/packages/python3.11/python3.11=/usr/src/python3.11-3.11.2-5"

[Andreas Hasenack (ahasenack)]

Somehow it's the $(filter-out ) makefile mechanism that is doing it:

Makefile:
dpkg_buildflags = DEB_BUILD_MAINT_OPTIONS="hardening=-pie $(DPKG_OPTIMIZE)" dpkg-buildflags
DPKG_CFLAGS := $(shell $(dpkg_buildflags) --get CFLAGS)
DEBUG_CFLAGS := $(patsubst -O%,-Og,$(DPKG_CFLAGS))
OPT_CFLAGS := $(filter-out -O%,$(DPKG_CFLAGS)) # default is -O3

all:
    DPKG_CFLAGS="$(DPKG_CFLAGS)" echo
    DEBUG_CFLAGS="$(DEBUG_CFLAGS)" echo
    OPT_CFLAGS="$(OPT_CFLAGS)" echo

$ make
DPKG_CFLAGS="-g -O2 -ffile-prefix-map=/home/ubuntu/x=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security" echo

DEBUG_CFLAGS="-g -Og -ffile-prefix-map=/home/ubuntu/x=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security" echo

OPT_CFLAGS="-g -ffile-prefix-map=/home/ubuntu/x=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security " echo

OPT_CFLAGS got the extra space, whereas the rest didn't.

[Andreas Hasenack (ahasenack)]

https://bugs.launchpad.net/ubuntu/+source/python3.11/+bug/2006738 made the sysconfig_substflags change

[Andreas Hasenack (ahasenack)]

Hah, @sergiodj nailed it. The extra space comes from this Makefile comment at the end of the line:

  OPT_CFLAGS := $(filter-out -O%,$(DPKG_CFLAGS)) # default is -O3

The space before the "#" is included in OPT_CFLAGS :)

As for the fix, I'm testing this:
--- a/debian/rules
+++ b/debian/rules
@@ -248,9 +248,9 @@ sysconfig_substflags = \
        -e 's/-O3/-O2/g' \
        -e 's/-fprofile-use *-fprofile-correction//g' \
        -e 's/-fstack-protector /-fstack-protector-strong /g' \
- -e "s/-specs=[^ ']*//g" \
- -e "s/-fdebug-prefix-map=[^ ']*//g" \
- -e "s/-ffile-prefix-map=[^ ']*//g" \
+ -e "s/-specs=[^ '\"]*//g" \
+ -e "s/-fdebug-prefix-map=[^ '\"]*//g" \
+ -e "s/-ffile-prefix-map=[^ '\"]*//g" \
        -e 's/-Wl,-z,relro//g' \
        -e 's,^RUNSHARED *=.*,RUNSHARED=,' \
        -e '/BLDLIBRARY/s/-L\. //'

[Andreas Hasenack (ahasenack)]

The fix worked:
$ python3-dbg-config --cflags
-I/usr/include/python3.11d -I/usr/include/python3.11d -Wsign-compare -g -Og -fstack-protector-strong -Wformat -Werror=format-security -DDYNAMIC_ANNOTATIONS_ENABLED=1 -g -Og -Wall

PPA at https://launchpad.net/~ahasenack/+archive/ubuntu/python-2009967/+packages (I used a "build" suffix because this package has weird handling of the maintainer header in d/control).

[Andreas Hasenack (ahasenack)]

This was fixed in https://launchpad.net/ubuntu/+source/python3.11/3.11.2-6

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.11 - 3.11.2-6

---------------
python3.11 (3.11.2-6) unstable; urgency=high

  [ Stefano Rivera ]
  * Explain more ways to pass --break-system-packages to pip.

  [ Matthias Klose ]
  * Fix syntax error in python3-dbg-config script. LP: #2009967.

 -- Matthias Klose <email address hidden> Mon, 13 Mar 2023 13:18:29 +0100