sysconfigdata__x86_64-linux-gnu.py SyntaxError: unterminated string literal

[Christian Ehrhardt ]

As discussed this morning on #ubuntu-release

[10:02] <ricotz> hello ubuntu-archive, looks like python3.11 3.11.2-2 is broken, its installation fails with
[10:02] <ricotz> python3.11-minimal (3.11.2-2) wird eingerichtet ...
[10:02] <ricotz> File "/usr/lib/python3.11/_sysconfigdata__x86_64-linux-gnu.py", line 46
[10:02] <ricotz> '
[10:02] <ricotz> ^
[10:02] <ricotz> SyntaxError: unterminated string literal (detected at line 46)
[10:04] <seb128> doko, ^
[10:16] <seb128> and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030818
[10:16] <seb128> http://launchpadlibrarian.net/650179842/python3.11_3.11.1-2_3.11.2-1.diff.gz
[10:16] <seb128> seems weird
[10:16] <seb128> ah, the changelog of -2 mentions that but just fail to reference the bug
[10:18] <slyon> doko: ginggs: I'm seeing the same python3.11 build issue with the single quote in _sysconfigdata.py as we had yesterday with the new 3.11.2-2 upload from a few hours ago. Is there anything we can do about it?
[10:18] <slyon> e.g.: https://launchpadlibrarian.net/650560179/buildlog_ubuntu-lunar-s390x.fuse3_3.13.0-2build1~ppa0_BUILDING.txt.gz
[10:24] <cpaelzer> slyon: seb128, ricotz: yep starting to see the same creping into all my builds as well. Interestingly only amd64, arm64 and s390x. On armhf and ppc64 it isn't showing the same
[10:25] <seb128> slyon, hey, is that a regression from proposed? if so I think we should delete that update
[10:25] <cpaelzer> Here an example on ppc64el https://launchpadlibrarian.net/650557966/buildlog_ubuntu-lunar-ppc64el.libvirt_9.0.0-1ubuntu1~lunarppa1_BUILDING.txt.gz which works
[10:25] <seb128> unless someone stops me I'm going to the delete the update for now, it can come back once it's fixed
[10:25] <cpaelzer> seb128: I'm very much on the same path/decision you are, just doing a few more checks
[10:26] <slyon> seb128: cpaelzer: yeah, it regressed in -proposed. The package was dropped yesterday from -proposed AFAIK, but a new python3.11 upload from this morning seems to trigger the same issue
[10:26] <cpaelzer> yes, 3.11.2-2 is proposed only and I ahve logs of 3.11.1-2 working fine up until recently
[10:26] <seb128> cpaelzer, do you want more time to finish your checks before I press the delete button?
[10:26] <cpaelzer> 3.11.1-2 is in lunar-release
[10:26] <cpaelzer> no seb128, push the button
[10:26] <cpaelzer> all I've looked at is still saying the same that we discussed here
[10:28] <seb128> removed
[10:28] <ginggs> seb128: thanks
[10:30] <cpaelzer> And for the bug itself, it was not a fluke the content really was broken - the file has a single ' in that line
[10:32] <seb128> if http://launchpadlibrarian.net/650179842/python3.11_3.11.1-2_3.11.2-1.diff.gz had the issue then it's probably not coming from the package itself
[10:33] <seb128> I mean it's python which end up buggy, but in that upload ^ there was no actual content change, so probably a build-depends impacting it?
[10:34] <cpaelzer> seb128: it might be generated at build time
[10:34] <ginggs> i don't think either upload had the problem in debian
[10:35] <seb128> cpaelzer, right, but if the same content used to generating right then it's probably impacted by something else that changes in the archive
[10:50] <cpaelzer> The change actually is https://paste.ubuntu.com/p/vV8DGj3sb2/ which is a dump of vars in python3.11/Lib/sysconfig.py (I can't see how that woudl miss a ') followed by sed magic in d/rules which very much could cause this if our default compiler arguments recently changed (and that would also explain why it might be different in Debian)

I've spun up a build in background that moment and later on had a look.

We know that the final broken one looks like this:
'CONFIGURE_CFLAGS: '
...
'

As mentioned above I expected the sed commands to break it.
After build before sed it looks like:

...
 'CFLAGS': '-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -g -Og -Wall -g -Og '
           '-ffile-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=. '
           '-flto=auto -ffat-lto-objects '
           '-specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector '
           '-Wformat -Werror=format-security '
           '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
           '-g -Og '
           '-ffile-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=. '
           '-flto=auto -ffat-lto-objects '
           '-specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector '
           '-Wformat -Werror=format-security '
           '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2',
...

All ok so far.
The first sed already breaks it:

sed -e 's,gcc-8,gcc,;s,g++-8,g++,' -e 's/-flto=auto//g' -e 's/-flto//g' -e 's/-fuse-linker-plugin//g' -e 's/-ffat-lto-objects//g' -e '/^OPT/s,-O3,-O2,' -e 's/-O3/-O2/g' -e 's/-fprofile-use *-fprofile-correction//g' -e 's/-fstack-protector /-fstack-protector-strong /g' -e 's/-specs=[^ ]*//g' -e 's/-fdebug-prefix-map=[^ ]*//g' -e 's/-ffile-prefix-map=[^ ]*//g' -e 's/-Wl,-z,relro//g' -e 's,^RUNSHARED *=.*,RUNSHARED=,' -e '/BLDLIBRARY/s/-L\. //' build-shdebug/build/lib.linux-x86_64-3.11-pydebug/_sysconfigdata_d_x86_64-linux-gnu.py

...
 'CFLAGS': '-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -g -Og -Wall -g -Og '
           ' '
           ' '
           ' -fstack-protector-strong '
           '-Wformat -Werror=format-security '
           ' '
           '-g -Og '
           ' '
           ' '
           ' -fstack-protector-strong '
           '-Wformat -Werror=format-security '
           '
...

Knowing the context we see that this is the breaking rule:
  -e 's/-fdebug-prefix-map=[^ ]*//g'

I do not have old data, maybe it used to have a trailing space like the others.
But it does no more and due to that the rule is too greedy.

But there are more and different variants of that in the file.
- Some with '' some with ""
- Some with trailing "," some without
- Some with whitespace, some without

# grep debug-prefix build-shdebug/build/lib.linux-x86_64-3.11-pydebug/_sysconfigdata_d_x86_64-linux-gnu.py
           '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
           '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2',
                     '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 ',
                "-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2' "
                    '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                    '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                    '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                    '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                             '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                             '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
              '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
              '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2',
                   '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                   '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                        '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '
                        '-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2 '

Due to that simple first thoughts like -e "s/-fdebug-prefix-map=.*'/'/g" are doomed by one or the other of these conditions.

Please admire the awesome entry #3 in this list
"-fdebug-prefix-map=/build/python3.11-vLLXL0/python3.11-3.11.2=/usr/src/python3.11-3.11.2-2' "
Other than all others guarded by " (because it contains a ') which is also a single ', not two.

AFAICS the rule should delete each of that to just an "empty whatever it was".
But deleting all lines which contain it is also dangerous as it might break structures which need a trailing , or not.

I'm not entirely sure what to use, but I guess that outlines the problem enough to tackle it.
Maybe someone just knows what has changed on the input data and might fix it there?

I'll attach the interim file for anyone to enjoy the fun of modifying sed rules ...

[Christian Ehrhardt  (paelzer)]

On this file it would execute:

sed -e 's,gcc-8,gcc,;s,g++-8,g++,' -e 's/-flto=auto//g' -e 's/-flto//g' -e 's/-fuse-linker-plugin//g' -e 's/-ffat-lto-objects//g' -e '/^OPT/s,-O3,-O2,' -e 's/-O3/-O2/g' -e 's/-fprofile-use *-fprofile-correction//g' -e 's/-fstack-protector /-fstack-protector-strong /g' -e 's/-specs=[^ ]*//g' -e 's/-fdebug-prefix-map=[^ ]*//g' -e 's/-ffile-prefix-map=[^ ]*//g' -e 's/-Wl,-z,relro//g' -e 's,^RUNSHARED *=.*,RUNSHARED=,' -e '/BLDLIBRARY/s/-L\. //' \
 /<<PKGBUILDDIR>>/build-shdebug/build/lib.linux-x86_64-3.11-pydebug/_sysconfigdata_d_x86_64-linux-gnu.py \
 > debian/tmp-dbg/usr/lib/python3.11/_sysconfigdata_d_x86_64-linux-gnu.py

sed -i -e 's,gcc-8,gcc,;s,g++-8,g++,' -e 's/-flto=auto//g' -e 's/-flto//g' -e 's/-fuse-linker-plugin//g' -e 's/-ffat-lto-objects//g' -e '/^OPT/s,-O3,-O2,' -e 's/-O3/-O2/g' -e 's/-fprofile-use *-fprofile-correction//g' -e 's/-fstack-protector /-fstack-protector-strong /g' -e 's/-specs=[^ ]*//g' -e 's/-fdebug-prefix-map=[^ ]*//g' -e 's/-ffile-prefix-map=[^ ]*//g' -e 's/-Wl,-z,relro//g' -e 's,^RUNSHARED *=.*,RUNSHARED=,' -e '/BLDLIBRARY/s/-L\. //' \
 debian/tmp-dbg/usr/lib/python3.11/_sysconfigdata_d_x86_64-linux-gnu.py