Python extension modules get built using wrong compiler flags with python2

[Nafees]

[ Impact ]

When compiling Python extensions using Python2, CFLAGS optimization flags are dropped.

This behavior has been caused by our update in this patch http://archive.ubuntu.com/ubuntu/pool/universe/p/python2.7/python2.7_2.7.18-1~20.04.3.diff.gz which differs from upstream.

The fix modifies the portion of code in Lib/distutils/sysconfig.py which gets the cflags from the environments, and includes the dropped OPT flag from get_config_vars().

[ Test Plan ]

There will be 2 separate tests for this bug:
* Ensuring no-change rebuilds are not changed
* Ensuring local builds are not changed unless environment variable is set

Test 1) No-change rebuilds

To test that no-change rebuilds are not changed, the package python-stdlib-extensions will be built against the new python2.7, and confirm the compiler flags are not altered. This will be a manual test and visual inspection of the build logs.

Test 2) Functional test

1. Create test container
$ lxc launch ubuntu:jammy jammy-2002043
$ lxc shell jammy-2002043

2. Install required packages
For Jammy
# apt update -y && apt install -y python2 python-pip
For Focal
# apt update -y && apt install -y python2 python-setuptools gcc

3. Create test files
# mkdir testprog
# cd testprog
# cat >setup.py <<EOL
from setuptools import setup, Extension

setup(
    name="test",
    ext_modules=[Extension("test", sources=["testmodule.c"])],
    zip_safe=False
)
EOL
# cat >testmodule.c <<EOL
#include <stdio.h>

int main(void)
{
        printf("This is test program");
        return 0;
}
EOL

4. Compile a test program
# python2 setup.py build_ext --inplace

5. Check CFLAGS
# python2 -c "import sysconfig; print(sysconfig.get_config_var('CFLAGS'))"
-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -ffile-prefix-map=/build/python2.7-W40Ff2/python2.7-2.7.18=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security

6. Check the flags used to compile the test wheel
# strings build/lib.linux-x86_64-2.7/test.so | grep -- -O
GNU GIMPLE 11.4.0 -mtune=generic -march=x86-64 -g -g -g -g -O2 -fno-openmp -fno-openacc -fcf-protection=full -fno-strict-aliasing -fwrapv -fstack-protector-strong -ffat-lto-objects -fstack-protector-strong -fPIC -fltrans

7. Install fixed python
Once updated, this will simply be an apt update && apt upgrade
# add-apt-repository ppa:mitchdz/python2.7-optimization-flags -y
# apt install -y python2.7
# dpkg -s python2.7 | grep Version:
Version: 2.7.18-13ubuntu1.2~jammy9

8. Clean build
# rm -rf build/ test.so

9. Enable opt-in environment variable
# export APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND=""

9. Rebuild with new python2.7 installed
# python2 setup.py build_ext --inplace

10. Check build flags
# strings build/lib.linux-x86_64-2.7/test.so | grep -- -O
GNU GIMPLE 11.4.0 -mtune=generic -march=x86-64 -g -g -g -g -O2 -O2 -fno-openmp -fno-openacc -fcf-protection=full -fno-strict-aliasing -fwrapv -fstack-protector-strong -ffat-lto-objects -fstack-protector-strong -fPIC -fltrans
GNU C17 11.4.0 -mtune=generic -march=x86-64 -g -g -O2 -fno-strict-aliasing -fwrapv -flto -flto -ffat-lto-objects -fstack-protector-strong -fPIC -fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection -fcf-protection

[ Where problems could occur ]

* Changing optimization flags can cause a myriad of unattended side effects, but the change being opt-in means the users should be aware a change is being made
* The change is opt-in, so an informative message is printed to the console to spread awareness of the issue and how to use the workaround. A questionably made CI/CD system might see the new console output and flag it as a failure, so it is important to choose wording carefully.

[Fabio Augusto Miranda Martins (fabio.martins)]

Hello Nafees,

Can you share this setup.py that we can use to reproduce the problem and investigate?

Regards,
Fabio Martins

[Nafees (nafeabd)]

Hi Fabio Martins,
Thanks for looking into this issue. Attached tar file contains setup.py and steps to reproduce the issue. In general, the behavior is same with any package building with extensions.

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Nafees,

Thank you for the reproducer and instructions.

Per your comments, I would expect this to work well with python3 on 20.04, however, my tests indicated that python2 and python3 are both behaving the same way. In the example below, I'm using python3 and I also don't see the -O2 optimization:

ubuntu@ip-172-31-80-78:~/python-bug/testprog-python3-pip$ python3 setup.py build_ext --inplace
running build_ext
building 'test' extension
creating build/temp.linux-aarch64-3.8
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c testmodule.c -o build/temp.linux-aarch64-3.8/testmodule.o
creating build/lib.linux-aarch64-3.8
aarch64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-aarch64-3.8/testmodule.o -o build/lib.linux-aarch64-3.8/test.cpython-38-aarch64-linux-gnu.so
copying build/lib.linux-aarch64-3.8/test.cpython-38-aarch64-linux-gnu.so ->

ubuntu@ip-172-31-80-78:~/python-bug/testprog-python3-pip$ strings test.so | grep -- -O

(output from the command above was empty)

On the other hand, I do see the -O2 with python2 in a 22.04 instance.

Can you clarify if this is the same for you and, if not, provide some details on versions being used?

Regards,
Fabio Martins

[Nafees (nafeabd)]

Thanks for looking into this. On python3, can we make sure if we are looking into right output file?

From your pasted output on Python3, I do see, there is -O2 but I see from next command the query is against test.so file which seems incorrect. From the compilation output, the file name when python3 is used is test.cpython-38-aarch64-linux-gnu.so.

On my machine (Ubuntu 20.04) with Python3:-
ubuntu@ip-172-31-18-56:~/test$ python3 setup.py build_ext --inplace
running build_ext
building 'test' extension
creating build
creating build/temp.linux-aarch64-3.8
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c testmodule.c -o build/temp.linux-aarch64-3.8/testmodule.o
creating build/lib.linux-aarch64-3.8
aarch64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-aarch64-3.8/testmodule.o -o build/lib.linux-aarch64-3.8/test.cpython-38-aarch64-linux-gnu.so
copying build/lib.linux-aarch64-3.8/test.cpython-38-aarch64-linux-gnu.so ->
ubuntu@ip-172-31-18-56:~/test$ strings test.cpython-38-aarch64-linux-gnu.so | grep -- -O
GNU C17 9.4.0 -mlittle-endian -mabi=lp64 -g -g -g -g -O2 -O2 -fstack-protector-strong -fwrapv -fstack-protector-strong -fPIC -fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection

[Fabio Augusto Miranda Martins (fabio.martins)]

@Nafees,

I'm sorry, I hadn't noticed that the file produced by python3 was different (and I did have the test.so in my directory because I had built it with python2 before, so I was just grepping the same file). I see the problem now and I'm checking it with our Engineering team.

Regards,
Fabio Martins

[Ghadi Rahme (ghadi-rahme)]

Hello Nafees,

The bug has been fixed and I am currently preparing a patch for it. Thank you for reporting the issue.

Regards,
Ghadi Rahme

[Nafees (nafeabd)]

Hi Ghadi Rahme,
I see this ticket is in progress, just want to check when the fix will be updated in Ubuntu's packages?

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Nafees,

I've discussed this case with our Engineering team and they are working on the SRU process to get this fix released, however this should take around 1 month before it gets published to -updates. Although we are prioritizing it, the fix still needs to go to the -proposed repository for QA / testing, before completing the SRU process, hence why the 1 month estimate (assuming that it doesn't hit a regression, that might delay things a bit more).

In the meantime, if a user is being impacted, although not really supported, they could modify the sysconfig.py file to "manually" fix it with the steps below:

1) open the following file: /usr/lib/python2.7/distutils/sysconfig.py
2) change the following lines:

(cc, cxx, cflags, extra_cflags, basecflags,
ccshared, ldshared, so_ext, ar, ar_flags,
configure_cppflags, configure_cflags, configure_ldflags) = \
get_config_vars('CC', 'CXX', 'CFLAGS', 'EXTRA_CFLAGS', 'BASECFLAGS',
'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS',
'CONFIGURE_CPPFLAGS', 'CONFIGURE_CFLAGS', 'CONFIGURE_LDFLAGS')

To:

(cc, cxx, cflags, extra_cflags, basecflags,
ccshared, ldshared, so_ext, ar, ar_flags,
configure_cppflags, configure_cflags, configure_ldflags, opt) = \
get_config_vars('CC', 'CXX', 'CFLAGS', 'EXTRA_CFLAGS', 'BASECFLAGS',
'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS',
'CONFIGURE_CPPFLAGS', 'CONFIGURE_CFLAGS', 'CONFIGURE_LDFLAGS', 'OPT')

3) change the following lines:

elif configure_cflags:
cflags = ' '.join(str(x) for x in (basecflags, configure_cflags, extra_cflags) if x)
ldshared = ldshared + ' ' + configure_cflags
if 'CPPFLAGS' in os.environ:

To:

elif configure_cflags:
cflags = ' '.join(str(x) for x in (basecflags, opt ,configure_cflags, extra_cflags) if x)
ldshared = ldshared + ' ' + configure_cflags
if 'CPPFLAGS' in os.environ:

4) Thats it! After these changes the optimization should work fine on ARM as well as x86

Note: watch out for the indentation! Make sure the indentation is the same line per line as the line being replaced.

Please, let me know if this helps.

Regards,
Fabio Martins

[Nafees (nafeabd)]

Thanks for the update Fabio Martins !

[Ghadi Rahme (ghadi-rahme)]

Won't fix Lunar nor Mantic since package is not available for these releases

[Ghadi Rahme (ghadi-rahme)]

Hello,

Changes have been compiled and tested. This is the Focal patch and I will be sending the jammy and kinetic patches in the subsequent comments. The Bionic patch should be ready soon.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "[focal]add-optimization-flags-to-cflags.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Heitor Alves de Siqueira (halves)]

Thanks for the debdiffs, Ghadi! I've marked Bionic as Won't Fix, as the release is not under standard support anymore. If this is needed for 18.04, please consider targeting it under the Ubuntu Pro project.

[Steve Langasek (vorlon)]

Reviewing this as part of the Sponsor process, but with my SRU Team hat on.

The only releases python2.7 is present in which are still eligible for SRUs are focal, jammy, and kinetic.

All of these releases post-date the demotion of python2.7 to universe.

No one should be deploying new code on python2.7 in Ubuntu 20.04 or later.

For the cflags to have been this way for years without anyone noticing, I do not think 'high' is a correct bug importance.

We would not rebuild the python extensions in the archive to correct the CFLAGS they were already built with when released.

For the uncommon case of a user building their extensions, the obvious workaround is to set any important flags directly instead of relying on them being inherited from python2.7.

I would not accept this as an SRU in Ubuntu. Therefore I do not think anyone should proceed with sponsorship. Unsubscribing the sponsors team and marking wontfix.

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Steve,

Thanks for looking into this SRU request.

Their original request was specifically to python2.7 on Ĵammy, which matches your comment on the eligible releases.

The request is not to rebuild the python extensions, but to fix the compiler so new modules getting compiled will inherit the right flags.

Based on the description and the proposed patches, this seems to be a legitimate bug, so shouldn't we fix this? Especially considering that Focal still have a long lifecycle ahead.

Also, if the issue is that python2.7 is in universe, can't we fix this through esm-apps?

Regards,
Fabio Martins

[Fabio Augusto Miranda Martins (fabio.martins)]

It's also worth mentioning that, a patch was backported mentioning "# DP: Allow setting BASECFLAGS, OPT and EXTRA_LDFLAGS (like, CC, CXX, CPP, CFLAGS, CPPFLAGS, CCSHARED, LDSHARED) from the environment." [1], but then OPT is not being added at all by the patch (which had just mentioned adding it), which seems to point that this is a bug.

https://git.launchpad.net/ubuntu/+source/python2.7/tree/debian/patches/distutils-sysconfig.diff?h=applied/ubuntu/focal

[Steve Langasek (vorlon)]

It is a legitimate bug.

"Legitimate bug" is not the threshold for an SRU. There are many legitimate bugs that we decline to fix.

> Stable release updates are automatically recommended to a very large number
> of users, and so it is critically important to treat them with great caution.
> Therefore, when updates are proposed, they must be accompanied by a strong
> rationale and present a low risk of regressions.

https://wiki.ubuntu.com/StableReleaseUpdates#Why

Fixing this bug carries the non-local risk that subsequent security updates of OTHER packages will have regressions because the build flags used to compile them have been "corrected" to a value previously untested by us.

And as I said, this can be worked around in local code.

[Fabio Augusto Miranda Martins (fabio.martins)]

Alright, thanks for the information @Steve.

Hi @Nafees,

With the information above, it seems there's a regression risk if we fix this bug in python2.7 in Focal, and the risk outstands the benefits of the fix, as it seems to be an uncommon use-case and there's a workaround for it, so we wouldn't be able to apply the patch.

As a workaround, you can:

1. Export the desired CFLAGS in ~/.bashrc:

i.e.:

export CFLAGS="-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security"

2. Source .bashrc and compile:

ubuntu@ip-172-31-43-65:~/testprog$ source ~/.bashrc
ubuntu@ip-172-31-43-65:~/testprog$ python2 setup.py build_ext --inplace
running build_ext
building 'test' extension
aarch64-linux-gnu-gcc -pthread -fno-strict-aliasing -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c testmodule.c -o build/temp.linux-aarch64-2.7/testmodule.o
creating build/lib.linux-aarch64-2.7
aarch64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC build/temp.linux-aarch64-2.7/testmodule.o -o build/lib.linux-aarch64-2.7/test.so
copying build/lib.linux-aarch64-2.7/test.so ->

3. You will have the desired CFLAGS:

ubuntu@ip-172-31-43-65:~/testprog$ strings build/lib.linux-aarch64-2.7/test.so | grep -- -O
GNU C17 9.4.0 -mlittle-endian -mabi=lp64 -g -g -O2 -fno-strict-aliasing -fwrapv -fstack-protector-strong -fPIC -fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection

Would that work for you moving forward?

Regards,
Fabio Martins

[Fabio Augusto Miranda Martins (fabio.martins)]

Another suggestion from one of our Engineers is that you can also set it inside your python script, something like this:

```
from setuptools import setup, Extension
import sysconfig

extra_flags=sysconfig.get_config_var('CFLAGS').split()

setup(
    name="test",
    ext_modules=[Extension("test", sources=["testmodule.c"],
 extra_compile_args=extra_flags
)],
    zip_safe=False
)
```

[Nafees (nafeabd)]

Thanks for providing the workarounds. While the workarounds work in cases when user is willing to compile extensions from the python package sources and aware of these problems, for the most common cases work arounds are not feasible. There are 100’s of python packages with extension modules and the most common use case is to install them using pip. The general expectation from end user is to have them compile with right flags and work without any extra work. So it’s going to be very challenging for us to communicate end users about this potential problem which only occurs with Python 2 on Ubuntu distributions.
I’m also not clear about the regression problem mentioned above. Why would we expect regressions if the patch itself is trying to fix the bug and use right optimizations (in this case O2)?

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Steve,

Would you and the SRU team reconsider the "won't fix" decision or further elaborate on the regression problem, based on the comment above?

[Robby Pocase (rpocase)]

@vorlon - adding to Fabio's question. I agree with your reasoning that the workarounds should be reasonably easy for a knowledgeable user. We should enable end users to apply the same fix since they may be affected without realizing it. While we shouldn't encourage users to stay on 2.7, there are apps that either won't transition or still have a large amount of work to do so. For Jammy, I believe these apps can be effectively guaranteed security patches until 2032.

I believe the path below avoids the regression concerns. Its a bit ugly, but it at least won't outlive Jammy. None of this is ideal, but I think solves both sides of the equation.

* Add a feature flag that applies the proposed fix when present. This should be the default to satisfy the initial bug report constraint. Maybe UBUNTU_APPLY_DEFAULT_PYTHON_CFLAGS_ON_EXTENSIONS? Extremely verbose, but very little chance of someone accidentally disabling it. We should also print when its used so users have a launchpad issue to reference if it happens to cause a regression for a out of distro package compilation.
* Prevent applying the above feature flag for packages that already exist in the archive. This is a static list that is very unlikely to change. This does assume that distutils.sysconfig has a mechanism to introspect the package currently being built. Public API hints that it MAY be possible, but I haven't done thorough inspection across the ecosystem of callers.

Would you support a solution similar to the above?

[Dimitri John Ledkov (xnox)]

We should also consider the global impact beyond ubuntu. I believe many binary wheels in pypi are builing using the ubuntu's toolchain. Inadvertently we are the curator of the toolchain used to build wheels used by everyone for some of the pypi projects.

W.r.t. rebuilding extensions in the Ubuntu Archive - note that most of extensions use pybuild, which correctly export and set CFLAGS meaning all Ubuntu built binary extensions are not affected by this, as they correctly built with -O2 (e.g. focal pyyaml build https://launchpadlibrarian.net/533170365/buildlog_ubuntu-focal-amd64.pyyaml_5.3.1-1ubuntu0.1_BUILDING.txt.gz clearly has -O2). So this toolchain change is unlikely to require rebuilding any Ubuntu shipped binary extensions.

This really is about self-built wheels, and wheels built on Ubuntu and uploaded to pypi.org.

[Nafees (nafeabd)]

I want to check if there is any update on this?

[Mitchell Dzurick (mitchdz)]

Hi Nafees, I'll be looking into this bug. Planning to have some more updates next week.

[Nafees (nafeabd)]

Hi Mitchell, Do you have any update for this bug?

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Nafees, I discussed this with Mitchell and we are still looking into the best possible way to move forward with this SRU. We'll keep this bug update as soon as we have some more details to share.

[Nafees (nafeabd)]

I just want to check if there are any updates with this bug?

[Mitchell Dzurick (mitchdz)]

Hi, I have the patch modified how I want it to, I'm still exploring ways to blocklist the current launchpad builds, as I'm not sure I can extract the necessary metadata in distutils.sysconfig.

I'll be exploring more next week and attempting to find a reliable way to determine the name of the python wheel being built in the file that we are making the change.

[Mitchell Dzurick (mitchdz)]

Nafees, while exploring adding this change, a priority is to not cause any regressions in packages already built in the archives. This is unfortunately a non-trivial change so I'm proposing making this an opt-in change, but make an informative message whenever someone compiles wheels, so they can be made aware of it. This will make it so there's minimal chances of regression, and spreads the word of this change.

How does this solution sound? I have the change staged in https://code.launchpad.net/~mitchdz/ubuntu/+source/python2.7/+git/python2.7/+merge/458533

[Nafees (nafeabd)]

Thanks Mitchell for looking into this. The proposed workaround looks good to me.

[Mitchell Dzurick (mitchdz)]

Thanks Nafees! I'll go ahead and prepare the SRU and move forward with it.

[Mitchell Dzurick (mitchdz)]

Description Updated. I'll work on getting the patch for Jammy approved and then I will create the patch for Focal. I expect (and would love) feedback on wording for the new print statements I made.

[Timo Aaltonen (tjaalton)]

Hello Nafees, or anyone else affected,

Accepted python2.7 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python2.7/2.7.18-13ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Maybe it's just me, but could we please clarify in the test plan exactly what flag changes are we expecting? Given the change is to honor or not the "OPT" flags, I was expecting an additional test plan that would set those, and then show that they are included when the workaround var is set. That's much easier to verify.

Keep the tests you have, please, but what do you think about this extra test? It would clearly show when OPT is used and not.

[Andreas Hasenack (ahasenack)]

Hello Nafees, or anyone else affected,

Accepted python2.7 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python2.7/2.7.18-1~20.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mitchell Dzurick (mitchdz)]

That's a great idea Andreas! It's a little difficult to pinpoint the exact OPT flags as they are bunched up with the other compiler flags, so I'll add a test in addition to the current ones which makes it easier to see what's happening.

[Andreas Hasenack (ahasenack)]

You can add something obvious, like -DMITCHELL_WAS_HERE :)

On Thu, 8 Feb 2024, 18:35 Mitchell Dzurick, <email address hidden>
wrote:

> That's a great idea Andreas! It's a little difficult to pinpoint the
> exact OPT flags as they are bunched up with the other compiler flags, so
> I'll add a test in addition to the current ones which makes it easier to
> see what's happening.
>
> --
> You received this bug notification because you are a member of Ubuntu
> Pythoneers, which is subscribed to python2.7 in Ubuntu.
> https://bugs.launchpad.net/bugs/2002043
>
> Title:
> Python extension modules get built using wrong compiler flags with
> python2
>
> Status in python2.7 package in Ubuntu:
> Invalid
> Status in python2.7 source package in Bionic:
> Won't Fix
> Status in python2.7 source package in Focal:
> Fix Committed
> Status in python2.7 source package in Jammy:
> Fix Committed
> Status in python2.7 source package in Kinetic:
> Invalid
> Status in python2.7 source package in Lunar:
> Invalid
> Status in python2.7 source package in Mantic:
> Invalid
>
> Bug description:
> [ Impact ]
>
> When compiling Python extensions using Python2, CFLAGS optimization
> flags are dropped.
>
> This behavior has been caused by our update in this patch
>
> http://archive.ubuntu.com/ubuntu/pool/universe/p/python2.7/python2.7_2.7.18-1~20.04.3.diff.gz
> which differs from upstream.
>
> The fix modifies the portion of code in Lib/distutils/sysconfig.py
> which gets the cflags from the environments, and includes the dropped
> OPT flag from get_config_vars().
>
> [ Test Plan ]
>
> There will be 2 separate tests for this bug:
> * Ensuring no-change rebuilds are not changed
> * Ensuring local builds are not changed unless environment variable is
> set
>
> Test 1) No-change rebuilds
>
> To test that no-change rebuilds are not changed, the package python-
> stdlib-extensions will be built against the new python2.7, and confirm
> the compiler flags are not altered. This will be a manual test and
> visual inspection of the build logs.
>
> Test 2) Functional test
>
> 1. Create test container
> $ lxc launch ubuntu:jammy jammy-2002043
> $ lxc shell ubuntu:jammy jammy-2002043
>
> 2. Install required packages
> For Jammy
> # apt update -y && apt install -y python2 python-pip
> For Focal
> # apt update -y && apt install -y python2 python-setuptools
>
> 3. Create test files
> # mkdir testprog
> # cd testprog
> # cat >setup.py <<EOL
> from setuptools import setup, Extension
>
> setup(
> name="test",
> ext_modules=[Extension("test", sources=["testmodule.c"])],
> zip_safe=False
> )
> EOL
> # cat >testmodule.c <<EOL
> #include <stdio.h>
>
> int main(void)
> {
> printf("This is test program");
> return 0;
> }
> EOL
>
> 4. Compile a test program
> # python2 setup.py build_ext --inplace
>
> 5. Check CFLAGS
> # python2 -c "import sysconfig;
> print(sysconfig.get_config_var('CFLAGS'))"
> -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
> -Wdate-time -D_FORTIFY_SOURCE=2 -g
> -ffile-prefix-map=/build/python2.7-W40Ff2/pytho...

[Mitchell Dzurick (mitchdz)]

Attached is my manual verification steps of testing the proposed package for Jammy.

Package version 2.7.18-13ubuntu1.2

Part of the verification process is to ensure no regressions happen in no-change rebuilds of the current wheels compiled with python2.7 in our archives. It was determined that the best way to do this was to make this feature opt-in, so a test is performed to ensure that functionally no changes happen for no-change rebuilds.

This was tested in this PPA - https://launchpad.net/~mitchdz/+archive/ubuntu/python-stdlib-extensions-py2-opt

Observe the build logs and see

The specific build log for Jammy is https://launchpadlibrarian.net/714574054/buildlog_ubuntu-jammy-amd64.python-stdlib-extensions_2.7.18-1ubuntu1~jammy1_BUILDING.txt.gz

Where you will see the following lines showing the workaround is not used:

cd 2.7 && python2.7 setup.py build
running build
running build_ext
There is a workaround to now inherit optimization CFLAGS when compiling wheels.
To enable this, set APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND in your
environment. See LP: https://launchpad.net/bugs/2002043 for further context.
APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND not detected.
building 'gdbm' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/Modules
x86_64-linux-gnu-gcc -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -ffile-prefix-map=/build/python2.7-RH0SVf/python2.7-2.7.18=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -I/usr/include/python2.7 -c Modules/gdbmmodule.c -o build/temp.linux-x86_64-2.7/Modules/gdbmmodule.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -ffile-prefix-map=/build/python2.7-RH0SVf/python2.7-2.7.18=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -ffile-prefix-map=/build/python2.7-RH0SVf/python2.7-2.7.18=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fPIC build/temp.linux-x86_64-2.7/Modules/gdbmmodule.o -lgdbm -o build/lib.linux-x86_64-2.7/gdbm.so
building '_tkinter' extension

Notice the lack of the OPT flags (-DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes) in the compiletime flags.

[Mitchell Dzurick (mitchdz)]

Attached is my manual verification steps of testing the proposed package for Focal.

Package version tested: 2.7.18-1~20.04.4

Part of the verification process is to ensure no regressions happen in no-change rebuilds of the current wheels compiled with python2.7 in our archives. It was determined that the best way to do this was to make this feature opt-in, so a test is performed to ensure that functionally no changes happen for no-change rebuilds.

This was tested in this PPA - https://launchpad.net/~mitchdz/+archive/ubuntu/python-stdlib-extensions-py2-opt

Observe the build logs and see

The specific build log for Focal is https://launchpadlibrarian.net/714574260/buildlog_ubuntu-focal-amd64.python-stdlib-extensions_2.7.18-1ubuntu1~focal1_BUILDING.txt.gz

Where you will see the following lines showing the workaround is not used:

cd 2.7 && python2.7 setup.py build
running build
running build_ext
There is a workaround to now inherit optimization CFLAGS when compiling wheels.
To enable this, set APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND in your
environment. See LP: https://launchpad.net/bugs/2002043 for further context.
APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND not detected.
building 'gdbm' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/Modules
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-CxOYiX/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -I/usr/include/python2.7 -c Modules/gdbmmodule.c -o build/temp.linux-x86_64-2.7/Modules/gdbmmodule.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-CxOYiX/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-CxOYiX/python2.7-2.7.18=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC build/temp.linux-x86_64-2.7/Modules/gdbmmodule.o -lgdbm -o build/lib.linux-x86_64-2.7/gdbm.so
building '_tkinter' extension

Notice the lack of the OPT flags (-DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes) in the compile time flags for the module.

[Mauricio Faria de Oliveira (mfo)]

Just a few observations before release.

1) The verification for Jammy apparently has a copy-paste error,
as the banner about the env var is present (step 5) _before_ the
package from -proposed -- which adds it -- is installed (step 6).

To be clear, this is not a problem and should not block the release,
as the _new_ behavior (with -proposed installed) is what matters,
and that looks correct in the output:
- env var undefined/no opt-in = old behavior;
- env var defined/opted-in = new behavior.

The verification for Focal is OK on that regard (does not show the banner in step 5).

2) The autopkgtests for python2.7/focal failed, but this is not due to these changes.
I re-run the tests with the trigger migration-reference/0, and they continue to fail.

This is expected, since the changes are opt-in / are not enabled by default.
The only change which _is_ enabled by default is the new banner during build,
but that output is likely not checked for, I'd imagine (ie, it's OK to change).

3) There's been a recent rebuild of the autopkgtest database, and I'd have to
verify the status (ie, finished?) and impact (eg, report pages) at the moment
to make sure this is OK on that regard.

[Mitchell Dzurick (mitchdz)]

mfo: thanks for catching the ctrlc+ctrlv error! It sounds like we are just waiting to re-run the autopkgtest then.

Let me know if you would like any additional testing from me!

[Mauricio Faria de Oliveira (mfo)]

Hey Mitchell,

No, the autopkgtests re-run for python2.7/focal are done and OK ('they continue to fail').
Link: [1] (see the Focal results on diff archs have 'migration-reference/0' and 'mfo').

The autopkgtests point pending is a clarification on the database/results, ie, whether we can trust the report pages (pending-sru, update_excuses).

We'll check that. Thanks!

[1] https://autopkgtest.ubuntu.com/packages/python2.7

[Mauricio Faria de Oliveira (mfo)]

Hi Mitchell,

Indeed, there are some autopkgtests results/failures that are not reported in pending-sru or update_excuses -- this is the case specially for Focal (Jammy looks good).

I downloaded the autopkgtest.db from autopkgtest.ubuntu.com, and locally queried it.
The test results with exitcode != 0 need checking.

Could you please take a look at those?
If you need any tests retried or other assistance, please let us know!
(You know, starting from https://autopkgtest.ubuntu.com/packages/$SRCPKG and going from there.)

Thanks!
Mauricio

[Mauricio Faria de Oliveira (mfo)]

RELEASE=jammy
TRIGGER=python2.7/2.7.18-13ubuntu1.2

sqlite3 autopkgtest.db -column -header \
"SELECT test.release, test.arch, test.package, "\
" result.version, result.exitcode, "\
" result.triggers, result.requester "\
"FROM test, result "\
"WHERE test.id = result.test_id "\
"AND test.release = '${RELEASE}' "\
"AND result.triggers LIKE '%${TRIGGER}%' "\
"ORDER BY test.release, test.package, test.arch"

release arch package version exitcode triggers requester
------- ------- ------------------ ------------------ -------- ---------------------------- ---------
jammy amd64 libgnatcoll-python 21.0.0-2 0 python2.7/2.7.18-13ubuntu1.2
jammy arm64 libgnatcoll-python 21.0.0-2 0 python2.7/2.7.18-13ubuntu1.2
jammy armhf libgnatcoll-python 21.0.0-2 0 python2.7/2.7.18-13ubuntu1.2
jammy ppc64el libgnatcoll-python 21.0.0-2 0 python2.7/2.7.18-13ubuntu1.2
jammy s390x libgnatcoll-python 21.0.0-2 0 python2.7/2.7.18-13ubuntu1.2
jammy amd64 python2.7 2.7.18-13ubuntu1.2 0 python2.7/2.7.18-13ubuntu1.2
jammy arm64 python2.7 2.7.18-13ubuntu1.2 0 python2.7/2.7.18-13ubuntu1.2
jammy armhf python2.7 2.7.18-13ubuntu1.2 0 python2.7/2.7.18-13ubuntu1.2
jammy i386 python2.7 2.7.18-13ubuntu1.2 12 python2.7/2.7.18-13ubuntu1.2
jammy ppc64el python2.7 2.7.18-13ubuntu1.2 0 python2.7/2.7.18-13ubuntu1.2
jammy s390x python2.7 2.7.18-13ubuntu1.2 0 python2.7/2.7.18-13ubuntu1.2

jammy/i386/python2.7: not a regression (unrelated failure)

[Mauricio Faria de Oliveira (mfo)]

RELEASE=focal
TRIGGER=python2.7/2.7.18-1~20.04.4

sqlite3 autopkgtest.db -column -header \
"SELECT test.release, test.arch, test.package, "\
" result.version, result.exitcode, "\
" result.triggers, result.requester "\
"FROM test, result "\
"WHERE test.id = result.test_id "\
"AND test.release = '${RELEASE}' "\
"AND result.triggers LIKE '%${TRIGGER}%' "\
"ORDER BY test.release, test.package, test.arch"

release arch package version exitcode triggers requester
------- ------- --------------------- ---------------------------- -------- -------------------------- ---------
focal amd64 dbus-python 1.2.16-1build1 0 python2.7/2.7.18-1~20.04.4
focal arm64 dbus-python 1.2.16-1build1 0 python2.7/2.7.18-1~20.04.4
focal armhf dbus-python 1.2.16-1build1 0 python2.7/2.7.18-1~20.04.4
focal i386 dbus-python 1.2.16-1build1 14 python2.7/2.7.18-1~20.04.4
focal ppc64el dbus-python 1.2.16-1build1 0 python2.7/2.7.18-1~20.04.4
focal s390x dbus-python 1.2.16-1build1 0 python2.7/2.7.18-1~20.04.4
focal amd64 kodi 2:18.6+dfsg1-2ubuntu1 0 python2.7/2.7.18-1~20.04.4
focal arm64 kodi 2:18.6+dfsg1-2ubuntu1 0 python2.7/2.7.18-1~20.04.4
focal armhf kodi 2:18.6+dfsg1-2ubuntu1 0 python2.7/2.7.18-1~20.04.4
focal ppc64el kodi 2:18.6+dfsg1-2ubuntu1 0 python2.7/2.7.18-1~20.04.4
focal amd64 libapache2-mod-python 3.3.1-11ubuntu5 0 python2.7/2.7.18-1~20.04.4
focal arm64 libapache2-mod-python 3.3.1-11ubuntu5 0 python2.7/2.7.18-1~20.04.4
focal armhf libapache2-mod-python 3.3.1-11ubuntu5 0 python2.7/2.7.18-1~20.04.4
focal ppc64el libapache2-mod-python 3.3.1-11ubuntu5 0 python2.7/2.7.18-1~20.04.4
focal s390x libapache2-mod-python 3.3.1-11ubuntu5 0 python2.7/2.7.18-1~20.04.4
focal amd64 libgnatcoll-bindings 19-2 0 python2.7/2.7.18-1~20.04.4
focal arm64 libgnatcoll-bindings 19-2 0 python2.7/2.7.18-1~20.04.4
focal armhf libgnatcoll-bindings 19-2 0 python2.7/2.7.18-1~20.04.4
focal ppc64el libgnatcoll-bindings 19-2 0 python2.7/2.7.18-1~20.04.4
focal s390x libgnatcoll-bindings 19-2 0 python2.7/2.7.18-1~20.04.4
focal amd64 libxml2 2.9.10+dfsg-5ubuntu0.20.04.6 0 python2.7/2.7.18-1~20.04.4
focal arm64 libxml2 2.9.10+dfsg-5ubuntu0.20.04.6 0 python2.7/2.7.18-1~20.04.4
focal armhf libxml2 2.9.10+dfsg-5ubuntu0.20.04.6 0 python2.7/2.7.18-1~20.04.4
focal i386 libxml2 2.9.10+dfsg-5ubuntu0...

[Mitchell Dzurick (mitchdz)]

Thanks a ton Mauricio! Would we be good to start phasing the Jammy package then while I investigate the Focal failures? I'll start taking a look at Focal today.

[Mauricio Faria de Oliveira (mfo)]

No problem, Mitchell!

Yes, the point pending from comment #44 (.3) is resolved for Jammy per #48 (so Jammy should be OK, AFAICT), and pending for Focal per #49 (being addressed per #50; thanks!).

Please note that other SRU vanguards may have a different opinion (or raise other points I may have missed); my shift is on Mondays, so this is to be checked by someone else during this week.

[Andreas Hasenack (ahasenack)]

I grabbed one of the focal failures from comment #49, mercurial:

focal amd64 mercurial 5.3.1-1ubuntu1 4 python2.7/2.7.18-1~20.04.4

That is indeed mostly red in [1], including the recent run with python2.7 from this SRU.

And it's not showing up as a failure in the excuses[2] report.

But it's hinted[3] on s390x:

# mercurial tests are flakey in focal https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956697
force-badtest mercurial/5.3.1-1ubuntu1/s390x

So that would explain it not being flagged for s390x, but not for amd64. I'm still puzzled: I didn't find another hint.

I retriggered the mercurial amd64 test, let's see if a new result refreshes the report.

1. https://autopkgtest.ubuntu.com/packages/m/mercurial/focal/amd64
2. https://ubuntu-archive-team.ubuntu.com/proposed-migration/focal/update_excuses.html#python2.7
3. https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu/commit/?id=8f4fd5462e98199c76909d3510911c573f0fa071

[Andreas Hasenack (ahasenack)]

> So that would explain it not being flagged for s390x, but not for amd64. I'm still puzzled: I didn't find
> another hint.

Ah, it's because of a migration-reference/0 run from last year:

5.3.1-1ubuntu1 migration-reference/0 None 2023-06-20 09:25:36 UTC 0h 37m 11s schopin fail

@mfo, I think your query is indeed showing the raw data, but not the interpretation of the results. The analysis seems to be missing:
- hints
- migration-reference/0 runs

I don't know if there is another db-like file that has "final" results that take all of this into account, or if it has to be analyzed each time by the excuses report when deciding if a failure should be flagged or not.

But at least on the above mercurial failure, I think it's a failure that is correctly not being shown in the excuses report.

[Andreas Hasenack (ahasenack)]

Taking a look at another focal result from the table in comment 49:

focal amd64 pam-python 1.0.7-1ubuntu1 4 python2.7/2.7.18-1~20.04.4

No hints in the hints-ubuntu repository, focal branch.

But there is a migration-reference/0 run that failed that exact same version:

1.0.7-1ubuntu1 migration-reference/0 None 2021-12-13 19:37:26 UTC 0h 02m 11s - fail

So the recent failure is not reported in the excuses page.

I'm starting to think that's the same for all cases (heh, sample of 2). @mfo, do you agree?

[Andreas Hasenack (ahasenack)]

Checking the jammy amd64 autopkgtest runs[1], we can see that the top most one is the run for this sru. Its log[2] is showing, at the end, a good run, but if you look more carefully, there are some test failures that apparently are being ignored.

In particular:
1910s ======================================================================
1910s FAIL: test_customize_compiler (distutils.tests.test_sysconfig.SysconfigTestCase)
1910s ----------------------------------------------------------------------
1910s Traceback (most recent call last):
1910s File "/usr/lib/python2.7/distutils/tests/test_sysconfig.py", line 110, in test_customize_compiler
1910s 'env_cc --sc-cflags --env-cflags --env-cppflags')
1910s AssertionError: 'env_cc -fno-strict-aliasing --env-cflags --env-cppflags' != 'env_cc --sc-cflags --env-cflags --env-cppflags'

That is exactly the function being touched by this patch. Could we have a bit of an investigation on what's going on? Was it failing before, and keeps failing now? Is this SRU patch affecting this test?

1. https://autopkgtest.ubuntu.com/packages/python2.7/jammy/amd64
2. https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/p/python2.7/20240202_171211_eb91c@/log.gz

[Mitchell Dzurick (mitchdz)]

Hi Andreas, the test_customize_compiler flag test has always failed. I'm curious about why it's failing but the autopkgtest as a whole is passing.

In fact, the same failures occur for both jammy/focal since forever - yet they seem to be the reason w hy the Focal autopkgtest fails, whereas Jammy happily passes the test even with these failures present.

[Mitchell Dzurick (mitchdz)]

i took a look at the other focal packages and put my thoughts.

RELEASE=focal
TRIGGER=python2.7/2.7.18-1~20.04.4

release arch package version exitcode triggers requester
------- ------- --------------------- ---------------------------- -------- -------------------------- ---------

focal i386 dbus-python 1.2.16-1build1 14 python2.7/2.7.18-1~20.04.4
Has always been giving the exit code 14 ever since glibc/2.31-0ubuntu6 back in 2020
In hints as force-badtest https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu/tree/ubuntu-release?h=focal#n230

focal i386 libxml2 2.9.10+dfsg-5ubuntu0.20.04.6 12 python2.7/2.7.18-1~20.04.4
Has been failing since glibc/2.30-0ubuntu3 in 2019
In hints as force-badtest https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu/tree/ubuntu-release?h=focal#n236

focal arm64 llvm-toolchain-7 1:7.0.1-12 4 python2.7/2.7.18-1~20.04.4
focal armhf llvm-toolchain-7 1:7.0.1-12 4 python2.7/2.7.18-1~20.04.4
focal ppc64el llvm-toolchain-7 1:7.0.1-12 4 python2.7/2.7.18-1~20.04.4
This package has just never passed on any of these architectures. I'm not sure why it's not passing.

focal amd64 mercurial 5.3.1-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal arm64 mercurial 5.3.1-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal armhf mercurial 5.3.1-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal i386 mercurial 5.3.1-1ubuntu1 12 python2.7/2.7.18-1~20.04.4
focal ppc64el mercurial 5.3.1-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal s390x mercurial 5.3.1-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
Mentioned by Andreas already

focal amd64 pam-python 1.0.7-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal arm64 pam-python 1.0.7-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal armhf pam-python 1.0.7-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal ppc64el pam-python 1.0.7-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
focal s390x pam-python 1.0.7-1ubuntu1 4 python2.7/2.7.18-1~20.04.4
Mentioned by Andreas already

focal i386 python-debian 0.1.36ubuntu1.1 12 python2.7/2.7.18-1~20.04.4
Fails due to unmet dependencies since glibc/2.30-0ubuntu3. I didn't notice anything in the hints.

focal i386 python-pbr 5.4.5-0ubuntu1 12 python2.7/2.7.18-1~20.04.4
Fails due to unmet dependencies since glibc/2.30-0ubuntu3. I didn't notice anything in the hints.

focal amd64 python2.7 2.7.18-1~20.04.4 4 python2.7/2.7.18-1~20.04.4
focal arm64 python2.7 ...

[Mitchell Dzurick (mitchdz)]

The only focal autopkgtest that really has me confused in the python2.7 failure. This is due to a unit test that fails. The strange thing is that this unit test actually fails in the Jammy autopkgtest, but the Jammy autopkgtest passes anyways. It has always been like this.

You will see

1643s FAIL: test_customize_compiler (distutils.tests.test_sysconfig.SysconfigTestCase)
1643s ----------------------------------------------------------------------
1643s Traceback (most recent call last):
1643s File "/usr/lib/python2.7/distutils/tests/test_sysconfig.py", line 110, in test_customize_compiler
1643s 'env_cc --sc-cflags --env-cflags --env-cppflags')
1643s AssertionError: 'env_cc -fno-strict-aliasing --env-cflags --env-cppflags' != 'env_cc --sc-cflags --env-cflags --env-cppflags'

Which has always been like this - my patch did not cause this change.

I think this should be safe to ignore simply because the package has always been like this, and if it is an issue we should've found a bug for it by now.

I tried to see if the Debian autopkgtest also sees this behavior, but unfortunately those logs have been removed due to the retention policy.

I'm curious if the change I am doing will fix this unit test... So I created a PPA with the change enabled by default and will come back when the results are ready.

[Mitchell Dzurick (mitchdz)]

Ran the autopkgtest on a new PPA with my changes enabled by default and the test still fails[0]. I believe this is fine enough to ignore simply because this test has always been failing like this.

[0] - https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-mitchdz-python2.7-opt-test-enabled-default/jammy/amd64/p/python2.7/20240313_174406_9bb00@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package python2.7 - 2.7.18-13ubuntu1.2

---------------
python2.7 (2.7.18-13ubuntu1.2) jammy; urgency=medium

  * Add d/p/add-optimization-flags-to-cflags.diff: Add optimization flags to
    cflags when compiling C modules. (LP: #2002043)

 -- Mitchell Dzurick <email address hidden> Sun, 15 Oct 2023 12:43:11 -0400

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for python2.7 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python2.7 - 2.7.18-1~20.04.4

---------------
python2.7 (2.7.18-1~20.04.4) focal; urgency=medium

  * Add d/p/add-optimization-flags-to-cflags.diff: Add optimization flags to
    cflags when compiling C modules. (LP: #2002043)

 -- Mitchell Dzurick <email address hidden> Wed, 31 Jan 2024 09:23:13 -0700