FTBFS and autopkgtest failure in Hirsute (badly generated symbols files)

[Balint Reczey]

https://autopkgtest.ubuntu.com/packages/t/talloc/hirsute/amd64

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/t/talloc/20210316_172513_0ef88@/log.gz
...
dh_makeshlibs -ppython3-talloc -Xtalloc. -- -c3
dpkg-gensymbols: error: some libraries disappeared in the symbols file: _pytalloc_get_name@PYTALLOC_UTIL_2.3.0 PYTALLOC_UTIL_2.1.10@PYTALLOC_UTIL_2.1.10 pytalloc_Check@PYTALLOC_UTIL_2.0.6 pytalloc_BaseObject_size@PYTALLOC_UTIL_2.1.6 PYTALLOC_UTIL_2.1.5@PYTALLOC_UTIL_2.1.5 _pytalloc_get_ptr@PYTALLOC_UTIL_2.1.6 PYTALLOC_UTIL_2.1.12@PYTALLOC_UTIL_2.1.12 PYTALLOC_UTIL_2.3.1@PYTALLOC_UTIL_2.3.1 PYTALLOC_UTIL_2.0.7@PYTALLOC_UTIL_2.0.7 PYTALLOC_UTIL_2.1.11@PYTALLOC_UTIL_2.1.11 PYTALLOC_UTIL_2.0.8@PYTALLOC_UTIL_2.0.8 PYTALLOC_UTIL_2.1.1@PYTALLOC_UTIL_2.1.1 PYTALLOC_UTIL_2.0.6@PYTALLOC_UTIL_2.0.6 PYTALLOC_UTIL_2.1.4@PYTALLOC_UTIL_2.1.4 pytalloc_GetObjectType@PYTALLOC_UTIL_2.0.6 _pytalloc_get_mem_ctx@PYTALLOC_UTIL_2.1.6 PYTALLOC_UTIL_2.1.16@PYTALLOC_UTIL_2.1.16 PYTALLOC_UTIL_2.1.2@PYTALLOC_UTIL_2.1.2 PYTALLOC_UTIL_2.1.13@PYTALLOC_UTIL_2.1.13 PYTALLOC_UTIL_2.1.7@PYTALLOC_UTIL_2.1.7 PYTALLOC_UTIL_2.2.0@PYTALLOC_UTIL_2.2.0 pytalloc_GetBaseObjectType@PYTALLOC_UTIL_2.1.6 PYTALLOC_UTIL_2.1.9@PYTALLOC_UTIL_2.1.9 PYTALLOC_UTIL_2.1.14@PYTALLOC_UTIL_2.1.14 pytalloc_BaseObject_check@PYTALLOC_UTIL_2.1.6 PYTALLOC_UTIL.CPYTHON_39_X86_64_LINUX_GNU_2.3.1@PYTALLOC_UTIL.CPYTHON_39_X86_64_LINUX_GNU_2.3.1 PYTALLOC_UTIL_2.3.0@PYTALLOC_UTIL_2.3.0 pytalloc_GenericObject_reference_ex@PYTALLOC_UTIL_2.1.9 pytalloc_BaseObject_PyType_Ready@PYTALLOC_UTIL_2.1.6 pytalloc_reference_ex@PYTALLOC_UTIL_2.0.6 PYTALLOC_UTIL_2.1.0@PYTALLOC_UTIL_2.1.0 PYTALLOC_UTIL_2.1.15@PYTALLOC_UTIL_2.1.15 _pytalloc_get_type@PYTALLOC_UTIL_2.1.6 _pytalloc_check_type@PYTALLOC_UTIL_2.1.9 pytalloc_steal@PYTALLOC_UTIL_2.0.6 PYTALLOC_UTIL_2.1.6@PYTALLOC_UTIL_2.1.6 PYTALLOC_UTIL_2.1.3@PYTALLOC_UTIL_2.1.3 pytalloc_GenericObject_steal_ex@PYTALLOC_UTIL_2.1.9 pytalloc_steal_ex@PYTALLOC_UTIL_2.0.6 PYTALLOC_UTIL_2.1.8@PYTALLOC_UTIL_2.1.8
dpkg-gensymbols: error: some new symbols appeared in the symbols file: see diff output below
dpkg-gensymbols: warning: debian/python3-talloc/DEBIAN/symbols doesn't match completely debian/python3-talloc.symbols
--- debian/python3-talloc.symbols (python3-talloc_2.3.1-2build1_amd64)
+++ dpkg-gensymbols3SpPRF 2021-03-16 17:25:03.424278193 +0000
@@ -1,42 +1,41 @@
-PYTALLOC_UTIL.CPYTHON_39_X86_64_LINUX_GNU_2.3.1@PYTALLOC_UTIL.CPYTHON_39_X86_64_LINUX_GNU_2.3.1 2.3.1
-* Build-Depends-Package: python3-talloc-dev
-PYTALLOC_UTIL_2.0.6@PYTALLOC_UTIL_2.0.6 2.3.0
-PYTALLOC_UTIL_2.0.7@PYTALLOC_UTIL_2.0.7 2.3.0
-PYTALLOC_UTIL_2.0.8@PYTALLOC_UTIL_2.0.8 2.3.0
-PYTALLOC_UTIL_2.1.0@PYTALLOC_UTIL_2.1.0 2.3.0
-PYTALLOC_UTIL_2.1.10@PYTALLOC_UTIL_2.1.10 2.3.0
-PYTALLOC_UTIL_2.1.11@PYTALLOC_UTIL_2.1.11 2.3.0
-PYTALLOC_UTIL_2.1.12@PYTALLOC_UTIL_2.1.12 2.3.0
-PYTALLOC_UTIL_2.1.13@PYTALLOC_UTIL_2.1.13 2.3.0
-PYTALLOC_UTIL_2.1.14@PYTALLOC_UTIL_2.1.14 2.3.0
-PYTALLOC_UTIL_2.1.15@PYTALLOC_UTIL_2.1.15 2.3.0
-PYTALLOC_UTIL_2.1.16@PYTALLOC_UTIL_2.1.16 2.3.0
-PYTALLOC_UTIL_2.1.1@PYTALLOC_UTIL_2.1.1 2.3.0
-PYTALLOC_UTIL_2.1.2@PYTALLOC_UTIL_2.1.2 2.3.0
-PYTALLOC_UTIL_2.1.3@PYTALLOC_UTIL_2.1.3 2.3.0
-PYTALLOC_UTIL_2.1.4@PYTALLOC_UTIL_2.1.4 2.3.0
-PYTALLOC_UTIL_2.1.5@PYTALLOC_UTIL_2.1.5 2.3.0
-PYTALLOC_UTIL_2.1.6@PYTALLOC_UTIL_2.1.6 2.3.0
-PYTALLOC_UTIL_2.1.7@PYTALLOC_UTIL_2.1.7 2.3.0
-PYTALLOC_UTIL_2.1.8@PYTALLOC_UTIL_2.1.8 2.3.0
-PYTALLOC_UTIL_2.1.9@PYTALLOC_UTIL_2.1.9 2.3.0
-PYTALLOC_UTIL_2.2.0@PYTALLOC_UTIL_2.2.0 2.3.0
-PYTALLOC_UTIL_2.3.0@PYTALLOC_UTIL_2.3.0 2.3.0
-PYTALLOC_UTIL_2.3.1@PYTALLOC_UTIL_2.3.1 2.3.1
-_pytalloc_check_type@PYTALLOC_UTIL_2.1.9 2.1.16
-_pytalloc_get_mem_ctx@PYTALLOC_UTIL_2.1.6 2.1.16
-_pytalloc_get_name@PYTALLOC_UTIL_2.3.0 2.3.0
-_pytalloc_get_ptr@PYTALLOC_UTIL_2.1.6 2.1.16
-_pytalloc_get_type@PYTALLOC_UTIL_2.1.6 2.1.16
 libpytalloc-util.cpython-39-x86-64-linux-gnu.so.2 python3-talloc #MINVER#
-pytalloc_BaseObject_PyType_Ready@PYTALLOC_UTIL_2.1.6 2.1.16
-pytalloc_BaseObject_check@PYTALLOC_UTIL_2.1.6 2.1.16
-pytalloc_BaseObject_size@PYTALLOC_UTIL_2.1.6 2.1.16
-pytalloc_Check@PYTALLOC_UTIL_2.0.6 2.1.16
-pytalloc_GenericObject_reference_ex@PYTALLOC_UTIL_2.1.9 2.1.16
-pytalloc_GenericObject_steal_ex@PYTALLOC_UTIL_2.1.9 2.1.16
-pytalloc_GetBaseObjectType@PYTALLOC_UTIL_2.1.6 2.1.16
-pytalloc_GetObjectType@PYTALLOC_UTIL_2.0.6 2.1.16
-pytalloc_reference_ex@PYTALLOC_UTIL_2.0.6 2.1.16
-pytalloc_steal@PYTALLOC_UTIL_2.0.6 2.1.16
-pytalloc_steal_ex@PYTALLOC_UTIL_2.0.6 2.1.16
+ PYTALLOC_UTIL.CPYTHON_39_X86_64_LINUX_GNU_2.3.1@PYTALLOC_UTIL.CPYTHON_39_X86_64_LINUX_GNU_2.3.1 2.3.1-2build1
+ PYTALLOC_UTIL_2.0.6@PYTALLOC_UTIL_2.0.6 2.3.1-2build1
+ PYTALLOC_UTIL_2.0.7@PYTALLOC_UTIL_2.0.7 2.3.1-2build1
+ PYTALLOC_UTIL_2.0.8@PYTALLOC_UTIL_2.0.8 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.0@PYTALLOC_UTIL_2.1.0 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.10@PYTALLOC_UTIL_2.1.10 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.11@PYTALLOC_UTIL_2.1.11 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.12@PYTALLOC_UTIL_2.1.12 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.13@PYTALLOC_UTIL_2.1.13 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.14@PYTALLOC_UTIL_2.1.14 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.15@PYTALLOC_UTIL_2.1.15 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.16@PYTALLOC_UTIL_2.1.16 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.1@PYTALLOC_UTIL_2.1.1 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.2@PYTALLOC_UTIL_2.1.2 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.3@PYTALLOC_UTIL_2.1.3 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.4@PYTALLOC_UTIL_2.1.4 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.5@PYTALLOC_UTIL_2.1.5 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.6@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.7@PYTALLOC_UTIL_2.1.7 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.8@PYTALLOC_UTIL_2.1.8 2.3.1-2build1
+ PYTALLOC_UTIL_2.1.9@PYTALLOC_UTIL_2.1.9 2.3.1-2build1
+ PYTALLOC_UTIL_2.2.0@PYTALLOC_UTIL_2.2.0 2.3.1-2build1
+ PYTALLOC_UTIL_2.3.0@PYTALLOC_UTIL_2.3.0 2.3.1-2build1
+ PYTALLOC_UTIL_2.3.1@PYTALLOC_UTIL_2.3.1 2.3.1-2build1
+ _pytalloc_check_type@PYTALLOC_UTIL_2.1.9 2.3.1-2build1
+ _pytalloc_get_mem_ctx@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ _pytalloc_get_name@PYTALLOC_UTIL_2.3.0 2.3.1-2build1
+ _pytalloc_get_ptr@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ _pytalloc_get_type@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ pytalloc_BaseObject_PyType_Ready@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ pytalloc_BaseObject_check@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ pytalloc_BaseObject_size@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ pytalloc_Check@PYTALLOC_UTIL_2.0.6 2.3.1-2build1
+ pytalloc_GenericObject_reference_ex@PYTALLOC_UTIL_2.1.9 2.3.1-2build1
+ pytalloc_GenericObject_steal_ex@PYTALLOC_UTIL_2.1.9 2.3.1-2build1
+ pytalloc_GetBaseObjectType@PYTALLOC_UTIL_2.1.6 2.3.1-2build1
+ pytalloc_GetObjectType@PYTALLOC_UTIL_2.0.6 2.3.1-2build1
+ pytalloc_reference_ex@PYTALLOC_UTIL_2.0.6 2.3.1-2build1
+ pytalloc_steal@PYTALLOC_UTIL_2.0.6 2.3.1-2build1
+ pytalloc_steal_ex@PYTALLOC_UTIL_2.0.6 2.3.1-2build1
dh_makeshlibs: error: failing due to earlier errors
...

[Dan Bungert (dbungert)]

Change the generated filename.

[Matthias Klose (doko)]

so the patch writes a python3-talloc.symbols.common file which then isn't used. I don't think that's the correct fix.

[Dan Bungert (dbungert)]

I appreciate the feedback Matthias.
Please see my new patch. I have seen some unexpected behavior from dh-exec, and this patch provides a workaround. Tomorrow I plan to play with dh-exec more and expect to be filing a bug.

[Matthias Klose (doko)]

dh-exec's lib/dh-exec-filter-build-profiles has some line trimming ...

[...]
    s/(^\s+|\s+$)//;
[...]

which is done unconditionally, removing all leading spaces. Apparently nobody thought about using dh-exec for symbols files. Not sure what will happen if we remove that substitution.

[Dimitri John Ledkov (xnox)]

If we only want to replace environment variables and nothing else, we can use

#!/usr/share/dh-exec/dh-exec-subst-env

then even with DEB_BUILD_PROFILES set, the lines preserve the leading whitespace.

It would be nice for DEB_BUILD_PROFILES to preserve leading whitespace and/or only strip it for files where it matters and not others. I guess .symbols is one of the few special ones where leading whitespace is required.

[Dimitri John Ledkov (xnox)]

somehow settings --with / --without did not help, to exclude the filter commands.

[Dan Bungert (dbungert)]

Here is my proposal:
* dh-exec today desires to remove whitespace after a profile, so preserve that
* dh-exec also removes trailing whitespace, which might be OK and is current behavior, so preserve that
* remove leading whitespace for lines that don't have the profile
My perl is rusty (to say the least) but nonetheless here is my proposal. I have not given this regression test yet beyond the built-in tests.

[Dan Bungert (dbungert)]

I have run some 40-odd packages that build-depend on dh-exec thru autopkgtest with my modified dh-exec and everything seems fine, including talloc using the version prior to my workaround. If my approach is agreeable I suggest uploading.

[Dimitri John Ledkov (xnox)]

patch #7 looks good to me!

[Launchpad Janitor (janitor)]

This bug was fixed in the package dh-exec - 0.23.4

---------------
dh-exec (0.23.4) unstable; urgency=medium

  * QA upload.

  [ Daniel Bungert ]
  * Fix unintended whitespace removal (LP: #1920825) Per the tests it's
    desired to remove the whitespace following a build profile - such as
    "<A> B" resulting in "B", but leave lines with no build profile alone.

 -- Matthias Klose <email address hidden> Wed, 31 Mar 2021 15:00:30 +0200

[Sebastien Bacher (seb128)]

Unsubscribing ubuntu sponsors, there doesn't seems to be anything left to upload there

[Launchpad Janitor (janitor)]

This bug was fixed in the package ldb - 2:2.2.0-3ubuntu1

---------------
ldb (2:2.2.0-3ubuntu1) hirsute; urgency=medium

  * Fix symbols generation (LP: #1920825). On hirsute, dh-exec is showing
    different behavior based on the value of DEB_BUILD_PROFILES. This is
    causing it to sometimes generate a file with the leading whitespace
    removed, which is resulting in a bad symbols file. Set DEB_BUILD_PROFILES
    to empty gives a good symbols file.

 -- Matthias Klose <email address hidden> Tue, 30 Mar 2021 16:11:59 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package talloc - 2.3.1-2ubuntu1

---------------
talloc (2.3.1-2ubuntu1) hirsute; urgency=medium

  * Fix symbols generation (LP: #1920825). On hirsute, dh-exec is showing
    different behavior based on the value of DEB_BUILD_PROFILES. This is
    causing it to sometimes generate a file with the leading whitespace
    removed, which is resulting in a bad symbols file. Set DEB_BUILD_PROFILES
    to empty gives a good symbols file, but what does this matter?

 -- Dan Bungert <email address hidden> Mon, 29 Mar 2021 17:23:59 -0600