FTBFS on armhf in Kinetic

[Christian Ehrhardt ]

https://launchpad.net/ubuntu/+source/xen/4.16.1-1

[Christian Ehrhardt  (paelzer)]

he haskell/pandoc things (bug 1982460) are out of the way.
They blocked more and were temporarily removed.

But armhf still has an issue:

debian/installsharedlibs
x dh_install -plibxencall1 usr/lib/*/libxencall.so.*
Failed to copy 'etc/bash_completion.d/xl': No such file or directory at /usr/share/dh-exec/dh-exec-install-rename line 68, <> line 3.
dh_install: error: debian/xen-utils-common.install (executable config) returned exit code 127

=> https://launchpadlibrarian.net/614734467/buildlog_ubuntu-kinetic-armhf.xen_4.16.1-1_BUILDING.txt.gz

in debian/* that is at:
debian/xen-utils-common.install:8:etc/bash_completion.d/xl => usr/share/bash-completion/completions/xl

Interstingly also at (form the past)
debian/not-installed:16:etc/bash_completion.d/xl

But either way this shouldn't be arch specific and works here
https://buildd.debian.org/status/fetch.php?pkg=xen&arch=armhf&ver=4.16.1-1&stamp=1652138474&raw=0

All log occurrences of bash_completion in good/bad logs are the same until the failure.
Rebuilding to see if that was a flaky build - since it is armhf only debugging will need an amrhf system/vm.

[Christian Ehrhardt  (paelzer)]

x86
/<<PKGBUILDDIR>>/tools/xl/../../tools/cross-install -m0644 -p bash-completion /<<PKGBUILDDIR>>/dist/install/etc/bash_completion.d/xl
/<<PKGBUILDDIR>>/tools/xl/../../tools/cross-install -m0644 -p bash-completion /<<PKGBUILDDIR>>/debian/tmp/etc/bash_completion.d/xl

arm64
/<<PKGBUILDDIR>>/tools/xl/../../tools/cross-install -m0644 -p bash-completion /<<PKGBUILDDIR>>/dist/install/etc/bash_completion.d/xl
/<<PKGBUILDDIR>>/tools/xl/../../tools/cross-install -m0644 -p bash-completion /<<PKGBUILDDIR>>/debian/tmp/etc/bash_completion.d/xl

armhf
/<<PKGBUILDDIR>>/tools/xl/../../tools/cross-install -m0644 -p bash-completion /<<PKGBUILDDIR>>/dist/install/etc/bash_completion.d/xl
/<<PKGBUILDDIR>>/tools/xl/../../tools/cross-install -m0644 -p bash-completion /<<PKGBUILDDIR>>/debian/tmp/etc/bash_completion.d/xl

The same steps are present in all builds, but then in armhf only:
Failed to copy 'etc/bash_completion.d/xl': No such file or directory at /usr/share/dh-exec/dh-exec-install-rename line 68, <> line 3.

[Christian Ehrhardt  (paelzer)]

I've inserted `find -name xl` in various places and ran a debug build.

In good and bad case after build and before dh_install it looks like:

8065 find -name xl
8066 ./tools/xl
8067 ./tools/xl/xl
8068 ./tools/ocaml/libs/xl
8069 ./debian/tmp/usr/lib/xen-4.16/bin/xl
8070 ./debian/tmp/usr/sbin/xl
8071 ./debian/tmp/etc/bash_completion.d/xl
8072 ./.pc/0019-tools-xl-bash-completion-also-complete-xen.patch/tools/xl
8073 ./dist/install/usr/sbin/xl
8074 ./dist/install/etc/bash_completion.d/xl

Line 8071 is the one that should be picke dup, and thereby (since it is available in both cases) might mean
a) it isn't missing the source of this copy but the target dir?
b) dh_install behaves different on that platform

I've added some potential fixes that came to my mind to d/rules/.install/.dirs files.
Test builds in https://launchpad.net/~paelzer/+archive/ubuntu/lp-1982460-xen-ftbfs-armhf/+packages - if one of them works I'll clean up and propose something - otherwise back to the drawing board.

[Christian Ehrhardt  (paelzer)]

Not nice :-/

With the fixes now arm64 fails the same way
https://launchpadlibrarian.net/616452230/buildlog_ubuntu-kinetic-arm64.xen_4.16.1-1ubuntu1~kineticppa2_BUILDING.txt.gz

But armhf works - need to apply them individually and enable more arches

[Christian Ehrhardt  (paelzer)]

It is odd, these all fail with the same error all the time.
But affected architectures changed - what seems like over time.

original
 => armhf fails
d/xen-utils-common.dirs: prep dir that is used for bash-completion
 => both arm builds fail
d/xen-utils-common.install: add leading/trailing slashes following the dh-exec examples
 => both arm builds fail
back to the original
 => Failing both arm builds now

Might there be something else that landed on armhf first and then arm64 later?
With that in mind I rechecked logs and found this:

All working arm builds have:
- dh-exec 0.23.4build2
All failing arm builds have
- dh-exec - 0.26

I do not know yet why this lands at so different times in the different
architectures.

In addition sadly it isn't so easy - as the working amd64 has dh-exec 0.26 as well.

[Christian Ehrhardt  (paelzer)]

Now that it is at least also broken on arm64 repro is easier.
Comparing Jammy vs Kinetic on arm64 systems ... working in both :-/

This is trying to be a heisenbug ...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

All I try locally (d/rules, debuild, sbuild) works fine, and in launchpad it continues to fail on arm64.

Trying a armhf container there now ... also both worked jammy and kinetic.

For now I'm out of ideas what it could be, why it would be arch dependent and how to debug :-/

[Luís Infante da Câmara (luis220413)]

Christian Ehrhardt, is your local arm64/armhf testing native or under emulation?

[Luís Infante da Câmara (luis220413)]

I am re-testing now in my PPA (https://launchpad.net/~luis220413/+archive/ubuntu/test-builds/+packages) with the proposed pocket enabled.

[Luís Infante da Câmara (luis220413)]

The build fails in amd64 due to new warnings introduced in GCC 12, that are made into errors by the -Werror flag.

[Luís Infante da Câmara (luis220413)]

Sorry, I uploaded a version with additional upstream changes. I will upload 4.16.1-1 now.

[Luís Infante da Câmara (luis220413)]

The build of this version on amd64 fails due to new warnings introduced in GCC 12, that are made into errors by the -Werror flag.

The builds of this version on arm64 and armhf fails due to the same issue.

[Christian Ehrhardt  (paelzer)]

Hi, my testing was on native arm64 (and armhf LXD on those).

Just FYI - I'm soon unavailable a lot for sprint and PTO.
I'd appreciate+++ it if you could continue to have a look to move this over the finishing line.

[Luís Infante da Câmara (luis220413)]

The version of dh-exec in the kinetic-proposed pocket is now 0.27. With this version, the build succeeds on arm64 and armhf, but fails on amd64 due to the issue in comment #15. The solution is to not pass the -Werror flag to gcc or to disable the warning with #pragma GCC warning disable("array-bounds")