apport-retrace crashed with subprocess.CalledProcessError in check_call(): Command '['dpkg', '-x', '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb', '/tmp/apport_sandbox_zj9wto2z']' returned non-zero exit status 2.

[Brian Murray]

I'm finding it impossible to retrace any crashes with the version of apport packaged in noble or with upstream commit id adfe3698731b69b0f19ca36ecca90d8e96001250.

The retracing process ends with the following lines:

Extracting downloaded debs...
tar: ./lib: Cannot create symlink to ‘usr/lib’: File exists
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2
Traceback (most recent call last):
  File "/usr/bin/apport-retrace", line 696, in <module>
    main()
  File "/usr/bin/apport-retrace", line 465, in main
    sandbox, cache, outdated_msg = apport.sandboxutils.make_sandbox(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/apport/sandboxutils.py", line 227, in make_sandbox
    outdated_msg = packaging.install_packages(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/apport/packaging_impl/apt_dpkg.py", line 1350, in install_packages
    subprocess.check_call(["dpkg", "-x", i.destfile, rootdir])
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['dpkg', '-x', '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb', '/tmp/apport_sandbox_zj9wto2z']' returned non-zero exit status 2.

I've tried cleaning the cache, /srv/vms/apport-retrace/Ubuntu 24.04/, but the retracing process continues to fail. The crashes I'm trying to retrace have been uploaded to Launchpad and still have the coredumps attached but I'd be happy to subscribe an apport developer to them.

ProblemType: Crash
DistroRelease: Ubuntu 24.04
Package: apport-retrace 2.28.1-0ubuntu3
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
NonfreeKernelModules: nvidia_modeset wl nvidia zfs
ApportLog:

ApportVersion: 2.28.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Fri May 24 16:03:45 2024
ExecutablePath: /usr/bin/apport-retrace
ExecutableTimestamp: 1713871810
InstallationDate: Installed on 2018-08-10 (2114 days ago)
InstallationMedia: Ubuntu-Server 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
InterpreterPath: /usr/bin/python3.12
JournalErrors: -- No entries --
PackageArchitecture: all
ProcCmdline: /usr/bin/python3 /usr/bin/apport-retrace -v -S /home/bdmurray/source-trees/daisy/watchtower-archive/ -C /srv/vms/apport-retrace --verbose --gdb-sandbox --auth /home/bdmurray/.launchpadlib/retracer-credentials-lpnet --confirm 2062390
ProcCwd: /home/bdmurray/source-trees/apport/apport
Python3Details: /usr/bin/python3.12, Python 3.12.3, python3-minimal, 3.12.3-0ubuntu1
PythonArgs: ['/usr/bin/apport-retrace', '-v', '-S', '/home/bdmurray/source-trees/daisy/watchtower-archive/', '-C', '/srv/vms/apport-retrace', '--verbose', '--gdb-sandbox', '--auth', '/home/bdmurray/.launchpadlib/retracer-credentials-lpnet', '--confirm', '2062390']
PythonDetails: N/A
SourcePackage: apport
Title: apport-retrace crashed with subprocess.CalledProcessError in check_call(): Command '['dpkg', '-x', '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb', '/tmp/apport_sandbox_zj9wto2z']' returned non-zero exit status 2.
UpgradeStatus: Upgraded to noble on 2024-04-16 (38 days ago)
UserGroups: adm cdrom dip kvm libvirt lpadmin lxd plugdev sambashare sbuild sudo

[Benjamin Drung (bdrung)]

Can you run the dpkg command manually to see the failure message?

```
dpkg -x '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb' /tmp/apport_sandbox_zj9wto2z
```

[Benjamin Drung (bdrung)]

The dpkg output was there before the stack trace:

```
tar: ./lib: Cannot create symlink to ‘usr/lib’: File exists
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2
```

So this is related to usrmerge.

[Sebastien Bacher (seb128)]

I'm tagging it rls-nn-incoming because it might be the reason why we don't have errors.ubuntu.com working normally for Noble

[Benjamin Drung (bdrung)]

Simple extraction works:

```
pull-lp-debs base-files 13ubuntu9
mkdir sandbox
dpkg -x base-files_13ubuntu9_amd64.deb sandbox
```

Let's check the Apport code.

[Benjamin Drung (bdrung)]

I tried to construct a test case locally since I do not have access to bug #2062390. The failure message was different, but also usrmerge related. Here is the test case:

```
$ divide-by-zero
$ mkdir -p "$PWD/sandbox-config/Ubuntu 24.04"
$ echo noble > "$PWD/sandbox-config/Ubuntu 24.04/codename"
$ cat > "$PWD/sandbox-config/Ubuntu 24.04/sources.list" <<EOF
deb http://de.archive.ubuntu.com/ubuntu/ noble main universe
deb-src http://de.archive.ubuntu.com/ubuntu/ noble main universe
EOF
$ apport-retrace -v -S "$PWD/sandbox-config" --gdb-sandbox -C "$PWD/sandbox-cache" /var/crash/_usr_bin_divide-by-zero.1000.crash
[...]
Extracting downloaded debs...
ERROR: [Errno 2] No such file or directory: '/tmp/apport_sandbox_8x704zqr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2'
```

Proposed fix for this test case: https://github.com/canonical/apport/pull/331

Can you test that possible fix if it is also the solution for your failures?

[Brian Murray (brian-murray)]

This was also necessary to fix the retracing failure although I imagine it only works for releases like Noble and Oracular.

index ae094606..052eb211 100644
--- a/apport/report.py
+++ b/apport/report.py
@@ -1922,7 +1922,7 @@ class Report(problem_report.ProblemReport):
                     "GCONV_PATH": f"{gdb_sandbox}/usr/lib/{native_multiarch}/gconv",
                 }
                 command.insert(
- 0, f"{gdb_sandbox}/lib/{native_multiarch}/ld-linux-x86-64.so.2"
+ 0, f"{gdb_sandbox}/usr/lib/{native_multiarch}/ld-linux-x86-64.so.2"
                 )
                 command += ["--ex", f"set data-directory {gdb_sandbox}/usr/share/gdb"]
             if not os.path.exists(sandbox + executable):

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matheus Carvalho Raimundo (mcarvalhor)]

(moved back to "New" for now, as I mistakenly marked it affected me)

[Benjamin Drung (bdrung)]

Maybe related fix finding source for /lib/x86_64-linux-gnu/libc.so.6: https://github.com/canonical/apport/pull/336

Brian, is adding `ld-linux-x86-64.so.2` to the gdb command really needed? My apport-retrace test cases succeed when I am removing the insert.

[Brian Murray (brian-murray)]

Yes it is really needed, apport-retrace with a gdb-sandbox fails without it.

 $ PYTHONPATH=/home/bdmurray/source-trees/apport/trunk ~/source-trees/apport/trunk/bin/apport-retrace -v -S ~/source-trees/daisy/watchtower-archive/ -C /srv/vms/apport-retrace --verbose --gdb-sandbox --stdout test-crashes/mantic/amd64/_usr_bin_cat.2001.crash
Hit http://192.168.10.7/ubuntu mantic InRelease
Hit http://192.168.10.7/ubuntu mantic-updates InRelease
Hit http://192.168.10.7/ubuntu mantic-security InRelease
Get:1 http://ddebs.ubuntu.com mantic InRelease [41.3 kB]
Get:2 http://ddebs.ubuntu.com mantic-updates InRelease [41.4 kB]
Fetched 82.7 kB in 0s (0 B/s)
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
dynamically loaded /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 needs package libc6, queueing
dynamically loaded /usr/lib/x86_64-linux-gnu/libc.so.6 needs package libc6, queueing
Installing extra package coreutils to get ExecutablePath
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
ERROR: Error: command ['/tmp/apport_sandbox_4hl714lr/usr/bin/gdb', '--ex', 'set debug-file-directory /tmp/apport_sandbox_4hl714lr/usr/lib/debug', '--ex', 'set solib-absolute-prefix /tmp/apport_sandbox_4hl714lr', '--ex', 'add-auto-load-safe-path /tmp/apport_sandbox_4hl714lr', '--ex', 'set solib-search-path /tmp/apport_sandbox_4hl714lr/lib/x86_64-linux-gnu:/tmp/apport_sandbox_4hl714lr/usr/lib/x86_64-linux-gnu', '--ex', 'set data-directory /tmp/apport_sandbox_4hl714lr/usr/share/gdb', '--ex', 'file "/tmp/apport_sandbox_4hl714lr/bin/cat"', '--ex', 'core-file /tmp/apport_core_pxcsb4wf', '--batch', '--ex', 'set backtrace limit 2000', '--ex', 'p -99', '--ex', 'info registers', '--ex', 'p -99', '--ex', 'x/16i $pc', '--ex', 'p -99', '--ex', 'bt full', '--ex', 'p -99', '--ex', 'thread apply all bt full', '--ex', 'p -99', '--ex', 'print __abort_msg->msg', '--ex', 'p -99', '--ex', 'print __glib_assert_msg', '--ex', 'p -99', '--ex', 'print (char*) __nih_abort_msg', '--ex', 'p -99'] failed with exit code -11:

[Benjamin Drung (bdrung)]

Thanks. I can reproduce this failure. The host release must be different to the sandbox one.

[Benjamin Drung (bdrung)]

Proposed fix: https://github.com/canonical/apport/pull/341

It adds four system tests that cover the host and jammy release. The CI runs those tests on jammy, mantic, and noble.