apport-retrace crashed with subprocess.CalledProcessError in check_call(): Command '['dpkg', '-x', '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb', '/tmp/apport_sandbox_zj9wto2z']' returned non-zero exit status 2.

[Brian Murray]

[ Impact ]

I'm finding it impossible to retrace any crashes with the version of apport packaged in noble or with upstream commit id adfe3698731b69b0f19ca36ecca90d8e96001250.

The retracing process ends with the following lines:

Extracting downloaded debs...
tar: ./lib: Cannot create symlink to ‘usr/lib’: File exists
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2
Traceback (most recent call last):
  File "/usr/bin/apport-retrace", line 696, in <module>
    main()
  File "/usr/bin/apport-retrace", line 465, in main
    sandbox, cache, outdated_msg = apport.sandboxutils.make_sandbox(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/apport/sandboxutils.py", line 227, in make_sandbox
    outdated_msg = packaging.install_packages(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/apport/packaging_impl/apt_dpkg.py", line 1350, in install_packages
    subprocess.check_call(["dpkg", "-x", i.destfile, rootdir])
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['dpkg', '-x', '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb', '/tmp/apport_sandbox_zj9wto2z']' returned non-zero exit status 2.

I've tried cleaning the cache, /srv/vms/apport-retrace/Ubuntu 24.04/, but the retracing process continues to fail. The crashes I'm trying to retrace have been uploaded to Launchpad and still have the coredumps attached but I'd be happy to subscribe an apport developer to them.

[ Test Plan ]

The fix for this bug adds four new system test cases. These system test cases are run in the autopkgtest. Check that those four test cases succeed on amd64 on autopkgtest.

[ Where problems could occur ]

The fix touches apport-retrace. This binary is not installed by default, but is used by us to retrace users reports and by users to retrace their own crashes. This fix might reveal new issues with apport-retrace and it will hopefully help retracing more bugs on Launchpad. This should lead to more bug reports.

[ Other Info ]

ProblemType: Crash
DistroRelease: Ubuntu 24.04
Package: apport-retrace 2.28.1-0ubuntu3
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
NonfreeKernelModules: nvidia_modeset wl nvidia zfs
ApportLog:

ApportVersion: 2.28.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Fri May 24 16:03:45 2024
ExecutablePath: /usr/bin/apport-retrace
ExecutableTimestamp: 1713871810
InstallationDate: Installed on 2018-08-10 (2114 days ago)
InstallationMedia: Ubuntu-Server 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
InterpreterPath: /usr/bin/python3.12
JournalErrors: -- No entries --
PackageArchitecture: all
ProcCmdline: /usr/bin/python3 /usr/bin/apport-retrace -v -S /home/bdmurray/source-trees/daisy/watchtower-archive/ -C /srv/vms/apport-retrace --verbose --gdb-sandbox --auth /home/bdmurray/.launchpadlib/retracer-credentials-lpnet --confirm 2062390
ProcCwd: /home/bdmurray/source-trees/apport/apport
Python3Details: /usr/bin/python3.12, Python 3.12.3, python3-minimal, 3.12.3-0ubuntu1
PythonArgs: ['/usr/bin/apport-retrace', '-v', '-S', '/home/bdmurray/source-trees/daisy/watchtower-archive/', '-C', '/srv/vms/apport-retrace', '--verbose', '--gdb-sandbox', '--auth', '/home/bdmurray/.launchpadlib/retracer-credentials-lpnet', '--confirm', '2062390']
PythonDetails: N/A
SourcePackage: apport
Title: apport-retrace crashed with subprocess.CalledProcessError in check_call(): Command '['dpkg', '-x', '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb', '/tmp/apport_sandbox_zj9wto2z']' returned non-zero exit status 2.
UpgradeStatus: Upgraded to noble on 2024-04-16 (38 days ago)
UserGroups: adm cdrom dip kvm libvirt lpadmin lxd plugdev sambashare sbuild sudo

[Benjamin Drung (bdrung)]

Can you run the dpkg command manually to see the failure message?

```
dpkg -x '/srv/vms/apport-retrace/Ubuntu 24.04/apt/var/cache/apt/archives//base-files_13ubuntu9_amd64.deb' /tmp/apport_sandbox_zj9wto2z
```

[Benjamin Drung (bdrung)]

The dpkg output was there before the stack trace:

```
tar: ./lib: Cannot create symlink to ‘usr/lib’: File exists
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2
```

So this is related to usrmerge.

[Sebastien Bacher (seb128)]

I'm tagging it rls-nn-incoming because it might be the reason why we don't have errors.ubuntu.com working normally for Noble

[Benjamin Drung (bdrung)]

Simple extraction works:

```
pull-lp-debs base-files 13ubuntu9
mkdir sandbox
dpkg -x base-files_13ubuntu9_amd64.deb sandbox
```

Let's check the Apport code.

[Benjamin Drung (bdrung)]

I tried to construct a test case locally since I do not have access to bug #2062390. The failure message was different, but also usrmerge related. Here is the test case:

```
$ divide-by-zero
$ mkdir -p "$PWD/sandbox-config/Ubuntu 24.04"
$ echo noble > "$PWD/sandbox-config/Ubuntu 24.04/codename"
$ cat > "$PWD/sandbox-config/Ubuntu 24.04/sources.list" <<EOF
deb http://de.archive.ubuntu.com/ubuntu/ noble main universe
deb-src http://de.archive.ubuntu.com/ubuntu/ noble main universe
EOF
$ apport-retrace -v -S "$PWD/sandbox-config" --gdb-sandbox -C "$PWD/sandbox-cache" /var/crash/_usr_bin_divide-by-zero.1000.crash
[...]
Extracting downloaded debs...
ERROR: [Errno 2] No such file or directory: '/tmp/apport_sandbox_8x704zqr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2'
```

Proposed fix for this test case: https://github.com/canonical/apport/pull/331

Can you test that possible fix if it is also the solution for your failures?

[Brian Murray (brian-murray)]

This was also necessary to fix the retracing failure although I imagine it only works for releases like Noble and Oracular.

index ae094606..052eb211 100644
--- a/apport/report.py
+++ b/apport/report.py
@@ -1922,7 +1922,7 @@ class Report(problem_report.ProblemReport):
                     "GCONV_PATH": f"{gdb_sandbox}/usr/lib/{native_multiarch}/gconv",
                 }
                 command.insert(
- 0, f"{gdb_sandbox}/lib/{native_multiarch}/ld-linux-x86-64.so.2"
+ 0, f"{gdb_sandbox}/usr/lib/{native_multiarch}/ld-linux-x86-64.so.2"
                 )
                 command += ["--ex", f"set data-directory {gdb_sandbox}/usr/share/gdb"]
             if not os.path.exists(sandbox + executable):

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matheus Carvalho (mcarvalhor)]

(moved back to "New" for now, as I mistakenly marked it affected me)

[Benjamin Drung (bdrung)]

Maybe related fix finding source for /lib/x86_64-linux-gnu/libc.so.6: https://github.com/canonical/apport/pull/336

Brian, is adding `ld-linux-x86-64.so.2` to the gdb command really needed? My apport-retrace test cases succeed when I am removing the insert.

[Brian Murray (brian-murray)]

Yes it is really needed, apport-retrace with a gdb-sandbox fails without it.

 $ PYTHONPATH=/home/bdmurray/source-trees/apport/trunk ~/source-trees/apport/trunk/bin/apport-retrace -v -S ~/source-trees/daisy/watchtower-archive/ -C /srv/vms/apport-retrace --verbose --gdb-sandbox --stdout test-crashes/mantic/amd64/_usr_bin_cat.2001.crash
Hit http://192.168.10.7/ubuntu mantic InRelease
Hit http://192.168.10.7/ubuntu mantic-updates InRelease
Hit http://192.168.10.7/ubuntu mantic-security InRelease
Get:1 http://ddebs.ubuntu.com mantic InRelease [41.3 kB]
Get:2 http://ddebs.ubuntu.com mantic-updates InRelease [41.4 kB]
Fetched 82.7 kB in 0s (0 B/s)
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
dynamically loaded /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 needs package libc6, queueing
dynamically loaded /usr/lib/x86_64-linux-gnu/libc.so.6 needs package libc6, queueing
Installing extra package coreutils to get ExecutablePath
Fetched 0 B in 0s (0 B/s)
Extracting downloaded debs...
ERROR: Error: command ['/tmp/apport_sandbox_4hl714lr/usr/bin/gdb', '--ex', 'set debug-file-directory /tmp/apport_sandbox_4hl714lr/usr/lib/debug', '--ex', 'set solib-absolute-prefix /tmp/apport_sandbox_4hl714lr', '--ex', 'add-auto-load-safe-path /tmp/apport_sandbox_4hl714lr', '--ex', 'set solib-search-path /tmp/apport_sandbox_4hl714lr/lib/x86_64-linux-gnu:/tmp/apport_sandbox_4hl714lr/usr/lib/x86_64-linux-gnu', '--ex', 'set data-directory /tmp/apport_sandbox_4hl714lr/usr/share/gdb', '--ex', 'file "/tmp/apport_sandbox_4hl714lr/bin/cat"', '--ex', 'core-file /tmp/apport_core_pxcsb4wf', '--batch', '--ex', 'set backtrace limit 2000', '--ex', 'p -99', '--ex', 'info registers', '--ex', 'p -99', '--ex', 'x/16i $pc', '--ex', 'p -99', '--ex', 'bt full', '--ex', 'p -99', '--ex', 'thread apply all bt full', '--ex', 'p -99', '--ex', 'print __abort_msg->msg', '--ex', 'p -99', '--ex', 'print __glib_assert_msg', '--ex', 'p -99', '--ex', 'print (char*) __nih_abort_msg', '--ex', 'p -99'] failed with exit code -11:

[Benjamin Drung (bdrung)]

Thanks. I can reproduce this failure. The host release must be different to the sandbox one.

[Benjamin Drung (bdrung)]

Proposed fix: https://github.com/canonical/apport/pull/341

It adds four system tests that cover the host and jammy release. The CI runs those tests on jammy, mantic, and noble.

[Timo Aaltonen (tjaalton)]

Hello Brian, or anyone else affected,

Accepted apport into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.28.1-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apport (2.28.1-0ubuntu3.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.28.1-0ubuntu3.1 (arm64)
apport/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#apport

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.29.0-0ubuntu1

---------------
apport (2.29.0-0ubuntu1) oracular; urgency=medium

  [ Benjamin Drung ]
  * New upstream release:
    - Ignore container crash reports from systemd-coredump (LP: #2063349)
    - apport: do not modify permission of existing /var/crash or
      create /var/crash with setgid permission (mode 3777) (LP: #2066995)
    - Fix retracing crashes on Ubuntu 24.04 (noble) (LP: #2067120)
  * ubuntu: fix possibly using dpkg_log_without_error before assignment
  * Drop python3-requests dependency
  * autopkgtest: add chaos-marmosets for system-tests
  * apport.postinst: Create /var/crash with mode 3777 (LP: #2066995)
  * Build-depend on mypy and python3-typeshed for type checking
  * Build-depend on systemd-dev to determine systemd/udev paths

  [ Chris Peterson ]
  * d/package-hooks/subiquity.py: update information collection (LP: #2067775)
    - Add information about snap update/refresh state
    - Attach logs and change key names to align with the information attached
      by subiquity's self-collection
    - Attach hardware information
    - Ensure bug reports are always private since there may be private
      information in the logs
    - Don't require sudo for logs collection
    - Add journal logs
    - Ensure snap revision is included in both deb and snap collection style

 -- Benjamin Drung <email address hidden> Fri, 14 Jun 2024 10:18:31 +0200

[Benjamin Drung (bdrung)]

The autopkgtest for apport/2.28.1-0ubuntu3.1 on amd64 succeeded: https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/a/apport/20240731_122144_72f7b@/log.gz

The four new tests/system/test_apport_retrace.py tests passed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.28.1-0ubuntu3.1

---------------
apport (2.28.1-0ubuntu3.1) noble; urgency=medium

  [ Chris Peterson ]
  * d/package-hooks/subiquity.py: update information collection (LP: #2067775)
    - Add information about snap update/refresh state
    - Attach logs and change key names to align with the information
      attached by subiquity's self-collection
    - Attach hardware information
    - Ensure bug reports are always private since there may be private
      information in the logs
    - Don't require sudo for logs collection
    - Add journal logs
    - Ensure snap revision is included in both deb and snap collection
      style
    - Handle version information when permission elevation fails.

  [ Benjamin Drung ]
  * apport: do not modify permission of existing /var/crash or
    create /var/crash with setgid permission (mode 3777) (LP: #2066995)
  * apport.postinst: Create /var/crash with mode 3777 (LP: #2066995)
  * whoopsie-upload-all: exit with 0 if whoopsie is disabled (LP: #2069360)
  * Fix retracing crashes on Ubuntu 24.04 (noble) and add chaos-marmosets as
    dependency for the newly added system-tests (LP: #2067120)

  [ Sebastien Bacher ]
  * Fix some issues in the hook for ubuntu-desktop-bootstrap (LP: #2072751)

 -- Benjamin Drung <email address hidden> Mon, 22 Jul 2024 16:59:07 +0200

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for apport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.