Debootstrap fails for Noble with base-files 13ubuntu7

Bug #2054925 reported by Åka Sikrom
34
This bug affects 3 people
Affects Status Importance Assigned to Milestone
base-files (Ubuntu)
Won't Fix
Undecided
Julian Andres Klode
Mantic
Won't Fix
Undecided
Unassigned
debootstrap (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Committed
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
Mantic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
The last couple of days, I have been unable to run a successful debootstrap for Noble Numbat.

Apparently this is caused by the addition of symlinks (/bin, /lib, /lib64 and /sbin) in base-files 13ubuntu7. According to debootstrap.log, it fails to extract said symlinks because they already exist at that point.

This can be reproduced on build hosts running Jammy Jellyfish against any up-to-date Ubuntu public archive mirror as of today.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy

# apt-cache policy debootstrap
debootstrap:
  Installed: 1.0.126+nmu1ubuntu0.5
  Candidate: 1.0.126+nmu1ubuntu0.5
  Version table:
 *** 1.0.126+nmu1ubuntu0.5 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages
        100 /var/lib/dpkg/status
     1.0.126+nmu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy/main i386 Packages

Attached shell output of two runs of debootstrap. First run uses mirror archive.ubuntu.com (which as of this report serves base-files version 13ubuntu7), and second run uses a local custom mirror (which serves base-files 13ubuntu6). First run fails, second run succeeds.

[Test plan]
Successfully for each of focal, jammy, mantic, noble
- debootstrap
- mk-sbuild
- pbuilder-dist $release create
- ubuntu-image, if we can pull debootstrap from proposed for this

as well as

- debootstrap noble --merged-usr

[Where problems could occur]
We do not anticipate any regressions as we replace the previous *) case for usrmerge for post-hirsute with a new jammy|kinetic|lunar|mantic one, and the new solution will only impact noble and onward.

That said, this is a different approach than mantic and newer take, and a different approach than Debian takes, where they moved to merging post-extraction, even in stable releases.

Revision history for this message
Åka Sikrom (akrosikam) wrote :
Steve Langasek (vorlon)
Changed in base-files (Ubuntu):
assignee: nobody → Julian Andres Klode (juliank)
Steve Langasek (vorlon)
tags: added: rls-nn-incoming
Revision history for this message
Julian Andres Klode (juliank) wrote :

Steve I'm sorry but I cannot reproduce this. I have verified this countless of times during working on this and I am also unable to verify this here:

# apt policy debootstrap
debootstrap:
  Installed: 1.0.134ubuntu1
  Candidate: 1.0.134ubuntu1
  Version table:
 *** 1.0.134ubuntu1 500
        500 https://debian.charite.de/ubuntu noble/main amd64 Packages
        500 https://debian.charite.de/ubuntu noble/main i386 Packages
        500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu noble/main i386 Packages
        100 /var/lib/dpkg/status

# rm -r /tmp/nobletest/ && sudo debootstrap --variant=minbase noble /tmp/nobletest &> debootstrap.log
# grep base-files debootstrap.log
I: Retrieving base-files 13ubuntu7
I: Validating base-files 13ubuntu7
I: Extracting base-files...
I: Unpacking base-files...
I: Configuring base-files...
# tail debootstrap.log
I: Configuring libtasn1-6:amd64...
I: Configuring libstdc++6:amd64...
I: Configuring libhogweed6:amd64...
I: Configuring libidn2-0:amd64...
I: Configuring libapt-pkg6.0:amd64...
I: Configuring libp11-kit0:amd64...
I: Configuring libgnutls30:amd64...
I: Configuring apt...
I: Configuring libc-bin...
I: Base system installed successfully.

Your debootstrap log looks fine, it extracts base-files first. At this point, /tmp/nobletest should be empty. Unless I suppose you use a old broken version of debootstrap that used to create the symlinks before extracting the packages?

debootstrap (1.0.130) unstable; urgency=medium

  [ Helmut Grohne ]
  * implement merged-/usr by post-merging. Closes: #1049898

 -- Luca Boccassi <email address hidden> Fri, 18 Aug 2023 16:02:59 +0100

Changed in base-files (Ubuntu):
status: New → Incomplete
no longer affects: base-files (Ubuntu Focal)
no longer affects: base-files (Ubuntu Jammy)
Changed in debootstrap (Ubuntu):
status: New → Fix Released
Revision history for this message
Julian Andres Klode (juliank) wrote :

Let me move this to debootstrap tasks instead, I guess we should fix the LTS debootstraps.

Changed in debootstrap (Ubuntu Focal):
status: New → Triaged
Changed in debootstrap (Ubuntu Jammy):
status: New → Triaged
Revision history for this message
Julian Andres Klode (juliank) wrote (last edit ):

I wrote instead but of course it's in addition, I think that's probably the easier outcome than trying to workaround it in the base-files preinst. Consulting with helmut

Revision history for this message
Julian Andres Klode (juliank) wrote :

I forgot we can't rely on preinst in bootstrap so backporting the debootstrap change it will have to be.

Changed in base-files (Ubuntu):
status: Incomplete → Won't Fix
Revision history for this message
Julian Andres Klode (juliank) wrote :

As a workaround, --no-merged-usr probably should work fine in old releases. It might be easier to add this to the script as a special case for > noble than backporting new logic we don't need anymore.

Revision history for this message
Åka Sikrom (akrosikam) wrote :

Apologies Julian, I completely forgot to include details of my (Jammy / 22.04) build host which would have avoided the confusion. Adding it to the report now (although I know it is sort of too late at this point).

description: updated
Revision history for this message
Åka Sikrom (akrosikam) wrote :

Also, many thanks for the workaround. I can confirm it works in my case.

description: updated
Changed in debootstrap (Ubuntu Jammy):
status: Triaged → In Progress
description: updated
description: updated
description: updated
Changed in debootstrap (Ubuntu Focal):
status: Triaged → In Progress
description: updated
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

TL;DR: The fix is not needed on Mantic.

This is just to expand a bit on the brief note in '[Where problems could occur]' ("moved to merging post-extraction"), for clarity.

The code in `scripts/gutsy` (used for noble) is different between Mantic and Jammy with regards to the order of package extraction and usr-merge setup, in first_stage_install().

The issue happens in Jammy because usr-merge setup runs _before_ extraction (thus extraction of base-files fails as the /{bin,lib,lib64,sbin} symlinks already exist):

        setup_merged_usr
        extract $required

But it doesn't happen in Mantic because usr-merge setup runs _after_ extraction (thus extraction of base-files works as the symlinks don't yet exist):

        extract $required
        merge_usr

Revision history for this message
Julian Andres Klode (juliank) wrote :

Slrry I can make this clearer too by adding a mantic task and marking it as fix released.

Changed in debootstrap (Ubuntu Mantic):
status: New → Fix Released
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

I confirmed that the resulting target dir of debootstrap in Jammy (with patch/MERGED_USR=no) and Mantic is the same, with a comparison of `ls -lR` (normalized for Month/Day/Time fields).

...

Test command for Jammy (patched) and Mantic:

# debootstrap --arch=amd64 --variant=minbase --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg noble /tmp/nobletest http://archive.ubuntu.com/ubuntu

Dir listing for Jammy (patched) and Mantic:

deboostrap-jammy # ls -lR /tmp/nobletest/ > list.jammy
deboostrap-mantic # ls -lR /tmp/nobletest/ > list.mantic

Normalization and comparison:

$ sed -i 's/[A-Z][a-z][a-z] .[0-9] [0-9][0-9]:[0-9][0-9]/DATETIME/' list.jammy list.mantic

$ diff -U0 list.jammy list.mantic
--- list.jammy 2024-02-26 19:39:49.102864808 -0300
+++ list.mantic 2024-02-26 19:39:49.114864569 -0300
@@ -76 +76 @@
--rw-r--r-- 1 root root 18 DATETIME hostname
+-rw-r--r-- 1 root root 19 DATETIME hostname
@@ -7093 +7093 @@
--rw-r--r-- 1 root root 60238 DATETIME bootstrap.log
+-rw-r--r-- 1 root root 60045 DATETIME bootstrap.log

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote : Please test proposed package

Hello Åka, or anyone else affected,

Accepted debootstrap into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debootstrap/1.0.126+nmu1ubuntu0.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in debootstrap (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (debootstrap/1.0.126+nmu1ubuntu0.7)

All autopkgtests for the newly accepted debootstrap (1.0.126+nmu1ubuntu0.7) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

debuerreotype/0.14-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#debootstrap

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Steve Langasek (vorlon)
description: updated
Revision history for this message
Åka Sikrom (akrosikam) wrote :

debootstrap 1.0.126+nmu1ubuntu0.7 tested and verified successfully on my amd64 Jammy install. Thanks!

Now, with the above ppc64el regression warning from ubuntu-sru-bot in mind, I will refrain from changing the tag to verification-done until devs address it and/or advise.

Revision history for this message
Paul Mars (upils) wrote :

Building an noble image with ubuntu-image, using debootstrap 1.0.126+nmu1ubuntu0.7, worked (on a amd64 Jammy).

Revision history for this message
Julian Andres Klode (juliank) wrote :

The regression in debuerreotype is expected due to the included change from the previous SRU (LP: #1990856), packages are now extracted in a different order for the test case, hence some time stamps shifted.

Revision history for this message
Julian Andres Klode (juliank) wrote (last edit ):

On

root@known-seal:~# apt policy debootstrap
debootstrap:
  Installed: 1.0.126+nmu1ubuntu0.7
  Candidate: 1.0.126+nmu1ubuntu0.7
  Version table:
 *** 1.0.126+nmu1ubuntu0.7 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.126+nmu1ubuntu0.5 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.0.126+nmu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

I ran
- debootstrap {focal,jammy,mantic,noble}
- mk-sbuild {focal,jammy,mantic,noble}
- pbuilder-dist {focal,jammy,mantic,noble} create

and with the ubuntu-image run from upils I think we can call that verified.

tags: added: verification-done verification-done-jammy
removed: rls-nn-incoming verification-needed verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debootstrap - 1.0.126+nmu1ubuntu0.7

---------------
debootstrap (1.0.126+nmu1ubuntu0.7) jammy; urgency=medium

  * Disable usrmerge code on noble+ as they ship symlinks in base-files,
    and the code is incompatible (and no longer needed) (LP: #2054925)

debootstrap (1.0.126+nmu1ubuntu0.6) jammy; urgency=medium

  * Support Packages files which are not ordered alphabetically by Package
    field, by backporting upstream commit
    86ca8bcc736ceba53ad4a7d9b10b4c2ab65d739d (LP: #1990856)

 -- Julian Andres Klode <email address hidden> Mon, 26 Feb 2024 10:41:03 +0100

Changed in debootstrap (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for debootstrap has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The changes for Focal had a syntax error (double ';;' lines, not present in Jammy) which causes debootstrap to fail early and silently, even for a previously working series.

Original:

 $ sudo debootstrap --variant=minbase focal /tmp/debootstrap.focal http://br.archive.ubuntu.com/ubuntu
 ...
 I: Base system installed successfully.
 $ echo $?
 0

Patched:

 $ sudo debootstrap --variant=minbase focal /tmp/debootstrap.focal http://br.archive.ubuntu.com/ubuntu
 $ echo $?
 2

Addressed this issue; verified the diff only changed in that regard; and re-uploaded.

 $ diff -U0 debootstrap_1.0.118ubuntu1.12_1.0.118ubuntu1.13.diff debdiff | grep -v '^@@' | sort | uniq -c | sort
       1 +++ debdiff 2024-03-04 21:01:43.500570118 +0000
       1 --- debootstrap_1.0.118ubuntu1.12_1.0.118ubuntu1.13.diff 2024-02-26 15:24:26.000000000 +0000
      32 +@@ -78,7 +78,14 @@
      32 -+ ;;
      32 -@@ -78,7 +78,15 @@

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote (last edit ):

Testing with the change applied:

 $ sudo debootstrap --variant=minbase noble /tmp/debootstrap.noble http://br.archive.ubuntu.com/ubuntu
 ...
 I: Base system installed successfully.
 $ echo $?
 0

 $ ls -l /tmp/debootstrap.noble/ | grep usr/
 lrwxrwxrwx 1 root root 7 Feb 22 14:39 bin -> usr/bin
 lrwxrwxrwx 1 root root 7 Feb 22 14:39 lib -> usr/lib
 lrwxrwxrwx 1 root root 9 Feb 22 14:39 lib64 -> usr/lib64
 lrwxrwxrwx 1 root root 8 Feb 22 14:39 sbin -> usr/sbin

Changed in debootstrap (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
removed: verification-done
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote : Please test proposed package

Hello Åka, or anyone else affected,

Accepted debootstrap into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debootstrap/1.0.118ubuntu1.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Apologies, I saw the same issue locally fixed it but must have forgotten to build a new .dsc:/

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

> Apologies, I saw the same issue locally fixed it but must have forgotten to build a new .dsc:/

No worries; thanks for mentioning!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (debootstrap/1.0.118ubuntu1.13)

All autopkgtests for the newly accepted debootstrap (1.0.118ubuntu1.13) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

debuerreotype/0.10-1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#debootstrap

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Brian Murray (brian-murray) wrote :

Ubuntu 23.10 (Mantic Minotaur) has reached end of life, so this bug will not be fixed for that specific release.

Changed in base-files (Ubuntu Mantic):
status: New → Won't Fix
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [debootstrap/focal] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for focal for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.