please add cnf support to debmirror

Bug #1821251 reported by Jamie Strandboge
92
This bug affects 16 people
Affects Status Importance Assigned to Milestone
debmirror (Ubuntu)
Medium
Unassigned
Focal
Undecided
Unassigned
Groovy
Undecided
Unassigned

Bug Description

[Impact]

 * Starting with Ubuntu Focal (20.04), the APT client expects APT sources to provide command-not-found (cnf) metadata files.

 * This package provides package mirrors, however does not support mirroring this cnf metadata up until upstream version 1:2.32, which is later than the version in Focal (20.04) and Groovy (20.10).

 * This upload cherry-picks upstream commits 292e9e9c, 39055366 and 9606535d, in order to add the cnf metadata support.

[Test Case]

 * Create a mirror of e.g. `bionic main` and `focal main`: debmirror -d bionic,focal -s main --passive --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror

 * Serve the root directory of /opt/debmirror as webroot.

 * Point APT `sources.list` at this webroot: deb http://mirror.example.com/ubuntu/ focal main

 * Attempt to run `apt update` on Focal (20.04) client to see that it fails to obtain the cnf metadata.

 * Optionally run `apt update` on Bionic (18.04) client to see that it works fine without the cnf metadata.

[Where problems could occur]

 * New cnf files are expected to be created and maintained by the debmirror package. The regression risk for incorrectly downloading these files does not serve more harm than not downloading the files whatsoever (current state of 20.04 release), as the APT client does not use the mirror in either case.

 * The change as uploaded was released in upstream as part of 1:2.32, tagged 23 Mar 2019, and is part of Debian stable. This serves as evidence of it having had testing carried out over time.

[Other Info]

Disco's apt started downloading cnf (command-not-found) files but debmirror does not download these files, so when pointing a disco machine at a mirrored archive, apt-get update errors. Eg, we need to mirror the Commands-*.xz files: http://archive.ubuntu.com/ubuntu/dists/disco/main/cnf/

The fix is straightforward since the files are quite similar to dep11, and debmirror already handles dep11. Here is a patch that simply copy/pastes the various dep11 code. It is lightly tested and appears to do the job.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :
tags: added: patch
Revision history for this message
Iain Lane (laney) wrote :

Thanks for this Jamie. I just had the same problem with my local mirror here at home.

I had to make some small changes to your patch for it to work here. I guess you were using an un-split (archive vs. ports) mirror to test? The Release files advertise Commands-* for all architectures even though they aren't actually available to download, so we need to limit to the arches requested.

This change uses perl smartmatching, which is apparently still(!) experimental. Colin might not like that. :-)

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Yes Iain, I don't mirror ports. Thanks for the updated patch. :)

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in debmirror (Ubuntu):
status: New → Confirmed
Changed in debmirror (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Brian Murray (brian-murray) wrote :

For what its worth this worked on my Ubuntu 18.04 with debmirror installed.

Revision history for this message
Colin Watson (cjwatson) wrote :
Changed in debmirror (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt-mirror (Ubuntu):
status: New → Confirmed
Revision history for this message
Michael (3-ueuntu-4) wrote :

Would be nice to have this into Ubuntu 20.04 so it can mirror itself...

Frikkie Botha (fbotha)
Changed in debmirror (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Frikkie Botha (fbotha) wrote :

Please change status back to Confirmed

Barry Price (barryprice)
Changed in debmirror (Ubuntu):
status: Fix Released → Fix Committed
Revision history for this message
Alastair Browne (alastairbrowne) wrote :

I applied this patch and it still doesn't work. Still no cnf directories.

Revision history for this message
Alastair Browne (alastairbrowne) wrote :

The patch is not correct.

In the function "sub cnf_from_release", there is a line which reads...

next if $filename =~ /^Commands-/;

This should actually read...

next if $filename =~ /^Commands-/ and $filename !~ /^Commands-$all_arches\./;

Please update the patch to take this into account.

Thanks

Revision history for this message
Olaf Seibert (oseibert-sys11) wrote :

This may be patched upstream, but it is not present in the version which today installs on Focal 20.04 (reported as 1:2.27ubuntu1).

One reason might be that https://code.launchpad.net/~vcs-imports/debmirror/trunk (which is about this package, right?) reports that "Last successful import was on 2014-11-07. "

Possibly this is caused by the fact that the upstream repo has moved from git://git.debian.org/collab-maint/debmirror.git (listed on that page) to https://salsa.debian.org/debian/debmirror as shown in comment #6.

Another reason why Focal cannot mirror itself successfully is reported in this bug: https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/1879348

tags: added: rls-ff-incoming
Changed in debmirror (Ubuntu):
status: Fix Committed → Triaged
tags: added: rls-gg-incoming
Revision history for this message
wiki (osvaldito91) wrote :

How do i patch the debmirror file please??

Dan Streetman (ddstreet)
tags: added: seg
Dan Streetman (ddstreet)
tags: added: sts-sponsor-volunteer
Revision history for this message
Valters Jansons (sigv) wrote :

For completeness sake here, there is a merge request at https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/1892110 which rebases the Ubuntu changes on top of the latest upstream debmirror version, in turn including the required `command-not-found` changes.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debmirror - 1:2.33ubuntu1

---------------
debmirror (1:2.33ubuntu1) hirsute; urgency=medium

  * Resynchronise with Debian. (LP: #1892110) Remaining changes:
    - debian/{control,rules}: Add quilt for patch management.
    - Debian bug 455082:
      + default-settings.patch: change rsync defaults.
      + silence-errors.patch: throw away find errors.
      + drop-redundant-rsync.patch: clean up logic in rsync batching.
      + check_file-return.patch: report why a file needs to sync.
    - Debian bug 576576:
      + skip-installer.patch: allow specific releases to be skipped.
    - ubuntu-settings.patch: Changed defaults to download Ubuntu, replaced
      most instances of Debian with Ubuntu in the documentation.

debmirror (1:2.33) unstable; urgency=medium

  * Use debhelper-compat instead of debian/compat.
  * Strip trailing spaces and tabs from the content of clearsigned files
    before verifying the signature, in accordance with RFC 4880 section 7.1
    (closes: #943970).

debmirror (1:2.32) unstable; urgency=medium

  [ Jamie Strandboge ]
  * Support downloading command-not-found metadata (LP: #1821251).

debmirror (1:2.31) unstable; urgency=medium

  [ Ondřej Nový ]
  * d/copyright: Use https protocol in Format field
  * d/changelog: Remove trailing whitespaces

  [ Colin Watson ]
  * Fetch Packages-all and Contents-all if they exist (closes: #904927).
  * Fix mirroring of suites that have InRelease but not Release.gpg.
  * Improve GPG verification: accept a signature file as long as there's at
    least one good signature from a trusted key, and borrow code from APT to
    explicitly verify the structure of InRelease files (closes: #918304).

debmirror (1:2.30) unstable; urgency=medium

  * Make the error message when mkdir fails more helpful.
  * Restart partial HTTP downloads (closes: #629922).
  * Use /usr/share/dpkg/pkg-info.mk rather than equivalent hand-written
    code.
  * Upgrade to debhelper v9.

debmirror (1:2.29) unstable; urgency=medium

  [ Stefan Kisdaroczi ]
  * Add version string and --version option.
  * Add additional trace info (closes: #863194).

  [ Colin Watson ]
  * Automatically update $version at binary package build time.

debmirror (1:2.28) unstable; urgency=medium

  * Change priority to optional, since "Priority: extra" is now deprecated.
  * Move VCS to salsa.debian.org.
  * Fix use of uninitialised value $buf in name_release.
  * Depend on xz-utils (LP: #1614779); xz is needed for all suites that
    publish .xz but not uncompressed index files, or .gz but not .xz without
    --slow-cpu, so I think that's common enough to rise above the level of a
    Recommends.
  * Always use "exists $file_lists{foo}{size}" rather than just "exists
    $file_lists{foo}". The latter can return false positives due to
    autovivification (closes: #868675).

 -- Valters Jansons <email address hidden> Sat, 22 Aug 2020 15:18:46 +0300

Changed in debmirror (Ubuntu):
status: Triaged → Fix Released
Dan Streetman (ddstreet)
tags: removed: sts-sponsor-volunteer
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

please add the usual SRU details in the description, thanks

Revision history for this message
Valters Jansons (sigv) wrote :

This was handled as a merge in #1892110 and is being handled as an SRU in #1906541 with the patch provided in that SRU ticket including further upstream commits.

Revision history for this message
Steve Langasek (vorlon) wrote :

> This was handled as a merge in #1892110 and is being handled as an SRU
> in #1906541 with the patch provided in that SRU ticket including
> further upstream commits.

The bug linked in the changelog of the SRU is this one, not bug #1906541. The SRU template needs to be on this bug (and, in general, we should not be opening separate bugs for an SRU).

Revision history for this message
Valters Jansons (sigv) wrote :

> The bug linked in the changelog of the SRU is this one, not bug #1906541.

Indeed, I am not sure of the proper formatting rules, considering this was my first merge request followed by SRU request of upstream changes. The upstream changelog included the details of the original owner along with this LP# considering the fix originated here.

> The SRU template needs to be on this bug

How should this be handled as of now? By who?

> in general, we should not be opening separate bugs for an SRU

In this case, this bug ticket is marked for multiple packages. It seemed more straight-forward to me for the individual changes to be split off in a separate ticket for the SRU itself. Would, instead, multiple SRU templates be expected on this ticket - one for each package? Or would the SRU potentially change depending on the changes required in the individual packages?

This process is fairly scary from the outside perspective as these real life scenarios are not covered. Considering this bug has not been fixed for one-and-a-half years, with a patch laying around, I was trying to get the ball rolling. I do ask once again for actionable items as of now, or clarity of whether something should be done whatsoever (as it may instead be handled in other communication channels without information visible to the 'outside world').

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hey! Ok, so let me help out with the formalities here and then accept the SRU. I'll update this bug with the description of LP: #1906541, as this is the bug number that is referenced in the upload. I'll then mark the other one as a duplicate of this bug. We should be good then!

It does seem scary but it's not that much really! It's just that we need SRU's to link in the changelog to the bug with the appropriate SRU information. This is crucial as it's necessary for bugfix validation and, in case of regressions, knowing about what's going on. Since not everyone has the power to update bug descriptions, I can do it for you. Normally we'd just expect the package upload to reference LP: #1906541 instead.

Anyway, on it now.

description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

I noticed that the version number is not perfect. Then again, I don't think it makes sense to re-upload just because of that. Let me accept it as is.

Changed in debmirror (Ubuntu Groovy):
status: New → Fix Committed
tags: added: verification-needed verification-needed-groovy
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Jamie, or anyone else affected,

Accepted debmirror into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debmirror/1:2.27ubuntu2.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in debmirror (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Jamie, or anyone else affected,

Accepted debmirror into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debmirror/1:2.27ubuntu2.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Valters Jansons (sigv) wrote :

TEST CASE:
1. Installed current non-SRU package: apt install debmirror=1:2.27ubuntu1
2. Created a 20.04 Focal mirror: /usr/bin/debmirror -a amd64,i386 -d focal,focal-security,focal-updates,focal-backports -s main,universe,restricted,multiverse --nosource --passive --method=http -h archive.ubuntu.com -r ubuntu --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror
3. Exposed /opt/debmirror as webroot on test.example.com
4. On a separate node, configured APT to use the mirror, via sources.list such as:
   deb http://test.example.com/ubuntu/ focal main universe restricted multiverse
   deb http://test.example.com/ubuntu/ focal-security main universe restricted multiverse
   deb http://test.example.com/ubuntu/ focal-updates main universe restricted multiverse
5. Tried to use mirror on the newly configured client: apt update
   E: Failed to fetch http://test.example.com/ubuntu/dists/focal/main/cnf/Commands-amd64 404 Not Found [IP: 10.10.10.10 80]
   E: Failed to fetch http://test.example.com/ubuntu/dists/focal-security/main/cnf/Commands-amd64 404 Not Found [IP: 10.10.10.10 80]
   E: Failed to fetch http://test.example.com/ubuntu/dists/focal-updates/main/cnf/Commands-amd64 404 Not Found [IP: 10.10.10.10 80]
   E: Some index files failed to download. They have been ignored, or old ones used instead.
6. Installed SRU package: apt install debmirror=1:2.27ubuntu2.20.04.1
7. Repeated `debmirror` invocation from step 2, in order to pull in cnf files.
8. Repeated `apt update` invocation as from step 5, in order to try using the mirror.

VERIFICATION DONE
APT client succeeds, and mirrored repository can be successfully used.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Valters Jansons (sigv) wrote :

TEST CASE:
1. Deploy fresh (up-to-date) Ubuntu 20.10 (Groovy) machine.
2. apt install -qqy debmirror=1:2.27ubuntu1
3. debmirror -a amd64 -d groovy -s main --nosource --passive --method=http -h archive.ubuntu.com -r ubuntu --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror
4. echo 'deb file:///opt/debmirror groovy main' > /etc/apt/sources.list.d/file.list
5. apt update # expected E: failed to fetch file:/opt/debmirror/dists/groovy/main/cnf/Commands-amd64 File not found
6. apt install -qqy debmirror=1:2.27ubuntu2.20.10.1
7. debmirror -a amd64 -d groovy -s main --nosource --passive --method=http -h archive.ubuntu.com -r ubuntu --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror
8. echo 'deb file:///opt/debmirror groovy main' > /etc/apt/sources.list.d/file.list
9. apt update # All packages are up to date.

VERIFICATION DONE
Mirror is usable with updated `debmirror` package.

tags: added: verification-done verification-done-groovy
removed: verification-needed verification-needed-groovy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debmirror - 1:2.27ubuntu2.20.10.1

---------------
debmirror (1:2.27ubuntu2.20.10.1) groovy; urgency=medium

  * Cherry-pick upstream command-not-found changes (version 1:2.32).

  [ Jamie Strandboge ]
  * Support downloading command-not-found metadata (LP: #1821251).

 -- Valters Jansons <email address hidden> Wed, 02 Dec 2020 18:36:09 +0200

Changed in debmirror (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for debmirror has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debmirror - 1:2.27ubuntu2.20.04.1

---------------
debmirror (1:2.27ubuntu2.20.04.1) focal; urgency=medium

  * Cherry-pick upstream command-not-found changes (version 1:2.32).

  [ Jamie Strandboge ]
  * Support downloading command-not-found metadata (LP: #1821251).

 -- Valters Jansons <email address hidden> Wed, 02 Dec 2020 18:36:09 +0200

Changed in debmirror (Ubuntu Focal):
status: Fix Committed → Fix Released
no longer affects: apt-mirror (Ubuntu)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers