apt-mirror in all versions (including disco and last from github) doesn't mirror Commands-* files

Bug #1825755 reported by laulau
122
This bug affects 22 people
Affects Status Importance Assigned to Milestone
apt-mirror (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Bionic
High
Unassigned
Eoan
High
Unassigned
Focal
High
Unassigned
Groovy
High
Unassigned

Bug Description

[Impact]

* since ubuntu disco, main repositories contains 'cnf' directory with 'Commands*' files. those files are compressed with xy, whereas apt-mirror 0.5.1 doesn't contain the patch which enable download of .xz compressed files

* Without this update, users are not able to mirror newer releases of Ubuntu.

[Test Case]

* On any release of Ubuntu install apt-mirror and add this line to /etc/apt/mirror.list:

deb http://archive.ubuntu.com/ubuntu eoan-updates main restricted universe multiverse

* sync the mirror using `apt-mirror`

* Attempt to run `apt update` against the mirror. The result will be similar to:

...
Err:7 http://10.3.0.41/ubuntu eoan-updates/main amd64 c-n-f Metadata
  404 Not Found [IP: 10.3.0.41 80]
Ign:14 http://10.3.0.41/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata
Ign:15 http://10.3.0.41/ubuntu eoan-updates/universe amd64 c-n-f Metadata
Ign:16 http://10.3.0.41/ubuntu eoan-updates/restricted amd64 c-n-f Metadata
Reading package lists... Done
E: Failed to fetch http://10.3.0.41/ubuntu/dists/eoan-updates/main/cnf/Commands-amd64 404 Not Found [IP: 10.3.0.41 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.
...

* Updating the apt-mirror package from ppa:slashd/test and then re-running the `apt-mirror` command will then allow a successful `apt update`.

[Regression Potential]

* The regression potential is low since the package doesn't currently work properly anyway.

* It's a patchset from a fork repo, if apt-mirror upstream becomes alive again in the future, it won't necessary mean that the fork will be commited/merged upstream, and Ubuntu package might differ from original upstream until next version bump.

[Other informations]

Upstream bug:
https://github.com/apt-mirror/apt-mirror/issues/118

Not much update has been made since Upstream become inactive as noted from the output of rmadison:

Since Bionic, it is a simple copy from release to active devel release and so on directly from debian, no patching whatsoever and nothing new to pull from Debian as well.

----
Ubuntu
 apt-mirror | 0.5.1-1ubuntu1 | xenial/universe | source, all
 apt-mirror | 0.5.4-1 | bionic/universe | source, all
 apt-mirror | 0.5.4-1 | disco/universe | source, all
 apt-mirror | 0.5.4-1 | eoan/universe | source, all
 apt-mirror | 0.5.4-1 | focal/universe | source, all
 apt-mirror | 0.5.4-1 | groovy/universe | source, all

Debian
apt-mirror 0.5.4-1 is found in sid/unstable
---

The required patchset is from a fork of upstream apt-mirror josbraden/apt-mirror (https://github.com/josbraden/apt-mirror)

Note that upstream apt-mirror is not active anymore, and is looking for a new maintainer as published in their github page: https://github.com/apt-mirror/apt-mirror

That's why we have to rely on the fork atm.

The required patchset are (tested by Chris in the form of a test pkg:
0001-Copy-DEP-11-functions-for-editing.patch
0001-Replace-DEP-11-with-cnf-in-DEP-11-code-copy.patch
0001-Fix-Regex.patch
0001-Improve-regex-for-cnf-file-matching.patch
0001-Remove-dev-note.patch
0001-Fix-missing-parenthesis.patch

I also talked with sil2100 (SRU verification team member) about the feasibilty of this SRU (from a fork repo):

"
<sil2100> Yeah, I'd say fixes from forked repositories in this case make sense Since there's no longer any 'one authority' for it.
"

Revision history for this message
laulau (olaulau) wrote :
Revision history for this message
laulau (olaulau) wrote :

seems that even using more recent apt-mirror perl script doesn't solve the problem, Commands files aren't mirrored.
any idea ?

Revision history for this message
laulau (olaulau) wrote :

tested on a fresh disco server install, apt-mirroring ubuntu disco :
# ll mirror/archive.ubuntu.com/ubuntu/dists/disco/main/
total 24
drwxrwxr-x 6 root root 4096 Apr 22 02:47 ./
drwxrwxr-x 6 root root 4096 Apr 22 02:44 ../
drwxrwxr-x 2 root root 4096 Apr 22 02:44 binary-amd64/
drwxrwxr-x 2 root root 4096 Apr 22 02:47 dep11/
drwxrwxr-x 2 root root 4096 Apr 22 02:45 i18n/
drwxrwxr-x 2 root root 4096 Apr 22 02:44 source/
# find . | grep Commands
-> nothing

so the bug is not specific to old apt-mirror.
by the way, apt-mirror provided with disco has a mirror.list that mirrors artful by default. should also be fixed ...

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt-mirror (Ubuntu):
status: New → Confirmed
Revision history for this message
int19h (int19h) wrote :

Might be worth updating the bug title and tags to indicate that this is not Xenial-specific, and also happens to the apt-mirror package in Disco.

laulau (olaulau)
summary: - apt-mirror in xenials doesn't mirror corrrectly Commands-amd64 file
+ apt-mirror in all versions (including disco and last from github)
+ doesn't mirror Commands-* files
Revision history for this message
Christian Grafe (fake4d) wrote :

I created an issue in the github from apt-mirror. Don't know whether it is still in maintenance!

https://github.com/apt-mirror/apt-mirror/issues/118

Revision history for this message
Chris Johnston (cjohnston) wrote :

I have tested the version of apt-mirror from ppa:slashd/test and can confirm that with the updates Eric added it now works on bionic.

description: updated
Changed in apt-mirror (Ubuntu Bionic):
importance: Undecided → High
Changed in apt-mirror (Ubuntu Eoan):
importance: Undecided → High
Changed in apt-mirror (Ubuntu Focal):
importance: Undecided → High
Changed in apt-mirror (Ubuntu Groovy):
importance: Undecided → High
status: Confirmed → Triaged
Changed in apt-mirror (Ubuntu Focal):
status: New → Triaged
Changed in apt-mirror (Ubuntu Eoan):
status: New → Triaged
Changed in apt-mirror (Ubuntu Bionic):
status: New → Triaged
Revision history for this message
Eric Desrochers (slashd) wrote :

If other impacted users want to try the test package before moving forward with the SRU:

sudo add-apt-repository ppa:slashd/test
sudo apt-get update
sudo apt-get install apt-mirror ### version 0.5.4-1ubuntu0.18.04.1+testpkg20200428b2

- Eric

description: updated
Revision history for this message
Eric Desrochers (slashd) wrote :

I would definitely appreciate more feedbacks if possible.

That would definitely bring more confidence in the patchset (at SRU perpsective) if more impacted folks test it pre-SRU.

- Eric

description: updated
Eric Desrochers (slashd)
description: updated
Eric Desrochers (slashd)
description: updated
Revision history for this message
Eric Desrochers (slashd) wrote :

I just left a comment in the apt-mirror upstream bug[0] seeking for volunteers to test.

[0] - https://github.com/apt-mirror/apt-mirror/issues/118

- Eric

Revision history for this message
Chris Johnston (cjohnston) wrote :

Eric, I think we may end up needing this commit as well:

https://github.com/josbraden/apt-mirror/commit/8c06384dd5a7c4c3c27ac4f5d37824cedb8634c6

E: Failed to fetch http://X.X.X.X/ubuntu/dists/focal-updates/universe/dep11/icons-64x64@2.tar 404 Not Found [IP: X.X.X.X 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

Revision history for this message
Jerome Cheviet (gharan) wrote :

Hi,

I testing your repository for Ubuntu 20.04 LTS.

My apt-mirror server running on Ubuntu 18.04 LTS upgraded since Ubuntu 14.04 LTS.

I have repository for Ubuntu 14.04, Ubuntu 18.04 and now Ubuntu 20.04 (and I using too for Debian 10).

I must use "slashd/test" with Focal and I have no more issue with cnf index, but when I run apt-mirror, I have some warning with indexes :

<pre>
Processing indexes: [SSSSSSSSSSSSSUse of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 1.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 2.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 3.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 4.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 5.
SPPPPPPPPPPPPPPPPPPPPPPPPPPPP]
</pre>

However, the download continue and I have my update.

Maybe it's nothing, but i prefer tell you my error.

Sorry for my bad English.

Regards,
Jerome.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt-mirror (Ubuntu Xenial):
status: New → Confirmed
Revision history for this message
Chris Wieringa (chriswier) wrote :

I tested the ppa:slashd/test apt-mirror 0.5.4-1ubuntu0.18.04.1+testpkg20200428b2 package on my Bionic 18.04.4 LTS mirror host, and it successfully got all applicable cnf directories for the 20.04 focal release. It looks good for me!

Revision history for this message
Eric Periard (ericperiard) wrote :

This is quite ridiculous, would it not be possible to apply the fix toe apt-mirror already?

I'd rather not use a modified third party apt-mirror tool if it can be helped.

Revision history for this message
Eric Desrochers (slashd) wrote :

@Eric

We are in discussion atm trying to resolve the maintainer transition upstream.
As soon as the maintainer aspect is figured out, hopefully the code will be merge in the official project and not only the fork.

And then I'll be happy to update the Ubuntu apt-mirror package in the Ubuntu archive.

Hope that answer your concern here.

- Eric

Revision history for this message
Rod Smith (rodsmith) wrote :

FWIW, I'm having this problem, too, specifically with the icons files, as noted in comment #12, on my mirror running on Ubuntu 18.04. Oddly, some installations using my mirror are fine with this, whereas others complain. Applying the patch linked in comment #12 fixed it for me.

Revision history for this message
Sam Morrison (sorrison) wrote :

I had this problem running apt-mirror on bionic, upgrading to the PPA version fixed it

Revision history for this message
Marcel Brouwers (marcelbrouwers) wrote :

Had this problem with apt-mirror on Xenial with Focal clients. Using apt-mirror 0.5.4-1ubuntu0.18.04.1+testpkg20200428b2 from ppa:slashd/test solved the problem for me. Thanks for the work on this.

Revision history for this message
Stefan Staeglich (staeglis) wrote :

I'm running in this issue too. When will the fix be releases for 18.04? In the PPP there are no packages anymore.

Revision history for this message
Brian Murray (brian-murray) wrote :

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

Changed in apt-mirror (Ubuntu Eoan):
status: Triaged → Won't Fix
chenrongwen (were0415)
information type: Public → Public Security
information type: Public Security → Public
Revision history for this message
Brian Turek (brian-turek) wrote :

Just ran into this issue on with a Focal host mirroring Focal repositories with Focal clients. Some searching led me to https://github.com/Stifler6996/apt-mirror which I can confirm works (i.e. fixes cnf, icons, and compression-related problems).

I don't know the politics behind how maintainers get chosen but I'm happy to help however I can.

Revision history for this message
Jo-Erlend Schinstad (joerlend.schinstad) wrote :

Any progress on this?

Revision history for this message
JamesGreenhalgh (james-greenhalgh) wrote :

Trying to mirror into a secure environment through a proxy which allows me to hit ubuntu.com repo servers.

I'm in a bind because the workaround here is to use some random code from github which I'm not allowed to bring into the secure environment. Therefore my plans of using Ubuntu in the secure environment are dashed.

I feel like this deserves some more attention as the fix is in existence and security vetting of a perl script shouldn't be too onerous. Please let me know if I can help.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.