apt-mirror in all versions (including disco and last from github) doesn't mirror Commands-* files

[laulau]

[Impact]

* since ubuntu disco, main repositories contains 'cnf' directory with 'Commands*' files. those files are compressed with xy, whereas apt-mirror 0.5.1 doesn't contain the patch which enable download of .xz compressed files

* Without this update, users are not able to mirror newer releases of Ubuntu.

[Test Case]

* On any release of Ubuntu install apt-mirror and add this line to /etc/apt/mirror.list:

deb http://archive.ubuntu.com/ubuntu eoan-updates main restricted universe multiverse

* sync the mirror using `apt-mirror`

* Attempt to run `apt update` against the mirror. The result will be similar to:

...
Err:7 http://10.3.0.41/ubuntu eoan-updates/main amd64 c-n-f Metadata
  404 Not Found [IP: 10.3.0.41 80]
Ign:14 http://10.3.0.41/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata
Ign:15 http://10.3.0.41/ubuntu eoan-updates/universe amd64 c-n-f Metadata
Ign:16 http://10.3.0.41/ubuntu eoan-updates/restricted amd64 c-n-f Metadata
Reading package lists... Done
E: Failed to fetch http://10.3.0.41/ubuntu/dists/eoan-updates/main/cnf/Commands-amd64 404 Not Found [IP: 10.3.0.41 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.
...

* Updating the apt-mirror package from ppa:slashd/test and then re-running the `apt-mirror` command will then allow a successful `apt update`.

[Regression Potential]

* The regression potential is low since the package doesn't currently work properly anyway.

* It's a patchset from a fork repo, if apt-mirror upstream becomes alive again in the future, it won't necessary mean that the fork will be commited/merged upstream, and Ubuntu package might differ from original upstream until next version bump.

[Other informations]

Upstream bug:
https://github.com/apt-mirror/apt-mirror/issues/118

Not much update has been made since Upstream become inactive as noted from the output of rmadison:

Since Bionic, it is a simple copy from release to active devel release and so on directly from debian, no patching whatsoever and nothing new to pull from Debian as well.

----
Ubuntu
 apt-mirror | 0.5.1-1ubuntu1 | xenial/universe | source, all
 apt-mirror | 0.5.4-1 | bionic/universe | source, all
 apt-mirror | 0.5.4-1 | disco/universe | source, all
 apt-mirror | 0.5.4-1 | eoan/universe | source, all
 apt-mirror | 0.5.4-1 | focal/universe | source, all
 apt-mirror | 0.5.4-1 | groovy/universe | source, all

Debian
apt-mirror 0.5.4-1 is found in sid/unstable
---

The required patchset is from a fork of upstream apt-mirror josbraden/apt-mirror (https://github.com/josbraden/apt-mirror)

Note that upstream apt-mirror is not active anymore, and is looking for a new maintainer as published in their github page: https://github.com/apt-mirror/apt-mirror

That's why we have to rely on the fork atm.

The required patchset are (tested by Chris in the form of a test pkg:
0001-Copy-DEP-11-functions-for-editing.patch
0001-Replace-DEP-11-with-cnf-in-DEP-11-code-copy.patch
0001-Fix-Regex.patch
0001-Improve-regex-for-cnf-file-matching.patch
0001-Remove-dev-note.patch
0001-Fix-missing-parenthesis.patch

I also talked with sil2100 (SRU verification team member) about the feasibilty of this SRU (from a fork repo):

"
<sil2100> Yeah, I'd say fixes from forked repositories in this case make sense Since there's no longer any 'one authority' for it.
"

[laulau (olaulau)]

seems that even using more recent apt-mirror perl script doesn't solve the problem, Commands files aren't mirrored.
any idea ?

[laulau (olaulau)]

tested on a fresh disco server install, apt-mirroring ubuntu disco :
# ll mirror/archive.ubuntu.com/ubuntu/dists/disco/main/
total 24
drwxrwxr-x 6 root root 4096 Apr 22 02:47 ./
drwxrwxr-x 6 root root 4096 Apr 22 02:44 ../
drwxrwxr-x 2 root root 4096 Apr 22 02:44 binary-amd64/
drwxrwxr-x 2 root root 4096 Apr 22 02:47 dep11/
drwxrwxr-x 2 root root 4096 Apr 22 02:45 i18n/
drwxrwxr-x 2 root root 4096 Apr 22 02:44 source/
# find . | grep Commands
-> nothing

so the bug is not specific to old apt-mirror.
by the way, apt-mirror provided with disco has a mirror.list that mirrors artful by default. should also be fixed ...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[int19h (int19h)]

Might be worth updating the bug title and tags to indicate that this is not Xenial-specific, and also happens to the apt-mirror package in Disco.

[Christian Grafe (fake4d)]

I created an issue in the github from apt-mirror. Don't know whether it is still in maintenance!

https://github.com/apt-mirror/apt-mirror/issues/118

[Chris Johnston (cjohnston)]

I have tested the version of apt-mirror from ppa:slashd/test and can confirm that with the updates Eric added it now works on bionic.

[Eric Desrochers (slashd)]

If other impacted users want to try the test package before moving forward with the SRU:

sudo add-apt-repository ppa:slashd/test
sudo apt-get update
sudo apt-get install apt-mirror ### version 0.5.4-1ubuntu0.18.04.1+testpkg20200428b2

- Eric

[Eric Desrochers (slashd)]

I would definitely appreciate more feedbacks if possible.

That would definitely bring more confidence in the patchset (at SRU perpsective) if more impacted folks test it pre-SRU.

- Eric

[Eric Desrochers (slashd)]

I just left a comment in the apt-mirror upstream bug[0] seeking for volunteers to test.

[0] - https://github.com/apt-mirror/apt-mirror/issues/118

- Eric

[Chris Johnston (cjohnston)]

Eric, I think we may end up needing this commit as well:

https://github.com/josbraden/apt-mirror/commit/8c06384dd5a7c4c3c27ac4f5d37824cedb8634c6

E: Failed to fetch http://X.X.X.X/ubuntu/dists/focal-updates/universe/dep11/[e-mail] 404 Not Found [IP: X.X.X.X 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

[Jerome Cheviet (gharan)]

Hi,

I testing your repository for Ubuntu 20.04 LTS.

My apt-mirror server running on Ubuntu 18.04 LTS upgraded since Ubuntu 14.04 LTS.

I have repository for Ubuntu 14.04, Ubuntu 18.04 and now Ubuntu 20.04 (and I using too for Debian 10).

I must use "slashd/test" with Focal and I have no more issue with cnf index, but when I run apt-mirror, I have some warning with indexes :

<pre>
Processing indexes: [SSSSSSSSSSSSSUse of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 1.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 2.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 3.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 4.
Use of uninitialized value $lines{"Files:"} in split at /usr/bin/apt-mirror line 936, <STREAM> line 5.
SPPPPPPPPPPPPPPPPPPPPPPPPPPPP]
</pre>

However, the download continue and I have my update.

Maybe it's nothing, but i prefer tell you my error.

Sorry for my bad English.

Regards,
Jerome.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Wieringa (chriswier)]

I tested the ppa:slashd/test apt-mirror 0.5.4-1ubuntu0.18.04.1+testpkg20200428b2 package on my Bionic 18.04.4 LTS mirror host, and it successfully got all applicable cnf directories for the 20.04 focal release. It looks good for me!

[Eric Periard (ericperiard)]

This is quite ridiculous, would it not be possible to apply the fix toe apt-mirror already?

I'd rather not use a modified third party apt-mirror tool if it can be helped.

[Eric Desrochers (slashd)]

@Eric

We are in discussion atm trying to resolve the maintainer transition upstream.
As soon as the maintainer aspect is figured out, hopefully the code will be merge in the official project and not only the fork.

And then I'll be happy to update the Ubuntu apt-mirror package in the Ubuntu archive.

Hope that answer your concern here.

- Eric

[Rod Smith (rodsmith)]

FWIW, I'm having this problem, too, specifically with the icons files, as noted in comment #12, on my mirror running on Ubuntu 18.04. Oddly, some installations using my mirror are fine with this, whereas others complain. Applying the patch linked in comment #12 fixed it for me.

[Sam Morrison (sorrison)]

I had this problem running apt-mirror on bionic, upgrading to the PPA version fixed it

[Marcel Brouwers (marcelbrouwers)]

Had this problem with apt-mirror on Xenial with Focal clients. Using apt-mirror 0.5.4-1ubuntu0.18.04.1+testpkg20200428b2 from ppa:slashd/test solved the problem for me. Thanks for the work on this.

[Stefan Staeglich (staeglis)]

I'm running in this issue too. When will the fix be releases for 18.04? In the PPP there are no packages anymore.

[Brian Murray (brian-murray)]

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

[Brian Turek (brian-turek)]

Just ran into this issue on with a Focal host mirroring Focal repositories with Focal clients. Some searching led me to https://github.com/Stifler6996/apt-mirror which I can confirm works (i.e. fixes cnf, icons, and compression-related problems).

I don't know the politics behind how maintainers get chosen but I'm happy to help however I can.

[Jo-Erlend Schinstad (joerlend.schinstad-deactivatedaccount)]

Any progress on this?

[JamesGreenhalgh (james-greenhalgh)]

Trying to mirror into a secure environment through a proxy which allows me to hit ubuntu.com repo servers.

I'm in a bind because the workaround here is to use some random code from github which I'm not allowed to bring into the secure environment. Therefore my plans of using Ubuntu in the secure environment are dashed.

I feel like this deserves some more attention as the fix is in existence and security vetting of a perl script shouldn't be too onerous. Please let me know if I can help.

[Brian Murray (brian-murray)]

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

[Peycho Dimitrov (peycho)]

This is broken in Ubuntu server 22.04 distribution.

E: Failed to fetch .........../ubuntu/dists/jammy/main/cnf/Commands-amd64 404 Not Found [NoSuchKey] The specified key does not exist.
E: Failed to fetch .........../ubuntu/dists/jammy-updates/main/cnf/Commands-amd64 404 Not Found [NoSuchKey] The specified key does not exist.
E: Failed to fetch .........../ubuntu/dists/jammy-backports/main/cnf/Commands-amd64 404 Not Found [NoSuchKey] The specified key does not exist.
E: Failed to fetch .........../ubuntu/dists/jammy-security/main/cnf/Commands-amd64 404 Not Found [NoSuchKey] The specified key does not exist.

[Utkarsh Gupta (utkarsh)]

Hiya, a fix was introduced in Debian and in Ubuntu Kinetic via 0.5.4-2. \o/

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt-mirror - 0.5.4-2

---------------
apt-mirror (0.5.4-2) unstable; urgency=medium

  [ Jelmer Vernooij ]
  * Migrate repository from alioth to salsa.

  [ Benjamin Drung ]
  * Update Homepage URL to github.io
  * Switch to debhelper 13
  * Support for xz translations (Closes: #932112, #932379)
  * Download 'cnf' Files (LP: #1825755)
  * Bump Standards-Version to 4.6.1
  * Set Rules-Requires-Root: no

 -- Benjamin Drung <email address hidden> Fri, 07 Oct 2022 13:04:08 +0200

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Heather Lemon (hypothetical-lemon)]

will start the backport process. Thanks.

[Heather Lemon (hypothetical-lemon)]

jammy debdiff

[Heather Lemon (hypothetical-lemon)]

focal debdiff

[Timo Aaltonen (tjaalton)]

Hello laulau, or anyone else affected,

Accepted apt-mirror into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt-mirror/0.5.4-1ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello laulau, or anyone else affected,

Accepted apt-mirror into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt-mirror/0.5.4-1ubuntu1.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Heather Lemon (hypothetical-lemon)]

### JAMMY VERIFICATION-DONE ###
*edited

Install a lxc jammy container
previous apt-mirror version: 0.5.4-1
sudo apt-get install apt-mirror
vim /etc/apt/mirror.list
deb http://archive.ubuntu.com/ubuntu eoan-updates main restricted universe multiverse
./apt-mirror
...
Failed to open Release file from http://archive.ubuntu.com/ubuntu/dists/eoan-updates/Release at /usr/bin/apt-mirror line 653.
....

** Running apt-mirror never successfully finishes and errors out **

### Proposed
- enable Proposed pocket
new apt-mirror version: 0.5.4-1ubuntu0.22.04.1

# show mirror.list
vim /etc/apt/mirror.list

# run apt-mirror without bug line
apt-mirror
Downloading 162 index files using 20 threads...
Begin time: Mon Feb 26 14:03:44 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 14:03:48 2024

Processing translation indexes: [TTT]

Downloading 558 translation files using 20 threads...
Begin time: Mon Feb 26 14:03:48 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 14:03:54 2024

Processing DEP-11 indexes: [DDD]

Downloading 64 dep11 files using 20 threads...
Begin time: Mon Feb 26 14:03:54 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 14:03:56 2024

Processing cnf indexes: [CCC]

Downloading 24 cnf files using 20 threads...
Begin time: Mon Feb 26 14:03:56 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 14:03:56 2024

Processing indexes: [SSSPPP]

425.7 GiB will be downloaded into archive.
Downloading 195380 archive files using 20 threads...
Begin time: Mon Feb 26 14:04:19 2024
[20]... ^C

425.7 GiB will be downloaded into archive. <--- this line is concerning, is this the correct behavior? *Note - we do see the new cnf files being collected and added.
* confirmed this is the correct behavior if you have all series listed with main/updates/universe/multiverse etc.

Just using multiverse turned on.
Downloading 14 index files using 14 threads...
Begin time: Mon Feb 26 16:56:38 2024
[14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 16:56:39 2024

Processing translation indexes: [T]

Downloading 105 translation files using 20 threads...
Begin time: Mon Feb 26 16:56:39 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 16:56:39 2024

Processing DEP-11 indexes: [D]

Downloading 8 dep11 files using 8 threads...
Begin time: Mon Feb 26 16:56:39 2024
[8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 16:56:39 2024

Processing cnf indexes: [C]

Downloading 2 cnf files using 2 thr...

[Heather Lemon (hypothetical-lemon)]

I forgot to mention, I saw c-n-f files in apt-update as well
Get:5 http://archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 c-n-f Metadata [1464 B]

*will re-test with just multiverse turned on just because the repos are so huge.

[Valentyna (valia0906)]

Please look at this bug https://bugs.launchpad.net/ubuntu/+source/apt-mirror/+bug/2044556 it`s easy to fix, it solves problem with missing dep11 icons.

[Heather Lemon (hypothetical-lemon)]

### FOCAL VERIFICATION-DONE ###

previous apt-mirror version: 0.5.4-1
sudo apt-get install apt-mirror
vim /etc/apt/mirror.list
deb http://archive.ubuntu.com/ubuntu eoan-updates main restricted universe multiverse
./apt-mirror
...
apt-mirror: can't open index archive.ubuntu.com/ubuntu//dists/artful-updates/main/source/Sources in process_index at /usr/bin/apt-mirror line 800.
....

** note we do not see any cnf files listed here in the output of apt-mirror.

# enabled proposed pocket
new apt-mirror version: 0.5.4-1ubuntu1.20.04.1
apt-get install --only-upgrade apt-mirror

# edit /etc/apt/mirror.list to use only multiverse
deb http://archive.ubuntu.com/ubuntu focal main multiverse

Downloading 22 index files using 20 threads...
Begin time: Mon Feb 26 17:16:45 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 17:16:47 2024

Processing translation indexes: [T]

Downloading 252 translation files using 20 threads...
Begin time: Mon Feb 26 17:16:47 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 17:16:49 2024

Processing DEP-11 indexes: [D]

Downloading 16 dep11 files using 16 threads...
Begin time: Mon Feb 26 17:16:49 2024
[16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 17:16:50 2024

Processing cnf indexes: [C]

Downloading 4 cnf files using 4 threads...
Begin time: Mon Feb 26 17:16:50 2024
[4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 17:16:50 2024

Processing indexes: [P]

11.6 GiB will be downloaded into archive.
Downloading 6903 archive files using 20 threads...
Begin time: Mon Feb 26 17:16:51 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Mon Feb 26 17:23:38 2024

0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.

Running the Post Mirror script ...
(/var/spool/apt-mirror/var/postmirror.sh)

Post Mirror script has completed. See above output for any possible errors.

# checking apt-get update
apt-get update
Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [970 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main Translation-en [506 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 c-n-f Metadata [29.5 kB]

Thanks,
Heather Lemon

### FOCAL VERIFICATION-DONE ###

[Andreas Hasenack (ahasenack)]

Hi Heather,

thanks for these verifications!

I just wanted to clarify some points.

Jammy verification:

> Failed to open Release file from http://archive.ubuntu.com/ubuntu/dists/eoan-updates/Release at
> /usr/bin/apt-mirror line 653.

I don't think this is expected to work today, since eoan is EOL and doesn't exist at that URL anymore. I would have changed the ubuntu release to mirror to something that is still available, like jammy itself.

So this conclusion:
> ** Running apt-mirror never successfully finishes and errors out **

Is not because of the bug that is being fixed, is just because that distro is not available anymore.

Later on, when verifying the proposed package:
> # show mirror.list
> vim /etc/apt/mirror.list

You edit the mirror list, but don't show what you edited it to.

Of interest to this bug, we can see that now it downloads cnf files, so that's good:
> Downloading 24 cnf files using 20 threads...

The last step of the test plan, however:
> apt-get update
> also returns no errors

Here I would like some clarification. The intention is to show that the mirror that was just created is working, and to do that we want to run "apt-get update" *against* *it*. I can't infer that from this verification: it could be that you just ran apt-get update with sources.list pointing at the ubuntu archive, and not the mirror you just created.

From the test plan, we can see that apt-get update is meant to run against the newly created mirror:

> * Attempt to run `apt update` against the mirror. The result will be similar to:
> ...
> Err:7 http://10.3.0.41/ubuntu eoan-updates/main amd64 c-n-f Metadata
> 404 Not Found [IP: 10.3.0.41 80]

[Heather Lemon (hypothetical-lemon)]

Hi Andreas
> I don't think this is expected to work today, since eoan is EOL and doesn't exist at that URL anymore. I would have changed the ubuntu release to mirror to something that is still available, like jammy itself.

ack, okay I was following the impact notes which list eoan, I didn't realize it's been open since 2020.

I did a re-test with jammy and running apt-mirror succeeds as well as apt-update.
*If there's a specific command to run please let me know since the impact description was not able to be completed during original testing.

### JAMMY VERIFICATION-DONE #2 ###

Re-test problem first
# /etc/apt/mirror.list - by default (no changes)
-------------
deb http://archive.ubuntu.com/ubuntu artful main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu artful-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu artful-updates main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu artful-proposed main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu artful-backports main restricted universe multiverse

deb-src http://archive.ubuntu.com/ubuntu artful main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu artful-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu artful-updates main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu artful-proposed main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu artful-backports main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu
-------------

update /etc/apt/mirror.list to include and comment everything else out:
deb http://archive.ubuntu.com/ubuntu jammy multiverse

update /etc/apt/sources.list and comment everything out.

-------------

# run apt-mirror
apt-mirror

Downloading 14 index files using 14 threads...
Begin time: Fri Mar 8 21:22:27 2024
[14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 8 21:22:30 2024

Processing translation indexes: [T]

Downloading 105 translation files using 20 threads...
Begin time: Fri Mar 8 21:22:30 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 8 21:22:31 2024

Processing DEP-11 indexes: [D]

Downloading 8 dep11 files using 8 threads...
Begin time: Fri Mar 8 21:22:31 2024
[8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 8 21:22:31 2024

Processing indexes: [P]

6.6 GiB will be downloaded into archive.
Downloading 881 archive files using 20 threads...
Begin time: Fri Mar 8 21:22:31 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 8 21:26:26 2024

0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.

Running the Post Mirror script ...
(/var/spool/apt-mirror/var/postmirror.sh)

Post Mirror script has completed. See...

[Andreas Hasenack (ahasenack)]

> I edited the sources.list again to include the other mirror
> https://ftp.uni-stuttgart.de/ubuntu
> then ran apt-get update

Sorry, what the test plan means with this:

> * Attempt to run `apt update` against the mirror. The result will be similar to:

Is to run apt-get update against the mirror you just made. Note the IP address in the sample output:

> Ign:14 http://10.3.0.41/ubuntu eoan-updates/multiverse amd64 c-n-f Metadata

In other words, you should setup apache or ftp or something like that and serve the mirror that was just created, and change sources.list to point at that. I think you might be able to get away with file:/// type urls in sources.list, so perhaps no need to setup apache.

[Heather Lemon (hypothetical-lemon)]

Ah okay, I was thinking something like this where they setup apache, but if I can get away with ftp / python server I can try that.

https://blog.programster.org/set-up-a-local-ubuntu-mirror-with-apt-mirror

Thanks,
Heather Lemon

[Heather Lemon (hypothetical-lemon)]

*In a Jammy lxc container

I setup an apache server with these instructions [0]

edited /etc/apt/mirror.list

deb https://ftp.uni-stuttgart.de/ubuntu/ jammy multiverse

pulled latest with apt-mirror
------------------
apt-mirror
Downloading 14 index files using 14 threads...
Begin time: Wed Mar 27 13:21:19 2024
[14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Mar 27 13:21:26 2024

Processing translation indexes: [T]

Downloading 105 translation files using 20 threads...
Begin time: Wed Mar 27 13:21:26 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Mar 27 13:21:30 2024

Processing DEP-11 indexes: [D]

Downloading 8 dep11 files using 8 threads...
Begin time: Wed Mar 27 13:21:30 2024
[8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Mar 27 13:21:31 2024

Processing cnf indexes: [C]

Downloading 2 cnf files using 2 threads...
Begin time: Wed Mar 27 13:21:32 2024
[2]... [1]... [0]...
End time: Wed Mar 27 13:21:32 2024

Processing indexes: [P]

5.4 GiB will be downloaded into archive.
Downloading 589 archive files using 20 threads...
Begin time: Wed Mar 27 13:21:32 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Wed Mar 27 13:27:25 2024

6.7 GiB in 1000 files and 562 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.

Running the Post Mirror script ...
(/var/spool/apt-mirror/var/postmirror.sh)

Post Mirror script has completed. See above output for any possible errors.

-----------------

double checked that we had something in /var/spool
/var/spool/apt-mirror/mirror/ftp.uni-stuttgart.de/ubuntu/dists/jammy

create symbolic link in apache - /var/wwww
ln -s /var/spool/apt-mirror/mirror/ftp.uni-stuttgart.de/ubuntu/ /var/www/ubuntu
cd /var/www
ls
ubuntu_de -> /var/spool/apt-mirror/mirror/ftp.uni-stuttgart.de/ubuntu//

edited /etc/apt/sources.list to get our apache hosted files
* only use this line in our sources.list, for readability

deb http://localhost/ubuntu_de/ jammy multiverse

# run apt-get update
root@jammy-list-mirror:/etc/apt# sudo apt-get update
Get:1 http://localhost/ubuntu_de jammy InRelease [270 kB]
Ign:2 http://localhost/ubuntu_de jammy/multiverse amd64 Packages
Ign:3 http://localhost/ubuntu_de jammy/multiverse Translation-en
Ign:4 http://localhost/ubuntu_de jammy/multiverse amd64 c-n-f Metadata
Get:2 http://localhost/ubuntu_de jammy/multiverse amd64 Packages [217 kB]
Get:3 http://localhost/ubuntu_de jammy/multiverse Translation-en [112 kB]
Get:4 http://localhost/ubuntu_de jammy/multiverse amd64 c-n-f Metadata [8372 B]

We can see our apache-hosted symlink works as expected with packages from our
mirror.list pointed to our upstream repo (https://ftp.uni-stuttgart.de/ubuntu/).

[0] https://blog.programster.org/set-up-a-local-ubuntu-mirror-with-apt-mirror

Thanks,
Heather Lemon

[Andreas Hasenack (ahasenack)]

Thank you for the updated jammy verification using the local apt mirror check. The focal verification should follow the same steps, so I'll tag focal as verification-needed still, but jammy looks good now in that regard.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt-mirror - 0.5.4-1ubuntu0.22.04.1

---------------
apt-mirror (0.5.4-1ubuntu0.22.04.1) jammy; urgency=medium

  * Download cnf files (LP: #1825755)
    - Adds support for xz files
    - d/p/Support-for-xz-translations.patch
    - d/p/Download-cnf-Files.patch

 -- Heather Lemon <email address hidden> Tue, 30 Jan 2024 22:50:26 +0000

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for apt-mirror has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Heather Lemon (hypothetical-lemon)]

great thanks Andreas, I'll start Focal soon. Cheers

[Heather Lemon (hypothetical-lemon)]

### VERIFICATION-DONE FOCAL ###

Install a lxc focal container or virt-manager focal vm
previous apt-mirror version: 0.5.4-1
sudo apt-get install apt-mirror

sudo vim /etc/apt/mirror.list
comment out all lines and add in
deb http://archive.ubuntu.com/ubuntu focal multiverse

apt-mirror
------------
Downloading 14 index files using 14 threads...
Begin time: Fri Mar 29 10:00:59 2024
[14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 29 10:01:02 2024

Processing translation indexes: [T]

Downloading 105 translation files using 20 threads...
Begin time: Fri Mar 29 10:01:02 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 29 10:01:04 2024

Processing DEP-11 indexes: [D]

Downloading 8 dep11 files using 8 threads...
Begin time: Fri Mar 29 10:01:04 2024
[8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 29 10:01:05 2024

Processing cnf indexes: [C]

Downloading 2 cnf files using 2 threads...
Begin time: Fri Mar 29 10:01:05 2024
[2]... [1]... [0]...
End time: Fri Mar 29 10:01:05 2024

Processing indexes: [P]

5.3 GiB will be downloaded into archive.
Downloading 813 archive files using 20 threads...
Begin time: Fri Mar 29 10:01:05 2024
[20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]...
End time: Fri Mar 29 10:06:29 2024

0 bytes in 0 files and 0 directories can be freed.
Run /var/spool/apt-mirror/var/clean.sh for this purpose.

Running the Post Mirror script ...
(/var/spool/apt-mirror/var/postmirror.sh)

Post Mirror script has completed. See above output for any possible errors.

--------

# setup apache

cd sites-enabled/
vim 000-default.conf

sudo echo '<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot /var/www
        <Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    ErrorLog ${APACHE_LOG_DIR}/error.log
    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>' | sudo tee /etc/apache2/sites-enabled/000-default.conf

-----------

enabled proposed pocket
apt-cache policy apt-mirror
Installed: 0.5.4-1ubuntu1.20.04.1

# edit /etc/apt/mirror.list
* only include this line
deb https://ftp.uni-stuttgart.de/ubuntu/ focal multiverse

# run apt-mirror
./apt-mirror

*this pulls packages from https://ftp.uni-stuttgart.de/
and puts them in /var/spool/apt-mirror
# double check /var/spool/
/var/spool/apt-mirror/mirror/

ln -s /var/spool/apt-mirror/mirror/ftp.uni-stuttgart.de/ubuntu/ /var/www/ubuntu_de
ll
# valid link
ubuntu_de -> /var/spool/apt-mirror/mirror/ftp.uni-stuttgart.de/ubuntu//

# edit sources.list
deb http://localhost/ubuntu_de/ focal multiverse

# run apt-get update
root@focal:/etc/apt# apt-get update
Get:1 http://localhost/ubuntu_de focal InRelease [265 kB]
Ign:2 http://localhos...

[Jonas Gamao (yamiyukisenpai)]

Are you also gonna check this:
https://bugs.launchpad.net/ubuntu/+source/apt-mirror/+bug/2044556

seems to be an easy fix as I'm using this apt-mirror to sync

[Jonas Gamao (yamiyukisenpai)]

"this apt-mirror" i meant
https://github.com/Stifler6996/apt-mirror

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt-mirror - 0.5.4-1ubuntu1.20.04.1

---------------
apt-mirror (0.5.4-1ubuntu1.20.04.1) focal; urgency=medium

  * Download cnf files (LP: #1825755)
    - Adds support for xz files
    - d/p/Support-for-xz-translations.patch
    - d/p/Download-cnf-Files.patch

 -- Heather Lemon <email address hidden> Wed, 31 Jan 2024 14:16:34 +0000

[Heather Lemon (hypothetical-lemon)]

@yamiyukisenpai I can take a look at the code next week.

Cheers,
Heather Lemon