Repositories created via apt-mirror do not support do-release-upgrade

[Paul Goins]

It appears that charm-apt-mirror's generated repository, or more specifically apt-mirror, does not clone enough of the apt repository to power do-release-upgrade.

We may need to leverage a postmirror.sh script to add code which will sync the important missing bits. This script will also need to be careful about proxy settings since a simple rsync may not be allowed; it may be worth investigating how apt-mirror does its mirroring and to use a similar method.

This page (as well as many other pages scattered online) seems to give some details regarding the missing bits: https://makandracards.com/makandra/12439-setup-an-ubuntu-mirror-that-enables-local-release-upgrades

Definition of done: apt-mirror should be capable of mirroring enough of the official Ubuntu repositories so as to allow for performing a do-release-upgrade using the mirror without errors. Any special parameters needed by do-release-upgrade to use the custom mirror during this process should be documented.

[Pedro Castillo (peterctl)]

Note that the changelogs.ubuntu.com directory on the mirror is not copied over when creating a snapshot. It looks like the _on_create_snapshot handler only looks at directories that contain `dists/` and `pool/` as subdirectories. The changelogs mirror does not have these subdirectories and is thus not copied over to the published snapshot, and is thus not exposed as part of the repo.

Workaround:
cd /var/spool/apt-mirror/ && cp -r mirror/changelogs.ubuntu.com publish/

[Pedro Castillo (peterctl)]

We also need to pull the dist-upgrader-all directory from the ${series}-updates distros. However, dist-upgrader-all is not a repository and cannot be added to the mirror list. Rather, it has to be downloaded via `wget --mirror` or `rsync`.

Another issue here is that the `cnf` directories are not being pulled from the repos. It looks like the apt-mirror master branch [0] already handles that, but the apt-mirror version installed by this charm does not include that fix and the `cnf` files need to be downloaded directly via `wget --mirror` or `rsync`.

I have been writing a postmirror.sh script to run after the repo is synced via the `synchronize` Juju action, however I haven't gotten DRU into a working state yet so I will wait to post the script here once it works correctly.

[Pedro Castillo (peterctl)]

I got DRU to work using the private repo exclusively. I attached a bash script to this comment which contains the necessary commands to run after the repo is synced with upstream in order to allow do-release-upgrades to work using the private repo without reaching out to upstream.

These are the steps taken by the script. Note that the script will not handle step 5 as that needs to happen when a snapshot is created, and the script is called when the repo is synced before the creation of the snapshot.
1. Synchronize the private repository.
2. Pull the meta-release files from `http://changelogs.ubuntu.com`. DRU needs these to know which series are available for the upgrade.
    2.1. Modify the meta-release files to point to the private repo. The directory structure of apt-mirror allows this change to be performed via a simple sed substitution.
        sed -i "s|http://|$REPO_URL/apt-mirror/|g"
3. Pull the dist-upgrader-all directory from `http://archive.ubuntu.com/ubuntu/dists/${series}-updates/main/dist-upgrader-all`. DRU downloads the actual series upgrade tool from this location.
4. Pull the `cnf` directories for the following repos. DRU needs these in order to know what commands to run after the packages are upgraded.
    # NOTE: apt-mirror does this by itself on the master branch but the charm doesn't install that version, so we need to get the cnf files ourselves.
    echo ${series}{,-updates,-backports}/{main,restricted,universe,multiverse} | sed 's| |\n|g'
        ${series}/main
        ${series}/restricted
        ${series}/universe
        ${series}/multiverse
        ${series}-updates/main
        ${series}-updates/restricted
        ${series}-updates/universe
        ${series}-updates/multiverse
        ${series}-backports/main
        ${series}-backports/restricted
        ${series}-backports/universe
        ${series}-backports/multiverse
5. When creating a repo snapshot, make sure to copy the changelogs directory into the snapshot as the charm does not do it on its own.
    cp -r mirror/changelogs.ubuntu.com $snapshot/
6. On the machine you want to upgrade, point update-manager to the meta-release files in the private repo. Modify the `/etc/update-manager/meta-release` file as such:
    URI = http://$PRIVATE_REPO_URL/changelogs.ubuntu.com/meta-release
    URI_LTS = http://$PRIVATE_REPO_URL/changelogs.ubuntu.com/meta-release-lts
7. Run do-release-upgrade.
    7.1. If DRU fails, it might be due to incorrect handling of Apt proxy settings. I had to manually add a URL to NO_PROXY that was set on the Apt config but it wasn't being picked up by DRU.
        If you have the following line on the Apt config files:
            Acquire::http::Proxy::"$URL" "DIRECT";
        Then you need to set NO_PROXY to contain $URL yourself:
            $ sudo env NO_PROXY=$URL,$NO_PROXY do-release-upgrade
    7.2. You can enable the DEBUG_UPDATE_MANAGER=1 env var when running DRU to get more verbose output.

[Kamal Bhaskar (kamalb)]

Hi Pedro, If we update the mirror.list to include these URL's before launching "apt-mirror" in the charm install hook, does that suffice the requirement here ?

[Launchpad Janitor (janitor)]

[Expired for charm-apt-mirror because there has been no activity for 60 days.]

[John Lettman (jplettman)]

Hello Kamal,

We have another instance of this.
I will check as requested.

[Eric Chen (eric-chen)]

Hello Kamal and John,

We put this project into maintainence mode. It's probably we won't handle this new feature.
We expect the best practice should convert to landscape in the future.
Please raise this issue in your team internal if you think it's really important. At the same time, let's think about how to convert it to landscape. thanks!

[Andrea Ieri (aieri)]

additionally, this is not a charm bug; it should be re-filed with the upstream project at https://github.com/apt-mirror/apt-mirror/issues/