ovn-source option doesn't clear the cloud pockets when moving to internal repository

[DUFOUR Olivier]

As it stands right now on ovn-chassis and ovn-central charms, starting from 22.03/stable channel, a new configuration option called "ovn-source", has been added.
This option add, on the units involved with OVN, a dedicated repository on Focal for OVN, to retrieve ovn 22.03 packages.

As seen on a newly deployed unit, it adds a repository from cloud-archive :
ubuntu@ovn-central:~$ cat /etc/apt/sources.list.d/cloud-archive-ovn.list
# Ubuntu Cloud Archive
deb http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/ovn-22.03 main

This is an issue for air-gapped environment since they cannot rely on public repositories.

Attempting to run a command like below, didn't update the file cloud-archive-ovn.list :
juju config ovn-central ovn-source="deb http://172.123.45.67/repository/ovn/ubuntu/ focal-updates/ovn-22.03 main"

[DUFOUR Olivier (odufourc)]

After experimenting a lot with ovn-chassis and ovn-central. It appears it has a relation with charm-helpers.

The configuration option for "source", "openstack-origin" or "ovn-source" are relying from the function add_source from charmhelpers.
( https://github.com/juju/charm-helpers/blob/dc0dc74ac2debd328bd7846dbf1d2044563e255a/charmhelpers/fetch/ubuntu.py#L634 )
The main issue here is that adding a cloud repository, the charm will consequently create a file in /etc/apt/sources.list.d/ with either a file called "cloud-archive.list" or "cloud-archive-ovn.list".

But if adding a "deb http://myrepository/ubuntu focal-updates/ovn-22.03 main"
charmhelpers will use the command "add-apt-repository" which will edit instead /etc/apt/sources.list directly to create the entry.
( https://github.com/juju/charm-helpers/blob/dc0dc74ac2debd328bd7846dbf1d2044563e255a/charmhelpers/fetch/ubuntu.py#L757 )

This gets tricky with a new deployment on Focal with ovn-central and ovn-chassis being empty will lead to have automatically and implicitely "cloud:focal-ovn-22.03". And then changing to an internal repository will lead in having both the old entry in /etc/apt/sources.list.d/cloud-archive-ovn.list and the new entry /etc/apt/sources.list to be present at the same time.

This doesn't concern only "ovn-source" but also "source" or "openstack-origin" configuration option.
It has been observed that when using an URL type entry ("deb http://myrepository/ubuntu XXX"), the previous entry is never cleared and the new repository is simply added in /etc/apt/sources.list .
So for instance, when upgrading a cloud with :
"deb http://myrepository/ubuntu focal-updates/victoria main"
then
"deb http://myrepository/ubuntu focal-updates/wallaby main"
and then
"deb http://myrepository/ubuntu focal-updates/xena main"

"/etc/apt/sources.list" will contain 3 times the previous value of repository without being cleared and will continue to reference old versions of pockets.

At first glance, this looks like a limitation of the charm, where ideally, if using add-apt-repository, we should run add-apt-repository --remove "the previous entry", before adding the new repository/entry, but it looks difficult to retrieve the previous value of a config entry from a charm.

An idea would be to avoid using "add-apt-repository" command, and create manually the file as /etc/apt/sources.list.d/ with the same name like "/etc/apt/sources.list.d/cloud-archive(-ovn).list". That way, whenever there is an update, the previous entry is properly cleared and when moving between a cloud pocket and an internal repository there is no leftover of configuration.