Comment 0 for bug 1855061

In charms.openstack we have this sequience of calls:

        source, key = os_utils.get_source_and_pgp_key(
            self.config[self.source_config_key])
        fetch.add_source(source, key)
        fetch.apt_update(fatal=True)

On a failed deployment I found this:
2019-12-03 15:07:10 INFO juju-log Reactive main running for hook install
2019-12-03 15:07:11 INFO juju-log Initializing Leadership Layer (is leader)
2019-12-03 15:07:11 DEBUG juju-log tracer: set flag leadership.is_leader
2019-12-03 15:07:11 DEBUG juju-log tracer>
tracer: starting handler dispatch, 8 flags set
tracer: set flag charms.openstack.do-default-charm.installed
tracer: set flag charms.openstack.do-default-config.changed
tracer: set flag charms.openstack.do-default-update-status
tracer: set flag charms.openstack.do-default-upgrade-charm
tracer: set flag config.changed
tracer: set flag config.changed.source
tracer: set flag config.set.source
tracer: set flag leadership.is_leader
2019-12-03 15:07:11 DEBUG juju-log tracer: hooks phase, 0 handlers queued
2019-12-03 15:07:11 DEBUG juju-log tracer>
tracer: main dispatch loop, 3 handlers queued
tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:109:broken:certificates
tracer: ++ queue handler reactive/layer_openstack.py:26:default_config_changed
tracer: ++ queue handler reactive/layer_openstack.py:9:default_install
2019-12-03 15:07:11 INFO juju-log Invoking reactive handler: reactive/layer_openstack.py:9:default_install
2019-12-03 15:07:12 INFO juju-log Installing ['ubuntu-cloud-keyring'] with options: ['--option=Dpkg::Options::=--force-confold']
2019-12-03 15:07:12 DEBUG install Reading package lists...
2019-12-03 15:07:12 DEBUG install Building dependency tree...
2019-12-03 15:07:12 DEBUG install Reading state information...
2019-12-03 15:07:12 DEBUG install The following package was automatically installed and is no longer required:
2019-12-03 15:07:12 DEBUG install libfreetype6
2019-12-03 15:07:12 DEBUG install Use 'apt autoremove' to remove it.
2019-12-03 15:07:12 DEBUG install The following NEW packages will be installed:
2019-12-03 15:07:12 DEBUG install ubuntu-cloud-keyring
2019-12-03 15:07:12 DEBUG install 0 upgraded, 1 newly installed, 0 to remove and 42 not upgraded.
2019-12-03 15:07:12 DEBUG install Need to get 6008 B of archives.
2019-12-03 15:07:12 DEBUG install After this operation, 22.5 kB of additional disk space will be used.
2019-12-03 15:07:12 DEBUG install Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ubuntu-cloud-keyring all 2018.09.18.1~18.04.0 [6008 B]
2019-12-03 15:07:13 DEBUG install Fetched 6008 B in 0s (14.7 kB/s)
2019-12-03 15:07:13 DEBUG install Selecting previously unselected package ubuntu-cloud-keyring.
(Reading database ... 35333 files and directories currently installed.)
2019-12-03 15:07:13 DEBUG install Preparing to unpack .../ubuntu-cloud-keyring_2018.09.18.1~18.04.0_all.deb ...
2019-12-03 15:07:13 DEBUG install Unpacking ubuntu-cloud-keyring (2018.09.18.1~18.04.0) ...
2019-12-03 15:07:13 DEBUG install Setting up ubuntu-cloud-keyring (2018.09.18.1~18.04.0) ...
2019-12-03 15:07:15 DEBUG install Err:1 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/train InRelease
2019-12-03 15:07:15 DEBUG install 503 Service Unavailable [IP: 10.219.3.124 3128]
2019-12-03 15:07:15 DEBUG install Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
2019-12-03 15:07:15 DEBUG install Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
2019-12-03 15:07:15 DEBUG install Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
2019-12-03 15:07:15 DEBUG install Hit:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
2019-12-03 15:07:17 DEBUG install Reading package lists...
2019-12-03 15:07:17 DEBUG install W: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/train/InRelease 503 Service Unavailable [IP: 10.219.3.124 3128]
2019-12-03 15:07:17 DEBUG install W: Some index files failed to download. They have been ignored, or old ones used instead.

At this point execution should have stopped, but instead it continued with the in-distro packages, leading to a very confusing result.

2019-12-03 15:07:17 INFO juju-log Installing ['ovn-central'] with options: ['--option=Dpkg::Options::=--force-confold']
2019-12-03 15:07:18 DEBUG install Reading package lists...
2019-12-03 15:07:18 DEBUG install Building dependency tree...
2019-12-03 15:07:18 DEBUG install Reading state information...
2019-12-03 15:07:18 DEBUG install The following package was automatically installed and is no longer required:
2019-12-03 15:07:18 DEBUG install libfreetype6
2019-12-03 15:07:18 DEBUG install Use 'apt autoremove' to remove it.
2019-12-03 15:07:18 DEBUG install The following additional packages will be installed:
2019-12-03 15:07:18 DEBUG install libpython-stdlib libpython2.7-minimal libpython2.7-stdlib openvswitch-common
2019-12-03 15:07:18 DEBUG install openvswitch-switch ovn-common python python-minimal python-six python2.7
2019-12-03 15:07:18 DEBUG install python2.7-minimal
2019-12-03 15:07:18 DEBUG install Suggested packages:
2019-12-03 15:07:18 DEBUG install openvswitch-doc python-doc python-tk python2.7-doc binfmt-support
2019-12-03 15:07:18 DEBUG install The following NEW packages will be installed:
2019-12-03 15:07:18 DEBUG install libpython-stdlib libpython2.7-minimal libpython2.7-stdlib openvswitch-common
2019-12-03 15:07:18 DEBUG install openvswitch-switch ovn-central ovn-common python python-minimal python-six
2019-12-03 15:07:18 DEBUG install python2.7 python2.7-minimal
2019-12-03 15:07:18 DEBUG install 0 upgraded, 12 newly installed, 0 to remove and 42 not upgraded.
2019-12-03 15:07:18 DEBUG install Need to get 7821 kB of archives.
2019-12-03 15:07:18 DEBUG install After this operation, 33.2 MB of additional disk space will be used.
2019-12-03 15:07:18 DEBUG install Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython2.7-minimal amd64 2.7.15-4ubuntu4~18.04.2 [336 kB]
2019-12-03 15:07:18 DEBUG install Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python2.7-minimal amd64 2.7.15-4ubuntu4~18.04.2 [1296 kB]
2019-12-03 15:07:19 DEBUG install Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-minimal amd64 2.7.15~rc1-1 [28.1 kB]
2019-12-03 15:07:19 DEBUG install Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython2.7-stdlib amd64 2.7.15-4ubuntu4~18.04.2 [1916 kB]
2019-12-03 15:07:19 DEBUG install Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python2.7 amd64 2.7.15-4ubuntu4~18.04.2 [239 kB]
2019-12-03 15:07:19 DEBUG install Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpython-stdlib amd64 2.7.15~rc1-1 [7620 B]
2019-12-03 15:07:19 DEBUG install Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 python amd64 2.7.15~rc1-1 [140 kB]
2019-12-03 15:07:19 DEBUG install Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-six all 1.11.0-2 [11.3 kB]
2019-12-03 15:07:20 DEBUG install Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openvswitch-common amd64 2.9.2-0ubuntu0.18.04.3 [814 kB]
2019-12-03 15:07:20 DEBUG install Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 openvswitch-switch amd64 2.9.2-0ubuntu0.18.04.3 [1506 kB]
2019-12-03 15:07:20 DEBUG install Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 ovn-common amd64 2.9.2-0ubuntu0.18.04.3 [821 kB]
2019-12-03 15:07:20 DEBUG install Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 ovn-central amd64 2.9.2-0ubuntu0.18.04.3 [706 kB]