`juju bootstrap` won't give up when failing to connect to ubuntu archive

[Jordan Barrett]

I tried to bootstrap a new lxd controller, but it got stuck on the `Running machine configuration script...` step. By SSHing into the controller and looking at /var/log/cloud-init-output.log, I determined that it was:
1. Attempting to install some packages
2. Failing to connect to archive.ubuntu.com
3. Repeating steps 1-2 ad infinitum.

Here is the log:

```
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  msr-tools
The following NEW packages will be installed:
  cpu-checker msr-tools
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 16.9 kB of archives.
After this operation, 66.6 kB of additional disk space will be used.
Err:1 http://archive.ubuntu.com/ubuntu focal/main amd64 msr-tools amd64 1.3-3
  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::15). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out
Err:2 http://archive.ubuntu.com/ubuntu focal/main amd64 cpu-checker amd64 0.7-1.1
  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::15). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/msr-tools/msr-tools_1.3-3_amd64.deb Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::15). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/cpu-checker/cpu-checker_0.7-1.1_amd64.deb Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::15). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1562::18). - connect (101: Network is unreachable)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
```

... and then it would just try the same thing again. Surely it should give up at some point (probably after the first failed connection)?

[Jordan Barrett (barrettj12)]

Apparently there is supposed to be a 20 minute timeout, but I observed mine running for much longer than that. Even then, 20 minutes might be too long.

[Jordan Barrett (barrettj12)]

To be clear, there are two separate issues here:
1. On my machine, Juju can't seem to connect to http://archive.ubuntu.com/, and hence it can't finish the bootstrapping process.
2. When Juju fails to connect, it just keeps retrying indefinitely, and gives no user feedback.

[Jordan Barrett (barrettj12)]

Issue 1 was caused by my firewall. I've now fixed it but maybe it would be possible for Juju to notify the user if there are firewall issues blocking a controller?

Issue 2 still stands.

[Jordan Barrett (barrettj12)]

Found the issue. The cloud-init script includes the following function:
---------------------------------------------------------
function package_manager_loop {
    local rc=
    while true; do
        if ($*); then
                return 0
        else
                rc=$?
        fi
        if [ $rc -eq 100 ]; then
                sleep 10s
                continue
        fi
        return $rc
    done
}
---------------------------------------------------------

which is invoked like this:
---------------------------------------------------------
package_manager_loop apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --assume-yes --quiet install cpu-checker
---------------------------------------------------------

If apt-get can't connect to the Ubuntu archive, then this will run in an infinite loop. We should probably limit the number of retries (if we haven't connected after 5-10 retries, we're probably unable to).

[Jordan Barrett (barrettj12)]

https://github.com/juju/packaging/pull/18