20.04 autoinstall restores apt config before security updates

[kfsone]

Using the 20.04.1 live-server iso and providing it with a user-data as follows:

    #cloud-config
    autoinstall:
      version: 1
      proxy: http://apt-cacher-ng:3142/
      refresh-installer: { "update": no }
    storage:
      layout:
        name: direct

and then booting with kvm:

    $ truncate -s 10G hd.img ; kvm -m 1024 -drive file=hd.img,format=raw,cache=none,if=virtio \
      -cdrom ubuntu-20.0.4-live-server.iso

runs thru the autoinstall as expected, but logs this:

> start: subiquity/InstallProgress/install/postinstall/restore_apt_config: restoring apt configuration
> finish: subiquity/InstallProgress/install/postinstall/restore_apt_config: SUCCESS: restoring apt configuration
> finish: subiquity/InstallProcess/install/postinstall: SUCCESS: final system configuration
> start: subiquity/InstallProgress/install/run_unattended_upgrades: downloading and installing security updates

A simple tcpdump "host apt-cacher-ng and port 3142" shows that the initial install calls to apt use the proxy, but the final security updates go out to the internet.

Expected behavior: the apt-proxy would be used for installing the final security updates, and/or the download and install of security updates could be disabled via `autoinstall`.

[Michael Hudson-Doyle (mwhudson)]

Hmm this is surprising, restoring the apt configuration should just be removing the packages on the ISO as a source, the proxy should still be configured after it runs. I'll have to try on my machine and see if I can reproduce.

[Nate Childers (nate-duke)]

This appears to have stuck around long enough to still be an issue in 22.04

[Mabine (mabine)]

I have exactly the same issue with automated installs of Ubuntu 22.04 LTS server. Does anybody know if there is maybe a workaround?

[Kodiak Firesmith (kfiresmith-whoi)]

+1 via 00336319, but maybe worse in that regular Subiquity installations done interactively are now failing for me in a proxied environment for 22.04 LTS.

This does block us on moving U22.04. We were not affected in 20.04.

[Dan Bungert (dbungert)]

A fix for this has been merged to the Subiquity snap beta channel.
If you would like to help test this fix, please do the following:
* Launch your installer ISO as you normally would
* During the install, when the network is configured enough that snap
  should be able to reach the snap store, please go to a command line.
  The easiest way to do this should be to use the "Enter Shell" option
  in the "Help Menu"
* Run command `sudo snap refresh --beta subiquity`
* Complete your install as you would normally

Autoinstall users would need to use:
  refresh-installer:
    update: yes
    channel: beta

This build differs from the released version with the following fixes:
https://github.com/canonical/probert/pull/116/commits/e4115e837ed9e48b714b17f7a1c57cc62e5f7abe
https://git.launchpad.net/curtin/commit/?id=15ecdeab1e3a41069e84ddcac42d8bd1747e0382

[Kodiak Firesmith (kfiresmith-whoi)]

Just got a chance to test this and it looks like trying to use the beta channel from a network that must use the proxy set in auto-install causes the entire installation to error out with an unknown error and die.

It's very hard to retrieve logs about this from a system that is broken and off the network so I have no idea how to get them into the open case I have with Canonical.

[Kodiak Firesmith (kfiresmith-whoi)]

I should note that I'm trying to use static networking and unfortunately since subiquity fails to honor DNS settings set in the kernel command line (eg: ip=10.10.10.100::10.10.10.1:255.255.255.0:foo.college.edu:::10.10.10.200:10.10.10.201), the installer seems to die on applying network config with a ton of tracebacks.
'ERROR subiquitycore.controller.network:223 unset_link_flags failed for ens160'
Then a traceback, then a failed attempt to check for snap updates, internal server error on the unix socket for snapd, then more failures.

[Kodiak Firesmith (kfiresmith-whoi)]

Tried this out a few more times, and it looks like the build makes it through to installing security updates, and the installer fails at that point.

It looks like it doesn't preserve the proxy settings for the security update. One more reason why I wish Canonical were more flexible about making build-time security updates mandatory. If this were not the case, we could finish out the build, and have Ansible take it from there.