subiquity curtin apt-get update fails

[MercSniper]

CI/CD pipeline was functional until I switch over to using 20.04.4 ISO. When it gets to Curtin in-target apt-get update, it crashes with error code 100. Google has not been useful in providing an answer. I checked NTP on the ESXi host, and it is synced. Attached is the subiquity crash report.

[Dan Bungert (dbungert)]

Thanks @MercSniper.

Sadly the crash file doesn't have the curtin log, and the details on what apt is unhappy about will be there. Are you able to share the contents of /var/log/curtin/install.log?

[MercSniper (mercsniper)]

/var/log/curtin does not exist at crash prompt. Did a redirect on journalctl to get the attached.

[Dan Bungert (dbungert)]

Mar 04 19:12:56 ubuntu-server subiquity_log.2013[2214]: E: Failed to fetch http://mirror.us-midwest-1.nexcess.net/ubuntu/dists/focal-backports/universe/i18n/Translation-en.xz File has unexpected size (15168 != 15416). Mirror sync in progress? [IP: 208.69.120.125 80]

Would you retry with a different mirror?

[MercSniper (mercsniper)]

Seems to be it. I've been fighting this all day. Can't believe the mirror hadn't synced in the last 6+ hours.

[Michael Hudson-Doyle (mwhudson)]

We need to do a better job of surfacing "your mirror is broken" errors to
the user.

On Sat, 5 Mar 2022, 13:50 MercSniper, <email address hidden> wrote:

> Seems to be it. I've been fighting this all day. Can't believe the
> mirror hadn't synced in the last 6+ hours.
>
> --
> You received this bug notification because you are subscribed to
> subiquity in Ubuntu.
> https://bugs.launchpad.net/bugs/1963725
>
> Title:
> subiquity curtin apt-get update fails
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/subiquity/+bug/1963725/+subscriptions
>
>

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paul Lambert (paul7l)]

I believe I have if not the same but a similar issue. OS is Ubuntu 22.04.
Logs attached.

Code "100" can be for permission denied.

I noticed also, that "geoip" setting is ignored, even if set to "false" is connecting to geoip.ubuntu.com.

[Dan Bungert (dbungert)]

> I noticed also, that "geoip" setting is ignored, even if set to "false" is connecting to geoip.ubuntu.com.

Multiple things ask for geoip results (timezone looks at it also), turning geoip off in the apt section only affects mirror selection.
I see the value in a top-level disable of geoip checks.

[Ling Zhou (zhouling12)]

we got the same issue with Ubuntu 22.04. Please see the attached for details. The connection failure to geoip.ubuntu.com caused the system stop installing. this issue caused our unattended deployment of ubuntu 2204 to Lenovo server not able to continue, we put a high priority to it, please help us to resolve it ASAP. Thank you very much!

[Dan Bungert (dbungert)]

@zhouling12 - The geoip tracebacks in the logs are just warnings, I know they look more serious. The bad one is at the end of the file, where `apt-get update` has failed. I would check that the network and mirror look OK.

[Ling Zhou (zhouling12)]

@Dan, Pls see the attached. When I run the command "apt-get update" in tty2, it worked properly, get "0" return value, but the issue still exist.

[Jimmy (sujj5)]

I can reproduce it on ubuntu 20.04.5.System will crash at curtin command in-target after configuring apt during installation.This issue occured when server connect network and the ip can access Internet.
Upload sos report log.
Please help to resolve it ASAP,thanks.

[boza (bozakov)]

Hitting the same error on on Ubuntu 20.04.5. Running apt-get from the crash prompt works fine. Is there any workaround for this?

[Cavelle Benjamin (thecb4)]

20.04.1, 20.04.5, and 22.10 all produce the same error for me.

[L1 (l0th1)]

Dunno if it helps, but I'm trying to create a custom autoinstall ubuntu 22.04.2 image and I'm experiencing this error. Inside the shell, by running

curtin in-target -t ... -- apt-get update

I get a file not found and permission denied errors on /cdrom/dists/jammy Packages files. When I look at the CDROM content there is only the Packages.gz file and the apt user is not allowed to read the file anyway.

I'm fixing it (brutally) by unpacking the Packages.gz files and by allowing _apt to read it

```
for f in $BUILD/dists/jammy/*/*/Packages.gz; do
  if [[ ! -f ${f/.gz/} ]]; then
    gunzip -k $f
    chmod o+r $f
    chmod o+r ${f/.gz/}
  fi
done
```

The custom ISO is built with the parameters obtained by running xorriso -indev ubuntu-22.04.2-live-server-amd64.iso report_el_torito as_mkisofs. The autoinstall user-data doesn't configure apt in any way.

[Vito Maria Domenico Nitti (vnitti)]

Still having this issue on 20.04.06.
The same ISO may/may not work on different PC and/or different countries.

[Noel Ashford (nashford77)]

Hitting the same i think in 23.10? Can't get autoinstall to work at all.

 Feb 17 19:51:42 ubuntu-server subiquity_log.1914[2391]: Running command ['umount', '/tmp/tmpje5tgw44/mount/proc'] with allowed return codes [0] (capture=False)
 Feb 17 19:51:42 ubuntu-server subiquity_log.1914[2391]: Running command ['mount', '--make-private', '/tmp/tmpje5tgw44/mount/dev'] with allowed return codes [0] (capture=False)
 Feb 17 19:51:42 ubuntu-server subiquity_log.1914[2391]: Running command ['umount', '/tmp/tmpje5tgw44/mount/dev'] with allowed return codes [0] (capture=False)
 Feb 17 19:51:42 ubuntu-server subiquity_log.1914[2391]: finish: cmd-in-target: FAIL: curtin command in-target
 Feb 17 19:51:42 ubuntu-server curtin_event.1914.3[2391]: finish: cmd-in-target: FAIL: curtin command in-target
 Feb 17 19:51:42 ubuntu-server systemd[1]: run-u30.service: Main process exited, code=exited, status=100/n/a
 Feb 17 19:51:42 ubuntu-server systemd[1]: run-u30.service: Failed with result 'exit-code'.
 Feb 17 19:51:42 ubuntu-server systemd[1]: run-u30.service: Consumed 12.156s CPU time.
 Feb 17 19:51:42 ubuntu-server subiquity_event.1914[1914]: subiquity/Install/install/configure_apt: Command '['systemd-run', '--wait', '--same-dir', '--property', 'SyslogIdentifier=subiquity_log.1914', '--property', 'PrivateMounts=yes', '--setenv', 'PATH=/snap/subiquity/5270/bin:/snap/subiquity/5270/usr/bin:/snap/subiquity/5270/usr/sbin:/snap/subiquity/5270/usr/bin:/snap/subiquity/5270/sbin:/snap/subiquity/5270/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/subiquity/5270/bin:/snap/subiquity/5270/sbin', '--setenv', 'PYTHONPATH=:/snap/subiquity/5270/lib/python3.10/site-packages', '--setenv', 'PYTHON=/snap/subiquity/5270/usr/bin/python3.10', '--setenv', 'SNAP=/snap/subiquity/5270', '--', '/snap/subiquity/5270/usr/bin/python3.10', '-m', 'curtin', '--showtrace', '-vvv', '--set', 'json:reporting={"subiquity": {"type": "journald", "identifier": "curtin_event.1914.3"}}', 'in-target', '-t', '/tmp/tmpje5tgw44/mount', '--', 'apt-get', 'update']' returned non-zero exit status 100.
 Feb 17 19:51:42 ubuntu-server subiquity_event.1914[1914]: subiquity/Install/install: Command '['systemd-run', '--wait', '--same-dir', '--property', 'SyslogIdentifier=subiquity_log.1914', '--property', 'PrivateMounts=yes', '--setenv', 'PATH=/snap/subiquity/5270/bin:/snap/subiquity/5270/usr/bin:/snap/subiquity/5270/usr/sbin:/snap/subiquity/5270/usr/bin:/snap/subiquity/5270/sbin:/snap/subiquity/5270/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/subiquity/5270/bin:/snap/subiquity/5270/sbin', '--setenv', 'PYTHONPATH=:/snap/subiquity/5270/lib/python3.10/site-packages', '--setenv', 'PYTHON=/snap/subiquity/5270/usr/bin/python3.10', '--setenv', 'SNAP=/snap/subiquity/5270', '--', '/snap/subiquity/5270/usr/bin/python3.10', '-m', 'curtin', '--showtrace', '-vvv', '--set', 'json:reporting={"subiquity": {"type": "journald", "identifier": "curtin_event.1914.3"}}', 'in-target', '-t', '/tmp/tmpje5tgw44/mount', '--', 'apt-get', 'update']' returned non-zero exit status 100.
 Feb 17 19:51:42 ubuntu-server sub...