Comment 4 for bug 2034656

Thanks for following up and commenting.

Currently, the largest request spike seems to be the 0600 - 0700 window. On Xenial at least, while it spreads downloads over a larger window (throughout the day), they are applied within that window, IIUC. That would be in `/lib/systemd/system/apt-daily-upgrade.timer` which then starts `apt-daily-upgrade.service`. An example on a Xenial host we have available:

| [hloeung@nernst ~]$ systemctl status apt-daily-upgrade.service
| ● apt-daily-upgrade.service - Daily apt upgrade and clean activities
| Loaded: loaded (/lib/systemd/system/apt-daily-upgrade.service; static; vendor preset: enabled)
| Active: inactive (dead) since Tue 2023-09-19 06:27:06 UTC; 3h 30min ago
| Docs: man:apt(8)
| Process: 14530 ExecStart=/usr/lib/apt/apt.systemd.daily install (code=exited, status=0/SUCCESS)
| Main PID: 14530 (code=exited, status=0/SUCCESS)
|
| Sep 19 06:26:57 nernst systemd[1]: Starting Daily apt upgrade and clean activities...
| Sep 19 06:27:05 nernst apt.systemd.daily[14530]: An error occurred: 503 Service Unavailable
| Sep 19 06:27:05 nernst apt.systemd.daily[14530]: The URI https://esm.ubuntu.com/infra/ubuntu/pool/main/b/binutils/binutils_2.26.1-1ubuntu1~16.04.8+esm7_amd64.deb failed to download, aborting
| Sep 19 06:27:06 nernst systemd[1]: Started Daily apt upgrade and clean activities.

Yeah, the 503 is due to the surge / request spike causing the service to not cope with the load, hence this bug.

I've added more metrics to our grafana dashboard to see in particular what release and pocket so we can shift our focus there. Will report back.