Currently, the largest request spike seems to be the 0600 - 0700 window. On Xenial at least, while it spreads downloads over a larger window (throughout the day), they are applied within that window, IIUC. That would be in `/lib/systemd/system/apt-daily-upgrade.timer` which then starts `apt-daily-upgrade.service`. An example on a Xenial host we have available:
| [hloeung@nernst ~]$ systemctl status apt-daily-upgrade.service
| ● apt-daily-upgrade.service - Daily apt upgrade and clean activities
| Loaded: loaded (/lib/systemd/system/apt-daily-upgrade.service; static; vendor preset: enabled)
| Active: inactive (dead) since Tue 2023-09-19 06:27:06 UTC; 3h 30min ago
| Docs: man:apt(8)
| Process: 14530 ExecStart=/usr/lib/apt/apt.systemd.daily install (code=exited, status=0/SUCCESS)
| Main PID: 14530 (code=exited, status=0/SUCCESS)
|
| Sep 19 06:26:57 nernst systemd[1]: Starting Daily apt upgrade and clean activities...
| Sep 19 06:27:05 nernst apt.systemd.daily[14530]: An error occurred: 503 Service Unavailable
| Sep 19 06:27:05 nernst apt.systemd.daily[14530]: The URI https://esm.ubuntu.com/infra/ubuntu/pool/main/b/binutils/binutils_2.26.1-1ubuntu1~16.04.8+esm7_amd64.deb failed to download, aborting
| Sep 19 06:27:06 nernst systemd[1]: Started Daily apt upgrade and clean activities.
Yeah, the 503 is due to the surge / request spike causing the service to not cope with the load, hence this bug.
I've added more metrics to our grafana dashboard to see in particular what release and pocket so we can shift our focus there. Will report back.
Thanks for following up and commenting.
Currently, the largest request spike seems to be the 0600 - 0700 window. On Xenial at least, while it spreads downloads over a larger window (throughout the day), they are applied within that window, IIUC. That would be in `/lib/systemd/ system/ apt-daily- upgrade. timer` which then starts `apt-daily- upgrade. service` . An example on a Xenial host we have available:
| [hloeung@nernst ~]$ systemctl status apt-daily- upgrade. service upgrade. service - Daily apt upgrade and clean activities system/ apt-daily- upgrade. service; static; vendor preset: enabled) /usr/lib/ apt/apt. systemd. daily install (code=exited, status=0/SUCCESS) daily[14530] : An error occurred: 503 Service Unavailable daily[14530] : The URI https:/ /esm.ubuntu. com/infra/ ubuntu/ pool/main/ b/binutils/ binutils_ 2.26.1- 1ubuntu1~ 16.04.8+ esm7_amd64. deb failed to download, aborting
| ● apt-daily-
| Loaded: loaded (/lib/systemd/
| Active: inactive (dead) since Tue 2023-09-19 06:27:06 UTC; 3h 30min ago
| Docs: man:apt(8)
| Process: 14530 ExecStart=
| Main PID: 14530 (code=exited, status=0/SUCCESS)
|
| Sep 19 06:26:57 nernst systemd[1]: Starting Daily apt upgrade and clean activities...
| Sep 19 06:27:05 nernst apt.systemd.
| Sep 19 06:27:05 nernst apt.systemd.
| Sep 19 06:27:06 nernst systemd[1]: Started Daily apt upgrade and clean activities.
Yeah, the 503 is due to the surge / request spike causing the service to not cope with the load, hence this bug.
I've added more metrics to our grafana dashboard to see in particular what release and pocket so we can shift our focus there. Will report back.