Apt updates that are uniformly spread across all timezones, with predictable application windows

Bug #1686470 reported by Dimitri John Ledkov on 2017-04-26
46
This bug affects 7 people
Affects Status Importance Assigned to Milestone
apt (Debian)
Fix Released
Unknown
apt (Ubuntu)
Status tracked in Artful
Xenial
High
Dimitri John Ledkov
Yakkety
High
Dimitri John Ledkov
Zesty
High
Dimitri John Ledkov
Artful
High
Dimitri John Ledkov
unattended-upgrades (Debian)
Fix Committed
Unknown
unattended-upgrades (Ubuntu)
Status tracked in Artful
Xenial
High
Dimitri John Ledkov
Yakkety
High
Dimitri John Ledkov
Zesty
High
Dimitri John Ledkov
Artful
High
Dimitri John Ledkov

Bug Description

[ Impact ]

 * unattended-upgrades are enabled by default in Ubuntu 16.04 and later

 * Currently the following three things happen as a monolithic event:
   - metadata updates: apt update
   - download of updates: apt upgrade --download-only
   - application of updates: apt upgrade

 * For the long running instances, all of the above happens at random
   times throughout the day.

 * If systems were poweredoff / suspended, this happens on boot / resume

 * End-users would like to have predictable timing, and control over when
   the updates happen.

Considering all of the above, the following new behavior is proposed which should address all concerns in question. It combines all the desired properties from both end-user and mirror perspectives.

[ Proposed Default Behavior ]

 * Decouple unattended-upgrades application, from apt update

 * apt update:
   - shall be a systemd timer based unit, triggered every 12h with a
     random delay of 12h, therefore executed randomly twice a day.
   - if unattened-upgrades (default on), or download-upgreadaeble-packages
     are enabled, it should result in updates being downloaded aka
     `apt upgrade --download-only`

 * unattended-upgrades:
   - shall be a separate systemd timer based unit triggered at 6am local
     time with a random delay of 1h, therefore executed between 6am and
     7am local time.

 * On boot / resume:
   - if we have missed one, or more, apt update timers,
     apt update / download upgrades / unattended-upgrade will happen in
     sequence. This may result in mirror spikes, but we do want to secure
     cold/stale-booted systems as soon as possible.

[Test Case]

 * Run system for more than 24h, and check that apt updates were
   automatically executed twice.

 * Check that unattended upgrades were triggered to be applied at
   6am..7am window, if any.

 * Poweroff the machine over the period when apt-get update was
   scheduled, poweron and observe that apt-get update / download /
   unattended upgrade are all performed on boot.

[Regression Potential]

 * The newly proposed behavior is a mix of Pre-xenial behavior of "do
   everything at 6am..6:30am window" and the xenial+ behavior of "do
   everything at random times throughout the day". If there are specific
   deployments that rely on the previous types of behaviour they will be
   able to adjust manually the systemd timers with the overrides to be
   executed exactly as they wish; or match the .0 release behaviour that
   they preffer.

 * If timers behavior is coded wrongly the proposed behaviour might not be
   executed as intended, thus requiring further SRUs to bring us in-line
   with the great expectations.

[Other Info]

  * Related bug reports and history:
    - bug #1615482
    - bug #1554848

Changed in apt (Ubuntu Xenial):
status: New → Triaged
Changed in apt (Ubuntu Yakkety):
status: New → Triaged
Changed in apt (Ubuntu Zesty):
status: New → Triaged
Changed in apt (Ubuntu Artful):
status: New → Triaged
Changed in apt (Ubuntu Xenial):
importance: Undecided → High
Changed in apt (Ubuntu Yakkety):
importance: Undecided → High
Changed in apt (Ubuntu Zesty):
importance: Undecided → High
Changed in apt (Ubuntu Artful):
importance: Undecided → High
description: updated
description: updated
Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Triaged
Changed in unattended-upgrades (Ubuntu Yakkety):
status: New → Triaged
Changed in unattended-upgrades (Ubuntu Zesty):
status: New → Triaged
Changed in unattended-upgrades (Ubuntu Artful):
status: New → Triaged
Changed in unattended-upgrades (Ubuntu Xenial):
importance: Undecided → High
Changed in unattended-upgrades (Ubuntu Yakkety):
importance: Undecided → High
Changed in unattended-upgrades (Ubuntu Zesty):
importance: Undecided → High
Changed in unattended-upgrades (Ubuntu Artful):
importance: Undecided → High
Julian Andres Klode (juliank) wrote :

I propose we simply split up the apt daily script and run that in two timers (one for fetching, the other for potentially destructive stuff like u-u and clean).

/usr/lib/apt/apt.systemd.daily.common.sh
  The shell functions used by both scripts (top part of script), plus the variables
  (so right until the # update package lists part).

/usr/lib/apt/apt.systemd.daily
  The middle part of the current script, until u-u

/usr/lib/apt/apt.systemd.destructive-daily

  The final part of the script handling u-u, clean, and autoclean

Julian Andres Klode (juliank) wrote :

I just do an if now instead of three files, and here is the implementation:

https://github.com/Debian/apt/compare/master...julian-klode:lp1686470?diff=split&expand=1&name=lp1686470

Julian Andres Klode (juliank) wrote :

(be?field.comment=(be

Julian Andres Klode (juliank) wrote :

Uploaded 1.4.1ubuntu1 to artful which introduces the apt-daily-upgrade timer and service matching the proposed default behavior.

Changed in apt (Ubuntu Artful):
status: Triaged → Fix Committed
Julian Andres Klode (juliank) wrote :

Hmm, seems I forgot to change the downloading to run when unattended-upgrades is configured. But this seems like a bad choice anyway with the current implementation.

Maybe let's do something like

    if which unattended-upgrade >/dev/null 2>&1 && check_stamp $DOWNLOAD_UPGRADEABLE_STAMP $UnattendedUpgradeInterval; then
 if unattended-upgrade -d $XUUPOPT; then
     update_stamp $UPGRADE_STAMP
     debug_echo "unattended-upgrade -d (success)"
 else
     debug_echo "unattended-upgrade -d (error)"
 fi
    else
 debug_echo "unattended-upgrade -d (not run)"
    fi

That is, don't download the upgrades with apt -d dist-upgrade, use unattended-upgrade -d. This uses the same stamp file as the current list-upgrade option, but compares it against the unattended-upgrades section, and contains the unattended-upgrade body, with -d appended.

Julian Andres Klode (juliank) wrote :

Fixed in ubuntu2.

Steve Langasek (vorlon) wrote :

Why should apt update be run twice a day if the updates are only applied to the system once a day? This would be:
- a no-op for any metadata that hasn't changed in the preceding 12h
- a wasteful, redundant download for metadata that *has* changed in the preceding 12h
- (in rare cases) very wasteful if two SRUs are published back-to-back in the same day, since the first will be downloaded but not applied.

If the purpose is to ensure the updates being applied are never more than 12 hours out of date, that can be achieved with a single apt update scheduled in the 12h before 6am machine time.

If the purpose is to ensure downloads are spread more evenly around the clock, that is better achieved by doing one apt update per day, with a 24h smear, instead of two per day. (And btw, we probably want locking between the two jobs, so that we don't try to run an 'apt update' /while/ unattended-upgrades are being applied, but instead wait for unattended-upgrades to finish)

If your goal is to try to achieve some sort of balance between these goals, that at least makes sense why the design would be this way; but I think overall it's better to just do a single 24h smear and not try to compromise in a way that increases overall load on the mirrors with no major benefit to the users.

Can you please also document this design on wiki.ubuntu.com, so that we have better history tracking of any further changes (vs. edits to a bug description) and can more easily refer to it rather than pointing to an (at some point) ancient bug?

Dimitri John Ledkov (xnox) wrote :

Justifications for twice a day:

 - to get motd updated & get upgrade popups on the desktop.
 - to have a better chance of having critical security fixes downloaded, before the next upgrade window.
 - in case of intermittent network losses, having two download attempts is better than one.
 - machines that are off overnight (office workers), with twice daily download have a better chance of downloading updates during office hours; instead of effectively always on boot.
 - i like it

Julian Andres Klode (juliank) wrote :

APT maintains a multitude of internal time stamp files preventing updates occuring more than once per day.

tags: added: patch
Julian Andres Klode (juliank) wrote :

I think the "locking" is already done by systemd. When units specify an ordering, it seems that one unit will only start when the other has shut down:

"Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up"

On Wed, Apr 26, 2017 at 08:39:36PM -0000, Julian Andres Klode wrote:
> I think the "locking" is already done by systemd. When units specify an
> ordering, it seems that one unit will only start when the other has shut
> down:

> "Given two units with any ordering dependency between them, if one unit
> is shut down and the other is started up, the shutdown is ordered before
> the start-up"

That language sounds very general; it's probably worth having someone
explicitly check this for our case before committing to rely on systemd :)

summary: - Apt updates that are uniformally spread across all timezones, with
+ Apt updates that are uniformly spread across all timezones, with
predictable application windows
Julian Andres Klode (juliank) wrote :

pabs just mentioned in #debian-apt that we could probably drop the ConditionACPower for the downloading service. I'm not sure why that was originally added, if only to protect upgrades or to save power when downloading, if anyone has an opinion, let me know.

Julian Andres Klode (juliank) wrote :

Re systemd conflict handling, I can ask systemd ML tomorrow.

On Wed, Apr 26, 2017 at 11:20:47PM -0000, Julian Andres Klode wrote:
> pabs just mentioned in #debian-apt that we could probably drop the
> ConditionACPower for the downloading service. I'm not sure why that was
> originally added, if only to protect upgrades or to save power when
> downloading, if anyone has an opinion, let me know.

Agreed, it should be ok to drop this. I believe the original rationale is
that you don't want to *apply* an upgrade while on battery, since running
out of power during the upgrade is Very Bad. But downloads should be ok.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.4.1ubuntu2

---------------
apt (1.4.1ubuntu2) artful; urgency=medium

  * Run unattended-upgrade -d in download part (LP: #1686470)

apt (1.4.1ubuntu1) artful; urgency=medium

  * Split apt-daily timer into two (LP: #1686470)

 -- Julian Andres Klode <email address hidden> Wed, 26 Apr 2017 21:41:05 +0200

Changed in apt (Ubuntu Artful):
status: Fix Committed → Fix Released
Julian Andres Klode (juliank) wrote :

I asked the systemd mailing list for some assistance on keeping the units from interfering with each other:

https://lists.freedesktop.org/archives/systemd-devel/2017-April/038772.html

Maybe they have a good solution to that issue.

Changed in apt (Debian):
status: Unknown → New
Julian Andres Klode (juliank) wrote :

OK, so 1.4.4 and friends actually enable debugging mode of unattended-upgrade instead of a download-only mode. --dry-run is documented as only doing downloads, but it also simulates the install and does verbose logging, so that's not useful.

In 1.4.6, I modified apt.systemd.daily to check unattended-upgrade --help for "download-only", and use that if it exists. This means that with 1.4.6, everything should work once this is implemented in unattended-upgrades.

I attached an early patch for unattended-upgrades (against the upstream version), there might be issues with it other than missing translations for --help.

Changed in unattended-upgrades (Debian):
status: Unknown → New
Changed in apt (Debian):
status: New → Fix Released
Łukasz Zemczak (sil2100) wrote :

Hello! I see uploads with fixes to this bug in various stable release queues - seeing that there was still some discussion after the uploads were done, I wanted to ask - what is the state of the fixes that are uploaded to the SRU queues? Are those fixes that are good to go or is this still in progress?

Julian Andres Klode (juliank) wrote :

Ahem, yes, sorry, I have to update those with the fix from 1.4.6. We also need updates for unattended upgrade that adds a --download-only option.

Julian Andres Klode (juliank) wrote :

Fixed updates are on their way, currently finishing CI tests.

Julian Andres Klode (juliank) wrote :

Uploaded 1.4.6~17.04.1, 1.3.9, and 1.2.24

Changed in apt (Ubuntu Xenial):
status: Triaged → In Progress
Changed in apt (Ubuntu Yakkety):
status: Triaged → In Progress
Changed in apt (Ubuntu Zesty):
status: Triaged → In Progress
Changed in unattended-upgrades (Ubuntu Artful):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in unattended-upgrades (Ubuntu Zesty):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in unattended-upgrades (Ubuntu Yakkety):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in unattended-upgrades (Ubuntu Xenial):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in apt (Ubuntu Artful):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in apt (Ubuntu Zesty):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in apt (Ubuntu Yakkety):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in apt (Ubuntu Xenial):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in unattended-upgrades (Debian):
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.