apt-get update triggers asynchronous task

[Scott Moser]

I have a work flow (curtin installation) that does something to the affect of:
 mount $DEVICE /target
 mount --bind /dev /target/dev
 chroot /target apt-get update
 umount /target/dev
 umount /target/

With xenial, I started having failures on the umount of /target/dev because of open file handles.
An lsof showed me that /usr/lib/update-notifier/update-motd-updates-available had open file handles on /dev/null and apt-check had handles on /dev/urandom.

I admit that part of this is my fault, because we do a selective apt-get update (only pulling deb sources, not deb-src). As a result, when /usr/lib/update-notifier/update-motd-updates-available there was possibly more work to be done then would normally have to be done... not sure on that.

For the record, what we do is in http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/curtin/util.py#L486 (apt_update method).

Related bugs:
 * bug 524674: apt-check hangs, preventing login via SSH
 * bug 1527710: apt-get update triggers asynchronous task
 * bug 1533243: preseeded installation fails on critical question: partman/unmount_active DISKS /dev/vda

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: update-notifier-common 3.164
ProcVersionSignature: User Name 4.3.0-2.11-generic 4.3.0
Uname: Linux 4.3.0-2-generic x86_64
ApportVersion: 2.19.3-0ubuntu2
Architecture: amd64
Date: Fri Dec 18 18:02:41 2015
Ec2AMI: ami-0000072f
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: None
Ec2Ramdisk: None
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: update-notifier
UpgradeStatus: No upgrade log present (probably fresh install)

[Steve Langasek (vorlon)]

Hi Scott,

I don't see what could be triggering this behavior. The update-notifier-common package includes hooks for the following:
 - apt
 - cron
 - /etc/kernel/postinst.d
 - update-motd

The only one of these that would be triggered by 'apt-get update' is the apt hook, which only does:

$ cat /etc/apt/apt.conf.d/99update-notifier
DPkg::Post-Invoke {"if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo > /var/lib/update-notifier/updates-available; fi "; };

And a) apt-get update doesn't invoke dpkg, and b) this should not be asynchronous in any way.

So, what is causing /usr/lib/update-notifier/update-motd-updates-available to be invoked in this environment?

[Christian Ehrhardt  (paelzer)]

Eventually the async apt-check is to avoid stalling humans, and there it makes total sense.

We could think to do something like

function checkit {
   *code of the check*
}

if [[ $- == *i* ]];
then
    checkit &
else
    checkit
fi

That would cause the check to be synchronous (as it was in the past), but only if it is a non interactive environment read "nobody directly cares about the sync delay"

Scott, let me know if you - as discussed on IRC in 2016 - could test a deb with that approach that I'd build for you.

[Christian Ehrhardt  (paelzer)]

deb for testing the approach, not very large so we can just attach it.

[Christian Ehrhardt  (paelzer)]

debdiff to show the suggested change, this is in no way an upload to be sponsored yet as it needs way more testing.
Just FYI so you see what was changed

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "xenial_update-notifier.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Michael Terry (mterry)]

Removed the sponsors team subscription per comment #5. Please re-add manually when you've tested it. :)

[Scott Moser (smoser)]

Its possibly something else that was doing this, but it was definitely happening.
We fixed the issue in curtin with:
 http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/revision/328

[Brian Murray (brian-murray)]

Wouldn't this change be causing the issue?

update-notifier (3.164) xenial; urgency=medium
  ...
  * debian/99update-notifier:
    - Now additionally triggers the an asynchronous background update of the
      cached info via update-motd-updates-available hooking into
      APT::Update::Post-Invoke-Success.

Here's the contents of 99update-notifier:

  1 DPkg::Post-Invoke {"if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi";};
  2 APT::Update::Post-Invoke-Success {"/usr/lib/update-notifier/update-motd-updates-available 2>/dev/null || true";};

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.165

---------------
update-notifier (3.165) xenial; urgency=medium

  * data/update-motd-updates-available
    - do not execute asynchronously as that causes problems with installers
      and generally produces unexpected behavior by default (LP: #1527710).
    - define NEEDS_CHECK to avoid unintentionally using value from inherrited
      environment.

 -- Scott Moser <email address hidden> Wed, 27 Jan 2016 17:04:44 -0500

[Scott Moser (smoser)]

In order to fix bug 525674, Christian did the following things in 3.164:
 a.) made /etc/update-motd.d/90-updates-available simply 'cat' a
     file /var/lib/update-notifier/updates-available
 b.) modified /usr/lib/update-notifier/update-motd-updates-available to
     background its execution of /usr/lib/update-notifier/apt-check
     which populates /var/lib/update-notifier/updates-available
 c.) added /etc/apt/apt.conf.d/99update-notifier to run
     /etc/apt.conf.d/99update-notifier so that
     /usr/lib/update-notifier/update-motd-updates-available is run
     each time 'apt-get update' finishes successfully.

The result of 'a' is that logins no longer trigger or block on execution
of a fairly slow and IO/CPU entensive task. That is wonderful and does a
good job of fixing bug 525674 and not triggering unnecessary activity on a
system simply as a result of a user logging in.

The result of 'c' is that is that running:
  apt-get update
populates keeps /var/lib/update-notifier/updates-available up to date.

The result of 'b' and 'c' is what caused the problem for me in curtin
and I believe for 'd-i' as described in bug 1533243.

My change was to un-do 'b'.

I can see a use case for a user wanting for 'apt-check' to run
backgrounded so that it doesn't make 'apt-get update' take any longer than
it already does. However, that should not be the default behavior as it
is quite unexpected. If someone wants to make this configurable in
the future I'm OK with that as long as it is done foreground by default.

An example of something that fails unexpectedly due to the backgrounding
is:
    mount-image-callback ubuntu-xenial.img --system-mounts -- \
       chroot _MOUNTPOINT_ apt-get update

Other things:
 * I'm not aware of anything in update-notifier-common that is running
   'apt-get update' via cron or otherwise keeping
   /var/lib/update-notifier/updates-available up to date other than
   when something manually runs 'apt-get update'
 * /var/lib/update-notifier/updates-available is not world readable
   which seems likely unintended as all it contains it it is something like.
     32 packages can be updated.
     0 updates are security updates.

[Scott Moser (smoser)]

I talked to juliank aboutn this, requesting for apt to maintain this data more itself (so that it wouldnt be out of date after 'apt-get update && apt-get dist-upgrade').

He pointed me at a debian apt feature request bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809441 .
If that bug gets fixed, we should hook into it and report status that way and drop apt-check.