excessive debconf use when triggered

Bug #1372673 reported by Colin Watson on 2014-09-22
This bug affects 171 people
Affects Status Importance Assigned to Milestone
man-db (Debian)
Fix Released
Unknown
man-db (Ubuntu)
High
Colin Watson
Lucid
High
Colin Watson
Precise
High
Colin Watson
Trusty
High
Colin Watson

Bug Description

SRU justification:

[Impact]

We see rather frequent hard-to-debug upgrade failures that amount to man-db's trigger failing in some way that has nothing to do with the mandb program itself, but rather in some kind of communication with the package management frontend. Almost all the open bugs on Ubuntu man-db come down to this in one way or another. The root cause is, I believe, that it is not really safe to use debconf from a dpkg trigger: while debconf itself behaves as if it were Essential, the debconf protocol is often mediated by various frontends with less stringent practices. It would certainly be a good idea to get man-db's trigger out of the loop here, as dpkg tends to run it at the end of an unpack phase when large numbers of packages are unconfigured, which is pretty much the worst case for having this work properly.

In the past I've tried to investigate why debconf fails in these situations. I've come to believe this is a wild goose chase, and that the common case of ordinary postinsts sourcing the debconf confmodule is much less likely to be a problem due to the nature of unpack/configure sequencing; I admit that I don't have proof of this, but having one fewer script using debconf is surely not going to make things worse, and since lots of unpack runs pull the man-db trigger it seems likely to improve things substantially.

The man-db trigger reads a single value from debconf to tell it whether to automatically update the manual page database (used by apropos and whatis). This is in an agreed location in debconf so that such things as package builders can save time by suppressing the database update. There is, however, no reason to read this value every time the trigger is activated; we can just cache it under /var/lib/man-db/ when man-db is configured and then do a much simpler file-level test in the trigger. I should have thought of this years ago.

To have the maximum benefit for upgraders, we should do what we can to ensure that they have a fixed version of man-db installed before the upgrade. I'd therefore like to apply this fix to all currently-supported releases.

[Test Case]

I don't have a reliable reproduction scenario, but I have two suggestions, which are in the sort of general vein of unit testing and integration testing respectively:

 * Install the new version of man-db and check that it causes /var/lib/man-db/auto-update to exist if and only if the debconf value for man-db/auto-update is true, and in turn that this still causes the database to be updated or not updated respectively when installing a package containing a manual page.
 * After installing the new version of man-db, upgrade to the next supported or development series using the upgrade tool of your choice.

[Regression Potential]

Installing packages that contain manual pages and running system upgrades should exercise this pretty thoroughly. Since in general this weakens the constraints on achieving successful upgrades, the main thing to watch out for would just be things like inverted logic that might cause the database not to be updated when it should be or vice versa.

Original report follows, imported from Debian bug http://bugs.debian.org/579075:

Package: man-db
Version: 2.5.7-2
Severity: wishlist

I noticed that man-db starts debconf when triggered. Given how
often triggers run, I think that should be optimised. Ie, move
the $1 = triggered test above the confmodule sourcing.

-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.32-3-686 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages man-db depends on:
ii bsdmainutils 8.0.10 collection of more utilities from
ii debconf [debconf-2.0] 1.5.32 Debian configuration management sy
ii dpkg 1.15.7.1 Debian package management system
ii groff-base 1.20.1-9 GNU troff text-formatting system (
ii libc6 2.10.2-6 Embedded GNU C Library: Shared lib
ii libgdbm3 1.8.3-9 GNU dbm database routines (runtime
ii zlib1g 1:1.2.3.4.dfsg-3 compression library - runtime

man-db recommends no packages.

Versions of packages man-db suggests:
ii 5.0.386.0~svn20100423r45407-0u Chromium browser
ii 2.30.2-1 Intuitive GNOME web browser
pn <none> (no description available)
ii 3.5.9-2 Web browser based on Firefox
ii 436-1 pager program similar to more
ii 0.5.2-4 WWW browsable pager with excellent

-- debconf information excluded

--
see shy jo

Colin Watson (cjwatson) on 2014-09-22
description: updated
Changed in man-db (Debian):
importance: Undecided → Unknown
status: New → Fix Released
Colin Watson (cjwatson) on 2014-09-23
Changed in man-db (Ubuntu Lucid):
importance: Undecided → High
Changed in man-db (Ubuntu):
importance: Undecided → High
Changed in man-db (Ubuntu Precise):
importance: Undecided → High
Changed in man-db (Ubuntu Trusty):
importance: Undecided → High
Changed in man-db (Ubuntu Precise):
status: New → In Progress
Changed in man-db (Ubuntu):
status: New → Fix Committed
Changed in man-db (Ubuntu Lucid):
status: New → In Progress
Changed in man-db (Ubuntu Trusty):
status: New → In Progress
Changed in man-db (Ubuntu):
assignee: nobody → Colin Watson (cjwatson)
Changed in man-db (Ubuntu Lucid):
assignee: nobody → Colin Watson (cjwatson)
Changed in man-db (Ubuntu Trusty):
assignee: nobody → Colin Watson (cjwatson)
Changed in man-db (Ubuntu Precise):
assignee: nobody → Colin Watson (cjwatson)
Gunnar Hjalmarsson (gunnarhj) wrote :

Hi Colin!

Does this fix cover bug #1364642 and its duplicates too?

On Tue, Sep 23, 2014 at 01:31:33PM -0000, Gunnar Hjalmarsson wrote:
> Does this fix cover bug #1364642 and its duplicates too?

Those bugs look entirely unrelated to me.

Gunnar Hjalmarsson (gunnarhj) wrote :

Ok, thanks.

Hello Colin, or anyone else affected,

Accepted man-db into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/man-db/2.6.7.1-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in man-db (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in man-db (Ubuntu Precise):
status: In Progress → Fix Committed
Chris J Arges (arges) wrote :

Hello Colin, or anyone else affected,

Accepted man-db into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/man-db/2.6.1-2ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in man-db (Ubuntu Lucid):
status: In Progress → Fix Committed
Chris J Arges (arges) wrote :

Hello Colin, or anyone else affected,

Accepted man-db into lucid-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/man-db/2.5.7-2ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Sorry, I would test, but my screen since cracked and computer is useless.
Thanks for your help.

On Tue, Sep 23, 2014 at 12:40 PM, Chris J Arges <email address hidden>
wrote:

> Hello Colin, or anyone else affected,
>
> Accepted man-db into trusty-proposed. The package will build now and be
> available at http://launchpad.net/ubuntu/+source/man-db/2.6.7.1-1ubuntu1
> in a few hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Your feedback will aid us getting this update
> out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, and change the tag
> from verification-needed to verification-done. If it does not fix the
> bug for you, please add a comment stating that, and change the tag to
> verification-failed. In either case, details of your testing will help
> us make a better decision.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance!
>
> ** Changed in: man-db (Ubuntu Trusty)
> Status: In Progress => Fix Committed
>
> ** Tags added: verification-needed
>
> ** Changed in: man-db (Ubuntu Precise)
> Status: In Progress => Fix Committed
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1198582).
> https://bugs.launchpad.net/bugs/1372673
>
> Title:
> excessive debconf use when triggered
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/man-db/+bug/1372673/+subscriptions
>

Colin Watson (cjwatson) wrote :

Fixed in utopic now:

man-db (2.7.0-1) unstable; urgency=medium

  * New upstream release:
    - Add systemd tmpfiles snippet to clean up old cat files after a week.
    - Run "col -b -p -x" over cat pages if possible before parsing them
      (closes: #751934).
    - Move database mtime out of the database into file metadata, making the
      database reproducible between installations (closes: #760895).
    - Use high-precision timestamps.
    - Order files by first physical extent before reading (closes: #574410).
    - Prioritise COLUMNS above TIOCGWINSZ (LP: #1315282).
    - Formatting improvements to man(1) (closes: #726266).
    - Don't use pointed-to name as title for database-located pages (closes:
      #709405).
    - Move zsoelim to /usr/lib/man-db/.
  * Remove unnecessary entries from debian/dirs.
  * Cache the value of man-db/auto-update in the file system, so that we
    don't have to talk to debconf when processing triggers (closes:
    #579075).
  * Add MIME handlers (thanks, Kevin Ryde; closes: #725157).
  * Override the long-standing Lintian warning for non-standard-dir-perm on
    /var/cache/man.
  * Adjust cron.daily to skip cat file cleanup if running under systemd,
    since the upstream-provided tmpfiles snippet now handles that.
  * Rebuild the database on upgrade to this version, since the format has
    changed.

 -- Colin Watson <email address hidden> Mon, 22 Sep 2014 19:43:40 +0100

Changed in man-db (Ubuntu):
status: Fix Committed → Fix Released
Ben Miller (benjamin-miller115) wrote :
Download full text (7.4 KiB)

What is this shit? Who is this?
On Sep 26, 2014 5:36 AM, "Colin Watson" <email address hidden> wrote:

> Fixed in utopic now:
>
> man-db (2.7.0-1) unstable; urgency=medium
>
> * New upstream release:
> - Add systemd tmpfiles snippet to clean up old cat files after a week.
> - Run "col -b -p -x" over cat pages if possible before parsing them
> (closes: #751934).
> - Move database mtime out of the database into file metadata, making
> the
> database reproducible between installations (closes: #760895).
> - Use high-precision timestamps.
> - Order files by first physical extent before reading (closes:
> #574410).
> - Prioritise COLUMNS above TIOCGWINSZ (LP: #1315282).
> - Formatting improvements to man(1) (closes: #726266).
> - Don't use pointed-to name as title for database-located pages
> (closes:
> #709405).
> - Move zsoelim to /usr/lib/man-db/.
> * Remove unnecessary entries from debian/dirs.
> * Cache the value of man-db/auto-update in the file system, so that we
> don't have to talk to debconf when processing triggers (closes:
> #579075).
> * Add MIME handlers (thanks, Kevin Ryde; closes: #725157).
> * Override the long-standing Lintian warning for non-standard-dir-perm on
> /var/cache/man.
> * Adjust cron.daily to skip cat file cleanup if running under systemd,
> since the upstream-provided tmpfiles snippet now handles that.
> * Rebuild the database on upgrade to this version, since the format has
> changed.
>
> -- Colin Watson <email address hidden> Mon, 22 Sep 2014 19:43:40 +0100
>
> ** Changed in: man-db (Ubuntu)
> Status: Fix Committed => Fix Released
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1201179).
> https://bugs.launchpad.net/bugs/1372673
>
> Title:
> excessive debconf use when triggered
>
> Status in “man-db” package in Ubuntu:
> Fix Released
> Status in “man-db” source package in Lucid:
> Fix Committed
> Status in “man-db” source package in Precise:
> Fix Committed
> Status in “man-db” source package in Trusty:
> Fix Committed
> Status in “man-db” package in Debian:
> Fix Released
>
> Bug description:
> SRU justification:
>
> [Impact]
>
> We see rather frequent hard-to-debug upgrade failures that amount to
> man-db's trigger failing in some way that has nothing to do with the
> mandb program itself, but rather in some kind of communication with
> the package management frontend. Almost all the open bugs on Ubuntu
> man-db come down to this in one way or another. The root cause is, I
> believe, that it is not really safe to use debconf from a dpkg
> trigger: while debconf itself behaves as if it were Essential, the
> debconf protocol is often mediated by various frontends with less
> stringent practices. It would certainly be a good idea to get man-
> db's trigger out of the loop here, as dpkg tends to run it at the end
> of an unpack phase when large numbers of packages are unconfigured,
> which is pretty much the worst case for having this work properly.
>
> In the past I've tried to investigate why debconf fails in these
> si...

Read more...

Colin Watson (cjwatson) wrote :

On Fri, Sep 26, 2014 at 09:40:39AM -0000, Ben Miller wrote:
> What is this shit? Who is this?
[...]
> > You received this bug notification because you are subscribed to a
> > duplicate bug report (1201179).
> > https://bugs.launchpad.net/bugs/1372673

If you don't want updates on the bug you filed some time back, then go
to this URL:

  https://bugs.launchpad.net/ubuntu/+source/man-db/+bug/1201179/+subscribe

... log in if necessary, select "unsubscribe me from this bug", and
press "Continue".

Please, however, do not send me abusive e-mail for doing my job.

Regards,

--
Colin Watson [<email address hidden>]

Colin Watson (cjwatson) wrote :

For each of lucid → precise, precise → trusty, and trusty → utopic, I tested as follows:

  1) Set man-db/auto-update to false and install man-db from -proposed. Check that /var/cache/man/index.db is not created. Install vim and check that it is still not created.
  2) Set man-db/auto-update to true and install man-db from -proposed. Check that /var/cache/man/index.db is created ("accessdb | head"). Install vim and check that it is updated ("whatis vim"). Edit /etc/apt/sources.list to point to the next release. Run apt-get dist-upgrade and check that everything works. (For lucid → precise I had to run "apt-get install apt" first.)

Everything looks good, so marking this as verification-done.

tags: added: verification-done
removed: verification-needed

The verification of the Stable Release Update for man-db has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package man-db - 2.6.1-2ubuntu2

---------------
man-db (2.6.1-2ubuntu2) precise; urgency=medium

  * Cache the value of man-db/auto-update in the file system, so that we
    don't have to talk to debconf when processing triggers (LP: #1372673).
 -- Colin Watson <email address hidden> Tue, 23 Sep 2014 11:58:23 +0100

Changed in man-db (Ubuntu Precise):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package man-db - 2.5.7-2ubuntu2

---------------
man-db (2.5.7-2ubuntu2) lucid; urgency=medium

  * Cache the value of man-db/auto-update in the file system, so that we
    don't have to talk to debconf when processing triggers (LP: #1372673).
 -- Colin Watson <email address hidden> Tue, 23 Sep 2014 11:56:37 +0100

Changed in man-db (Ubuntu Lucid):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package man-db - 2.6.7.1-1ubuntu1

---------------
man-db (2.6.7.1-1ubuntu1) trusty; urgency=medium

  * Cache the value of man-db/auto-update in the file system, so that we
    don't have to talk to debconf when processing triggers (LP: #1372673).
 -- Colin Watson <email address hidden> Tue, 23 Sep 2014 11:59:32 +0100

Changed in man-db (Ubuntu Trusty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.