status_update not sent any more, causes eternal hang on conffile prompts in aptdaemon

[Martin Pitt]

On November 23, aptdaemon started to fail its tests [1]. In particular, it now eternally hangs in the conffile tests, which is reproducible locally perfectly well (on current trusty):

$ PYTHONPATH=. python3 tests/test_configfileprompt.py ConfigFilePromptTestCase.test_keep
[...]
13:04:03 AptDaemon.Trans [DEBUG]: Emitting PropertyChanged: StatusDetails, silly-config wird konfiguriert
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: Setting up silly-config (0.1-0) ...
13:04:03 AptDaemon.Worker.Terminal [DEBUG]:
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: Configuration file `/etc/silly-packages.cfg' (actually `/tmp/tmp8q_w7btmp/etc/silly-packages.cfg')
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: ==> File on system created by you or by a script.
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: ==> File also in package provided by package maintainer.
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: What would you like to do about it ? Your options are:
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: Y or I : install the package maintainer's version
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: N or O : keep your currently-installed version
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: D : show the differences between the versions
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: Z : start a shell to examine the situation
13:04:03 AptDaemon.Worker.Terminal [DEBUG]: The default action is to keep your current version.

After that it hangs eternally, you need to run "killall dpkg" to finish this.

This coincides with the large apt update from 0.9.9.1~ubuntu5 to 0.9.13~exp1ubuntu1 [2]; dpkg or aptdaemon itself haven't been uploaded in a while, so I'm fairly certain that either this is a regression in apt, or some deliberate new behaviour for which aptdaemon needs to be updated.

I already ensured that the fix for bug 1254696 (Apt::Keep-Fds) does not help, and apparently aptdaemon does not even use that.

[1] https://jenkins.qa.ubuntu.com/job/trusty-adt-aptdaemon/?
[2] https://launchpad.net/ubuntu/+source/apt/0.9.13~exp1ubuntu1

[Martin Pitt (pitti)]

I believe aptdaemon/progress.py _on_status_update() is responsible for intercepting conffile prompts. However, this does not get called *at all* any more, I checked this by adding a raise RuntimeError('caught!') at the top, and also some log.warning() (which never appears). So it seems recent apt breaks the status-fd handling?

[Martin Pitt (pitti)]

I bisected this down to http://anonscm.debian.org/gitweb/?p=apt/apt.git;a=commitdiff;h=11ef54810d

[Martin Pitt (pitti)]

Tentatively moving this to apt as this doesn't seem to be an intended change.

[Michael Vogt (mvo)]

Thanks Martin! There is currently no apt-progress-fd testcase for conffile (we have one for error though). I will work on one now.

[Michael Vogt (mvo)]

Here is the fix:

diff --git a/apt-pkg/deb/dpkgpm.cc b/apt-pkg/deb/dpkgpm.cc
index b4bfd14..e77dd54 100644
--- a/apt-pkg/deb/dpkgpm.cc
+++ b/apt-pkg/deb/dpkgpm.cc
@@ -608,7 +608,7 @@ void pkgDPkgPM::ProcessDpkgStatusLine(char *line)
          WriteApportReport(list[1].c_str(), list[3].c_str());
          return;
       }
- else if(action == "conffile")
+ else if(action == "conffile-prompt")
       {
          d->progress->ConffilePrompt(list[1], PackagesDone, PackagesTotal,
                                      list[3]);

I'm working on a proper testcase now to ensure this does not break again. Sorry for the trouble and thanks a bunch
for your help!

[Martin Pitt (pitti)]

Many Thanks Michael, you are a star!

[Martin Pitt (pitti)]

Michael, btw, if you upload this please note that apt currently fails its autopkgtest. Primarily it produces a lot of stderr and fails to build during the test, but when I do that manually in a VM I also get one test case failure.

But this autopkgtest doesn't help much in the first place: it only tests the build tree, but it must test the actually installed apt (i. e. /usr/bin/). Can you run the tests against the installed package?

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.9.14.1ubuntu1

---------------
apt (0.9.14.1ubuntu1) trusty; urgency=low

  * Merge from debian/sid
  * disable autopkgtest for now until it can test the installed
    packages and not only the build-tree (thanks to Martin Pitt)

apt (0.9.14.1) unstable; urgency=medium

  * fix apt-get source -t dist regression (closes: #731853)
    and add testcase
  * clarify error message when apt-get source=ver fails
    (thans to David Kalnischkies)
  * Fix conffile prompt regression (LP: #1260297)
    and add testcase
  * improve error message for apt-get source pkg:arch{=ver,/release}
 -- Michael Vogt <email address hidden> Thu, 12 Dec 2013 21:19:49 +0100

[Martin Pitt (pitti)]

Hey Michael, many thanks for the fix!

[Michael Vogt (mvo)]

On Thu, Dec 12, 2013 at 01:49:46PM -0000, Martin Pitt wrote:
> Michael, btw, if you upload this please note that apt currently fails
> its autopkgtest. Primarily it produces a lot of stderr and fails to
> build during the test, but when I do that manually in a VM I also get
> one test case failure.
>
> But this autopkgtest doesn't help much in the first place: it only tests
> the build tree, but it must test the actually installed apt (i. e.
> /usr/bin/). Can you run the tests against the installed package?

Indeed, sorry for that. I disabled autopkgtest for the last upload
again and prepared a branch that redirects stderr to a file (output is
mostly noise) and test the installed files instead of the build file.

One question - whats the best practise here, should the test also
check against the build tree? Or is it better to run the tests against
the build tree during the package build? I guess the later but would
be nice if you could confirm :)

Cheers,
 Michael

[Martin Pitt (pitti)]

Indeed, run unit tests against the source tree during package build (failing the build if they fail), and integration tests in autopkgtest against the system-installed package. The latter don't necessarily need to be the same upstream tests; coverage of fine details is already handled during package build, it's sufficient to have some kind of smoke test that "apt-get update", "apt-cache search", "apt-get install" etc. work in general: I. e. binaries are installed, dependencies are correct, expected directories in /var exist, and so on.