lucid -> precise main all failed to upgrade: dpkg: dependency problems prevent configuration of kde-runtime

[Jean-Baptiste Lallement]

lucid to precise upgrade - main all - amd64/i386

upgrade failed with:
Setting up plasma-scriptengine-javascript (4:4.8.0-0ubuntu1) ...
dpkg: dependency problems prevent configuration of kde-runtime:
 kde-runtime depends on libsmbclient (>= 2:3.2.0); however:
  Package libsmbclient is not configured yet.
dpkg: error processing kde-runtime (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 kde-runtime

http://10.189.74.2:8080/job/precise-upgrade-lucid-main/ARCH=i386,LTS=lts,PROFILE=main-all,label=upgrade-test/33/

[Steve Langasek (vorlon)]

This is a bug with out-of-order configuration of packages on upgrade and as such is not feasible to fix for beta-1 as it requires an update to the apt in lucid-updates to address.

[Steve Langasek (vorlon)]

This seems to be fallout from the recent reorganization of the kubuntu seed. However, the fact that configuration is attempted out of order is definitely an apt bug. Since the backported upgrader apt is meant to address precisely these problems, I suspect this has the same underlying cause as bug #927993. Marking as a duplicate.

[Martin Pitt (pitti)]

For the record, bug 927993 marks packages for configuration before unpack. This one is trying to configure a package before its dependencies are configured, so they are distinct (Michael confirmed).

[Martin Pitt (pitti)]

Confirmed by bug 892630, which was an user report (not an auto-test), where libglib2.0-0 was forgotten to be configured before its reverse dependencies.

[Michael Vogt (mvo)]

In a log with Debug::pkgPackageManager=1 I can see:

around lines 1576:
  SmartConfigure plasma-dataengines-workspace (4:4.8.0b-0ubuntu6)^M^M
    SmartConfigure kdepim-runtime (4:4.8.0a-0ubuntu2)^M^M
    Unpacking kde-runtime to avoid loop^M^M
      SmartUnPack kde-runtime (install version 4:4.8.0-0ubuntu1)^M^M
        Unpacking kdebase-runtime to avoid break^M^M
        SmartUnPack kdebase-runtime (replace version 4:4.4.5-0ubuntu1.1 with 4:4.8.0-0ubuntu1)^M^M
        Unpacking plasma-scriptengine-javascript to avoid break^M^M
        SmartUnPack plasma-scriptengine-javascript (replace version 4:4.4.5-0ubuntu1.1 with 4:4.8.0-0ubuntu1)^M^M
        SmartConfigure kde-runtime (4:4.8.0-0ubuntu1)^M^M
        Unpacking libattica0.3 to avoid loop^M^M
...
but quite a bit later (l 1707):
                PreDepends order for libkrb5-3^M^M
                  Unpacking libsmbclient to avoid break^M^M
                  SmartUnPack libsmbclient (replace version 2:3.4.7~dfsg-1ubuntu3.8 with 2:3.6.3-1ubuntu1)^M^M
                  PreDepends order for libsmbclient^M^M
                  SmartConfigure libkrb5-3 (1.10+dfsg~beta1-2)^M^M
and then in l 2631:
  SmartConfigure libsmbclient (2:3.6.3-1ubuntu1)^M^M

So it appears that the first "SmartConfigure" for kde-runtime misses the smbclient requirement most likely because
at the point it was run the package did not need anything. But then the later libkrb5-3 pre-depends caused smbclient
to get unpacked at a bad time.

[Michael Vogt (mvo)]

It appears there is a loop here also in kde-runtime,kdelibs5-plugins,libgssapi-krb5-2,libsmbclient that is not broken quite right.

[Michael Vogt (mvo)]

This looks like a excerpt of the relevant stuff that is going on:
...
  SmartConfigure plasma-dataengines-workspace (4:4.8.0b-0ubuntu6)^M^M
    SmartConfigure kdepim-runtime (4:4.8.0a-0ubuntu2)^M^M
    Unpacking kde-runtime to avoid loop^M^M
      SmartUnPack kde-runtime (install version 4:4.8.0-0ubuntu1)^M^M
        Unpacking kdebase-runtime to avoid break^M^M
        SmartUnPack kdebase-runtime (replace version 4:4.4.5-0ubuntu1.1 with 4:4.8.0-0ubuntu1)^M^M
        SmartConfigure kde-runtime (4:4.8.0-0ubuntu1)^M^M
...
        Unpacking kdelibs5-plugins to avoid loop^M^M
          SmartUnPack kdelibs5-plugins (install version 4:4.8.0-0ubuntu2)^M^M
            SmartConfigure kdelibs5-plugins (4:4.8.0-0ubuntu2)^M^M
              SmartConfigure libgssapi-krb5-2 (1.10+dfsg~beta1-2)^M^M
            SmartConfigure kdelibs5-plugins (4:4.8.0-0ubuntu2)^M^M
              SmartConfigure libgssapi-krb5-2 (1.10+dfsg~beta1-2)^M^M
                SmartConfigure libk5crypto3 (1.10+dfsg~beta1-2)^M^M
                  SmartConfigure libkrb5support0 (1.10+dfsg~beta1-2)^M^M
              Unpacking libkrb5-3 to avoid loop^M^M
                SmartUnPack libkrb5-3 (replace version 1.8.1+dfsg-2ubuntu0.10 with 1.10+dfsg~beta1-2)^M^M
                PreDepends order for libkrb5-3^M^M
                  Unpacking libsmbclient to avoid break^M^M
                  SmartUnPack libsmbclient (replace version 2:3.4.7~dfsg-1ubuntu3.8 with 2:3.6.3-1ubuntu1)^M^M
                  PreDepends order for libsmbclient^M^M
                  SmartConfigure libkrb5-3 (1.10+dfsg~beta1-2)^M^M
...
much later (l. 2631)
...
  SmartConfigure libsmbclient (2:3.6.3-1ubuntu1)^M^M

[Michael Vogt (mvo)]

Another data point:

Package: libsmbclient
Depends: libkrb5-3 (>= 1.10+dfsg~)

Package: libkrb5-3
Breaks: libsmbclient (<= 2:3.6.1-2)

Package: kde-runtime
Depends: libsmbclient (>= 2:3.2.0)

[Michael Vogt (mvo)]

I can reproduce this on a minimal server install with:
$ ./auto-upgrade-tester profile/lts-server --additional-pkgs=libkrb5-3,libsmbclient,kcalc

(the later to get the kde-runtime dependency on upgrade). Same error, much quicker result.

[Michael Vogt (mvo)]

The sources.list needs to be:
deb http://archive.ubuntu.com/ubuntu lucid main restricted

[Michael Vogt (mvo)]

I can not currently make the test-case work, but I did verify that the install/remove/upgrade packages are identical for both the VM and the test-case apt-get dist-upgrade output (using wdiff on the main.log extract and the apt-get dist-upgrade output in the testcase).

[Michael Vogt (mvo)]

Just FYI, the jenkins log show that its both a issue with 0.8.16~exp12ubuntu1~upgrader3 and ~upgrade1 so its not a regression from the recent changes.

[Michael Vogt (mvo)]

(Build #33 in the jenkins instance shows the failure with the previous apt backport)

[James Hunt (jamesodhunt)]

I'm starting to suspect a multi-arch issue here...

$ dpkg-query -W -f='${Package}\t${Multi-Arch}\n' | sort -k1,1n|\
   egrep "libgssapi-krb5-2|libsmbclient|libglib2.0-0|kde-runtime|kdelibs5-plugins"|sort -k2,2r|expand
libgssapi-krb5-2 same
libsmbclient same
libglib2.0-0 same
kde-runtime no
kdelibs5-plugins no

All the dependencies that are missed are "Multi-Arch: same". The method pkgPackageManager::SmartConfigure() special-cases multi-arch support right at the end, but it's looking like it should be handled within the main dependency loop to avoid this scenario.

@slangasek/@mvo: Could you comment?

[Michael Vogt (mvo)]

Some more thoughts on this. The log shows that the following is happening:

1. kde-runtime is unpacked at line 2454, at this point libsmbclient is unchanged so its not unpacked nor configured (as this is not required by the kde-runtime dependencies)
2. libsmbclient is unpacked in line 2504 but its not configured (yet)
3. kde-runtime is looked at again in line 2548 but because its already configured its skipped
4. libsmbclient is finally configured in line 2654 (when its too late)

So it seems like the bug either in
 (3) that he dependencies are not checked if they are all ok
 (2) when libsmbclient is unpacked (but not configured) it needs to RmFlag() its rdependences configured flag

Thoughts?

[Michael Vogt (mvo)]

I should add that in 1. its unpacked and right after that SmatConfigured.

[Michael Vogt (mvo)]

I can finally reproduce this now with the lp:~mvo/+junk/test-case-lp940396 branch and the lp:~mvo/apt/debian-experimental2 branch: "./run.sh /path/to/debian-experimental2/build/bin/apt-get" triggers it for me.

There are various workaround:
- add versionized dependency of libsmbclient (>= 2:3.6.3) to kde-runtime that ensures libsmbclient is unpacked early
- add breaks to kde-runtime against libsmbclient (<= 2:3.6.1) (just like libkrb5-3)

But of course those are workaround not fixes.

One fix that works for me is to do:
=== modified file 'apt-pkg/packagemanager.cc'
--- apt-pkg/packagemanager.cc 2012-03-07 08:20:31 +0000
+++ apt-pkg/packagemanager.cc 2012-03-12 15:08:19 +0000
@@ -742,7 +742,7 @@
        }
        if (PkgLoop == false)
    List->Flag(Pkg,pkgOrderList::Loop);
- SmartUnPack(BrokenPkg, false, Depth + 1);
+ SmartUnPack(BrokenPkg, true, Depth + 1);
        if (PkgLoop == false)
    List->RmFlag(Pkg,pkgOrderList::Loop);
     }

So essentially calling unpack with immediately configure to ensure that the package is fully ready early. But that is
slightly risky as it will have side-effects beyond this one bug. It does work for me though.

I'm looking into more possible solutions.

[James Hunt (jamesodhunt)]

Re-assigning after discussion with mvo.

[David Kalnischkies (donkult)]

First A simple workaround for this specific case:
Move the libsmbclient dependency in kde-runtime from somewhere in the middle to the end of the dependency list.

The problem:
Imagine a package A depending on B and C. B depends on nothing and C breaks the "old" installed B.
In some situations (not many, usually the ordering prevents this, but "loops" can trigger this) APT will look at package A first:
Dependency B is satisfied with the version installed. Fine. Next dependency is C - this dependency is broken as an old B is installed. We fix this by unpacking the new version of B. Dependency C is fine now, too. But hang on a minute: The first dependency we checked isn't satisfied anymore, just that APT doesn't notice this and tries to configure A now as it thinks all dependencies are satisfied…

Michaels hot-fix above works here - as you might guess - as it will force APT to not only unpack B for C, but also configure it. Problem solved - but as Michael hinted this isn't side-effect free (it would do this for dependency D of C, too, even through it's not needed for D and therefore increases the complexity of the solution - if we are able to find one with this added).

In a few seconds launchpad should link my branch to this bug. Rev 2229 and 2230 (subject to change after merge/pull) fixes this problem by re-checking the dependencies in case we have "changed" anything (damn, this sounds so easy…). This shouldn't have a side-effect, but as this is quiet a bit of a change source-wise Michael will push it through a few testcases before we throw that at the general public…

[Steve Langasek (vorlon)]

my understanding is that the fix for this is included in the release-upgrader package currently in lucid-proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.8.16~exp12ubuntu6

---------------
apt (0.8.16~exp12ubuntu6) precise; urgency=low

  * cherry pick from
    http://bzr.debian.org/bzr/bzr/apt/apt/debian-experimental2/:
    * apt-pkg/packagemanager.cc:
      - fix bug in predepends handling - ensure that packages that needs
        unpackaging are unpacked before they are configured (LP: #927993)
    * apt-pkg/packagemanager.cc:
      - do not try to a void a breaks if the broken package pre-depends
        on the breaker, but let dpkg auto-deconfigure it
    * apt-pkg/packagemanager.cc:
      - recheck all dependencies if we changed a package in SmartConfigure
        as this could break an earlier dependency (LP: #940396)
    * recheck dependencies in SmartUnpack after a change, too
    * add Debug::pkgAcqArchive::NoQueue to disable package downloading
 -- Michael Vogt <email address hidden> Thu, 15 Mar 2012 19:46:08 +0100