Non-pinned available package in -proposed not chosen

[Iain Lane]

In autopkgtest we just had this situation:

root@humorous-bullfrog:~# cat /etc/apt/preferences
Package: *
Pin: release a=bionic
Pin-Priority: 990

Package: python3.6
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: python3.6-venv
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: libpython3.6-stdlib
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: python3.6-minimal
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: libpython3.6-minimal
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: libpython3.6
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: python3.6-examples
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: python3.6-dev
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: libpython3.6-dev
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: libpython3.6-testsuite
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: idle-python3.6
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: python3.6-doc
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: python3.6-dbg
Pin: release a=bionic-proposed
Pin-Priority: 995

Package: libpython3.6-dbg
Pin: release a=bionic-proposed
Pin-Priority: 995
root@humorous-bullfrog:~# apt -oDebug::pkgProblemResolver=true full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 -> 3.6.5-3 @ii umU Ib >
Broken libpython3.6-stdlib:amd64 Breaks on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mK > (< 3.6.5-2)
  Considering python3-distutils:amd64 7 as a solution to libpython3.6-stdlib:amd64 22
  Added python3-distutils:amd64 to the remove list
  Fixing libpython3.6-stdlib:amd64 via remove of python3-distutils:amd64
Investigating (0) dh-python:amd64 < 3.20180325ubuntu2 @ii mK Ib >
Broken dh-python:amd64 Depends on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mR >
  Considering python3-distutils:amd64 7 as a solution to dh-python:amd64 15
  Added python3-distutils:amd64 to the remove list
  Fixing dh-python:amd64 via keep of python3-distutils:amd64
Investigating (1) libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 -> 3.6.5-3 @ii umU Ib >
Broken libpython3.6-stdlib:amd64 Breaks on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mK > (< 3.6.5-2)
  Considering python3-distutils:amd64 7 as a solution to libpython3.6-stdlib:amd64 22
  Added python3-distutils:amd64 to the remove list
  Fixing libpython3.6-stdlib:amd64 via remove of python3-distutils:amd64
Investigating (1) dh-python:amd64 < 3.20180325ubuntu2 @ii mK Ib >
Broken dh-python:amd64 Depends on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mR >
  Considering python3-distutils:amd64 7 as a solution to dh-python:amd64 15
  Added python3-distutils:amd64 to the remove list
  Fixing dh-python:amd64 via keep of python3-distutils:amd64
Investigating (2) libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 -> 3.6.5-3 @ii umU Ib >
Broken libpython3.6-stdlib:amd64 Breaks on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mK > (< 3.6.5-2)
  Considering python3-distutils:amd64 7 as a solution to libpython3.6-stdlib:amd64 22
  Added python3-distutils:amd64 to the remove list
  Fixing libpython3.6-stdlib:amd64 via remove of python3-distutils:amd64
Investigating (2) dh-python:amd64 < 3.20180325ubuntu2 @ii mK Ib >
Broken dh-python:amd64 Depends on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mR >
  Considering python3-distutils:amd64 22 as a solution to dh-python:amd64 15
  Removing dh-python:amd64 rather than change python3-distutils:amd64
Investigating (3) python3:amd64 < 3.6.4-1 @ii mK Ib >
Broken python3:amd64 Depends on dh-python:amd64 < 3.20180325ubuntu2 @ii mR >
  Considering dh-python:amd64 22 as a solution to python3:amd64 40
  Added dh-python:amd64 to the remove list
  Fixing python3:amd64 via keep of dh-python:amd64
Investigating (3) dh-python:amd64 < 3.20180325ubuntu2 @ii mK Ib >
Broken dh-python:amd64 Depends on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mR >
  Considering python3-distutils:amd64 22 as a solution to dh-python:amd64 40
  Added python3-distutils:amd64 to the remove list
  Fixing dh-python:amd64 via keep of python3-distutils:amd64
Investigating (4) libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 -> 3.6.5-3 @ii umU Ib >
Broken libpython3.6-stdlib:amd64 Breaks on python3-distutils:amd64 < 3.6.5~rc1-1 @ii mK > (< 3.6.5-2)
  Considering python3-distutils:amd64 40 as a solution to libpython3.6-stdlib:amd64 22
  Removing libpython3.6-stdlib:amd64 rather than change python3-distutils:amd64
Investigating (4) python3.6:amd64 < 3.6.5~rc1-1 -> 3.6.5-3 @ii umU Ib >
Broken python3.6:amd64 Depends on libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 | 3.6.5-3 @ii umR > (= 3.6.5-3)
  Considering libpython3.6-stdlib:amd64 40 as a solution to python3.6:amd64 21
  Removing python3.6:amd64 rather than change libpython3.6-stdlib:amd64
Investigating (4) libpython3-stdlib:amd64 < 3.6.4-1 @ii mK Ib >
Broken libpython3-stdlib:amd64 Depends on libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 | 3.6.5-3 @ii umR > (>= 3.6.4-1~)
  Considering libpython3.6-stdlib:amd64 40 as a solution to libpython3-stdlib:amd64 15
  Removing libpython3-stdlib:amd64 rather than change libpython3.6-stdlib:amd64
Investigating (4) libpython3.6:amd64 < 3.6.5~rc1-1 -> 3.6.5-3 @ii umU Ib >
Broken libpython3.6:amd64 Depends on libpython3.6-stdlib:amd64 < 3.6.5~rc1-1 | 3.6.5-3 @ii umR > (= 3.6.5-3)
  Considering libpython3.6-stdlib:amd64 40 as a solution to libpython3.6:amd64 3
  Removing libpython3.6:amd64 rather than change libpython3.6-stdlib:amd64
Investigating (5) python3:amd64 < 3.6.4-1 @ii mK Ib >
Broken python3:amd64 Depends on python3.6:amd64 < 3.6.5~rc1-1 | 3.6.5-3 @ii umR > (>= 3.6.4-1~)
  Considering python3.6:amd64 40 as a solution to python3:amd64 40
  Removing python3:amd64 rather than change python3.6:amd64
Investigating (5) dh-python:amd64 < 3.20180325ubuntu2 @ii mK Ib >
Broken dh-python:amd64 Depends on python3:any:any < none @un H > (>= 3.3.2-2~)
  Considering python3:amd64 40 as a solution to dh-python:amd64 40
  Removing dh-python:amd64 rather than change python3:any:any
Investigating (5) netplan.io:amd64 < 0.34.1 @ii mK Ib >
Broken netplan.io:amd64 Depends on python3:amd64 < 3.6.4-1 @ii mR >
  Considering python3:amd64 40 as a solution to netplan.io:amd64 11
  Removing netplan.io:amd64 rather than change python3:amd64
Investigating (5) vim:amd64 < 2:8.0.1401-1ubuntu3 @ii mK Ib >
Broken vim:amd64 Depends on libpython3.6:amd64 < 3.6.5~rc1-1 | 3.6.5-3 @ii umR > (>= 3.6.4~rc1)
  Considering libpython3.6:amd64 40 as a solution to vim:amd64 8
  Removing vim:amd64 rather than change libpython3.6:amd64
Investigating (5) python3-yaml:amd64 < 3.12-1build2 @ii mK Ib >
Broken python3-yaml:amd64 Depends on python3:amd64 < 3.6.4-1 @ii mR > (< 3.7)
  Considering python3:amd64 40 as a solution to python3-yaml:amd64 8
  Removing python3-yaml:amd64 rather than change python3:amd64
Investigating (5) nplan:amd64 < 0.34.1 @ii mK Ib >
Broken nplan:amd64 Depends on netplan.io:amd64 < 0.34.1 @ii mR >
  Considering netplan.io:amd64 40 as a solution to nplan:amd64 7
  Removing nplan:amd64 rather than change netplan.io:amd64
Investigating (5) lsb-release:amd64 < 9.20170808ubuntu1 @ii mK Ib >
Broken lsb-release:amd64 Depends on python3:any:any < none @un H > (>= 3.4~)
  Considering python3:amd64 40 as a solution to lsb-release:amd64 7
  Removing lsb-release:amd64 rather than change python3:any:any
Investigating (5) python3-distutils:amd64 < 3.6.5~rc1-1 @ii mK Ib >
Broken python3-distutils:amd64 Depends on python3:amd64 < 3.6.4-1 @ii mR > (>= 3.6.4-1~)
  Considering python3:amd64 40 as a solution to python3-distutils:amd64 40
  Removing python3-distutils:amd64 rather than change python3:amd64
Investigating (5) python3-lib2to3:amd64 < 3.6.5~rc1-1 @ii mK Ib >
Broken python3-lib2to3:amd64 Depends on python3:amd64 < 3.6.4-1 @ii mR > (>= 3.6.4-1~)
  Considering python3:amd64 40 as a solution to python3-lib2to3:amd64 7
  Removing python3-lib2to3:amd64 rather than change python3:amd64
Investigating (5) ubuntu-minimal:amd64 < 1.414 @ii mK Ib >
Broken ubuntu-minimal:amd64 Depends on lsb-release:amd64 < 9.20170808ubuntu1 @ii mR >
  Considering lsb-release:amd64 40 as a solution to ubuntu-minimal:amd64 3
  Removing ubuntu-minimal:amd64 rather than change lsb-release:amd64
Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  vim-runtime
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
  dh-python libpython3-stdlib libpython3.6 libpython3.6-stdlib lsb-release netplan.io nplan python3 python3-distutils python3-lib2to3 python3-yaml python3.6 ubuntu-minimal vim
The following packages will be upgraded:
  libpython3.6-minimal python3.6-minimal
2 upgraded, 0 newly installed, 14 to remove and 0 not upgraded.
Need to get 1954 kB of archives.
After this operation, 19.3 MB disk space will be freed.
Do you want to continue? [Y/n]

autopkgtest merrily went ahead and removed all of those packages, which it turns out breaks package extraction, causing a tmpfail and then destroying all workers in turn.

---

libpython3.6-stdlib has Breaks: python3-distutils (<< 3.6.5-2)

root@humorous-bullfrog:~# apt policy python3-distutils
python3-distutils:
  Installed: 3.6.5~rc1-1
  Candidate: 3.6.5~rc1-1
  Version table:
     3.6.5-2 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
 *** 3.6.5~rc1-1 990
        990 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

There is a version available which would satisfy this requirement:

root@humorous-bullfrog:~# apt install libpython3.6-stdlib python3-distutils/bionic-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '3.6.5-2' (Ubuntu:18.04/bionic-proposed [all]) for 'python3-distutils'
The following additional packages will be installed:
  libpython3.6 libpython3.6-minimal python3.6 python3.6-minimal
Suggested packages:
  python3.6-venv python3.6-doc binfmt-support
The following packages will be upgraded:
  libpython3.6 libpython3.6-minimal libpython3.6-stdlib python3-distutils python3.6 python3.6-minimal
6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 5377 kB of archives.
After this operation, 13.3 kB disk space will be freed.
Do you want to continue? [Y/n]

Should apt have selected this version to upgrade, or is there something we should have done differently in autopkgtest to get around this problem?

[Julian Andres Klode (juliank)]

APT, by design, only picks candidate versions when installing dependencies, and the non-pinned proposed version is not the candidate. apt install foo/target has some logic to switch dependencies to target too, which works sometimes, but it's nothing that happens when using pinning.

[Iain Lane (laney)]

On Tue, Apr 03, 2018 at 09:26:55AM -0000, Julian Andres Klode wrote:
> APT, by design, only picks candidate versions when installing
> dependencies, and the non-pinned proposed version is not the candidate.
> apt install foo/target has some logic to switch dependencies to target
> too, which works sometimes, but it's nothing that happens when using
> pinning.

Then this bug is about finding a way for autopkgtest to do what we mean
here. If there's a solution available in -proposed, we want to find it -
in this case it'd have been upgrading python3-distutils.

(In a perfect world we'd want to then tell proposed-migration to enforce
this constraint, but we aren't really ready for that.)

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Julian Andres Klode (juliank)]

sbuild passes:

--solver aspcud -o APT::Solver::Strict-Pinning=false -o 'APT::Solver::aspcud::Preferences=-removed,-changed,-new,+sum(solution,apt-pin)'

That probably works here too. I guess I'd try without, and then fall back to it if pure APT can't handle it.

[Paul Gevers (paul-climbing)]

@juliank: I tried your command (with --simulate) on my own laptop and with this option it says that gnome can't be installed, while plain apt will happily install it.

For the history I'll attach my logging.

[Paul Gevers (paul-climbing)]

After discussion on IRC (#debian-devel) I filed Debian bug 896506¹ about this failing command.

I'll continue to pursue a solution in this area. Maybe (suggested by josch):
 --solver aspcud -o APT::Solver::Strict-Pinning=false -o 'APT::Solver::aspcud::Preferences=-removed,-changed,-new,count(solution,APT-Release:=/a=$unstable/)'

¹ https://bugs.debian.org/896506

[Paul Gevers (paul-climbing)]

Hmm, maybe it was just the syntax being wrong. The following line at least succeeded:

--solver aspcud -o APT::Solver::Strict-Pinning=false -o APT::Solver::aspcud::Preferences='-removed,-changed,-new,+count(solution,APT-Pin:=/995/)'

Next up, checking that it does what I want.

Just a worry that came up thinking about the request to aspcud: I think it is allowed to come up with solutions that don't involve any package from -proposed/unstable. So I think that if we go this route (and I think we should/must, not only for the issue encountered in this bug), autopkgtest must check that packages from the triggers are NOT from !-proposed/testing.

[Julian Andres Klode (juliank)]

Oh, you got a working result? I tried aspcud in various variants, but I don't get any solutions. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897330

Of course the autopkgtest-sat-dep package should have versioned dependencies and/or on the command-line, versions should be specified IMO.

[Paul Gevers (paul-climbing)]

On 18-05-18 22:23, Julian Andres Klode wrote:
> Oh, you got a working result? I tried aspcud in various variants, but I
> don't get any solutions. https://bugs.debian.org/cgi-
> bin/bugreport.cgi?bug=897330

I think it is wise to provide your command line. (I did one that worked
on that bug.

> Of course the autopkgtest-sat-dep package should have versioned
> dependencies and/or on the command-line, versions should be specified
> IMO.

I'll check, but I think autopkgtest will happily take it. So it is the
providing infrastructure that needs to be updated for that. I'll add
this to my TODO list for the Debian infrastructure.

Paul

[Paul Gevers (paul-climbing)]

> I'll check, but I think autopkgtest will happily take it.

I don't think this is supported by autopkgtest yet. I'll file a bug about that.

[Paul Gevers (paul-climbing)]

Please give feedback on my proposed implementation in autopkgtest. It also adds support for submission of versioned packages that need to come from -proposed.

https://salsa.debian.org/ci-team/autopkgtest/merge_requests/17