Disco: Please remove old version 17.11 binaries that are left

[Christian Ehrhardt ]

Hi,
dpdk is a vast assortment of libraries.
Since this is ever growing but almost no one is able to test/support all of these we split the packaging to depend on the main function but only suggest the special/experimental cases.

DPDK is held in main by:
platform-git/supported-misc-servers:107: * openvswitch-switch-dpdk

Note: also showing OVS 2.11* from a PPA which will be uploaded to Disco by Jamespage soon.
There will also be no-change rebuilds of collectd and virtio-forwarder which don't pull it in main but are dependencies.

Current openvswitch-switch-dpdk depends on:
- dpdk
- librte-eal17.11 (>= 16.04), librte-ethdev17.11 (>= 17.05), librte-mbuf17.11 (>= 16.04), librte-mempool17.11 (>= 16.07~rc1), librte-meter17.11 (>= 16.04), librte-pdump17.11 (>= 16.07~rc1), librte-pmd-ring17.11 (>= 16.04), librte-ring17.11 (>= 16.04), librte-vhost17.11 (>= 17.08)

The coming rebuilds of openvswitch will then depend on:
- dpdk
- librte-eal18.11 (>= 18.11), librte-ethdev18.11 (>= 18.11), librte-mbuf18.11 (>= 16.04), librte-mempool18.11 (>= 16.07~rc1), librte-meter18.11 (>= 18.08), librte-pmd-ring18.11 (>= 16.04), librte-ring18.11 (>= 16.04), librte-vhost18.11 (>= 17.08)

The librte things have a few internal cross dependencies, but there is something like a core lib and "the rest" this is what we have split.
So from Package: dpdk the dependencies now are partially suggests:

Recommends: librte-mempool18.11, librte-mempool-ring18.11, librte-pmd-af-packet18.11, librte-pmd-bond18.11, librte-pmd-e1000-18.11, librte-pmd-fm10k18.11, librte-pmd-i40e18.11, librte-pmd-ixgbe18.11, librte-pmd-kni18.11, librte-pmd-mlx4-18.11, librte-pmd-mlx5-18.11, librte-pmd-netvsc18.11, librte-pmd-pcap18.11, librte-pmd-tap18.11, librte-pmd-thunderx18.11, librte-pmd-vdev-netvsc18.11, librte-pmd-vhost18.11, librte-pmd-virtio18.11, librte-pmd-vmxnet3-18.11

Suggests: dpdk-doc, dpdk-igb-uio-dkms (= 18.11-5), dpdk-rte-kni-dkms (= 18.11-5), librte-mempool-bucket18.11 (= 18.11-5), librte-mempool-dpaa18.11 (= 18.11-5), librte-mempool-dpaa2-18.11 (= 18.11-5), librte-mempool-octeontx18.11 (= 18.11-5), librte-mempool-stack18.11 (= 18.11-5), librte-pmd-aesni-gcm18.11 (= 18.11-5), librte-pmd-aesni-mb18.11 (= 18.11-5), librte-pmd-ark18.11 (= 18.11-5), librte-pmd-atlantic18.11 (= 18.11-5), librte-pmd-avf18.11 (= 18.11-5), librte-pmd-avp18.11 (= 18.11-5), librte-pmd-axgbe18.11 (= 18.11-5), librte-pmd-bbdev-null18.11 (= 18.11-5), librte-pmd-bnx2x18.11 (= 18.11-5), librte-pmd-bnxt18.11 (= 18.11-5), librte-pmd-caam-jr18.11 (= 18.11-5), librte-pmd-ccp18.11 (= 18.11-5), librte-pmd-crypto-scheduler18.11 (= 18.11-5), librte-pmd-cxgbe18.11 (= 18.11-5), librte-pmd-dpaa-event18.11 (= 18.11-5), librte-pmd-dpaa-sec18.11 (= 18.11-5), librte-pmd-dpaa18.11 (= 18.11-5), librte-pmd-dpaa2-18.11 (= 18.11-5), librte-pmd-dpaa2-cmdif18.11 (= 18.11-5), librte-pmd-dpaa2-event18.11 (= 18.11-5), librte-pmd-dpaa2-qdma18.11 (= 18.11-5), librte-pmd-dpaa2-sec18.11 (= 18.11-5), librte-pmd-dsw-event18.11 (= 18.11-5), librte-pmd-ena18.11 (= 18.11-5), librte-pmd-enetc18.11 (= 18.11-5), librte-pmd-enic18.11 (= 18.11-5), librte-pmd-failsafe18.11 (= 18.11-5), librte-pmd-ifc18.11 (= 18.11-5), librte-pmd-ifpga-rawdev18.11 (= 18.11-5), librte-pmd-liquidio18.11 (= 18.11-5), librte-pmd-nfp18.11 (= 18.11-5), librte-pmd-null-crypto18.11 (= 18.11-5), librte-pmd-null18.11 (= 18.11-5), librte-pmd-octeontx-compress18.11 (= 18.11-5), librte-pmd-octeontx-crypto18.11 (= 18.11-5), librte-pmd-octeontx-event18.11 (= 18.11-5), librte-pmd-octeontx18.11 (= 18.11-5), librte-pmd-opdl-event18.11 (= 18.11-5), librte-pmd-openssl18.11 (= 18.11-5), librte-pmd-qat18.11 (= 18.11-5), librte-pmd-qede18.11 (= 18.11-5), librte-pmd-ring18.11 (= 18.11-5), librte-pmd-sfc18.11 (= 18.11-5), librte-pmd-skeleton-event18.11 (= 18.11-5), librte-pmd-skeleton-rawdev18.11 (= 18.11-5), librte-pmd-softnic18.11 (= 18.11-5), librte-pmd-sw-event18.11 (= 18.11-5), librte-pmd-virtio-crypto18.11 (= 18.11-5), librte-pmd-zlib18.11 (= 18.11-5)

It might not be the full list due to some cross dependencies, but I'd expect most of the binaries listed as "suggests" to be demotable.

Once all dependencies are rebuilt and in proposes I'll subscribe archive-admins and assign it to them. I'd ask the AA's to:
- check the demotions suggested above in case they are not auto-demoted.
  Please get in touch with me if I have misses a dependency somewhere so I can clean it up.
- by that also resolve the current hang of DPDK in updates_excuses by
  libdpdk-dev/amd64 unsatisfiable Depends: libipsec-mb-dev
  librte-pmd-aesni-gcm18.11/amd64 unsatisfiable Depends: libipsec-mb0
  librte-pmd-aesni-mb18.11/amd64 unsatisfiable Depends: libipsec-mb0
  All three packages on the left are from src:dpdk but after the change no more held in main.
  The pmd's are demoted directly, and libdpdk-dev only is a build-depends which also should not
  hold it in main.

Furthermore I'd ask the Archive admin to remove all librte*17.11 packages which should have no remaining dependencies to them anymore.

[Christian Ehrhardt  (paelzer)]

Collectd is already ok
Version 5.8.1-1.2 https://launchpad.net/ubuntu/+source/collectd/5.8.1-1.2
already build against DPDK 18.11-3 being in proposed back then and picked up:
  librte-eal18.11 (>= 16.04), librte-ethdev18.11 (>= 16.07~rc1)

virtio-forwarder has the fix to build fine against 18.11 but needs to be rebuilt.

[Christian Ehrhardt  (paelzer)]

Now virtio-forwarder is in disco-proposed and rebuild against DPDK 18.11
=> https://launchpad.net/ubuntu/+source/virtio-forwarder/1.1.99.51-1~unstable+build1
That has real depends:
librte-eal18.11 (>= 18.11), librte-ethdev18.11 (>= 18.11), librte-mbuf18.11 (>= 16.04), librte-mempool18.11 (>= 16.07~rc1), librte-vhost18.11 (>= 17.05)

So it will hang in -proposed being uninstallable until DPDK itself is made ready by demoting the binaries as requested.

[Christian Ehrhardt  (paelzer)]

Openvswitch now also built it's new version (thanks @James Page for prepareing that with me the last few days)
=> https://launchpad.net/ubuntu/+source/openvswitch/2.11.0~git20190121.4e4f80ec2-0ubuntu1

As expected it build with the following dependencies:
Depends: dpdk, openvswitch-switch (= 2.11.0~git20190121.4e4f80ec2-0ubuntu1), ibverbs-providers (>= 15), libatomic1 (>= 4.8), libbsd0 (>= 0.0), libc6 (>= 2.28), libcap-ng0 (>= 0.7.9), libibverbs1 (>= 1.1.6), libmnl0 (>= 1.0.3-4~), libnl-3-200 (>= 3.2.7), libnl-route-3-200 (>= 3.2.7), libnuma1 (>= 2.0.11), librte-eal18.11 (>= 18.11), librte-ethdev18.11 (>= 18.11), librte-mbuf18.11 (>= 16.04), librte-mempool18.11 (>= 16.07~rc1), librte-meter18.11 (>= 18.08), librte-pmd-ring18.11 (>= 16.04), librte-ring18.11 (>= 16.04), librte-vhost18.11 (>= 17.08), libssl1.1 (>= 1.1.0), libunbound8 (>= 1.8.0), zlib1g (>= 1:1.1.4)

That was the last dependency on librte-*17.11 being gone (at lest in Disco-proposed).

I think we are ready for an archive admin to clean out the old 17.11 binaries as well as demoting some of the binaries of dpdk 18.11 as requested.

[Christian Ehrhardt  (paelzer)]

Subscribed ~ubuntu-archive to help us resolve all that isn't done automatically.

@ubuntu-archive here a TL;DR for you:
- new DPDK 18.11 is in Disco-proposed
- all dependencies are rebuilt in Disco-proposed, those are
  openvswitch, virtio-forwarder (and collectd but that already is in -release)
- DPDK is held back for a false positive component mismatch
- In fact many binaries of src:dpdk are supposed to drop from main->universe now
  It is in main via openvswitch -> some-libs/dpdk -> libs and that latter one now
  has many old Recommends dropped to just a Suggests, therefore they should demote.
  I thought it would be automative, but it seems not could you please make that happen?
- this should make all of them migrate together to Disco-release
- then please remove any binaries left of DPDK 17.11

See the detailed description and comments in this bug for more and let me know if anything is left unclear.

[Christian Ehrhardt  (paelzer)]

I just learned (thanks Steve) that we have one more thing that needs to be resolved.
Our -dev package pulls int libdpdk-dev -> intel-ipsec-mb (which is correct).
We want libdpdk-dev to be demoted as well as part of all of this (you can build universe and main dependencies out of it, we want it to have the deps for both and build dependencies don't have to be in main).

But to achieve that we need to resolve that "-dev packages are autopromoted so long as any binary from the source is in main;" That is our case we will have binaries (what is pulled from "dpdk" via openvswitch) in main.
Therefore I learned that "So if you want libdpdk-dev out of main, you need to add it to Extra-Exclude in the seed"

That is in https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/tree/supported
So I opened an MP to add libdpdk-dev there.
=> https://code.launchpad.net/~paelzer/ubuntu-seeds/+git/ubuntu/+merge/362656

Thanks Steve for telling me about that.
@AA's - in case there is anything else missing let me know.

[Christian Ehrhardt  (paelzer)]

FYI - the MP for the non-auto-promotion of libdpdk-dev is reviewed, acked and pushed now, so that inhibitor is cleared

[Christian Ehrhardt  (paelzer)]

After clearing the auto-promotions of libdpdk-dev and dpdk-dev the case resolved itself as then dependencies were correctly detected.
Thanks for the help with that.

The one AA task which is left it to remove the 17.11 binaries which (so far) were not auto-removed.
This list [1] should have no depends anymore and be removable.

[1]: http://paste.ubuntu.com/p/cyRnwFPtxK/

[Christian Ehrhardt  (paelzer)]

Packages got deletes, I don't find any reverse dependencies left.
Done - thanks to everybody involved.

[Christian Ehrhardt  (paelzer)]

dpdk-doc is useful, but mostly is:
a) a collection of the upstream docs as you'd read them on [1] as well
b) an API reference (generated)
c) a collection of usage examples (code)

For (a) and (b) it is not important to be in main.
A main+security system can work just fine without dpdk-doc.
But vice versa I'm concerned about the promises people might have on the code examples being in main. So we should remote them.

They are currently auto-promoted as the -dev packages were that we discussed above.
I'm extending the Extra-Exclude in that regard.

[1]: https://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html

[Christian Ehrhardt  (paelzer)]

Ok, revert that change, it is not needed.

Quote form IRC:
    <vorlon>:
     but why do you specifically want to demote these docs?
     [...] Reading https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1814060
     doesn't seem like a good reason to me to demote the docs just because the
     examples might be poor.
    <cpaelzer>:
     yes it is not dependencies, you think having those examples declared to be
     supported while not being particularly great or reliable is not an issue?
     I really think that puts too much expectations on them wich they might not hold
    <vorlon>:
     I don't think code examples in docs are ever "supported" and I have never had
     a bug report from someone who assumed they were
     I think you're trying to hold this package to a much different standard than
     the rest of the docs packages in main
    <cpaelzer>:
     Your experience on these cases clearly beats mine, so I'm ok following you
     I'll remove the Extra-exclude then