[MIR] soupsieve (dependency of beautifulsoup4)

[Stefano Rivera]

[Availability]
From Debian. Bootstrapped in disco.

[Rationale]
beautifulsoup4 4.7 introduced soupsieve as a new dependency. This is replacing the same functionality that used to be part of beautifulsoup4 itself.

[Security]
This is a Python library, with no binaries on PATH.
It's relatively young, with a spotless security history, so far.

[Quality assurance]
It's a library. No configuration, no debconf questions.

There's a fairly extensive test-suite, run at build-time and as autopkgtests.
http://autopkgtest.ubuntu.com/packages/soupsieve
(Currently failing because this MIR isn't through yet and there's a missing versioned dependency in the autopkgtest. Fixed in -3)

[Dependencies]
The Python2 binary packages depend on backports.functools-lru-cache. This is a backport of the same functionality from Python 3, and is trivially maintained.

[Standards compliance]
It's a Python library, lintian-clean.

[Maintenance]
Expected to just be synced from Debian.

[Background information]
Probably promote python-soupsieve, python-backports.functools-lru-cache and python3-soupsive, but not pypy-soupsieve.

[Christian Ehrhardt  (paelzer)]

FYI some references:
- initial beautifulsoup MIR https://bugs.launchpad.net/ubuntu/+source/beautifulsoup/+bug/492560
- switching to beautifulsoup4 https://bugs.launchpad.net/ubuntu/+source/beautifulsoup4/+bug/1252623

Ack on the rationale, here the ref's of both projects [1][2] for this switch.
But this will need to go through the security Team's review (again) as it is not copying the former code from beautifulsoup4 as-is but is "a more complete CSS selector implementation".

I agree that the autopkgtests are extensive (good) but currently fail on all architectures.
That should be resolved so that there is a good baseline and broken uploads will be gated.
Seems to be the same set of errors in py2 and the py3 case.

The License was confusing at first using the ambiguous MIT license term in the project itself, but the package correctly identified it as the Expat license so things are ok here.

There is some minor packaging issues which would be nice to be resolved, but are not critical.
=> source-contains-empty-directory docs/theme/ (and it makes the tarball mismatch the packaging git)
The upstream tarball at [3] has content in that directory
$ ll docs/theme/
-rw-rw-r-- 1 paelzer paelzer 1168 Jan 23 07:16 extra-0b9b22dd13.js
-rw-rw-r-- 1 paelzer paelzer 7006 Jan 23 07:16 extra-83f68d2c59.css
So I assume that is part of the +dfsg packaging and should be improved just to be sure.

Further ok checks:
- Since the new beautifulsoup drops that function I see no code duplication issue.
- no embedded remote sources nor static linking
- dh-python is used

I currently see both main packages in main:
- python-bs4
- python3-bs4
But the new python-bs4 will pull python2 elements into main.
py2 dependencies in main are actively removed fromt he archive one by one and it is discouraged for new MIRs. And python-bs4 would depend on the py2 python-soupsieve.
I only found this in the seeds (referring to the old MIR)
  ubuntu-git/development:54: * python3-webtest
And that only pulls in python3-bs4 which would be ok.
I checked and currently (disco) python-lxml is pulling python-bs4 into main.
That dependency should be broken if possible to not add (semi-)new python2 dependencies.

I saw no team subscriber to the package yet, but that is a requirement for the MIR process.
Please get a Team to own (state it here) and subscribe to the package for maintenance.

Other than that this LGTM and IMHO this could go on as a MIR once the findings above are resolved.

[1]: https://facelessuser.github.io/soupsieve/
[2]: https://bazaar.launchpad.net/~leonardr/beautifulsoup/bs4/view/head:/CHANGELOG#L16
[3]: https://pypi.debian.net/soupsieve/soupsieve-1.7.3.tar.gz

Summary:
- @requestor: please resolve the autopkgtest failures
- @requestor: get a team to ack owning and subscribing to the package
- @requestor: break the dependency python-lxml -> python-bs4 -> python-soupsieve to not pull new py2 code into main
- Once the above is resolved it can enter the review queue of the security Team

[Stefano Rivera (stefanor)]

> That should be resolved so that there is a good baseline and broken uploads will be gated.
> Seems to be the same set of errors in py2 and the py3 case.

They're resolved in -3 and the tests are passing.

> So I assume that is part of the +dfsg packaging and should be improved just to be sure.

Yes, I stripped those files to avoid having to dig out their full history. They are minified JS, that the upstream uses in multiple projects. As I'm not building the docs, it seemed easier to just strip them for now, and figure out a better solution, later.

> @requestor: break the dependency python-lxml -> python-bs4 -> python-soupsieve to not pull new py2 code into main

If I understand correctly, you want lxml to not depend on python-bs4, so that python-bs4 can be demoted to universe? doko: This is your package.

[Christian Ehrhardt  (paelzer)]

Thanks Stefano already for the test fixups in -3.

I think we will discuss the "no-py2" rule in regard to this later to day in the MIR team meeting.
Maybe python-lxml already has a strategy to get out of main for 20.04 and we can it ignore it in regard to this MIR here as it will just "follow" that.
From what I found (see below) it might depend on the openstack off-of-py2 plan and be ok for now?
Feel free to join in #ubuntu-meeting in about ~5 hours from now.

The subscriber is what puzzles me most atm. I found that atm [1] that is actually the server Team.
Checking the old MIRs it was Chuck and James - maybe it is actually for openstack still?
Or is it actually no more needed?
Of the full reverse depends only:
- python-lxml (<- python-keystoneauth1 ...)
- python3-lxml (<- python3-ceilometer / python3-cinder / ...)
- python3-webtest (<- python3-pecan <- ceph-mgr / python3-aodh / python3-barbican / python3-neutron ...)
are in main.
That all seems like openstack to me which matches Chuck/James working on the past MIRs.
I'll discuss it with people longer in the Team than myself if they remember more of the history behind this - after all the subscriber for the new package might have to be the server team again ?!.

[1]: https://bugs.launchpad.net/ubuntu/+source/beautifulsoup4/+subscriptions
[2]: https://bugs.launchpad.net/ubuntu/+source/beautifulsoup/+bug/492560
[3]: https://bugs.launchpad.net/ubuntu/+source/beautifulsoup4/+bug/1252623

[Christian Ehrhardt  (paelzer)]

While the subscribers will eventually be resolved between server and openstack team I asked powersj to subscribe the server Team for now as we are the beautifulsoup4 owner atm.

I haven't reached any of the more experienced MIR members last week on IRC, will write a mail.
While my py2 question still needs to be solved still let me subscribe security already to have this going forward.

[Christian Ehrhardt  (paelzer)]

Discussed with the more experienced members of the MIR Team.
Since this is a new leaf package to a chain of already existing python2 dependencies it does not hit the red flag of "new py2 dependencies" as defined by the MIR process.

Mid Term it will be demotable once Openstack is off of python2.

Approve from the MIR Team's POV, waiting for the security Team who is already assigned.
We moved it up in the security teams queue after the discussion to get this resolved for 19.04 in time.

[Seth Arnold (seth-arnold)]

Because this package is derived from code that was previously in main, we decided to give this package a very light review.

I reviewed version 1.8+dfsg-1 from disco. I needed to enable the -proposed pocket in order to build this package. (Probably that's the whole point of this process.)

The code looked dense but careful. Errors were checked, there were extensive tests. pre/post inst/rm scripts were generated.

There was a lintian error during my build; is this an issue?

Setting up sbuild-build-depends-lintian-dummy (0.invalid.0) ...
E: soupsieve changes: bad-distribution-in-changes-file unstable

E: Lintian run failed (policy violation)

Security team ACK for promoting soupsieve to main.

Thanks

[Christian Ehrhardt  (paelzer)]

Thanks Seth for the review.
The Lintian warning is fine, that is just because it has no Ubuntu Delta at the moment.

I think overall this MIR concluded and is approved.

Since the code change to trigger is already waiting in disco-proposed:
beautifulsoup4 (4.6.3-2 to 4.7.1-1build1) in proposed for 25 days
  Unsatisfiable depends:
    python-soupsieve: amd64
    python3-soupsieve: amd64

The right state per [1] is "Fix Committed" and we now just need an AA to do the promotion.
Setting state and pinging in ubuntu-release ...

[1]: https://wiki.ubuntu.com/MIRTeam#Process_states

[Christian Ehrhardt  (paelzer)]

I almost forgot to state it here, this also covers the extra dep to backports.functools-lru-cache.

TL;DR:
- I reviewed it and package content and quality looks fine
- we will be able to demote it "soon" once Openstack has let go python2 which is WIP

Details:
[Duplication]
ok - Parth of the py3/py2 world, not a dup

[Embedded sources and static linking]
ok - no embedded sources or static linking
ok - no golang

[Security]
ok - this code is already acked as py3 code, the conversion does not imply a new re-review
ok - no daemons, services, setuid, ...

[Common blockers]
ok - builds fine atm
acceptable - it has rather stripped down tests as it is a broken out subfunction for something bigger - due to that those won't gate the build, not perfect but acceptable.
ok - server Team will be the subscriber who currently own beautifulsoup4
ok - no translations needed
ok - dh_python is used
ok - it is a py2 dependency, but no new one and it will be demoted onye Openstack has converted to py3

[Packaging red flags]
ok - no delta atm
ok - no symbols tracking for python
ok - has watch file
ok - updates are done regularly (by Debian)
ok - the current release is packaged
ok - no massive lintian warnings
ok - rather clean debian/rules
ok - no awkward build tweaks

[Upstream red flags]
ok - builds fine, no errors in the build
ok - no incautious use of malloc/sprintf (python after all)
ok - no use of sudo, gksu, pkexec, or LD_LIBRARY_PATH
ok - no Important bugs open, one minor already kindly handled by the Debian maintainer
ok - no dependency on webkit, qtwebkit, seed or libgoa-*
ok - no Embedded source copies

[Summary]
This can be promoted as well

Notes/TODOs:
For both packages we defined that the server Team will own them but were waiting with the actual subscription until the process completes.
Therefore I pinged @powersj to do so, then this is really ready for promotion.

[Christian Ehrhardt  (paelzer)]

Thanks btw Stefano for all the great packaging!

[Christian Ehrhardt  (paelzer)]

Sorry, but I have found that this is in the responsibility nimbus for now, give it a few days so that Powersj can sort that out. Then we can mark it ready and do the promotion.

[Joshua Powers (powersj)]

ubuntu-server subscribed

[Christian Ehrhardt  (paelzer)]

For now server Team is subscribed, all is in place and ready.
Status: Fix Committed
Next: AA to promote the packages

[Steve Langasek (vorlon)]

Override component to main
backports.functools-lru-cache 1.5-2 in disco: universe/misc -> main
pypy-backports.functools-lru-cache 1.5-2 in disco amd64: universe/python/optional/100% -> main
pypy-backports.functools-lru-cache 1.5-2 in disco arm64: universe/python/optional/100% -> main
pypy-backports.functools-lru-cache 1.5-2 in disco armhf: universe/python/optional/100% -> main
pypy-backports.functools-lru-cache 1.5-2 in disco i386: universe/python/optional/100% -> main
pypy-backports.functools-lru-cache 1.5-2 in disco ppc64el: universe/python/optional/100% -> main
pypy-backports.functools-lru-cache 1.5-2 in disco s390x: universe/python/optional/100% -> main
python-backports.functools-lru-cache 1.5-2 in disco amd64: universe/python/optional/100% -> main
python-backports.functools-lru-cache 1.5-2 in disco arm64: universe/python/optional/100% -> main
python-backports.functools-lru-cache 1.5-2 in disco armhf: universe/python/optional/100% -> main
python-backports.functools-lru-cache 1.5-2 in disco i386: universe/python/optional/100% -> main
python-backports.functools-lru-cache 1.5-2 in disco ppc64el: universe/python/optional/100% -> main
python-backports.functools-lru-cache 1.5-2 in disco s390x: universe/python/optional/100% -> main
Override [y|N]? y
13 publications overridden.

[Steve Langasek (vorlon)]

Override component to main
soupsieve 1.8+dfsg-1 in disco: universe/misc -> main
pypy-soupsieve 1.8+dfsg-1 in disco amd64: universe/python/optional/100% -> main
pypy-soupsieve 1.8+dfsg-1 in disco arm64: universe/python/optional/100% -> main
pypy-soupsieve 1.8+dfsg-1 in disco armhf: universe/python/optional/100% -> main
pypy-soupsieve 1.8+dfsg-1 in disco i386: universe/python/optional/100% -> main
pypy-soupsieve 1.8+dfsg-1 in disco ppc64el: universe/python/optional/100% -> main
pypy-soupsieve 1.8+dfsg-1 in disco s390x: universe/python/optional/100% -> main
python-soupsieve 1.8+dfsg-1 in disco amd64: universe/python/optional/100% -> main
python-soupsieve 1.8+dfsg-1 in disco arm64: universe/python/optional/100% -> main
python-soupsieve 1.8+dfsg-1 in disco armhf: universe/python/optional/100% -> main
python-soupsieve 1.8+dfsg-1 in disco i386: universe/python/optional/100% -> main
python-soupsieve 1.8+dfsg-1 in disco ppc64el: universe/python/optional/100% -> main
python-soupsieve 1.8+dfsg-1 in disco s390x: universe/python/optional/100% -> main
python3-soupsieve 1.8+dfsg-1 in disco amd64: universe/python/optional/100% -> main
python3-soupsieve 1.8+dfsg-1 in disco arm64: universe/python/optional/100% -> main
python3-soupsieve 1.8+dfsg-1 in disco armhf: universe/python/optional/100% -> main
python3-soupsieve 1.8+dfsg-1 in disco i386: universe/python/optional/100% -> main
python3-soupsieve 1.8+dfsg-1 in disco ppc64el: universe/python/optional/100% -> main
python3-soupsieve 1.8+dfsg-1 in disco s390x: universe/python/optional/100% -> main
19 publications overridden.