[SRU] python-botocore out of date

Bug #1875485 reported by Cody Shepherd on 2020-04-27
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-botocore (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Eoan
Undecided
Unassigned
Focal
Undecided
Unassigned

Bug Description

[Impact]

 * the python-botocore package is out of date, and out of sync with python3-botocore in debian [1], for all pockets except groovy-proposed.

 * The various outdated versions in, e.g. xenial and bionic, leave out lots of useful botocore functionality.

 * SRUing the current version back through supported suites will normalize sdk functionality across supported ubuntu releases.

[Test Case]

 * On a machine running xenial, bionic, eoan, or focal:
  `apt install python-botocore`
  `dpkg -s python-botocore`
  verify the package is installed, at its version matches the latest upstream release version, 1.15.46.

[Regression Potential]

 * apt-cache rdepends python-botocore reveals no dependent packages, so that should not be an issue.

 * however, the awscli tool source does require this package; there are probable configurations in use (e.g. pip install of awscli which use the local version of this package) wherein updates to this package will break users' expectations, especially where the version of botocore is pinned.

 * Any users on releases who are pinned to, or otherwise dependent on old versions of python-botocore being in the archive will likely be broken at some point by the package being updated in the archive.

 * New bugs introduced by botocore version releases will exist in the updated package

[Other Info]

 * The Hardware Enablement exception likely applies here, as updating this package version will enable using functionality of the AWS cloud which are impossible to use with the versions of the cli currently in the archive.

[1] https://packages.debian.org/sid/python3-botocore

description: updated
tags: added: id-5ea6eb3d73f4d32753382fc6
description: updated
description: updated
Łukasz Zemczak (sil2100) wrote :

I will sponsor the backports in a moment as these updates do make sense. That being said, the python-botocore binary packages do seem to have a few reverse-dependencies from what I see, so we'd have to make sure we didn't regress those as well:

$ reverse-depends src:python-botocore
Reverse-Recommends
* python3-mapproxy (for python3-botocore)
* snakemake (for python3-botocore)

Reverse-Depends
* awscli (for python3-botocore)
* python3-boto3 (for python3-botocore)
* python3-s3transfer (for python3-botocore)
* python3-sahara (for python3-botocore)

$ reverse-depends -b src:python-botocore
Reverse-Build-Depends-Indep
* awscli (for python3-botocore)
* python-scrapy (for python3-botocore)
* sahara (for python3-botocore)

Reverse-Build-Depends
* python-aws-xray-sdk (for python3-botocore)
* snakemake (for python3-botocore)

Changed in python-botocore (Ubuntu):
status: New → Fix Released
Łukasz Zemczak (sil2100) wrote :

Ok, uploaded backports to focal, eoan, bionic and xenial. Note to the CPC team and the SRU reviewers: it is a fact that this upload removes the python2 packages from the source, so this means that we need to make sure no package in either series will get broken - or if it will, the damage will be minimal. I leave it up to the CPC team to do the investigation.

Steve Langasek (vorlon) wrote :

$ reverse-depends -r eoan python-botocore
Reverse-Depends
* python-boto3
* python-s3transfer

Packages without architectures listed are reverse-dependencies in: amd64, arm64, armhf, i386, ppc64el, s390x
$ reverse-depends -r eoan python-botocore -a source
Reverse-Build-Depends
* python-aws-xray-sdk

$

There is no common package that python-botocore depends on, so it is possible for this binary package to be out of date and still be installable. But this does make it impossible to provide security support for the python2 package going forward, which makes me somewhat uncomfortable. Does the new upstream version of python-botocore no longer support python2? Or does something else prevent us from continuing to provide the python2 package?

Changed in python-botocore (Ubuntu Eoan):
status: New → Incomplete
Changed in python-botocore (Ubuntu Focal):
status: New → Fix Committed
Changed in python-botocore (Ubuntu Bionic):
status: New → Incomplete
Changed in python-botocore (Ubuntu Xenial):
status: New → Incomplete
tags: added: verification-needed verification-needed-focal

Hello Cody, or anyone else affected,

Accepted python-botocore into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.15.46+repack-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Brian Murray (brian-murray) wrote :

Hello Cody, or anyone else affected,

Accepted python-botocore into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Brian Murray (brian-murray) wrote :

Hello Cody, or anyone else affected,

Accepted python-botocore into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-botocore (Ubuntu Eoan):
status: Incomplete → Fix Committed
tags: added: verification-needed-eoan
Brian Murray (brian-murray) wrote :

Hello Cody, or anyone else affected,

Accepted python-botocore into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-botocore (Ubuntu Bionic):
status: Incomplete → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Cody, or anyone else affected,

Accepted python-botocore into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-botocore (Ubuntu Xenial):
status: Incomplete → Fix Committed
tags: added: verification-needed-xenial
David Krauser (davidkrauser) wrote :

Performed verification steps on a fresh bionic VM:

- Enable proposed
- apt install python-botocore/bionic-proposed and python3-botocore/bionic-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

tags: added: verification-done-bionic
removed: verification-needed-bionic
David Krauser (davidkrauser) wrote :

Performed verification steps on a fresh xenial VM:

- Enable proposed
- apt install python-botocore/xenial-proposed and python3-botocore/xenial-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

tags: added: verification-done-xenial
removed: verification-needed-xenial
David Krauser (davidkrauser) wrote :

Performed verification steps on a fresh focal VM:

- Enable proposed
- apt install python3-botocore/focal-proposed
- dpkg -s python3-botocore shows v1.16.19

tags: added: verification-done-focal
removed: verification-needed-focal
David Krauser (davidkrauser) wrote :

Performed verification steps on a fresh eoan VM:

- Enable proposed
- apt install python-botocore/eoan-proposed and python3-botocore/eoan-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

tags: added: verification-done-eoan
removed: verification-needed verification-needed-eoan
Brian Murray (brian-murray) wrote :

Before this package gets released to -updates we need to check and ensure that any packages which depend on it (see comment #1) still work. Subsequently, I'm setting the tags back to verification needed.

tags: added: verification-needed-bionic verification-needed-eoan verification-needed-focal verification-needed-xenial
removed: verification-done-bionic verification-done-eoan verification-done-focal verification-done-xenial
David Krauser (davidkrauser) wrote :

Hi Brian,

I've run autopkgtests for each package listed by reverse-depends:

reverse-depends -lsb python3-botocore

Against python3-botocore/bionic-proposed and python3-s3transfer/bionic using this helper script:

https://git.launchpad.net/~davidkrauser/+git/scripts/plain/autopkhttps://git.launchpad.net/~davidkrauser/+git/scripts/plain/autopkgtest-rdepends.sh?id=7cb005185744047f30e6f275eec19509eda50182gtest-rdepends.sh

With the command:

autopkgtest-rdepends.sh --package-under-test python3-botocore \
    --enable-proposed-for-package python3-s3transfer \
    --enable-proposed-for-package python3-botocore

This gives me the following results:

PASS: kombu
PASS: swift
FAIL: python-scrapy

To compare to the baseline results (run with python3-botocore/bionic-updates and python3-s3transfer/bionic-updates), I ran this:

autopkgtest-rdepends.sh --package-under-test python3-botocore

And got the following results:

PASS: kombu
PASS: swift
FAIL: python-scrapy

Looking at the full autopkgtest logs (linked below), these results appear to match the results I got with the packages from proposed.

Full autopackagetest logs:

https://git.launchpad.net/~davidkrauser/+git/logs/tree/LP1875485/20200710/bionic?id=a60928f005e87785719b58d519b87582e435616d

tags: added: verification-done-bionic
removed: verification-needed-bionic
David Krauser (davidkrauser) wrote :

There was a small bug in the script posted previously - it ran tests for kombu and swift that are unnecessary. The python-scrapy tests are the only relevant autopkgtests to run for python3-botocore

David Krauser (davidkrauser) wrote :

The tests in `python-scrapy-1.5.0/tests/test_downloader_handlers.py` run during python-scrapy's autopkgtests exercise botocore and s3transfer by generating mocked requests and comparing to known good values. They don't make any real API requests over the network.

David Krauser (davidkrauser) wrote :

The python-boto3 unit and functional tests seem to do a pretty good job exercising python-botocore and python-s3transfer. I ran those tests against botocore/updates and s3transfer/updates for each suite, and compared those results to the same tests run against botocore/proposed and s3transfer/proposed for each suite. The raw results are available here, alongside the script used to run the tests:

https://git.launchpad.net/~davidkrauser/+git/logs/tree/LP1875485/20200714?id=892a5e450a359e1f7aa0a43310311129ebd48518

The summary is as follows:

==> bionic-proposed-py.log <==
OK (SKIP=1)

==> bionic-proposed-py3.log <==
OK

==> bionic-updates-py.log <==
OK (SKIP=1)

==> bionic-updates-py3.log <==
OK

==> eoan-proposed-py.log <==
OK (SKIP=1)

==> eoan-proposed-py3.log <==
OK

==> eoan-updates-py.log <==
OK (SKIP=1)

==> eoan-updates-py3.log <==
OK

==> xenial-proposed-py.log <==
FAILED (SKIP=1, errors=10, failures=1)

==> xenial-proposed-py3.log <==
FAILED (errors=10, failures=1)

==> xenial-updates-py.log <==
FAILED (SKIP=1, errors=1, failures=1)

==> xenial-updates-py3.log <==
FAILED (errors=1, failures=1)

Seeing that the awscli verification for xenial against this package failed, and the test results are worse here against proposed for xenial, I'm marking this as failed verification for xenial. The other suites look to be in good order.

tags: added: verification-done-eoan verification-failed-xenial
removed: verification-needed-eoan verification-needed-xenial
David Krauser (davidkrauser) wrote :

Oops, I missed focal. Updated the repository to include logs for focal too:

https://git.launchpad.net/~davidkrauser/+git/logs/tree/LP1875485/20200714?id=17b08c1f3ea389dd3f49f2a511f6ed6ed45c6ac4

==> focal-proposed-py3.log <==
OK

==> focal-updates-py3.log <==
OK

I also forgot to mention: the documentation related python-boto3 tests were skipped for these runs. I couldn't get them to pass reliably, even with the dependencies from updates.

tags: added: verification-done-focal
removed: verification-needed-focal
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.20.04.1

---------------
python-botocore (1.16.19+repack-1ubuntu0.20.04.1) focal; urgency=medium

  * SRU the latest package version to 20.04 (LP: #1875485).

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 17 Jun 2020 02:27:41 +0200

Changed in python-botocore (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for python-botocore has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.19.10.1

---------------
python-botocore (1.16.19+repack-1ubuntu0.19.10.1) eoan; urgency=medium

  * SRU the latest package version to 19.10 (LP: #1875485).
  * debian/control: re-add the python-botocore py2 package.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 17 Jun 2020 02:17:33 +0200

Changed in python-botocore (Ubuntu Eoan):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.18.04.1

---------------
python-botocore (1.16.19+repack-1ubuntu0.18.04.1) bionic; urgency=medium

  * SRU the latest package version to 18.04 (LP: #1875485).
  * debian/control: adjust debhelper requirements to the one in bionic.
  * debian/control: re-add the python-botocore py2 package.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 17 Jun 2020 02:17:33 +0200

Changed in python-botocore (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers