[SRU] python-botocore out of date

[Cody Shepherd]

[Impact]

 * the python-botocore package is out of date, and out of sync with python3-botocore in debian [1], for all pockets except groovy-proposed.

 * The various outdated versions in, e.g. xenial and bionic, leave out lots of useful botocore functionality.

 * SRUing the current version back through supported suites will normalize sdk functionality across supported ubuntu releases.

[Test Case]

 * On a machine running xenial, bionic, eoan, or focal:
  `apt install python-botocore`
  `dpkg -s python-botocore`
  verify the package is installed, at its version matches the latest upstream release version, 1.15.46.

[Regression Potential]

 * apt-cache rdepends python-botocore reveals no dependent packages, so that should not be an issue.

 * however, the awscli tool source does require this package; there are probable configurations in use (e.g. pip install of awscli which use the local version of this package) wherein updates to this package will break users' expectations, especially where the version of botocore is pinned.

 * Any users on releases who are pinned to, or otherwise dependent on old versions of python-botocore being in the archive will likely be broken at some point by the package being updated in the archive.

 * New bugs introduced by botocore version releases will exist in the updated package

[Other Info]

 * The Hardware Enablement exception likely applies here, as updating this package version will enable using functionality of the AWS cloud which are impossible to use with the versions of the cli currently in the archive.

[1] https://packages.debian.org/sid/python3-botocore

[Łukasz Zemczak (sil2100)]

I will sponsor the backports in a moment as these updates do make sense. That being said, the python-botocore binary packages do seem to have a few reverse-dependencies from what I see, so we'd have to make sure we didn't regress those as well:

$ reverse-depends src:python-botocore
Reverse-Recommends
* python3-mapproxy (for python3-botocore)
* snakemake (for python3-botocore)

Reverse-Depends
* awscli (for python3-botocore)
* python3-boto3 (for python3-botocore)
* python3-s3transfer (for python3-botocore)
* python3-sahara (for python3-botocore)

$ reverse-depends -b src:python-botocore
Reverse-Build-Depends-Indep
* awscli (for python3-botocore)
* python-scrapy (for python3-botocore)
* sahara (for python3-botocore)

Reverse-Build-Depends
* python-aws-xray-sdk (for python3-botocore)
* snakemake (for python3-botocore)

[Łukasz Zemczak (sil2100)]

Ok, uploaded backports to focal, eoan, bionic and xenial. Note to the CPC team and the SRU reviewers: it is a fact that this upload removes the python2 packages from the source, so this means that we need to make sure no package in either series will get broken - or if it will, the damage will be minimal. I leave it up to the CPC team to do the investigation.

[Steve Langasek (vorlon)]

$ reverse-depends -r eoan python-botocore
Reverse-Depends
* python-boto3
* python-s3transfer

Packages without architectures listed are reverse-dependencies in: amd64, arm64, armhf, i386, ppc64el, s390x
$ reverse-depends -r eoan python-botocore -a source
Reverse-Build-Depends
* python-aws-xray-sdk

$

There is no common package that python-botocore depends on, so it is possible for this binary package to be out of date and still be installable. But this does make it impossible to provide security support for the python2 package going forward, which makes me somewhat uncomfortable. Does the new upstream version of python-botocore no longer support python2? Or does something else prevent us from continuing to provide the python2 package?

[Steve Langasek (vorlon)]

Hello Cody, or anyone else affected,

Accepted python-botocore into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.15.46+repack-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Cody, or anyone else affected,

Accepted python-botocore into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Cody, or anyone else affected,

Accepted python-botocore into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Cody, or anyone else affected,

Accepted python-botocore into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Cody, or anyone else affected,

Accepted python-botocore into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[David Krauser (davidkrauser)]

Performed verification steps on a fresh bionic VM:

- Enable proposed
- apt install python-botocore/bionic-proposed and python3-botocore/bionic-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

[David Krauser (davidkrauser)]

Performed verification steps on a fresh xenial VM:

- Enable proposed
- apt install python-botocore/xenial-proposed and python3-botocore/xenial-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

[David Krauser (davidkrauser)]

Performed verification steps on a fresh focal VM:

- Enable proposed
- apt install python3-botocore/focal-proposed
- dpkg -s python3-botocore shows v1.16.19

[David Krauser (davidkrauser)]

Performed verification steps on a fresh eoan VM:

- Enable proposed
- apt install python-botocore/eoan-proposed and python3-botocore/eoan-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

[Brian Murray (brian-murray)]

Before this package gets released to -updates we need to check and ensure that any packages which depend on it (see comment #1) still work. Subsequently, I'm setting the tags back to verification needed.

[David Krauser (davidkrauser)]

Hi Brian,

I've run autopkgtests for each package listed by reverse-depends:

reverse-depends -lsb python3-botocore

Against python3-botocore/bionic-proposed and python3-s3transfer/bionic using this helper script:

https://git.launchpad.net/~davidkrauser/+git/scripts/plain/autopkhttps://git.launchpad.net/~davidkrauser/+git/scripts/plain/autopkgtest-rdepends.sh?id=7cb005185744047f30e6f275eec19509eda50182gtest-rdepends.sh

With the command:

autopkgtest-rdepends.sh --package-under-test python3-botocore \
    --enable-proposed-for-package python3-s3transfer \
    --enable-proposed-for-package python3-botocore

This gives me the following results:

PASS: kombu
PASS: swift
FAIL: python-scrapy

To compare to the baseline results (run with python3-botocore/bionic-updates and python3-s3transfer/bionic-updates), I ran this:

autopkgtest-rdepends.sh --package-under-test python3-botocore

And got the following results:

PASS: kombu
PASS: swift
FAIL: python-scrapy

Looking at the full autopkgtest logs (linked below), these results appear to match the results I got with the packages from proposed.

Full autopackagetest logs:

https://git.launchpad.net/~davidkrauser/+git/logs/tree/LP1875485/20200710/bionic?id=a60928f005e87785719b58d519b87582e435616d

[David Krauser (davidkrauser)]

Sorry, link to helper script was mangled in previous message:

https://git.launchpad.net/~davidkrauser/+git/scripts/plain/autopkgtest-rdepends.sh?id=7cb005185744047f30e6f275eec19509eda50182

[David Krauser (davidkrauser)]

There was a small bug in the script posted previously - it ran tests for kombu and swift that are unnecessary. The python-scrapy tests are the only relevant autopkgtests to run for python3-botocore

[David Krauser (davidkrauser)]

The tests in `python-scrapy-1.5.0/tests/test_downloader_handlers.py` run during python-scrapy's autopkgtests exercise botocore and s3transfer by generating mocked requests and comparing to known good values. They don't make any real API requests over the network.

[David Krauser (davidkrauser)]

The python-boto3 unit and functional tests seem to do a pretty good job exercising python-botocore and python-s3transfer. I ran those tests against botocore/updates and s3transfer/updates for each suite, and compared those results to the same tests run against botocore/proposed and s3transfer/proposed for each suite. The raw results are available here, alongside the script used to run the tests:

https://git.launchpad.net/~davidkrauser/+git/logs/tree/LP1875485/20200714?id=892a5e450a359e1f7aa0a43310311129ebd48518

The summary is as follows:

==> bionic-proposed-py.log <==
OK (SKIP=1)

==> bionic-proposed-py3.log <==
OK

==> bionic-updates-py.log <==
OK (SKIP=1)

==> bionic-updates-py3.log <==
OK

==> eoan-proposed-py.log <==
OK (SKIP=1)

==> eoan-proposed-py3.log <==
OK

==> eoan-updates-py.log <==
OK (SKIP=1)

==> eoan-updates-py3.log <==
OK

==> xenial-proposed-py.log <==
FAILED (SKIP=1, errors=10, failures=1)

==> xenial-proposed-py3.log <==
FAILED (errors=10, failures=1)

==> xenial-updates-py.log <==
FAILED (SKIP=1, errors=1, failures=1)

==> xenial-updates-py3.log <==
FAILED (errors=1, failures=1)

Seeing that the awscli verification for xenial against this package failed, and the test results are worse here against proposed for xenial, I'm marking this as failed verification for xenial. The other suites look to be in good order.

[David Krauser (davidkrauser)]

Oops, I missed focal. Updated the repository to include logs for focal too:

https://git.launchpad.net/~davidkrauser/+git/logs/tree/LP1875485/20200714?id=17b08c1f3ea389dd3f49f2a511f6ed6ed45c6ac4

==> focal-proposed-py3.log <==
OK

==> focal-updates-py3.log <==
OK

I also forgot to mention: the documentation related python-boto3 tests were skipped for these runs. I couldn't get them to pass reliably, even with the dependencies from updates.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.20.04.1

---------------
python-botocore (1.16.19+repack-1ubuntu0.20.04.1) focal; urgency=medium

  * SRU the latest package version to 20.04 (LP: #1875485).

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 17 Jun 2020 02:27:41 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-botocore has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.19.10.1

---------------
python-botocore (1.16.19+repack-1ubuntu0.19.10.1) eoan; urgency=medium

  * SRU the latest package version to 19.10 (LP: #1875485).
  * debian/control: re-add the python-botocore py2 package.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 17 Jun 2020 02:17:33 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.18.04.1

---------------
python-botocore (1.16.19+repack-1ubuntu0.18.04.1) bionic; urgency=medium

  * SRU the latest package version to 18.04 (LP: #1875485).
  * debian/control: adjust debhelper requirements to the one in bionic.
  * debian/control: re-add the python-botocore py2 package.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 17 Jun 2020 02:17:33 +0200

[Brian Murray (brian-murray)]

Hello Cody, or anyone else affected,

Accepted python-botocore into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-botocore/1.16.19+repack-1ubuntu0.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lukas Märdian (slyon)]

Hi Brian,

I performed the verification steps on a fresh xenial LXC, using python-botocore 1.16.19+repack-1ubuntu0.16.04.2:

- Enable proposed
- apt install python-botocore/xenial-proposed and python3-botocore/xenial-proposed
- dpkg -s python-botocore shows v1.16.19
- dpkg -s python3-botocore shows v1.16.19

root@namely-premium-longhorn:~# dpkg -l | grep botocore
ii python-botocore 1.16.19+repack-1ubuntu0.16.04.2 all Low-level, data-driven core of boto 3 (Python 2)
ii python3-botocore 1.16.19+repack-1ubuntu0.16.04.2 all Low-level, data-driven core of boto 3 (Python 3)

Additionally, I executed David's test script, to run the python-boto3 unit and functional tests exercising python-botocore and python-s3transfer. I ran those tests against botocore/updates and s3transfer/updates for xenial, and compared those results to the same tests run against botocore/proposed and s3transfer/proposed for xenial. The failures observed in -proposed are equal to the baseline (-updates) failures. The errors observed in David's previous test runs on xenial are gone with 1.16.19+repack-1ubuntu0.16.04.2.

The raw results are available here, alongside the (modified) script used to run the tests:
https://git.launchpad.net/~slyon/+git/files/tree/LP1875485

==> xenial-proposed-py.log <==
test_serialize_bytes (tests.unit.dynamodb.test_types.TestSerializer) ... SKIP
ERROR: test_resource_lazy_loads_properties
FAIL: tests.functional.test_smoke.test_api_versions_synced_with_botocore
FAILED (SKIP=1, errors=1, failures=1)

==> xenial-proposed-py3.log <==
ERROR: test_resource_lazy_loads_properties
FAIL: tests.functional.test_smoke.test_api_versions_synced_with_botocore
FAILED (errors=1, failures=1)

==> xenial-updates-py.log <==
test_serialize_bytes (tests.unit.dynamodb.test_types.TestSerializer) ... SKIP
ERROR: test_resource_lazy_loads_properties
FAIL: tests.functional.test_smoke.test_api_versions_synced_with_botocore
FAILED (SKIP=1, errors=1, failures=1)

==> xenial-updates-py3.log <==
ERROR: test_resource_lazy_loads_properties
FAIL: tests.functional.test_smoke.test_api_versions_synced_with_botocore
FAILED (errors=1, failures=1)

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-botocore - 1.16.19+repack-1ubuntu0.16.04.2

---------------
python-botocore (1.16.19+repack-1ubuntu0.16.04.2) xenial; urgency=medium

  * Fix SRU of the latest package version for 16.04 (LP: #1875485)
  * d/p/0002-Avoid-ssl_context-and-chunked-arguments-with-legacy-.patch
    - Xenial is shipping urllib3 1.13
    - Avoid the 'ssl_context' argument, only available as of urllib3 >= 1.17
    - Avoid the 'chunked' argument, only available as of urllib3 >= 1.15

python-botocore (1.16.19+repack-1ubuntu0.16.04.1) xenial; urgency=medium

  * SRU the latest package version to 16.04 (LP: #1875485).
  * debian/control: adjust debhelper requirements to the one in xenial.
  * debian/control: re-add the python-botocore py2 package.

python-botocore (1.16.19+repack-1) unstable; urgency=medium

  * New upstream version 1.16.19+repack

python-botocore (1.15.46+repack-1) unstable; urgency=medium

  * New upstream version 1.15.46+repack

python-botocore (1.15.26+repack-1) unstable; urgency=medium

  * New upstream version 1.15.26+repack

python-botocore (1.15.21+repack-1) unstable; urgency=medium

  * New upstream version 1.15.21+repack
  * Refresh patch 0001-Don-t-use-duplicated-modules.patch

python-botocore (1.14.14+repack-1) unstable; urgency=medium

  * New upstream version 1.14.14+repack
  * Set upstream metadata fields: Bug-Database, Bug-Submit.
  * Bump Standards-Version to 4.5.0

python-botocore (1.13.37+repack-1) unstable; urgency=medium

  * Set upstream metadata fields: Repository, Repository-Browse.
  * New upstream version 1.13.37+repack

python-botocore (1.13.19+repack-1) unstable; urgency=medium

  * New upstream version 1.13.19+repack

python-botocore (1.12.241+repack-2) unstable; urgency=medium

  [ Ondřej Nový ]
  * Bump Standards-Version to 4.4.1.
  * d/changelog: Remove trailing whitespaces.

  [ Sandro Tosi ]
  * Drop python2 support; Closes: #937619

python-botocore (1.12.241+repack-1) unstable; urgency=medium

  * New upstream version 1.12.241+repack
  * Set upstream metadata fields: Repository.
  * Add debian/salsa-ci.yml.

python-botocore (1.12.208+repack-1) unstable; urgency=medium

  * New upstream version 1.12.208+repack

python-botocore (1.12.200+repack-1) unstable; urgency=medium

  * New upstream version 1.12.200+repack
  * Use debhelper-compat instead of debian/compat.
  * Bump Standards-Version to 4.4.0.

python-botocore (1.12.103+repack-1) unstable; urgency=medium

  * New upstream version 1.12.103+repack

python-botocore (1.12.71+repack-1) unstable; urgency=medium

  * New upstream version 1.12.71+repack
  * Bump Stanrads-Version to 4.3.0

python-botocore (1.12.53+repack-1) unstable; urgency=medium

  * New upstream version 1.12.53+repack
  * debian/patches/*: Refresh patches.

python-botocore (1.12.16+repack-1) unstable; urgency=medium

  * New upstream version 1.12.16+repack
  * debian/patches: Refresh patch files.
  * Bump Stanrads-Version to 4.2.1

python-botocore (1.10.78+repack-1) unstable; urgency=medium

  [ Ondřej Nový ]
  * Convert git repository from git-dpm to gbp layout

  [ TANIGUCHI Takaki ]
  * New upstream version 1.10.78+repack
  * Bump Stanrads-Version to 4.2.0.1

py...

[Marcin Sulikowski (msulikowski)]

I found python-botocore 1.16.19+repack-1ubuntu0.16.04.2 in xenial-updates to cause problems, see LP #1905069