Resolve autopkgtest failures Pep8/pyflakes on Xenial and Bionic

[Chad Smith]

[Impact]

- autopkgtest failures on for Xenial(pep8) and Bionic(pep8 & pyflakes)

- Cherry-picking upstream changes for ESM Product renaming and messaging
  for bug 1901627 trigger these issues.

- Also changes in the Xenial base cloud-images used for testing dropped
  pep8 package from the build environment and unittests rely on calling
  pep8 on Xenial and Bionic. minimally a build-depends on pep8 needs to be
  added and pep8 test failures addresssed for Xenial. Bionic also needs
  minor pyflakes fixes from master.

[Test Case]

 * Testing is "implicit", build time and autopkgtest tests have to
   succeed as we'd expect anyway.

[Where problems could occur]

 * These are whitespace/indent/style changes that should (tm) be totally
   without an effect other than for the code-checker. Never the less by
   changing code always something could happen. In this case the changes
   are local to apt_check, package-data-downloader, backend_helper and
   hooks. So in these subfunctions of update-notifier issue would be
   expected.

[Other Info]

 * This does not directly affect Focal or later because update-notifier
   moved to pycodestyle validation in tests instead of using pep8.
 * The Xenial upload will throw errors of some epxect-to-fail tests that
   are misleading. We've spent too many iterations on looking at these so
   they are silenced while fixing the test anyway.

--- ---

Initial pep8 build-dependency error on Xenial/Bionic autopkgtests:

Fetched 4 B in 0s (0 B/s)
....Get:1 http://localhost:17195/canary-file.txt [4 B]
Fetched 4 B in 0s (0 B/s)
.E
======================================================================
ERROR: test_pep8_clean (test_pep8.TestPep8Clean)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.RQGkx9/build.3cC/src/tests/test_pep8.py", line 32, in test_pep8_clean
    ["pep8", "--ignore={0}".format(IGNORE_PEP8), py_file])
  File "/usr/lib/python3.5/subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'pep8'

Once resolving the build-depends: example pep8 test failures: seen running nosetests3 on update-notifier on Xenial amd Bionic

./tmp/autopkgtest.wN63go/build.IFg/src/tests/../data/apt_check.py:157:21: W503 line break before binary operator ....

autopackage test failure logs showing missing pep8 dependency:

xenial: update-notifier/3.168.11
   - autopkg logs: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/amd64/u/update-notifier/20201016_043912_51752@/log.gz

bionic: update-notifier/3.192.1.8
    - autopkg failures: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/u/update-notifier/20201016_045656_8c126@/log.gz

focal: update-notifier/3.192.30.1
  - rejected this upload in Focal to reduce SRU thrashing because a followup will be provided for Focal anyway to sync with Xenial and Bionic.

[Timo Aaltonen (tjaalton)]

Hello Chad, or anyone else affected,

Accepted update-notifier into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.1.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Chad, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.168.12) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

update-notifier/3.168.12 (ppc64el, amd64, arm64, s390x, armhf, i386)
update-manager/1:16.04.17 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Timo Aaltonen (tjaalton)]

Hello Chad, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.168.13) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

update-manager/1:16.04.17 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Chad Smith (chad.smith)]

Xenial and Bionic autopkgtest results now succeed without flake errors
xenial: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/amd64/u/update-manager/20201221_145330_55dfc@/log.gz

bionic: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/u/update-manager/20201204_193308_8f594@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.168.13

---------------
update-notifier (3.168.13) xenial; urgency=medium

  * Fix pep8 autopkgtest failures in the right control file (LP: #1906436)
    - d/control: remove pep8 from Build-Depends as no tests run by
      override_dh_auto_test use pep8.
    - d/tests/control: add pep8 to Depends as it is only used by
      python-based autopkgtests via subprocess in test_motd.py.
    - d/tests/test_package-data-downloader.py: mock subprocess calls to
      apt-helper for expected failure tests to avoid emitting error messages
      to console

update-notifier (3.168.12) xenial; urgency=medium

  * data/apt_check.py: Update UA Infra: ESM product name and doc url
    (LP: #1901627)
    - data/apt_check.py: Update name and URL
    - tests/test_motd.py: adapt unittests to match new behavior
    - po/*.po: translation files with intltool-update -r
  * Fix pep8 autopkgtest failures (LP: 1906436)
    - d/control: add expilicit pep8 build-requires dependency
    - data/apt_check.py: fix over-indent pep8 errors
    - data/apt_check & data/backend_helper: resolve underindent pep8 issues
      backport of 9e0f7ee50 [ Brian Murray ]
    - data/apt_check.py, data/package-data-downloader, tests/test_pep8.py:
      + update the code formating to be not hit W504 warnings,
        change to ignore W503 and be consistent with update-manager.
        [ Andrea Azzarone ]
    - INSTALL, data/hooks.py, tests/test_package-data-downloader.py:
      Fix E117 over-indented pep issues. [ Gianfranco Costamagna ]

update-notifier (3.168.11) xenial; urgency=medium

  [ Julian Andres Klode ]
  * Handle missing cases of LP: #1822340, where we told people ESM is not
    enabled, but not how to enable it.
  * Fix multiple disabled ESM repositories being counted as enabled ones.

  [ Brian Murray ]
  * data/apt_check.py: modify wording and output regarding ESM support.
    (LP: #1842508)

  [ Andreas Hasenack ]
  * data/apt_check.py: Update ESM security pockets names (LP: #1881632)
    - the UbuntuESM pocket was renamed from <distro>-security to
      <distro>-infra-security
    - new origin UbuntuESMApps, with a corresponding pocket of
      <distro>-apps-security

 -- Chad Smith <email address hidden> Thu, 10 Dec 2020 14:40:50 -0700

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.1.9

---------------
update-notifier (3.192.1.9) bionic; urgency=medium

  * data/apt_check.py: Update UA Infra: ESM product name and doc url
    (LP: #1901627)
    - data/apt_check.py: Update name and URL
    - tests/test_motd.py: adapt unittests to match new behavior
    - po/*.po: translation files with intltool-update -r
  * Fix pep8/pyflakes autopkgtest failures (LP: #1906436)
    - d/control: add explicit pep8 build-requires dependency
    - test_motd.py: remove unused imports
    - pyflakes: data/backend_helper fix unittests for unreferenced
      variables
    - data/apt_check & data/backend_helper: resolve underindent pep8 issues
      backport of 9e0f7ee50 [ Brian Murray ]
    - data/apt_check.py, data/package-data-downloader, tests/test_pep8.py:
      + update the code formating to be not hit W504 warnings,
        change to ignore W503 and be consistent with update-manager.
        [ Andrea Azzarone ]
    - INSTALL, data/hooks.py, tests/test_package-data-downloader.py:
      Fix E117 over-indented pep issues. [ Gianfranco Costamagna ]
    - test_motd.py: Remove unused imports [ Julian Andres Klode ]
    - pyflakes: data/backend_helper fix unittests for unreferenced variables

update-notifier (3.192.1.8) bionic; urgency=medium

  [ Andreas Hasenack ]
  * data/apt_check.py: Update ESM security pockets names (LP: #1881632)
    - the UbuntuESM pocket was renamed from <distro>-security to
      <distro>-infra-security
    - new origin UbuntuESMApps, with a corresponding pocket of
      <distro>-apps-security

  [ Brian Murray ]
  * data/apt_check.py: modify wording and output regarding ESM support.
    (LP: #1842508)

  [ Julian Andres Klode ]
  * Handle missing cases of LP: #1822340, where we told people ESM is not
    enabled, but not how to enable it.
  * Fix multiple disabled ESM repositories being counted as enabled ones.

 -- Chad Smith <email address hidden> Mon, 30 Nov 2020 14:25:35 -0700