python3.4 autopkg test failures

[Matthias Klose]

see
https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/job/trusty-adt-python3.4/24/

disabled these in the autopkg tests for now. need some investigation

[Matthias Klose (doko)]

test_compileall fails on the autopkg test setup only

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4~b1-4ubuntu4

---------------
python3.4 (3.4~b1-4ubuntu4) trusty; urgency=medium

  * Disable test_compileall for the autopkg tests, fails only there.
 -- Matthias Klose <email address hidden> Sat, 28 Dec 2013 01:47:57 +0100

[Matthias Klose (doko)]

won't fix anymore for 3.3, removed in trusty

[Matthias Klose (doko)]

need to investigate: test_importlib, test_non_namespace_package_takes_precedence:

FAIL: test_non_namespace_package_takes_precedence (test.test_importlib.test_namespace_pkgs.LegacySupport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/scratch/packages/python/3.4/python3.4-3.4.0/Lib/test/test_importlib/test_namespace_pkgs.py", line 207, in test_non_namespace_package_takes_precedence
    import foo.two
AssertionError: ImportError not raised

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4.0-2ubuntu1

---------------
python3.4 (3.4.0-2ubuntu1) trusty; urgency=medium

  * Fix a distutils test error, skip a Solaris distutils test error.
  * Fix the importlib test failures, caused by moving around the test data.
    Delay looking at one remaining test failure, see LP: #1264554.
  * Skip the test_platform encoding test, failing with the lsb-release patch.
  * d/p/distutils-install-layout.diff, d/p/site-locations.diff: Adjust the
    "am I in a virtual environment" tests to include checking
    sys.base_prefix != sys.prefix. This is the definitive such test for
    pyvenv created virtual environments (Barry Warsaw).
  * Don't yet install the ensurepip module, requires further work.
    ensurepip wants to install bundled modules setuptools and python-pip,
    which should be built from the Ubuntu packages instead of using the
    bundled code.
 -- Matthias Klose <email address hidden> Fri, 11 Apr 2014 13:44:05 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4.1~rc1-1ubuntu3

---------------
python3.4 (3.4.1~rc1-1ubuntu3) utopic; urgency=medium

  * Set a temporary home directory for the autopkg tests.
  * Fix issue #17752, test_distutils failures in the installed location.
 -- Matthias Klose <email address hidden> Wed, 07 May 2014 00:10:07 +0200

[Steve Langasek (vorlon)]

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

Still fails: https://jenkins.qa.ubuntu.com/job/trusty-adt-python3.4/83/

======================================================================
ERROR: test_dh_params (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/test/test_ssl.py", line 2728, in test_dh_params
    chatty=True, connectionchatty=True)
  File "/usr/lib/python3.4/test/test_ssl.py", line 1866, in server_params_test
    s.connect((HOST, server.port))
  File "/usr/lib/python3.4/ssl.py", line 846, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python3.4/ssl.py", line 837, in _real_connect
    self.do_handshake()
  File "/usr/lib/python3.4/ssl.py", line 810, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSL_NEGATIVE_LENGTH] dh key too small (_ssl.c:600)

[Martin Pitt (pitti)]

Same error in the cloud tests: http://autopkgtest.ubuntu.com/packages/p/python3.4/trusty/amd64

[Martin Pitt (pitti)]

This is caused by a recent OpenSSL update in trusty-security. This needs https://hg.python.org/cpython/rev/1ad7c0253abe backported, I confirmed that this fixes the test.

[Adam Conrad (adconrad)]

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Matthias Klose (doko)]

confirmed that 3.4.3-1ubuntu1~14.04.1 passes the test_ssl test

[Martin Pitt (pitti)]

Confirmed, http://autopkgtest.ubuntu.com/packages/p/python3.4/trusty/amd64/ (and i386) passing now. Thanks Matthias!

[Páll Haraldsson (pall-haraldsson)]

The changelog says "replacing the 512 bit dh key with a 2014 bit one" coming from this:

"This is caused by a recent OpenSSL update in trusty-security. This needs https://hg.python.org/cpython/rev/1ad7c0253abe backported, I confirmed that this fixes the test."

That commit says the same, but looking (closer at it) at the file it confirms, 2014 (an odd number) is a typo(s) for 1024. [I wander if the changelog here (or in Debian and Python) needs to be updated, or even if it is a possibility to change retroactively. I guess you could add a correction entry..]

This also does:

http://bugs.python.org/issue23844

and confirms the change was also made for Python 2.7. I can't see a similar update done to it (or [lib]python2.7-minimal). All python2.7 updates I see are from the month before.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4.3-1ubuntu1~14.04.1

---------------
python3.4 (3.4.3-1ubuntu1~14.04.1) trusty-proposed; urgency=medium

  * Backport issue #23844 from the 3.4 branch, replacing the 512 bit dh key
    with a 2014 bit one. Triggered by OpenSSL security update in
    trusty-security. LP: #1264554.
  * Fix expansion of makefile macros for _sysconfigdata. Issue #24705.

python3.4 (3.4.3-1ubuntu1~14.04) trusty-proposed; urgency=medium

  * SRU: Update Python3 for trusty. LP: #1348954.

python3.4 (3.4.3-1ubuntu1) vivid; urgency=medium

  * debian/tests: Use init system agnostic "service" command instead of
    upstart specific "stop". Also drop unnecessary "status" call right after
    stopping apport.

python3.4 (3.4.3-1) experimental; urgency=medium

  * Python 3.4.3 release.
  * Changes since 20141202 (3.4.2-4):
    - Issue #22896: Avoid using PyObject_AsCharBuffer(),
      PyObject_AsReadBuffer(), and PyObject_AsWriteBuffer().
    - Issue #21295: Revert some changes (issue #16795) to AST line numbers and
      column offsets that constituted a regression.
    - Issue #21408: The default __ne__() now returns NotImplemented if __eq__()
      returned NotImplemented.
    - Issue #23321: Fixed a crash in str.decode() when error handler returned
      replacment string longer than mailformed input data.
    - Issue #23048: Fix jumping out of an infinite while loop in the pdb.
    - Issue #23165: Perform overflow checks before allocating memory in the
      _Py_char2wchar function.
    - Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
      prevent corrupting exported buffer.
    - Issue #23363: Fix possible overflow in itertools.permutations.
    - Issue #23364: Fix possible overflow in itertools.product.
    - Issue #23366: Fixed possible integer overflow in itertools.combinations.
    - Issue #23369: Fixed possible integer overflow in
      _json.encode_basestring_ascii.
    - Issue #23353: Fix the exception handling of generators in
      PyEval_EvalFrameEx(). At entry, save or swap the exception state even if
      PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception
      state is now always restored or swapped, not only if why is WHY_YIELD or
      WHY_RETURN.
    - Issue #18518: timeit now rejects statements which can't be compiled
      outside a function or a loop (e.g. "return" or "break").
    - Issue #23094: Fixed readline with frames in Python implementation of
      pickle.
    - Issue #23268: Fixed bugs in the comparison of ipaddress classes.
    - Issue #21408: Removed incorrect implementations of __ne__() which didn't
      returned NotImplemented if __eq__() returned NotImplemented. The default
      __ne__() now works correctly.
    - Issue #19996: :class:`email.feedparser.FeedParser` now handles
      (malformed) headers with no key rather than amusing the body has started.
    - Issue #23248: Update ssl error codes from latest OpenSSL git master.
    - Issue #23098: 64-bit dev_t is now supported in the os module.
    - Issue #23250: In the http.cookies module, capitalize "HttpOnly" and
      "Secure" as they are written in the standard.
    - Issue #23...

[Chris J Arges (arges)]

The verification of the Stable Release Update for python3.4 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Langasek (vorlon)]

python3.4 3.4.3-1ubuntu1~14.04.1 has been removed from trusty-updates due to bug #1500768 which is a regression introduced in this SRU. This SRU can be reconsidered once this regression has been resolved.

[Chris J Arges (arges)]

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Something has changed between the ~14.04.1 upload and the ~14.04.3 upload, and these tests are no longer passing on the autopkgtest infrastructure. So this bug is unfortunately verification-failed.

However, this is not a regression and the tests in question have passed as part of the package build; so this may be a regression in the autopkgtest environment rather than in the package. So since the python3.4 SRU is somewhat urgent after the previous one has been withdrawn, I'm going to go ahead with releasing this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4.3-1ubuntu1~14.04.3

---------------
python3.4 (3.4.3-1ubuntu1~14.04.3) trusty; urgency=medium

  * Remove the config file from the package, as there is no handling in
    place to deal with this config file on upgrade and it is not appropriate
    for inclusion in an urgent SRU.

 -- Steve Langasek <email address hidden> Wed, 14 Oct 2015 12:52:19 -0700

[Martin Pitt (pitti)]

For the record, this reproduces perfectly well in local QEMU:

  adt-run --apt-pocket=proposed -U -s --testname testsuite python3.4 --- qemu /srv/vm/adt-trusty-amd64-cloud.img

so this is not specific to the production CI environment. -s starts a shell after failure; after that the failure can be reproduced more quickly and directly:

$ python3.4 -W default -bb -E -R -m test -j 1 -w -uall,-network,-urlfetch,-gui test.test_socket
[...]
======================================================================
ERROR: testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/test/test_socket.py", line 2363, in testRecvmsgPeek
    socket.MSG_PEEK)
  File "/usr/lib/python3.4/test/test_socket.py", line 1907, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 14] Bad address

======================================================================
ERROR: testRecvmsgPeek (test.test_socket.RecvmsgIntoUDP6Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/test/test_socket.py", line 2363, in testRecvmsgPeek
    socket.MSG_PEEK)
  File "/usr/lib/python3.4/test/test_socket.py", line 1998, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 14] Bad address

However, downgrading the packages to 3.4.3-1ubuntu1~14.04.1 fails on the exact same issue. So this indeed does not look like a regression in python3.4 itself but in one of its underlying libraries or perhaps a changed kernel.

[Martin Pitt (pitti)]

Even faster:

$ python3.4 -W default -bb -E -R -m test -j 1 -v -m testRecvmsgPeek -uall,-network,-urlfetch,-gui test.test_socket

Attaching strace for one of the failed tests. The interesting part:

[pid 10253] socket(PF_INET6, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 4
[pid 10253] bind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
[pid 10253] getsockname(4, {sa_family=AF_INET6, sin6_port=htons(48419), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
[pid 10253] getsockname(4, {sa_family=AF_INET6, sin6_port=htons(48419), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
[pid 10253] recvmsg(4, <unfinished ...>
[pid 10253] <... recvmsg resumed> 0x7ffef846d890, MSG_PEEK) = -1 EFAULT (Bad address)

EFAULT is documented as "The receive buffer pointer(s) point outside the process's address space", hmm.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.4 - 3.4.3-1ubuntu1~14.04.5

---------------
python3.4 (3.4.3-1ubuntu1~14.04.5) trusty-security; urgency=medium

  * SECURITY UPDATE: StartTLS stripping attack
    - debian/patches/CVE-2016-0772.patch: raise an error when
      STARTTLS fails in Lib/smtplib.py.
    - CVE-2016-0772
  * SECURITY UPDATE: use of HTTP_PROXY flag supplied by attacker in CGI
    scripts (aka HTTPOXY attack)
    - debian/patches/CVE-2016-1000110.patch: if running as CGI
      script, forget HTTP_PROXY in Lib/urllib.py, add test to
      Lib/test/test_urllib.py, add documentation.
    - CVE-2016-1000110
  * SECURITY UPDATE: Integer overflow when handling zipfiles
    - debian/patches/CVE-2016-5636-pre.patch: check for negative size in
      Modules/zipimport.c
    - debian/patches/CVE-2016-5636.patch: check for too large value in
      Modules/zipimport.c
    - CVE-2016-5636
  * SECURITY UPDATE: CRLF injection vulnerability in the
    HTTPConnection.putheader
    - debian/patches/CVE-2016-5699.patch: disallow newlines in
      putheader() arguments when not followed by spaces or tabs in
      Lib/httplib.py, add tests in Lib/test/test_httplib.py
    - CVE-2016-5699

 -- Steve Beattie <email address hidden> Wed, 16 Nov 2016 12:38:40 -0800