FTBFS due to broken unit tests

[Łukasz Zemczak]

In the wily test-rebuild [1] python-pysaml2 seems to fail to build due to multiple errors and failures in the unit tests. The log:

============================= test session starts ==============================
platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /«PKGBUILDDIR», inifile:
collected 493 items / 1 errors

tests/test_00_xmldsig.py ..............................................
tests/test_01_xmlenc.py ......
tests/test_02_saml.py ..............................................................................
tests/test_03_saml2.py .....................
tests/test_04_samlp.py ........................
tests/test_05_md.py ..........................................................
tests/test_10_time_util.py .................
tests/test_12_s_utils.py ..................................
tests/test_13_validate.py .......
tests/test_19_attribute_converter.py .............
tests/test_20_assertion.py ................................
tests/test_22_mdie.py .
tests/test_30_mdstore.py .......
tests/test_30_mdstore_old.py .......
tests/test_31_config.py ............
tests/test_32_cache.py ..........
tests/test_33_identifier.py ......
tests/test_34_population.py ....
tests/test_37_entity_categories.py ......
tests/test_40_sigver.py ...............
tests/test_41_response.py ..FF
tests/test_42_enc.py ....
tests/test_43_soap.py ..
tests/test_44_authnresp.py ....
tests/test_50_server.py ................
tests/test_51_client.py ...............
tests/test_60_sp.py ss
tests/test_62_vo.py ........
tests/test_63_ecp.py .
tests/test_64_artifact.py ...
tests/test_65_authn_query.py ..
tests/test_66_name_id_mapping.py ..
tests/test_67_manage_name_id.py ..
tests/test_68_assertion_id.py .
tests/test_69_discovery.py .....
tests/test_70_redirect_signing.py .
tests/test_72_eptid.py ..
tests/test_75_mongodb.py .F
tests/test_76_metadata_in_mdb.py .
tests/test_77_authn_context.py ......
tests/test_81_certificates.py ....
tests/test_88_nsprefix.py ..
(...)

[1] http://people.ubuntuwire.org/~wgrant/rebuild-ftbfs-test/test-rebuild-20151001-wily.html

[Łukasz Zemczak (sil2100)]

At least parts of the failures are due to the new pymongo, which for instance deprecated the 'Connection' object. Looking if we can easily distro-patch it to work. Debian also doesn't have it changed for the new pymongo. Maybe we could sync python-pysaml2 3.0.0-1 from experimental? Looking into that still.

[Łukasz Zemczak (sil2100)]

Looks like we can get this fixed without pulling in the new upstream version with this:

https://github.com/rohe/pysaml2/commit/5035a58137f7ba5ad79e4e52703b789b2a726955

[Łukasz Zemczak (sil2100)]

After applying the patch there are still test failures. The first two (in tests/test_41_response.py) can be fixed by:

https://github.com/rohe/pysaml2/commit/ee3d828f80ff18082f89e219e0b48faf0dab45ca

The final failure needs to be fixed manually. The test_eptid_mongo_db test from tests/test_75_mongodb.py fails as the test should not run if a MongoDB is not running - but the test is invalidly written and times out, causing a failure. The change is trivial and after applying it it builds fine, but I want to forward it upstream before proposing a debdiff.

[Łukasz Zemczak (sil2100)]

Ok, prepared a debdiff fixing all FTBFS issues. Almost all the fixes that are part of distro patches in this version are cherry-picked from upstream (will contact them about certain aspects of the testing suite). Debian seems to prepare for switching to a newer python-pysaml2 (3.0.0 in experimental), not sure if it make sense to forward there.

Could someone sponsor this for me? Thanks!

[Barry Warsaw (barry)]

There's a typo in the changelog. It's "d/control" not "d/rules". I'll fix that when I sponsor. Other than that, the debdiff looks great and a local build succeeds. Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-pysaml2 - 2.4.0-0ubuntu2

---------------
python-pysaml2 (2.4.0-0ubuntu2) wily; urgency=medium

  * debian/control:
    - Depend on python-pymongo versions 3.0 and higher
  * debian/patches/fix-build-against-new-pymongo.patch:
    - Cherry-pick fix pysaml2 with python-pymongo 3.0
  * debian/patches/fix-test-failures.patch:
    - Fix the newly failing test failures (LP: #1503698)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Thu, 08 Oct 2015 20:16:42 +0200

[Łukasz Zemczak (sil2100)]

Oh, right! Yeah, I think I was not thinking when typing that. pysaml2 really needs a better test suite. I generally think that relying on actual time when having some pre-generated timestamps is a bad idea in overall. They should mock their time methods and stop relying on real time during tests, otherwise the test suite will have to be updated again soon.

Thanks for sponsoring!