testsuite failures in release 0.9.37; test_test_module, test_test_suite

[IAN DELANEY]

deps emerged.

mimeparse, extras, twisted

run from setup.py test;

```python
running test
Tests running...
======================================================================
FAIL: testtools.tests.test_distutilscmd.TestCommandTest.test_test_module
----------------------------------------------------------------------
stdout: {{{
running test
Tests running...

Ran 2 tests in 0.000s
OK
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-0.9.37/work/testtools-0.9.37/testtools/tests/test_distutilscmd.py", line 73, in
test_test_module
    """)))
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-0.9.37/work/testtools-0.9.37/testtools/testcase.py", line 423, in assertThat
    raise mismatch_error
MismatchError: u'running test\nTests running...\n\nRan 2 tests in 0.000s\nOK\n' does not match /Tests running...\n\nRan 2 tests in \d.\d\d\ds\nOK\n/
======================================================================
FAIL: testtools.tests.test_distutilscmd.TestCommandTest.test_test_suite
----------------------------------------------------------------------
stdout: {{{
running test
Tests running...

Ran 2 tests in 0.000s
OK
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-0.9.37/work/testtools-0.9.37/testtools/tests/test_distutilscmd.py", line 95, in
test_test_suite
    """)))
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-0.9.37/work/testtools-0.9.37/testtools/testcase.py", line 423, in assertThat
    raise mismatch_error
MismatchError: u'running test\nTests running...\n\nRan 2 tests in 0.000s\nOK\n' does not match /Tests running...\n\nRan 2 tests in \d.\d\d\ds\nOK\n/

Ran 1192 tests in 1.309s
FAILED (failures=2)

```

under all pythons including pypy. system; gentoo.
Can you replicate (Considering travis I expect not)? Do you require any further info?

[Robert Collins (lifeless)]

Please try with a current release (1.1.0)

[IAN DELANEY (johneed)]

Point 1. Does the 0.9.37 not warrant fixing?
Point 2. I thought this was the current release. Our euscan app presented new release 0.9.37.
Point 3. Now I know there is one I can try it.
Point 4. repeat Point 1.

[IAN DELANEY (johneed)]

same;

 /path/yo/dev-python/testtools-1.1.0/work/testtools-1.1.0 $ python setup.py test

```python
running test
Tests running...
======================================================================
FAIL: testtools.tests.test_distutilscmd.TestCommandTest.test_test_module
----------------------------------------------------------------------
stdout: {{{
running test
Tests running...

Ran 2 tests in 0.000s
OK
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/tests/test_distutilscmd.py", line 73, in test_test_module
    """)))
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/testcase.py", line 433, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'running test\nTests running...\n\nRan 2 tests in 0.000s\nOK\n' does not match /Tests running...\n\nRan 2 tests in \d.\d\d\ds\nOK\n/
======================================================================
FAIL: testtools.tests.test_distutilscmd.TestCommandTest.test_test_suite
----------------------------------------------------------------------
stdout: {{{
running test
Tests running...

Ran 2 tests in 0.000s
OK
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/tests/test_distutilscmd.py", line 95, in test_test_suite
    """)))
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/testcase.py", line 433, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'running test\nTests running...\n\nRan 2 tests in 0.000s\nOK\n' does not match /Tests running...\n\nRan 2 tests in \d.\d\d\ds\nOK\n/

Ran 1206 tests in 0.367s
FAILED (failures=2)
```

~/cvsPortage/gentoo-x86/dev-python/testtools $ ebuild testtools-1.1.0.ebuild clean test

ditto

all python impls. Appears to be a unicode issue

 u'running test\nTests running .......

I thought unicode issues in Cpython were long ago sorted, but then it's likely something else.

[Robert Collins (lifeless)]

Hi, thanks for confirming this is a current issue. Not sure why your euscan isn't reporting our current versions, they are all on pypi.

And no, we can't fix 0.9.37 - if we change it it would no long be 0.9.37. There are no API incompatibilities that we know of between 0.9.37 and 1.1.0 so fixing trunk is best.

It might be a unicode issue indeed. Python < 3 still suffers from huge unicode issues. What python version are you using?

How can I reproduce this? (Is there a qcow2 of a failing environment or some such?).

[IAN DELANEY (johneed)]

Hello Mr. Collins

<reporting our current versions, they are all on pypi>

Well that's why. For reasons not of relevance here, the SRC_URI in the various ebuilds in portage point to various locations. When it's set to a repo, bumps often get missed if they're only in pypi and not being picked up in the repo.

<And no, we can't fix 0.9.37 - if we change it it would no long be 0.9.37.>

Sure. No prob. We prefer the latest at gentoo also.

< What python version are you using?>

Re-running;

~/cvsPortage/gentoo-x86/dev-python/testtools $ ebuild testtools-1.1.0.ebuild clean test
yields

 * python2_7: running distutils-r1_run_phase python_compile_all
>>> Source compiled.
>>> Test phase: dev-python/testtools-1.1.0
 * pypy: running distutils-r1_run_phase python_test
/usr/bin/pypy setup.py test
running test
Tests running...

Ran 1204 tests in 1.478s
OK
 * python3_3: running distutils-r1_run_phase python_test
/usr/bin/python3.3 setup.py test
running test
Tests running...

Ran 1204 tests in 0.378s
OK
 * python3_4: running distutils-r1_run_phase python_test
/usr/bin/python3.4 setup.py test
running test
Tests running...

Ran 1204 tests in 0.545s
OK
 * python2_7: running distutils-r1_run_phase python_test
/usr/bin/python2.7 setup.py test
running test
Tests running...

Ran 1204 tests in 1.291s
OK

So; since ~ 2 weeks ago I did a massive system update. This is good news since there's now nothing to replicate. By rights the versions of what I had then ought to have been fine, so it suggests a version sensitivity. In the ebuild, written from specs of the source, I have no version bordering for dev-python/mimeparse or dev-python/extras and I don't have a convenient record of what they were on the 15th. However if it's only me declaring this at all, it suggests it's not easily triggered. Py2.7 may have been 2.7.6 back then but again I can't be sure. Currently

python2.7; $ python -V \ Python 2.7.8
python3.3; $ python -V \ Python 3.3.5
python3.4; $ python -V \ Python 3.4.2.
add pypy

<(Is there a qcow2 of a failing environment or some such?)>

If you mean a vm you can ssh into, I was going to offer one, however seems not needed. I have done this on previous occasions for upstream dev for packages that didn't play nice in my system. However, in gentoo systems aren't guaranteed consistent between devs either. Gentoo is all about customising and refining so while there are strict methods to ensure consistence they are equally offset by the variables on offer to the the whim or preference of dev or user alike.

Anyway, thx for your reply

[IAN DELANEY (johneed)]

Oh silly me. I had the failing tests seded out in the ebuild and didn't discover until after I clicked Post Comment cmd button. You don't appear to have an edit facility in launchpad here like github. Once it's in it's in.

* python2_7: running distutils-r1_run_phase python_test
/usr/bin/python2.7 setup.py test
running test
Tests running...
======================================================================
FAIL: testtools.tests.test_distutilscmd.TestCommandTest.test_test_module
----------------------------------------------------------------------
stdout: {{{
running test
Tests running...

Ran 2 tests in 0.000s
OK
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/tests/test_distutilscmd.py", line 73, in test_test_module
    """)))
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/testcase.py", line 433, in assertThat
    raise mismatch_error
MismatchError: u'running test\nTests running...\n\nRan 2 tests in 0.000s\nOK\n' does not match /Tests running...\n\nRan 2 tests in \d.\d\d\ds\nOK\n/
======================================================================
FAIL: testtools.tests.test_distutilscmd.TestCommandTest.test_test_suite
----------------------------------------------------------------------
stdout: {{{
running test
Tests running...

Ran 2 tests in 0.000s
OK
}}}

Traceback (most recent call last):
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/tests/test_distutilscmd.py", line 95, in test_test_suite
    """)))
  File "/mnt/gen2/TmpDir/portage/dev-python/testtools-1.1.0/work/testtools-1.1.0/testtools/testcase.py", line 433, in assertThat
    raise mismatch_error
MismatchError: u'running test\nTests running...\n\nRan 2 tests in 0.000s\nOK\n' does not match /Tests running...\n\nRan 2 tests in \d.\d\d\ds\nOK\n/

Ran 1206 tests in 1.305s
FAILED (failures=2)
>>> Completed testing dev-python/testtools-1.1.0

ditto py3.3 3.4 Sorry for the confusion.

Perhaps it is time to setup a gentoo env for you to see this first hand. I have an account in which I can 'spin up ' a gentoo server from rackspace. Say if you wish to pursue this. Being a corp. body it means we have to co-ordinate timing to have it up and running when you're ready.

[Robert Collins (lifeless)]

Sure. My work hours are roughly 2200-0530 UTC (I'm in NZ), if you want to pick a time we'll overlap we can try to track this down.

[Robert Collins (lifeless)]

Since this appears to be pretty rare,and not core functionality (e.g. just UI) I'm going to downgrade it to High.

[IAN DELANEY (johneed)]

right, well it seems this is another (rare) event where there's something not quite perfect in my system. After emerging the deps in the vm and running it the suite passed. Also used an alternate approach to run the suite which also passed in my own system. The method I used still draws these 2 fails although it is supposed to be an equivalent cmd. (Oh well). However it's more than enough to prove the suite actually passes all fine. I could do with more helpers to cross test occasionally. This is certainly not the norm.

[Jonathan Lange (jml)]

Marking resolved based on https://bugs.launchpad.net/testtools/+bug/1380918/comments/9. Please re-open if it's still an issue.