unicode tests failing since yuitest landing

[Robert Collins]

19:44 < wgrant> I guess it's probably related to sinzui's YUITestLayer thing.
19:44 < wgrant> But I'm not sure when it first appeared...
19:44 < wgrant> A-ha.
19:44 < wgrant> Yes.
19:45 < wgrant> Those three tests first failed in the build with the new YUITestLayer.

The three tests are:
lib/canonical/launchpad/doc/unicode_csv.txt
canonical.launchpad.webapp.tests.test_encryptor.TestSSHADigestEncryptor.test_nonunicode_password
lp.codehosting.vfs.tests.test_transport.TestLaunchpadTransportImplementation.test_put_file_unicode

[Curtis Hovey (sinzui)]

Since YUITestLayer is still not run, I do not think this is an issue with setup/teardown. I will look at the code for tangential changes.

[Curtis Hovey (sinzui)]

I do not see anything specific to the changes I made to the YUI test infrastructure that can cause a failure. These tests do not fail for me. My change may have caused a test order change even though none of the tests are run. I have seen failures in the past where the test order changed and tests failed because someone was not doing a proper setup/teardown, it was relying the the test before or after to do the right thing.

[Curtis Hovey (sinzui)]

I speculate if everyone is using the ec2image I built for lucid, and the gtk imports from html5browser cause problems, then that would explain why the failure is only in ec2, not on our desktops or in buildbot.

[Curtis Hovey (sinzui)]

I can confirm that the html5browser's gtk imports for lucid break the three tests. There is no error on maverick because the imports use modern libraries from gi.repository.

[Curtis Hovey (sinzui)]

using the harness, I walked though the imports to see when str.encode() breaks. It breaks the moment gtk is imported. I need to learn if pygtk is redefining encode() or if there is a global python config being set that makes encode() return ridiculous strings.

[Curtis Hovey (sinzui)]

Importing gtk sets the default encoding to utf-8
>>> sys.getdefaultencoding()
'ascii'
>>> import gtk
>>> sys.getdefaultencoding()
'utf-8'

pygtk's gtk is importing the obsolete gnomevfs lib which has a mode line
    # -*- mode: python; encoding: utf-8; -*-

I think the fix for this is to explicitly set the encoding to ascii, but my initial efforts to do
    # This encoding hack can be removed when lucid's "import gtk" is removed
    # from the code.
    sys.setdefaultencoding('ascii')
in lp_sitecustomize.py do not seem to work

[Curtis Hovey (sinzui)]

The root cause is bug 27112, the only known work around is to reload(sys) to undo the damage (https://bugzilla.gnome.org/show_bug.cgi?id=132040#c10)

[Curtis Hovey (sinzui)]

While I could do a quick hack to Lp's code to ensure that the default encoding is ascii, this will be making needless changes for natty and oneiric users.

I will correct the default encoding in html5-browser only when gtk is imported from pygtk.
I will create a new ami image for launchpad that uses the fixed version of html5-browser.

[Curtis Hovey (sinzui)]

I have delete ami ec2test516 so no new test runs will be affected. I am building a new image with the fixed library.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13250 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13250>.