Sane JavaScript testing?

[Jason Gerard DeRose]

Unit testing with JavaScript is apparently such a pain that few projects do much of it. As a Python guy, this blows my mind. In my perfect world, I could:

1) Run JavaScript tests with `setup.py test` along with the Python tests, and get an overall pass/fail from the setup.py exit code... very important for running tests automatically once we finally start using Tarmac.

2) Run JavaScript tests in embedded WebKit and log results with AJAX calls to a simple WSGI app that collects the results - don't want to require user to open a browser

3) Declare the JavaScript tests in Python and be able to do arbitrary setUp/tearDown from Python - say stuff like creating an empty CouchDB database and populating it with test data. Obviously the *definition* of the test will require JavaScript also, but I want the tests to be discovered according to a TestCase subclass

Some UI level JavaScript can't really be tested without human participation, but that's fine. We're going to have plenty non-UI JavaScript that needs to be very robust - which in my mind requires automatic testing.

This might be wishful thinking, but I'm going play with this for a day or so and see what I come up with. Or if someone else wants to spearhead this or work on it with me, fantastic.

[Jason Gerard DeRose (jderose)]

Little progress report... it works! It's awesome! I didn't know if I was chasing windmills with this one, but I'm so glad I gave it ago... totally bitchin.

I started thinking about how nice the Python unittest.TestCase API is, how I'd like to have basically the same thing available in JavaScript... and then it dawned on me, I might as well just forward calls to TestCase.assertEqual() etc using XMLHttpRequest. That gives us the rich Python API, all while requiring only a tiny amount of JavaScript.

This is the entire JavaScript test harness:

var py = {
    /* Synchronously POST results to ResultsApp */
    post: function(path, obj) {
        var request = new XMLHttpRequest();
        request.open('POST', path, false);
        if (obj) {
            request.setRequestHeader('Content-Type', 'application/json');
            request.send(JSON.stringify(obj));
        }
        else {
            request.send();
        }
    },

    /* Initialize the py.assertFoo() functions */
    init: function() {
        py.data.assertMethods.forEach(function(name) {
            py[name] = function() {
                var args = Array.prototype.slice.call(arguments);
                py.post('/', {method: name, args: args});
            };
        });
    },

    /* Run the test function indicated by py.data.methodName */
    run: function() {
        try {
            py.init();
            var method = py[py.data.methodName];
            method();
        }
        catch (e) {
            py.post('/error', e);
        }
        finally {
            py.post('/complete');
        }
    },
};

The test harness add a bit of dynamic info also, something like this:

py.data = {
    "assertMethods": [
        "assertTrue",
        "assertFalse",
        "assertEqual",
        "assertNotEqual",
        "assertAlmostEqual",
        "assertNotAlmostEqual",
        "assertGreater",
        "assertGreaterEqual",
        "assertLess",
        "assertLessEqual",
        "assertRegexpMatches",
        "assertNotRegexpMatches",
        "assertIn",
        "assertNotIn",
        "assertItemsEqual"
    ],
    "methodName": "test_self"
};

And tests can be defined in separate files like this:

/* Run a selftest on the tester */
py.test_self = function() {
    py.assertTrue(true);
    py.assertFalse(false);
    py.assertEqual('foo', 'foo');
    py.assertNotEqual('foo', 'bar');
    py.assertAlmostEqual(1.2, 1.2);
    py.assertNotAlmostEqual(1.2, 1.3);
    py.assertGreater(3, 2);
    py.assertGreaterEqual(3, 3);
    py.assertLess(2, 3);
    py.assertLessEqual(2, 2);
    py.assertIn('bar', ['foo', 'bar', 'baz']);
    py.assertNotIn('car', ['foo', 'bar', 'baz']);
    py.assertItemsEqual(['foo', 'bar', 'baz'], ['baz', 'foo', 'bar']);
};

And there is <body onload="py.run()"> to start the test. Luckily, embedded WebKit will execute the JavaScript even if gtk.Window.show_all() wasn't called... so the tests can run totally headless.

I'll have to clean things up a bit, but I think this is *the* way to test JavaScript.