What are the best web application test frameworks people have seen? Any language, though I'm thinking more about unit tests, not as much about through-HTTP acceptance testing.

I'm curious mostly because I want to see other useful features for me to borrow. Right now the test framework in Paste is pretty simple, but IMHO in a good way.

There's some other features I've seen that I'm unsure of. For instance, the ibofobi framework for Django uses Beautiful Soup for extracting response strings for testing. Do people think this is useful? Personally I'm happy enough testing for the simple existence of strings, without caring about where in the structure of the document they appear. Also it works better for negative assertions, i.e., asserting that a string doesn't appear anywhere in the document; I do a lot of these in my tests.

Also, what do people think about doctest in this context? I started out using that, but ended up reverting to py.test because my tests quickly became more complicated than doctest could really handle elegantly. Doctest still worked, I just wasn't using any of the features that make it convenient (and I was missing the py.test features). Note that paste.fixture doesn't really require py.test, since py.test is a kind of null framework; I just don't go out of my way to add any special unittest hooks (like, say, a custom TestCase subclass).

I haven't looked closely at Twill since shortly after it split from PBP. I like in-process tests a lot, so Twill doesn't really test in the way I'd want it to. Though I bet I could tweak it to test in-process too... anyway, a feature it has is how it fills in forms nicely. I just added form filling to Paste, but it's very rough still. And I feel bad that I'm using regexes and not Mechanize or something similar. One thing Paste can do is require reasonably well-formed markup, because it's not intended for testng applications you can't control (but I'm not willing to require XHTML, just reasonably unambiguous HTML 4).

Zope 3 emphasizes testing, but I have to admit that I am unimpressed by the functional testing (and doctest functional testing) there. It seems long-winded. There might also be better systems people have set up there. Paste creates a wrapper around the "application," which seems much better than the kind of manual request setup I see for many frameworks (like in TurboGears).

One thing I've done in paste.fixture is to expect success unless told otherwise. So you don't confirm a request returned a 200 status code -- it's an error if it doesn't, unless you explicitly say you expect something else. I think explicitly checking for a 200 status code is crazy.

What about patterns for matching markup? I test strings against a normalized form of the page (mostly whitespace normalized), but sometimes you do want to test the structure, and I want to avoid fragile tests that test for more than they should (and will fail, for example, due to small design changes). Regular expressions aren't good here; instead it's good to have something that knows about the structure of markup, knows to ignore attribute order, knows when to ignore whitespace (and where not to), etc. I've played with this before, but I'd be very happy if there was someone else's code I could use here.