VWUnit, a Framework for GUI Testing, David Buck, Simberon
David did some work for a company whose software manages bonds, etc.: when to issue coupons, make payments and so on for trades, swaps, etc. A significant number of people spent 2 weeks every release just looking at screen snapshots and comparing them to the current release.
Screen snapshots did not capture order of entry issues (e.g. enter USD then CND and get conversion rate was not the same process as enter USD then conversion rate and get CDN, and could induce changes further down the line of the overall process due to rounding issues, etc., but looked the same on the screen), so snapshot pages got covered with little notes. This had to be fixed. They also wanted to use XP for UI testing. They were already using SUnit for non-UI and now wanted tests that would match exactly the screen values that were currently being used in the screenshot by-hand comparisons. The test framework also had to handle pop-up dialogs.
They liked SUnit and so built a framework around it. They had to run tests in a given order since some tests created states for other tests (e.g. test on swap requires trade and instrument to be already there). Thus they overrode buildSuite to create an ordered suite. Another problem was the fact that tests tended to save to the database, thus second run would encounter the object already there from the first run. The solved this by adding unique timestamp suffixes to object names so reruns would not clash.
Their apps had many levels of subcanvases within subcanvases. They found these hard to navigate so decided to locate items on the screen based on name only, e.g.
harness rightOfLabel: 'Name:' enter: 'John Doe'
(Reinout has added a feature to locate by name or id.) The method allLeafWidgets gets the collection from which they search in the above methods.
Widgets are sometimes are not there yet (screen still opening) so they try a few times before conceding failure. There are various kinds of methods:
- locating methods: locateListBox (finds first, useful on simple screen), locateLabel: aString, locateRadioButton: aLabelString, locateTopLeftWidget, etc.)
- they fast flash widgets as they are located to verify behaviour (with 20 ms delay flash instead of standard 120ms flash to speed tests; this was an example of the 'profile, don't guess, about performance' rule; he thought allLeafWidgets was the problem till he profiled)
- Action methods: press{Radio}Button, pressCheckBox{On/Off}, atTableCell:enter:
- Combination methods: these combine location, navigation and entry: rightOfLabel: aLabelString enter: aDataString
- Group Boxes: withinGroupBox:do: can help to find a widget whose name is reused in several places.
- Dialogs: handling dialogs was hard; the following works but may not handle dialogs that themselves popup dialogs - harness performAction: aBlock onDialogPerform: anotherBlock
- Tables: VWUnit supports tables but not datasets. They will add cell-by-cell comparison. To write a test to check a table, start by checking against an empty array. Run test, it fails, do cut and paste of table in failing assertion, now test passes; it's an easy way to get the table.
They can attach to new windows. They can print out the screen snapshots they test against by calls to self printScreen during the test to provide an audit trail for the end-users. They can scroll down when printing but the scrolling is not yet in the version in the Open Repository as it needs work for datasets, etc.
The can replace the test harness with a reporting harness which rewrites the smalltalk calls to user-oriented reports. As it happened, the users were more interested in the screen snapshots. They can also export to excel files to do file comparison. They use performUIBlock: to do all UI calls (I was reminded of my inDialectUI: aBlock for Custom Refactoring UI tests.) They now have 100+ tests and the whole suite runs in 2 hours.
David would rather have written non-UI tests than UI tests but his client wanted UI so he did the work for them.
Limitations: it can't tell whether a widget is or should be enabled or disabled. He wants to add dataset support, cell-by-cell comparison, better locate by name of widgets, menu testing support, ...
David ended with a demo. He ran a test case, a screen appeared, ran through widgets and closed. The test tool updates with the result as each test runs. You can halt runs, etc. (The tool's UI was roughly SUnitBrowser-like. David has given it an 'abort run' feature. I have also found such a feature useful, so added it to SUnitBrowser). Dave also showed the report generation, and changed the flash time to show how that affected speed.
Q. How do you validate? We have locate...:check: methods.
Q. Notebook tabs? We handle our Notebooks but not VW ones and ours are not in public version. Locating is the hard issue.
Q. Excel macros found by location? Some thought on this. They could have a script that captured the values to be checked.
Q. I asked David why he overrode buildSuite instead of building the ordered TestSuite he needed. David said it was just easier to use class names. (I think that the fact that TestRunner is not set up to accept Suites naturally tends to point people away from using them when they first encounter SUnit and the habit lingers. My tweaks to TestRunner to make it accept suites are in the Cincom Open Repository, in the StarBrowser / SUnit Extensions. I believe TestSuite construction is the right approach.)
Q. I suggested handling Dialogs by using method wrappers (this is how I handle dialogs in my own tests). Wrap the appropriate dialog method with one that instead gives the effect of its being clicked and closed, etc.
Q. The case sounded like one suitable for my deep comparison test approach (see my talk at last year's Smalltalk Solutions), albeit the scale was smaller so did not compel such an approach. David had noted my talk but was concerned about two issues.
- Resilience to additions: would adding a widget fail the meta-test whereas an ordinary test that ignored it would still pass? The latter was the behaviour they wanted. (I think default customising new widgets to not compare should be straightforward.)
- Widget comparison time-delay: as noted, they had to retry the comparison a few times and wondered if the meta-approach could handle this. (I thought it could without difficulty, but will consider.)
Q. Maybe open window and write to graphics context that does nothing, so you do all comparisons without the time delay of actually displaying. Interesting idea; Dave will think about this.
Q. Why not use Test Mentor? Dave wanted them to use test mentor but the last bureaucrat would never sign off so he had to do it under the covers. Dave thinks they would have been much better off with Test Mentor.
Previous Topic
