This post originated from an RSS feed registered with Agile Buzz by James Robertson. Original Post: Automated Builds Feed Title: Travis Griggs - Blog Feed URL: http://www.cincomsmalltalk.com/rssBlog/travis-rss.xml Feed Description: This TAG Line is Extra Latest Agile Buzz Posts Latest Agile Buzz Posts by James Robertson Latest Posts From Travis Griggs - Blog

Ours just went through some changes, I thought I'd give a quick overview. We currently build no less than 7 deliverable applications from VisualWorks 7.2 (soon to be change to 7.2.1). All of them are composed from layers of packages that are prerequisite linked together.

Parked on the same nominally headless server that the Store Postgres database runs on, is a VNC server hosting a VW session. It is called ChooseControl. A directory called Projects houses each build target as a self contained directory. Each project directory uses a set of files to define things like what the "top level" packages are, who gets email messages, who get's error messages, etc. The process goes something like this:

1. Change Detection: Every 30 seconds, the image builds a prerequsite tree for the latest trunk version (trunk versions are those with pure digit versions, anything published with versions that contain other than digits is considered a branch), and looks to see if it differs from the last time it built, and if the current tree has been stable for a minute or more. If this is the case, then it starts a new build.
2. Initiating a New Build: The build creates a temporary directory to do the run in. It copies some static files in there, and generates a couple of othes, such as a file indicating the load (packages and versions). Oh yeah, the process is forked off at this point. That's new and has had some interesting implications.
3. Package to Parcel Conversion: This is the overweighty part. VisualWorks/Store has no way to turn Store'd packages into deployed parcel files, without having loaded the package first. parcelBuilder.im is a minimal development image with Store preloaded into it. UnixProcess is used to launch a totally new VW session with this image. The image is configured to connect to the database, read the load lists, load those packages from Store, write the equivalent parcels, and then quit.
4. Run Tests: One additional package that is added to the list of top packages is called "TestLogger". This is a simple class that gathers all of the TestCases in the image, randomizes them in one huge suite, and then runs them. It "overrides" the SUnit error:/passed:/failed: messages to log the results to temporary log files too. By using some thisContext walking and services add with PDP, it even attempts to log the exact line of code that presents an assert: failure, just like using the debugger interactively would do. Another aspect of this process is that Key uses a single image and VM for all of its end products. We build deployment images ONCE. Said image is prepared using RTP, but is not really "stripped". One modification, is that on startup, it uses SystemEventInterest to check for the existance of a .doit file. If it finds one, it will do a Compiler evaulate: on it. The .doit file usually looks something like "load these top level parcel files" and a line which "starts up" the application. So to run tests, and run them in as similiar of an environment as the end product, a .doit file is written which looks the same as the end application will, but instead runs the TestLogger as the last step, rather than opening the application. Like the parcelBuilder.im, UnixProcess is again used to run an entirely new VW session to do this. So at this point, in addition to the ChooseControl root scanning image, we've run two separate other VW invokations.
5. Debian Deployment: Many of our applications are deployed as a debian package. This optional step is used to take the parcels, provided all of the tests passed, and ball them up as a debian package, and submit it to our own debian repository.
6. Email Report: Finally, the test logs are parsed and an email is prepared and sent to the team. An example would be something like what's below. The email is all green if it passes, all red if there are any failures. All the differences between this and the last build are tabulated. Believe it or not, one of the important parts is the fortune cookie at the top. It gives people one more reason to look at the build messages.

766 run, 765 passed, 1 failed, 0 errors
ERRORS
FAILURES
	BaseExtensionsTests	testFinishInOrElseCanPropagateError
BaseExtensionsTests>>#testFinishInOrElseCanPropagateError
self should: [[] finishIn: 0 orElse: [ZeroDivide raise]] raise: ZeroDivide

Implications of Running Parallel

Running the builds in parallel led to some interesting problems. It improved some of the tests we had that assumed we could manage the mouse cursor. Also some of our tests that assumed various timing had to be rechecked. Ultimately, I think we're writing even more robust tests now.

The amusing thing about running the parallel builds is watching the parcelBuilder run for 6 or 7 images concurrently. I've whined about the progress dialogs of hell under X11 before. Try having 7 images, each of which might load up to 40 packages constantly opening new dialogs, and said dialogs constantly catapulting themselves to the front so they don't get "lost". All concurrently. The IceWM toolbar at the bottom goes positively hyperactive trying to show the current top window.