There's been another flurry of discussion in various forms about Store (the code control system for VisualWorks).
I've been working on algorithms this week to automate software building, from specification of target package to set of parcels. One of the things that is frustrating for me to work with is the "dual prereq stream" as I call it. Why are there two streams? And how did they come to be?
In VisualWorks Smalltalk, each module (where module is a Package or a Bundle) specifies TWO lists of prerequisites. The system presents these as deployment and development prerequisites. It's kind of misleading, because what they really refer to is binary loading or compilation loading. By default Store loads code and compiles it incrementally as if it were a big filein. Even with shadow compilation turned on, it's compiling the code. It just defers the "linking" til the last. Binary loading on the other hand, is a means of loading code and linking it into the system, without having to engage any compilation facilities. This puts the compilation loading mechanism at a disadvantage with regard to loading code that requires compilation services that are outside of the normal services provided by a basic system. The most obvious example here is code that takes advantage of services provided by DLLCC. You can load and use the methods which are specified with just fine without a special DLLCC compiler. But when you're "filing these in" as a means of adding the behavior, you need to have DLLCC loaded. So you need additional modules loaded into the system beforehand to make this kind of loading required.
How did we end up with this solution? I'm stepping into the fuzzy way back machine of memory here. I recall using Store in its very early days and noting that packages that had ExternalInterfaces in them, required more stuff to load. I recall mentioning this in a mailing list, and hearing then about the intention to have an alternate stream of prereqs. I have no recollection of what my reaction was then. I know what it would be today:
- It take three times to get it right. Smalltalkers love to build general solutions to immediate problems. The ability to use indirection and abstraction is just so fluid, that it's enjoyable for some to create full blown systems for one-off problems. I would opt to special case the DLLCC case. It is very common and very particular. It would go something like "if this package has ExternalInterface subclasses in it and I'm loading from Store, I'm loading DLLCC in as an implied prerequisite."
- I've been pondering how the Debian packaging/distribution system accomplishes this. They don't have the exact same problem. I can load the binary version of firefox. I can also load the source for a version of firefox. But loading the source there is just copying additional text files. I don't need to compile it. I may just browse it. A similiar step though, is if I want to use it for other development efforts, is to have a package (almost always suffixed with -dev) which includes the binary version of the component and any header files I may need to compile it into other programs. This could have been done instead. If you need to load a package with ExternalInterfaces via source, you load a -dev version of it. Not only can you load it via compilation, but then make further changes to it. Remember, a Store package can be a binary load as well. So the non-dev variants would be binary.
- At least, exploit the "additional" nature of these prerequisites. These type s of additional resources for compilation loading are always additional. So compilation loading could have been the original prerequisites plus additional prerequisites.
A couple of months ago, I cajoled Alan to help me set up some Storp (Store + Glorp) queries and did an analysis of both the Cincom and Open Repositories. I searched for all packages where the compilation (development) prerequisites were not the same as the binary (deployment) prerequisites. It was more than 10%. What was interesting were the different types of errors observed:
-
Complete ommission. As in, someone defined one list, and left the other complete empty. An operator error. There were cases where I could see why this would never catch up with them (e.g. it's an add on to Store, unlikely it will be parcel loaded).
- Partial ommisions. There were quite a few of these. Sampling many of them, what appeared had happened, was that while the lists may have been originally the same, updates to the list (removals or additions) had been reflected in only one one list.
- Reordings. In some cases there were order differences between the two, though they contained the same elements. In practice, this could lead to hard to find bugs where the "development" version works fine, but due to load order issues, the "deployed" version behaved differently.
- DLLCC. Of course, there were a number of these. This is ostensibly what started it all.