Summary
The JCP requires a reference implementation for each specification. Where the specification is vague, the reference implementation can be checked to determine the proper behavior. Is there a better way to deal with the problem of vague specifications?
At first glance JCP specs look OK; but once you really start digging into one at the level you need to write a book about one or implement it, you rapidly discover huge areas of unspecified behavior.
Harold explains that as he attempts to write about an API, he ends up using one or more implementations as the reference implementation is intended to be used, to clarify portions of the written specification where the proper behavior is not clear:
...I have to test the implementations to see what they actually do. I rarely test more than one or two, and then I write down its behavior.... Then other people come along and read my book to figure out what they’re supposed to. The behavior that gets fixed is chosen almost by accident.
Harold suggests that the JCP has a culture of "implementation as specification," and contrasts the JCP's approach with that of the W3C:
I blame this on the JCP’s culture of implementation as specification. Many specs are nothing but the JavaDoc compiled from the reference implementation and pointlessly encoded in PDF and/or a zip file....
By contrast W3C specs are normally written independently of implementations. Then the working group checks to see if the specs can actually be implemented in a compatible fashion. Since they require two implementations of each feature, this tends to identify any grey areas in the spec, especially when the people doing the implementing are not the people who wrote the spec.
The issue of whether to require a reference implementation also came up in the Jini community, which decided that the written specification was the truth. To submit a proposal to the Jini Decision Process, you must submit "specification code," .java files that contain the classes and interfaces defined in the specification, and possibly some implementing code. You may also submit one or more working implementations and clients. But there is nothing referred to as a "reference implementation." If the specification turns out to be vague on a particular question, the specification itself must be revised to clarify the answer to that question.
Various standards bodies take different approaches to dealing with the vague specification problem, which at its heart exists because it is very difficult to fully specify an API or protocol. It is difficult to think of all the questions that will need to be answered as a technology is used over time. The advantage of the JCP's reference implementation approach is that you can generally get an answer to your question, and you can get it quickly.
The disadvantage, however, is that the reference implementation's behavior, if it was "chosen by accident," may not be the behavior considered most desirable by the specification authors. Nevertheless, if an implementer must wait to get an official answer until the next version of the specification, in practice every implementer is going to make a best guess.
What's your opinion on the best way to prevent and, when not prevented, to deal with vagueness in specifications?
I don't see the problem here. Isn't the point of the reference implementation that you refer to it when the meaning of the specification isn't clear? It seems like the situation you're describing is ideal.
I know that when I write HTML and CSS, the W3C's specs be damned, I have to test on every browser that I expect to work; if I were implementing a browser I would carefully look at their existing behavior first. Specifications are useful only insofar as they provide a framework for understanding the observed behavior.
> I don't see the problem here. Isn't the point of the > reference implementation that you refer to it when > the meaning of the specification isn't clear? It seems > like the situation you're describing is ideal. > That is the point of a reference implementation. And I think I described the tradeoffs. I wouldn't call it ideal. The benefit is that you get an answer immediately. The drawback is that the answer you get may not be the "right" one, where "right" means what the specifiers would have decided if they'd actually considered the question.
> I know that when I write HTML and CSS, the W3C's specs be > damned, I have to test on every browser that I expect to > work; if I were implementing a browser I would carefully > look at their existing behavior first. Specifications are > useful only insofar as they provide a framework for > understanding the observed behavior. > Well, I think the whole point of a specified standard is that you can write against the standard, and then just switch the implementations. Things are rarely as easy as that, but they can be better or worse. For example, JavaScript is a standard (ECMAScript), but we still have to select a subset of browsers to support, and test JavaScript against each of those. By contrast the Java code I've written has pretty much worked on every JVM I've run it on. The specification and/or adherence is stronger in Java than JavaScript in my experience.
> the better way is to add to the specification the lessons > learned from the reference implementation!
I think they usually develop the spec and reference implementation together. It certainly helps a spec be more based in reality if it is informed by experience actually building an implementation. But it still doesn't mean that you'll think of every question other implementors may have in the future.
When we did the ServiceUI spec, we actually made one decision on purpose that someone later, by asking a simple question, pointed out was wrong. There was a bug in the spec and in the implementation. So I had to do a version 1.1 to fix that.
Rusty Harold in his blog post that inspired this item on Artima complains about JavaDoc. I think that JavaDoc is a great way to specify the details of an API spec, and I did so in the ServiceUI spec. But the JavaDoc stuff was at the end of the spec, and before that were several sections of prose.
One other way to think about the trouble of using reference implementation as backup specification is that just because a particular "reference" implementation does something a certain way doesn't tell you anything more (assuming the behvior is acceptable) than that is one acceptable behavior. It doesn't say it is the only one. The specification could be more abstract.
One problem I've had with various API documentation is that behavior is underspecified. The docs don't say, for example, what should happen if you pass in null as a particular argument of a particular method. Looking to see what the code does doesn't mean it will always do that in future versions. It just tells me what this particular version of this particular implementation does.
/* The benefit is that you get an answer immediately. The drawback is that the answer you get may not be the "right" one, where "right" means what the specifiers would have decided if they'd actually considered the question. */
Kind of reminds me of <i>Mythical Man-Month</i>'s advice to have an odd number of specs. That is, have a single official specification, or three, but never two. Doesn't solve the whole problem, but it gets rid of some of it.
I'd agree with Eliotte that the problem is in the fundamentally flawed way the JCP mixes the development of the reference implementation and the specification. In the end both are of worse quality than they'd be if the JCP followed W3C's model, as they suffer from inherent coupling.
Part of the problem is that the JCP is an oligarchy of companies content with the way the current rather rubberstamping, intransparent approach to specifications works, and complicit in the creation of resulting low-quality specifications. The quality of the J2SE specs hardly matters to a RI licensee, it's the developers and independent implementors that get to suffer. They, in turn, are rarely represented on the J2SE JSRs, so the things stay year in, year out, the way they have been before.
Previous Topic
