Weblogs Forum - Are Tests First Class Clients?

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Are Tests First Class Clients?

57 replies on 4 pages. Most recent reply: Aug 2, 2005 9:23 AM by Jim Cakalic

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 57 replies on 4 pages [ 1 2 3 4 | » ]

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Are Tests First Class Clients? (View in Weblogs)

Posted: Jan 26, 2005 12:15 AM

Summary
Automated tests can make demands on an API that are not made by any other client of the API. For example, you may find yourself wanting to add methods to a class that are not needed by any other, non-test clients. To what extent do you feel automated tests should be allowed change the API being tested?

Writing tests can help improve the usability design of an API, because when you write tests, you put on the shoes of a client trying to use your API. The simple act of writing a test can help you see problems with the usability of your API design, which encourages you to improve the design. But tests can also influence API designs in less desirable ways, because tests can make demands on the API that aren't made by any other client. For example, you may find yourself making private methods package access just so you can test them.

I always felt a bit dirty when making changes to an API solely to accomodate tests. I felt that the tests should be testing the API that would exist if there were no tests, because that's the "true" API. That's the API that serves the "real" clients. However, recently I started questioning that attitude. Should I consider tests as first class citizens of the application? Should I see tests as just another client that needs to be served, equal in status to other, non-test clients?

A couple months ago I was writing tests for a Java class called Merger, which contains a Velocity template name and a context Map. Instances of this class are returned by the controllers in Artima's new architecture. Mergers contain a merge method that renders the named Velocity template with the contained Map. I was writing tests that verified that the template name and context Map returned from our controllers were as expected. I first added get methods that allowed the tests to grab the template name and context Map. The tests were the only client that needed access to these private fields, so I didn't feel quite right about adding the get methods. But I did it anyway, because I wanted these tests. I soon found myself creating helper methods in the test class for comparing two context Maps and two template names.

After finding I needed to call the helper methods from a different test class, I started feeling the object-oriented urge to move the helper methods into the Merger class itself and get rid of the get methods I had added before. This would move the code that's using Merger's private data into class Merger, where the data resides. I struggled with this a while, then finally decided to go ahead and do it. Now there are no get methods, but there are two verify methods (formerly the helper methods) in the public API of class Merger. These verify methods are called only by tests.

To what extent do you feel automated tests should be allowed change the API being tested? What would you say in a design review of class Merger?

Jason Yip

Posts: 31
Nickname: jchyip
Registered: Mar, 2003

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 1:39 AM

> To what extent do you feel automated tests should be
> allowed change the API being tested? What would you say in
> a design review of class <code>Merger</code>?

I think that doing Test-Driven Development should definitely influence the design. However, in this case, I suspect it's more an issue of interaction versus state testing.

If I were to review Merger, the first thing I'd say would be: "May I see the code?" :)

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

It black v. white box testing.

Posted: Jan 26, 2005 2:11 AM

The problem isn't unit testing per se, but in the meaning and definition of the unit and the effects of that definition.

In testing, the unit is a 'black box' into which we feed input and examine the output. The unit cannot be subdivided.

With JUnit (I'm assuming here that we're basically talking about testing Java code with JUnit.) that unit is the 'public' interface presented by that class, so that the inputs are the method parameters and the outputs are the method return values (not necessarily from the same methods).

In theory, this is all very logical and simple. We write a set of tests that comprehensively test the class and we write them before we write the class (so truly the class is a black box). Once they are in place, developers (and, more importantly, maintainers) are free to develop and modify the class that are required, provided that the tests always work. What happens inside the class is unimportant provided the interface works.

Great theory and - in theory - it can work too. Unfortunately this gives most users of JUnit a bit of a problem. JUnit and 'test first' development is an interface verification tool not a code debugging tool. For most people writing software the unit of interest is the code construct, be it a loop or a conditional statement or a question such as "What exactly is this dodgy bit of SQL statement actually retrieving from the database?". What the developer is interested in are things like boundary conditions that may exist only within an obscure chunk of code but is irrelevant at the interface level.

The majority of software developers actually maintain existing code. They are only interested in code known to be broken in some way. Ideally, there should be a unit test to hand that confirms "doing x produces z where we expected y". But, of course, if the test had existed then the code wouldn't have been shipped. In the 'real world', not only is that test missing but "x doesn't produce y" is the symptom not the problem.

The weakness of JUnit is that it is good at what it is supposed to do (identify symptoms in the unit) but not so good at what we want it to do (identify problems at the sub-unit level). As a result we find that we have to 'cheat' in two important ways so that we can use it as a debugging tool. First, we modify the class by adding to its 'public' interface so that internal sub-unit information is exposed at the unit/interface level. Second, we write the 'unit' tests after the source code and based on the fact that we know what has been written but want to confirm what it does (i.e. 'code last' development).

This isn't necessarily a bad thing. There is a good case for arguing that there is frequently no need for methods to be private, since package protection still renders methods invisible to unrelated classes outside the package, which is generally all we want. This allows test classes that are compiled into the package to 'see' those methods and verify them. There is also a good case for arguing about which is better, a more testable class with a more complex interface or a less testable class with a interface. I would argue that interface simplicity is more important for API that are published for public use (which is a small minority of all classes written) and testability is more important for 'in-house' and non-API classes (the majority).

Of course, the argument against all this is that, if the unit tests are testing a different interface to the public one, then what they are verifying is a different interface to the public one and the results produced apply to a different interface to the public one. Breaking this 'other interface' may have no bearing on the public one. Therefore we still need a set of tests that apply only to the public interface.

Ideally, we need someone to develop a JSubUnit Debugger. Meanwhile, we'll get by with JUnit and common sense and frig the code occaisionally.

Vince.

Alexander Jerusalem

Posts: 36
Nickname: ajeru
Registered: Mar, 2003

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 2:16 AM

I think there are really two issues, that are linked to these two important API design principles:

(a) APIs provide a means of abstraction. They describe the interface to some encapsulated code whose inner workings they must not leak. Since encapsulation is _the_ most important means by which we cope with complexity, it is absolutely crucial that testing never[!] compromises encapsulation.

So, in a code review, I would ask two questions: Is the Merger class meant to encapsulate a generic template merging facility or is it a utility class to support Velocity specific coding? If the first is the case, then I would insist that the testing related methods must not leak Velocity specific types or semantics.

(b) An API should be as small as possible to make it easy for people to learn and understand and to reduce the chances of having to introduce incompatible modifications in later versions. Of course this is not a hard science. Sometimes it would seem unreasonably not to include one or the other convenience method. For me, this is a matter of practicality not ideology.

Still, as the user of an API, I find the prospect of having to wade through loads of purely testing related methods to find the actual stuff, rather scary. So my position would be to say, hey, I'm only the user of this API, it's not my job to test it, so why do I have to look at verifyThis() and verifyThat()?

As the implementer of an API, I have to ask myself, what I am actually testing? Am I testing if a particular implementation of the API does conform to the contract of that API? In that case, it should not be necessary to add anything to the API itself (quite the contrary, because theoretically this would lead to an infinite loop of testing requirements). It is black box testing after all. If I'm testing the specifics of a particular implementation and I want to check the intermediary state of that particular implementation, then it would seem all the more inappropriate to put it in the API.

The problem is, that sometimes it is simply hard to do actual black box testing, because the represenation of the output is inconvenient. Like the text output of a merged template. That's what causes the urge to hook into intermediary representations of the eventual output and simply infer the correctness of the real output. That may be legitimate, but I think it shouldn't pollute the API.

So, my conclusion is that the danger of breaking encapsulation and polluting the API is high and therefore I'd rather not see any testing related code in an API.

-Alexander

Edoardo Comar

Posts: 7
Nickname: edoardo
Registered: Apr, 2003

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 3:24 AM

> So, my conclusion is that the danger of breaking
> encapsulation and polluting the API is high and therefore
> I'd rather not see any testing related code in an API.

I agree in theory so if I can, I avoid test code polluting production code.
When I can't e.g. when testing would be impossible without polluting code or without reflection,
after many years of indecision, now I unashamedly add methods like

/**
* for testing only
*/
public void test_thisThingy()

that don't mix up with the rest of the interface when writing other clients.

Ideally, I'd need the friend keyword - tests should not be ashamed of beying friends of the testee.

To reiterate, ideally tests should just be clients of the API, but if I can't write them as such, then I pragmatically turn them into friends of the testee and I am still happy to have them.

Maarten Hazewinkel

Posts: 32
Nickname: terkans
Registered: Jan, 2005

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 3:54 AM

> So, my conclusion is that the danger of breaking
> encapsulation and polluting the API is high and therefore
> I'd rather not see any testing related code in an API.
>
> -Alexander

I agree with the point raised by Alexander.

One idea that crossed my mind to allow a bit more access to internals without completely breaking open the class:

Would it be possible to subclass the to-be-tested class, with the subclass living in the testing source tree? You could then add some test helper methods to this subclass.

It's just an off-the-cuff idea, and I can already see that it isn't the perfect solution, but it might help. Feel free to shoot some holes in it.

Maarten

Malcolm Sparks

Posts: 1
Nickname: xsumeman
Registered: Jan, 2005

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 6:23 AM

I completely agree with the thrust of Bill's article- writing tests first makes for better APIs. Generally, I find that writing good tests against the API itself is sufficient.

However, as Bill and others have already mentioned, sometimes there are some thorny internal algorithms that you want to write test first. In this instance, I don't think it is cheating to make a private method (or field) a 'package protected' method or field. The purpose of encapsulation in OO is to protect your fields and methods from external access. Just redefine external access to mean 'outside the package', rather than just 'outside the class'. I find packages are a more useful 'object' than classes, especially because I prefer to keep classes really small.

A good habit to get into (imho) is to keep your tests in the same package as the source code it is testing. However, that doesn't necessarily mean in the same directory- Java allows you to have a separate directory structure for tests. Keeping them separate from production source code can often make packaging simpler, and directories less cluttered.

Rick Kitts

Posts: 48
Nickname: rkitts
Registered: Jan, 2003

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 6:44 AM

Can you show the code?

It has proven true to me, after some few hundred tests written, that if you are adding methods solely for tests (i.e. they only can or need to be called by test code) then one is either testing the wrong thing or one has not created a fully testable design. The wrong thing is usually, perhaps always, something to do with the implementation and not the operation.

Rob

Posts: 2
Nickname: sarsipius
Registered: Oct, 2004

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 6:51 AM

I'm personally against tests affecting the public API in any way. It just opens up a whole host of problems, especially if the API will be used by third parties. I know that if I see a public method in an API, I want to see what it is and what I can do with it. If no client is ever supposed to see the method, then clients can just get confused, or worse, can get access to something they shouldn't and mess things up. (i.e. change the Map).

Also, changing a method/field from private to package or protected access still allows the client to get access by just creating a class in the same package or a subclass respectively. So I don't like this solution for myself either.

I read an article about a year ago (I forget from where) that mentioned writing tests in inner classes so that they would have full access to the outer class. Then you can just exclude the class files generated for the inner class tests when deploying or building your release jar. If using JUnit, I assume you could just instantiate the outer class, then the inner class from the outer class, and call test methods from the inner class. I haven't actually used this method myself, but it should work in theory.

One problem with that is that you pollute your code with tests instead of keeping the tests completely external.

Another option is to use reflection to access the private methods/fields you want to test. It's probably cleaner to do it this way, but again, I haven't tested it.

Jared MacDonald

Posts: 13
Nickname: jared
Registered: Oct, 2002

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 7:02 AM

In "Unit Testing In Java: How Tests Drive the Code," Johannes Link writes that the test code is a client of the API just like any other client. In his example of a Dictionary class that translates English words to German, he adds an "isEmpty()" method that the unit tests use, but that might never have appeared under traditional development.

But, as Bill said, merely adding getters for every private variable that you want to examine pollutes the API. So, the "verify()" method that you ended up with could be in a similar category to the "isEmpty" method, above: a useful method that just happens to be called solely from tests. (Perhaps "verify()" would be more similar to a "checkNotEmpty()" method.)

I'm curious about how this Merger class had two fields that couldn't be verified in any fashion -- certainly Merger itself used the fields... there were no public methods that would somehow indicate their value?

Mike Dunbar

Posts: 12
Nickname: mikedunbar
Registered: Jun, 2004

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 7:54 AM

Since I use DI/IoC just for the sake of testing (inserting mocks), I suppose that means I am open to letting tests influence the design of an API.

However, I still generally try to limit API changes solely for testability. I prefer tools like PrivateAccessor (JUnit add-on) to work around issues of private access. But, when I can't find a work around, I say go ahead and alter the design to facilitate testing.

I think that in the example you gave, I would put those comparison methods in a testing helper class...

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 8:20 AM

Hi Alexander,

> (b) An API should be as small as possible to make it easy
> for people to learn and understand and to reduce the
> chances of having to introduce incompatible modifications
> in later versions. Of course this is not a hard science.
> Sometimes it would seem unreasonably not to include one or
> the other convenience method. For me, this is a matter of
> practicality not ideology.
>
Well said. The usability minus of adding those verify methods is they increase the surface area for client programmers to wade through when they are trying to figure out how to use the class.

> Still, as the user of an API, I find the prospect of
> having to wade through loads of purely testing related
> methods to find the actual stuff, rather scary. So my
> position would be to say, hey, I'm only the user of this
> API, it's not my job to test it, so why do I have to look
> at verifyThis() and verifyThat()?
>
And this is the crux of the question. Even if you're not writing the tests, some other programmer is, and are they a less important a client of the class than you. If you had to write lots of extra code to use the class, you might want some help from the class, like a single method to call.

It sounds to me like everyone here has basically had the same feeling about tests that I have traditionally held: that their influence on the API design should be limited to usability benefits to non-test client, not minuses.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 8:47 AM

Hi Rick,

> Can you show the code?
>
Yes, but I'll post it a bit later today because of other demands on time this morning.

> It has proven true to me, after some few hundred tests
> written, that if you are adding methods solely for tests
> (i.e. they only can or need to be called by test code)
> then one is either testing the wrong thing or one has not
> created a fully testable design. The wrong thing is
> usually, perhaps always, something to do with the
> implementation and not the operation.

In general I agree with this. In certain cases in practice, though, I am nagged by the feeling that testing through the public interface isn't worth the time spent doing it. I have a very pragmatic, kind of business-oriented philosophy about software development. I do things I think will maximize return on investment in the business context. I think most people do this. When they decide not to write a test for a get method, for example, they are doing that because they judge the return to not be worth the investment in time to write the test, because the judge the risk of a get method ever breaking to be extremely low.

In the case of Merger, the functionality could be tested by testing the output of the merge method:

public void merge(Writer writer)

Right now the only page on Artima that's being served up this way is the Sign In page.

http://www.artima.com/signin.html

When Merger is being used to serve up web pages, I pass in a Writer that sends the output of the merge to the browser client.

I can test a controller through Merger's public interface, then, by passing a StringWriter into the merge method, and then just comparing the outputted string with a correct output. The trouble is that it is a pain to do this, because the template changes all the time. And that also wouldn't test whether there are extra things being stuffed in the context. The difficulty of this approach is compounded by the fact that Mergers can contain other Merger's in their context, all of which refer to templates that change fairly regularly.

So although I could test Mergers through the public interface, I find myself judging that the return on investment of doing that isn't worth the cost (in time spent doing it) compared to the return on investment of just testing the template name and context Map. The latter approach also has a cost, which includes the usability cost of polluting the public interface of Merger with two verify methods. I made the judgement that I get a better return on investment using the latter approach.

But it got me questioning why I felt that adding those verify methods to Merger was necessarily so bad. I want to have very good test coverage in the new architecture. The tests will always be there. Artima programmers will be writing them. Why are tests second class clients compared to all other clients? When you design a chip or a piece of hardware, you often put stuff on board to facilitate testing. Test points. And that's what those verify methods feel like to me.

I'll post the actual code a bit later...

Alex Dong

Posts: 1
Nickname: dongxun
Registered: Jun, 2003

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 9:01 AM

> Would it be possible to subclass the to-be-tested class,
> with the subclass living in the testing source tree? You
> could then add some test helper methods to this subclass.
>
> It's just an off-the-cuff idea, and I can already see that
> it isn't the perfect solution, but it might help. Feel
> free to shoot some holes in it.
>
> Maarten

Yes, that's one not-bad solution for C++ and C# tests.

In C++, you could make those methods protect and let your tes case derived from the class you want to test. That way, we could do all the state verification in the subclass.

In C#, we could also do so. Although C# doesn't allow multiple inheritance, you could derive from the class and annotate that class using attribute [TestFixture].

But in Java, you have no multiple inheritance, no friend, and annotation has just been introduced. So unless we change JUnit using annotation, I could hardly think out of any "real" solution to this problem.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Are Tests First Class Clients?

Posted: Jan 26, 2005 9:03 AM

There are really two questions here:

1. Should tests be first class citizens, in general?

2. Is that principle still valid if it causes private
data/methods to become more visible when, in theory
at least, they shouldn't be?

The answer to #1, I believe, is a resounding "Yes!"
I'll get to why, in a moment. As to #2, I believe that
Malcom Sparks made a most astute observation in his post when he said that package-protected is a reasonable approximation of private, in the sense that it does not permit "external" access, where external means "outside the package".

It makes sense that someone testing the package may want to access functions that aren't visible to external users. By the same token, developers who extend the package may find those functions useful when doing so. (It might even be plausible to say that, as a general rule, any function which is sufficiently granular to be testable is *likely* to useful when extending the package someone, sometime, for purposes that weren't dreamt of in the original design.

Now then, to return to question #1, when testing externally available APIs, I believe that testing is not only a first class citizen, but KING of the API. I say this out of experience as both an API designer and as a user.

When I'm designing an API, I think in terms of design consistency. I frequently use terminology that relates to the architecture and I'll do things that, at the time, seem to me to be effective and efficient.

But when I'm *using* the API, I'm not thinking about the internal architecture. I'm trying to recall from memory what the name of the function is, what arguments it takes, and what order they're in. In the process, I'll frequently "guess wrong"--the name that makes sense to me as a user of the API may not have been the name I chose as a designer. Similarly, the arguments I want to supply as a user may not be supported, or the "natural order" may be different from the one I designed, because other operations I'm doing have a different order, or because the code just naturally wants to generate item a, and then item b, so it just seems intrinsically logical to invoke "function(a,b)" rather than "function(b,a)".

Those kinds of thoughts invariably occur to me when I'm using an API, or testing it, and they frequently run counter to the ideas I had when designing it.

When that kind of discrepancy occurs--especially if it happens more than once--I invariably change the API to match my expectations as a user. (It helps that I have a poor memory. I can't remember what I coded last week, much less last month. That makes it easy to "become a user".)

I think of the world in terms of "user interface". With that perspective, an API is an interface to a library, a programming language is an interface to the computer's operations. For me, the user is king, always. When I'm designing an API, I'm frequently too caught up in the "how" (how I'm going to make things work) to keep my eye on the simple "what" (what it's supposed to do).

Testing an API can bring such oversights to light--and in my view, those are "API bugs" every bit as much as a functional faux pas.

Flat View: This topic has 57 replies on 4 pages [ 1 2 3 4 | » ]

Previous Topic

Next Topic