Teams that adopt agile practices often adopt Test Driven Development (TDD), which means, of course, that they end up writing a lot of tests. In general, thats great but there is a failure case for teams that attempt to get test infected; you can end up writing very slow tests that take so long to run that they essentially start to feel like baggage even though they help you catch errors.
This issue with unit tests isnt a new issue, its been around for a while, and there are a couple of ways of handling it. In Extreme Programming, the typical way of handling it is to periodically go back and optimize your tests as they get too slow. In many cases this works well, but the amount of optimization that you have to do can be rather large if you havent been conscious of how long your tests run during development. In one case that stands out in my memory, I visited a team on the east coast about four years ago that wrote oodles of tests against their EJB environment. The tests hit a server and went through session beans, entity beans, down to the bowels of the database and then up again. Their refrain? We dont like writing unit tests any more; they take too long to run. I didnt blame them for feeling that way, but I also didnt agree that they had written any unit tests.
The problem is rather common. Ive spoken to other XPers about it over the years and I sort of figured that the way that I handled it was common, but I was surprised to discover (on the XP yahoo group this week) that it was also a bit contentious. Heres what I typically say when I run into teams that have this problem.
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing
config files) to run it.
Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.
That might sound a little severe, but it is medicine for a common problem. Generally, unit tests are supposed to be small, they test a method or the interaction of a couple of methods. When you pull the database, sockets, or file system access into your unit tests, they arent really about those methods any more; they are about the integration of your code with that other software. If you write code in a way which separates your logic from OS and vendor services, you not only get faster unit tests, you get a binary chop that allows you to discover whether the problem is in your logic or in the things are you interfacing with. If all the unit tests pass but the other tests (the ones not using mocks) dont, you are far closer to isolating the problem.
Frankly, we need both kinds of tests, but these "pure" unit tests are undervalued.
> <i><p>A test is not a unit test if:</p> > <li>It talks to the database > <li>It communicates across the network > <li>It touches the file system > <li>It can't run at the same time as any of your other > unit tests > <li>You have to do special things to your environment > (such as editing > config files) to run it.
Sometimes, your unit test must do some of these things: 1. You test a class who's function is to interact with a db, write/read a file, or write/read the Windows Registry. 2. You develop a client; to make sure it works, it must interact with a server. At my work, for example, our client must exchange timestamped, signed, XML messages with a server. The test will not be complete without server interaction. Actually, you simply cannot test this without a server.
You want to say these are not unit tests? I'll argue that the tests can't be any simpler.
> Basically, these rules mean that 90% of the tests written > with JUnit are not unit tests :-) > > (I'm certainly not disagreeing with you, being the creator > of TestNG and all, but I just wanted to clarify...)
Yeah, depends on where you are, I guess. I usually convince teams to work like this rather early so the world looks better from where I sit. It's sort of like saying, "oh, grass never grows high" when you habitually mow your lawn.
I hope it's better than that, though. I mean, seriously, if every unit test talks to the database because that's the way the production code works, well, either those are big tests, there's not much layering in the system, or you're using a magical object mapping system. If the latter works speedily enough against the real store, more power to you, but in systems without that, well, it's just scary.
> Sometimes, your unit test must do some of these things: > 1. You test a class who's function is to interact with a > db, write/read a file, or write/read the Windows > Registry.
I'm pig headed, I write them but I don't call them UTs when I'm with a team. Say that we are working test first and we discover we need to save things in the registry. To me it's best to think about what we need more abstractly, we just need a place to store things and chances are it doesn't need to be hierarchical in the beginning. So we can develop that abstraction (which probably looks as simple as a map) and it's guaranteed to be easier to use than the Windows registry.
If we go that route, we make an implementation of that abstraction that works in memory for testing and one that translates its calls to the Windows registry calls.
> 2. You develop a client; to make sure it works, it > must interact with a server. At my work, for > example, our client must exchange timestamped, signed, XML > messages with a server. The test will not be complete > without server interaction. Actually, you simply > cannot test this without a server. > > You want to say these are not unit tests? I'll argue that > the tests can't be any simpler.
I agree that they are necessary, but I really like to see what can be done to strip logic away from I/O for the UTs. Other tests can handle the collaboration and the tests that actually touch the database or the the server, they have tests too, but there are in some netherworld in my view. In general, I don't think there should be many of them.
I think this is in keeping with real unit tests which should have as few dependencies as possible. I know when I figured out methods to mock or stub out dependencies on databases especially my unit testing running time went down by a factor of 10 or 100. And because I could close the feedback loop more quickly I wrote more tests and ended up with a lot less sneaky defects. Now if I see some dependency on the filesystem or some other object I'll try to find the fastest way to remove it so I can right true unit tests instead of less useful integration tests. Understanding and writing true unit tests has really moved my skills as a developer along. Now I just need to impart that experience to many of my direct reports. And I still write some integration, functional, or end to end GUI tests, I just let Cruisecontrol run them for me.
> If we go that route, we make an implementation of that > abstraction that works in memory for testing and one that > translates its calls to the Windows registry calls.
Abstraction is great, we use interfaces all the time. But at some point, you just have to implement (and test) your RegistryStorage and FileStorage classes (two implementations of the IStorage interface). To test these, you'll *have* to touch the filesystem and the registry (which is actually a db).
> > Basically, these rules mean that 90% of the tests > written > > with JUnit are not unit tests :-) > > > > (I'm certainly not disagreeing with you, being the > creator > > of TestNG and all, but I just wanted to clarify...) > > Yeah, depends on where you are, I guess. I usually > convince teams to work like this rather early so the world > looks better from where I sit. It's sort of like saying, > "oh, grass never grows high" when you habitually mow your > lawn. > > I hope it's better than that, though. I mean, seriously, > if every unit test talks to the database because that's > the way the production code works
Well, sometimes, the purpose of a class *is* to access the database, so one might argue that accessing the database still qualifies as a unit test.
But again, it really depends on your definition of what a unit test is, and yours is simply different from the one that is commonly used ("a unit test is a test that only exercises one class in isolation of all others", or something like that).
> If the > latter works speedily enough against the real store, more > power to you, but in systems without that, well, it's just > scary.
Speed can indeed be an issue but deciding that speed is the only factor to decide what a unit test is is, in my opinion, short-sighted.
Sometimes, unit tests go fast, sometimes they go slow. Sometimes, functional tests go fast and sometimes they go slow. There is no mutual exclusion between the concept of speed and that of unit tests.
Once I had accepted this idea, it was clear to me that the right way to tackle this problem in TestNG was to let users specify exactly what they wanted to run and give them a chance to "tag" tests appropriately.
Using groups in TestNG, you can therefore tag your test methods as "database" or "front-end" but you can also make them belong to the group "long" or "short". This effectively blurs the vague notion of "unit tests" and lets the users decide what they want to do pragmatically.
Want a quick sanity check before committing code? Run all the "short" tests, regardless of whether they exercise the database or the network.
Want a full functional coverage to make sure absolutely nothing is broken in your product? Run the "long" tests (or the "integration" ones).
You modified a schema and want to make sure you didn't break anything? Just run the "database" tests.
Groups are a very powerful and addictive feature once you start using them...
Well then we need some more nomenclature. The reason people generally use the term "unit test" for code that tests code, is that it is succinct. I've had trouble convincing some people that big automated integration tests shouldn't be called unit tests.
The common terms used seem to be "unit tests" (code testing code, often with some xunit or similar tools/framework) "automated tests" (using some automated tools that send messages, mouse clicks, etc.) and manual tests (black, gray and white box).
I think if you can come up with a short, clean term or phrase that clearly describes these other kinds of tests, you'll be able to promulgate the purer concept of unit tests. If not, then people will continue to call pretty much any code testing code a "unit test."
It's difficult to write tests for an application which uses a database as its persistent store if you have a rule that you can't access the database.
One application I'm working on currently has about two hundred tests. Each test which accesses the database takes around a tenth of a second. Running the suite takes twenty five seconds.
Each of the database tests' SetUp methods contacts SQL Server, creates a new database with an unique name, builds the structure of the database using a script, then populates the database with enough data for the test to run. TearDown simply drops the test database.
Many of the tests access the filesystem.
I run my suite of tests approximately once every ten minutes. I can let the tests run while I carry on working (I use TestDriven.NET) or, as I do more usually, sit back and decide what to do next.
I think it's fine for a test run to take a minute or even two, so long as it can be done in the background. Any longer and it might feel like you weren't getting the quick feedback you want. If you've broken something, you don't want to have made five minutes' worth of edits before you discover it.
I agree with the rules, with one exception: "1. It talks to the database" is redundant. Using a database is only not a unit test when it also violates one of the other rules.
I unit test Hibernate mapping files using HSQLDB, an embedded, in-memory database. The database doesn't live on another server, it never touches the file system, it requires not configuration, and it gives complete seperation (one db per JVM). It takes a little time to initialize the test, but it is all due to Hibernate setting up.
As far as I am concerned, these tests are unit tests. If the database that they touched wasn't in-memory, they probably wouldn't be.
Other than this point, I am very much in agreement with the post.
Isn't this just a case of 'how do we name things'?
Assuming that we adopt your definition of a Unit Test ...
Given a specific test, do we spend time to catagorize it as a 'Unit Test' or some other type of test? What is the point of making such a distinction?
If a unit of code, for example a function, is given one or more tests that attempt to prove that its logic is working, why is it important that we define these tests as 'Unit' or not? For example, a function that reads an entire text file into a RAM buffer may have one or more tests written to show that that is exactly what is achieves when run. Why split hairs about calling them Unit Tests or not? They are useful and important tests, whether or not they fit your Unit test definition.
How would my coding or testing behaviour change by calling some tests "Unit Tests" and other tests something else?
These are not rhetorical questions as I honestly don't know the answers and would like to improve my knowlege and practice of programming.
Flat View: This topic has 50 replies
on 4 pages
[
1234
|
»
]
Previous Topic
