Summary
TestNG creator Cedric Beust notes that while unit tests have been receiving the most amount of attention, functional tests can be more useful than unit tests.
Advertisement
Cedric Beust, author of the TestNG testing framework, notes in his latest blog post, Unit or functional? that,
While unit tests have been receiving a lot of exposure these past years, functional tests are ultimately the only way to guarantee that your application works the way your customers expect.
Beust illustrates this point with a simple example, showing that while individual unit tests can succeed, a functional test can still fail, indicating an error in the application. Since end-users care only about a functioning application, Beust notes that,
Ideally, you want your application to be covered by both unit and functional tests, but if you are pressed for time and you need to choose between implementing a unit test or a functional test, I ... recommend going for the latter (because it serves your users) and then down the road, implement more unit tests as you see fit (for your own comfort).
Beust also points out that in a testing suite consisting of both functional and unit tests, the unit tests often serve to pinpoint why a functional test fails: in other words, unit tests are useful mainly for the developer, but functional test exercise aspects of the code most relevant to users.
A test suite of both functional and unit tests can result in potentially more than 100% test coverage, according to Beust:
More than 100% coverage? Is this even possible? I know it sounds a bit strange, but it simply means that running all the tests in this class will result in certain portions of our main application to be run several times, which is not always a bad thing. Another way to look at it is that running only the [functional tests] will result in 100% coverage (as will running only the [unit tests]).
When you are pressed for time in your projects, what kinds of tests do you think are acceptable to skimp? And what sorts of tests would you not release an application without?
I agree that the current emphasis on unit tests over functional tests isn't entirely justified by the facts.
I think the fact that unit tests are core components of certain agile methodologies like TDD and XP are a large part of the reason. XP in particular has a low tolerance for deviations from the "laws" and any doubt about the value of unit tests would significantly endanger the whole edifice.
The practice of writing unit tests before coding seems most venerable since only the final iteration of a unit has any relevance to correct functionality as the customer sees it.
I agree that functional tests are necessary to make sure that requirements are fulfilled, and that you can do that without unit tests, but I do not think that anyone will save much time by such a decision.
To quote Steve McConnel: "A decision early in a project not to focus on defect detection amounts to a decision to postpone defect detection until later in the project, when it will be much more expensive and time consuming. That's not a rational decision when time is at a premium."
I read an interesting article in an Engineering magazine that caught my attention because I see a strong relationship to this general topic.
The article, in a nutshell, describes how there is a commonly held myth that Japanese car makers design the parts for their cars to be very precise and this is what allows them to create very precise end products. The fact is that the precision on their parts is, in general, lower than American car makers and that creating highly precise parts does not necessarily make it easier to create precise end products. The Japanese makers' cars are more precise as a whole than the American makers' cars. What is important is how the parts come together.
How this relates to software is that I think the focus on unit tests makes this same logical error. That eliminating defects in all the parts of the system will eliminate defects in the whole system. This isn't the case and it's also true that a system with defects in it's components can perform perfectly.
You will need the same amount of functional testing whether you write unit tests or not. But unit tests will reduce the amount of time spent debugging. For reusable components, they will pay off in spades. What concerns me is that the focus is on the unit tests when really functional testing and regression testing are crucial. I think perhaps it's because unit testing is simple and closed and functional testing is complex and open. It's not generally possible to get 100% coverage with functional tests. I don't think it's really possible with unit tests either but that's a different topic.
In my understanding, what is unique about functional tests is that they test interactions between units (usually classes), while unit-tests change refactoring from being too high-risk (acidentally introduced bugs) to the only viable solution to software rot. Beyond this, both have much to offer.
> In my understanding, what is unique about functional tests > is that they test interactions between units (usually > classes), while unit-tests change refactoring from being > too high-risk (acidentally introduced bugs) to the only > viable solution to software rot. Beyond this, both have > much to offer.
Unit tests will help with very simple refactoring within units. Non-trivial refactoring will generally require changing the unit tests. Regression testing, on the other hand, should remain static as long as the requirements are stable.
To me, the example does not look like a good case for functional tests. Depending on the user of code to ensure the right order of execution? Unit tests should catch that - filterFile() should either throw or open the file if it's not opened.
> To me, the example does not look like a good case for > functional tests. Depending on the user of code to ensure > the right order of execution? Unit tests should catch that > - filterFile() should either throw or open the file if > it's not opened.
I don't understand your point. I believe that filterFile() does throw an exception. The problem is that the application doesn't work. Can you clarify what you mean here?
It's amazing how a well-reasoned introduction of an interesting discussion veers off track so quickly into a "I don't agree with Bob Martin" track.
My two cents: I have never worked on a project that did sufficient functional or unit testing. Questioning the relative value of functional and units testing is a pragmatic act. Bob Martin's comment about trading quality for speed has nothing to with this question. I think that a lot of Bob Martins advice is credible, relevant and influential, particularly the papers about class and interface design, dependency inversion etc. I suspect that the arguments in the influential "End to end arguments in system design" paper can also be applied to this question and they support Cedric's assertion.
My experience isn't enough for em to have a strong evidence based view on this. My instinct is that Cedric is correct and that unit tests are currently over-emphasized compared to functional test
I think the first error is in assuming that "Unit or Functional Tests?" is a sensible question in the first place. To me, it's like asking, "Does my car need a gas pedal or a steering wheel?" The two are techniques intended to solve completely different problems.
Functional tests are the only way to be confident that your application will work for your customers as you intend it to. Consequently, they tend to be rather exhuastive and slow, and are primarily intended for acceptance testing. Unit tests are a way for developers to verify that the code works as they intend, and are a quick way to predict whether that code will pass the functional test suite. If I can tell with 98% certainty after one minute of unit testing that my code will pass acceptance tests that take ten hours, I am way ahead of someone who relies solely on functional tests. But if I only write unit tests, I am never really testing my application as customers will use it.
>I don't understand your point. I believe that filterFile() does throw an exception. The problem is that the application doesn't work. Can you clarify what you mean here?
What I meant is that, through defensive coding and unit testing, the described app can be brought to correctness (filterFile() opens the file if its not opened, and openFile() does nothing if the file is opened).
I'm not arguing against functional tests and Cedric brings a valid point - clearly to have functional tests is necessary because they are the last line of defense - software has to function or else it's useless. However, I see unit tests as fundamental, rather than optional. And they serve not only for testing purposes - they have a significant role in design, too.
Sorry if I have diverted the discussion in the wrong direction. I did not mean to.
> Functional tests are the only way to be confident that > your application will work for your customers as you > intend it to. Consequently, they tend to be rather > exhuastive and slow, and are primarily intended for > acceptance testing.
Personally, I think saying functional tests can serve as acceptance tests is worse than saying unit tests can replace functional tests.
Functional tests should focus on things that are measureable. Functional tests passing should generally be a prerequisite to passing acceptance tests, but they are different. It's just like a system can be completely useless but meet all of its functional requirements. Acceptance testing covers the less tangible aspects.
Flat View: This topic has 18 replies
on 2 pages
[
12
|
»
]
Previous Topic
