Unit Tests vs. Integration Tests

by breeve 23. November 2009 15:32

Unit tests are all the rage today. Talk to anyone who has been reading the unit test literature and they will talk with such passion you might for a split second think you were sitting in a town hall health care meeting.

To understand the arguments for unit tests, we must first understand how code is typically tested. Code is usually tested by an integration test. Integration tests run against the entire stack of dependencies when testing your code. Integration tests are hard to set up because all dependencies must be installed correctly for the test to run. They are associated with long test times. This can be especially bad for continuous integration systems which are brought to their knees by running them on every check in. If a test fails, the bug might be in a dependency or in the component under test making it hard to tell where the issue really is.

Unit tests aim to fix these issues. A unit test is written to test one specific unit of code and differ from integration tests in one critical way; unit tests do not test dependant code. To eliminate the dependencies, they are mocked—often with mock frameworks—to return certain hard coded values to your code. Your code is tested to function properly given the mocked implementation of the dependencies. The tests run fast and if a test fails it is guaranteed to be in your code.

Let’s say I have a library that is responsible for storing data. It stores the data in the cloud by calling a web service. Unit test advocates want the web service dependency eliminated for testing purposes. To accomplish this, the web service dependency is made into an interface—or what is typically called a seam—allowing me to plug in a mock implementation. Something like:

public interface IStorage
{
    public void SaveAsync(string name, Stream stream, Action<Boolean> callback);

Now, I can mock up an implementation of the IStorage interface for my tests without having to go to the actual network. If the behavior of different parameter values into the function are well defined, I can write many unit tests for all combinations of input. Technically, with all combination of inputs covered, when the code goes into production with the real web service implementation it should just work.

This seems splendid. In reality, chances are the real web service has a bug in it because the author doesn’t believe in unit testing or the API behavior was not well defined because you thought passing a null name was ok but apparently it isn’t because the server barfed when it couldn’t find the name in the XML. This brings me to my first negative about unit tests.

Unit tests desperately depend on strong API contracts

There are two types of API contracts: compile time and runtime behavior. The first, compile time, is self explanatory. The second, runtime behavior, simply means the function has well defined behavior for certain values that are passed in. For example, if a function returns 0 for errors and then is changed to return negative numbers for errors it is a behavior change and will break code that uses it.

Behavior changes are devastating for unit tests. Remember, a mock simply mimics what the dependency is supposed to do. If behavior is changing in the dependencies, a unit test is essentially worthless until the mocks are updated with the changed behavior. Worse still, if communication is poor between API owners about behavior changes, out of date mocks can remain in unit tests creating false negatives (tests don’t fail when they should).

When false negatives exist in unit tests—hiding like an explosive IED alongside an Iraqi roadway—there might be 100% unit test coverage in your code but that won’t prevent your boss from hunting your butt down when your code doesn’t work with the product as a whole. Even the slightest behavior changes in an API can bring an entire product down.

Mocking up dependencies is not always trivial

It is possible to mock every external dependency to your component if you have a lot of time. Many components have not just one but many dependencies with large API footprints. To make matters worse, many external components are not easily mocked.

If developers simply apply a brute force approach to mock all dependencies and create wrappers for ones that aren’t easily mocked, weeks will go by without much to show. Meanwhile your boss is in his office looking at budget numbers scratching his head wondering how you are creating customer value.

Unit tests don’t catch funky issues

Because unit tests run against a fake implementation of the dependencies, real product integration issues are never found. I am talking about those hard core issues that everyone pretends don’t exist like having 10 concurrent threads running in God knows what through 15 different API layers implemented in 15 different programming languages.

So what’s a time deprived developer to do?

This is not an easy question to answer. For reasons stated above, even if you have extensive unit tests you will still need integration tests. I believe the balance between the two depends heavily on stability of dependant APIs and how well they are tested. Developing against unstable buggy APIs, cover your butt with integration tests. Stable well tested API’s like the .NET framework, unit test away.

Tags:

Essays

Powered by BlogEngine.NET 1.5.0.7
Theme by Mads Kristensen

About Me

I am a Principal Engineer with 13 years experience developing and releasing software products. I started developing in C/C++ then moved into .NET and C# and have tech lead multiple projects. I have developed products in Windows Forms, ASP.NET/MVC, Silverlight, and WPF. I currently reside in Austin, Texas.

Own Projects

Pickaxe - An easy to use web page scraper. If you know a little SQL and how basic CSS selectors work, no web scraping product will be easier to use.

Download Page

Source Code

Created ASP.NET MVC forum originally targeting home owner associations but now in use by an investor group.

http://vtss.brockreeve.com

A language for querying PGATour golf strokes.

http://pga.brockreeve.com/

Real time bidder for car ads demo

http://cargawk.com/

Simple front end tic tac toe

http://tictac.brockreeve.com/

Currently Reading