Unitils started as a discussion group on unit testing. The result of these discussions is the list of guidelines found on this page.
3. Database testing guidelines
4. O/R mapping testing guidelines
The guidelines are work in progress and are intended to be open source. We have a guidelines forum on which guidelines can be discussed and new ones can be proposed. We hope to hear from you. Feel free to post your comments.
A unit test is a piece of code written by a developer that tests another piece of code. Usually a unit test executes some particular method in a particular context, and it checks if the outcome matches the expected one. Unit tests are independent of each other, so that the sequence in which they are executed is of no importance. They can be executed without user intervention, which makes them very useful as automatic regression tests.
Unit tests ensure that a piece of code really works at the time of writing, and that it keeps on working after subsequent changes to the code. Unit testing will reduce the time you spend on debugging, and it will improve your designs. It will make sure you have confidence in the code. It is a great help when refactoring, since it gives you direct feedback in case you break something. Unit tests also act as documentation, since they show how a method should be used and what boundary conditions it can handle.
The number one argument against unit testing is that it takes too much time. This not true, it will save you time instead, because of its immediate and constant feedback, and since it greatly reduces the debugging efforts you have to spend afterwards. It will make sure you fix bugs early that you would have found late when following a no-unit testing approach. This doesn't mean we ignore the fact unit testing takes time. It does take longer to write both code and unit tests, but the complete, correctly functioning product will be delivered earlier, and it will be much more maintainable afterwards.
The unit testing strategy is part of your project methodology and should be discussed in advance, in cooperation with the customer. The customer must be aware of and agree with the fact that you choose the unit testing approach. He has to be aware of the fact that daily coding will take some more time, but that the eventual stable product will be delivered sooner.
Each of the unit-tests test only a small part of the code. There should also be other kinds of tests on the project that test the code as whole. Examples are integration tests, user acceptance tests and performance tests.
There are many possible levels of doing unit-testing. Some people want to reach 100% code coverage: every line of code is used in at least one test. Of course this requires you to write lots and lots of tests, probably spending more time writing and maintaining tests than writing code.
The other extreme is writing no tests at all. We think the ideal lies somewhere in between. Adding a few unit tests for some piece of code already catches most of the bugs in that code while staying maintainable and keeping focus on the project code.
A possible set of unit tests per piece of code can include:
The main focus of the guidelines is to make unit-testing easy and maintainable. Writing a test shouldn't take much time and a test should not hinder you when code is being altered and refactored.
Deciding on a unit testing strategy means deciding which layers of the software system will be tested (GUI, business, database), and what tools and libraries will be used. What is best chosen highly depends on the project, but typically you would at least test the business and database layers.
We've chosen to use following libraries to aid us in writing unit tests:
Aside from these libraries there is also a library that we created our own, that helps implementing much of the functionality described in these guidelines: Unitils
A unit test generally consists of following parts:
For each test method:
Test classes typically contain a data setup part and a test part. The setup part is performed in the setUp() method and contains the instantiation of the class under test instance and all objects that will be used as data during the test (See Create test data during setup ). The class under test instance is the object on which the tested methods are called. Typically there is only one such instance per test class and it should be created in the setUp() method.
Verification of results is performed by using assert statements. You should only add asserts if they are of value for the test (See Keep test methods small ). If objects need to be compared, use Unitils' ReflectionAssert instead of a regular assertEquals() (See Use reflection for equality checking ). Assert statements can have messages as arguments that will be used when the assertion fails. These messages do not have much added-value and can therefore be left out. E.g. both assert statements in the example below give the same result, but it's less work to use the second version:
assertEquals("Name is not equal", "John", result.getFirstName());
assertEquals("John", result.getFirstName());All bad smells that you can have for regular code are also applicable to test code. Test classes can for example contain duplicate code or become too big because too many tests were added to them. You should react to these smells as you would with normal code: refactor the code.
package org.unitils.util;
import junit.framework.TestCase;
/**
* Test class for {@link UserAdmin}.
* Contains tests with ...
*/
public class UserAdminTest extends TestCase {
/* Class under test */
private UserAdmin userAdmin; // Class under Test
/* A simple test user */ //
private User user; // Test Data (Fixture)
/* An administrator role */ //
private Role adminRole; //
/**
* Initializes the test fixture.
*/
protected void setUp() throws Exception {
super.setUp(); // Superclass Setup
userAdmin = new UserAdmin(); // Create Class under Test
user = new User("John", "Doe"); //
adminRole = new Role("Administrator"); // Test Data Setup
}
/**
* Test for method with ...
*/
public void testAddUser_oldEnough() {
user.setAge(18); // Extra Test Setup
userAdmin.addUser("jdoe", user, adminRole); // Perform Test
User result = userAdmin.getUser("jdoe"); //
assertEquals("John", result.getFirstName()); // Verify Results
assertEquals("Doe", result.getLastName()); //
}
}
For the test classes:
For the test methods:
Proper naming of unit tests is important: it enables other developers and tools to easily locate the test classes and methods.
When creating a new test class, name the new class after the class that is being tested and add the suffix 'Test' . If there is more than 1 test class for a certain class (see Keep your test fixture simple ) use the same convention, but add an extra name in front of 'Test'. This extra name should give an indication of the kind of tests in this class.
The naming of test methods is similar, but instead of using a suffix, we use the prefix 'test' , followed by the name of the method under test (capitalize the first letter as with other method names). If there is more than 1 test for a certain method name all special cases by adding an underscore ('_') and an extra name. This extra name should again give an indication of the subject of the test.
| Class/method under test | Test class/method | |
| Typical | ArrayList | ArrayListTest |
| Multiple tests | ArrayList | ArrayListTest |
| ArrayListSortingTest | ||
| ArrayListIteratorTest | ||
| Happy path | add(int index, Object element) | test Add() |
| Special cases | add(int index, Object element) | test Add_nullElement () |
| test Add_negativeIndex () |
Put test classes in the same package as the tested code but store them in a separate source tree.
Test classes should be easy to locate. Creating test classes in the same package as the class under test and using the correct naming conventions enables you to find all tests of a class very quickly. Furthermore, sharing the same package will enable you, if necessary, to test protected and package-protected methods.
Test artifacts are only of use during development. They should not be included in the end-deliverables. Putting test code in a different source directory gives a clean separation of test code and production code. It will make it easier to create deliverables such as JARs and EARs that do not include any of the test classes.
src/main/java
/org/unitils/util/StringUtils.java
src/main/resources
/project.properties
src/test/java
/org/unitils/util/StringUtilsTest.java
src/test/resources
/unitils.properties
Put all non-trivial test data creation in the setUp() method of your test.
Test methods
If you need non-trivial test data in your test methods, put them in instance variables of the test class. Initialize the instance variables during the setup phase of the test. This set of test data objects is called the fixture of the test. Trivial test data such as strings and integers are typically not added to the fixture and used directly as arguments in the test methods.
Putting test data in instance variables will promote reuse of test data and will keep test methods small and maintainable.
If needed, a test method can perform modifications to the test data. This extra setup should however be used as an exception and be limited to changing a few values of the already created test data. JUnit and TestNG execute the fixture methods before each of the test methods so changing the test data will not cause problems for other test methods.
If the set of test data becomes too large (too many instance variables) and is incoherent, split the tests into multiple test classes (see Keep your test fixture simple ). Because not all test data is used by every test, splitting up tests will also avoid creation of too much unused test data.
Note: When using JUnit 3, do not forget to call super.setUp(), since superclasses can perform initialization in the setUp() method. If your forget the call to super.setUp(), this initialization will be omitted.
public class UserAdminTest extends TestCase {
/* Class under test */
private UserAdmin userAdmin;
/* A simple test user */ //
private User user; // Test Data (Fixture)
/* An administrator role */ //
private Role adminRole; //
/**
* Initializes the test fixture.
*/
protected void setUp() throws Exception {
super.setUp();
userAdmin = new UserAdmin();
user = new User("John", "Doe"); // Test Data Setup
adminRole = new Role("Administrator"); //
}
/**
* Test for method with ...
*/
public void testAddUser_oldEnough() {
user.setAge(18); // Extra Test Setup
userAdmin.addUser("jdoe", user, adminRole); // Use Test Data
User result = userAdmin.getUser("jdoe");
assertEquals("John", result.getFirstName());
assertEquals("Doe", result.getLastName());
}
}
If the test fixture becomes too large and incoherent, split the test into multiple smaller tests. As names for these extra test classes, use <class name> + <description of tested behavior> + 'Test'.
Grouping too many (incoherent) tests in the same test class can cause a test fixture to become large and incoherent. If this is the case, split up the test fixture by dividing the test methods over multiple test classes. Because test data is put in instance variables (See Create test data during setup ) you can easily spot incoherent test classes: they contain lots of instance variables that are only used by some of the test methods.
Try to find a logical group of test methods and move them to a new test class. Name this new test class as follows: the original test class name, followed by a name that characterizes this new group, followed by 'Test'.
Suppose you have a test class for a class named UserAdmin that contains code for maintaining users and their roles.
public class UserAdminTest extends TestCase {
/* A simple test user 1 */ //
private User user1; //
... //
/* A simple test user 10 */ // Test Data (Fixture)
private User user10; // that is too large
/* An administrator role 1 */ //
private Role adminRole1; //
... //
/* An administrator role 10 */ //
private Role adminRole10; //
/**
* Test for method with ...
*/
public void testAddUser1() {
}
/**
* Test for method with ...
*/
public void testAddRole1() {
}
...
}
This class clearly contains a fixture that is too large, so we should split the test class into multiple test classes. We could for example create a new class named UserAdminRoles Test and move all roles related tests and test data out of the UserAdminTest class to this new class.
public class UserAdminTest extends TestCase {
/* A simple test user 1 */ //
private User user1; //
... // User Test Data (Fixture)
/* A simple test user 10 */ //
private User user10; //
/** //
* Test for method with ... //
*/ //
public void testAddUser1() { // User Tests
} //
... //
}
public class UserAdminRolesTest extends TestCase {
/* An administrator role 1 */ //
private Role adminRole1; //
... // Roles Test Data (Fixture)
/* An administrator role 10 */ //
private Role adminRole10; //
/** //
* Test for method with ... //
*/ //
public void testAddRole1() { // Roles Tests
} //
... //
}
Write tests for all boundary conditions of the tested method. Only boundary conditions that a method can handle according to its interface (javadoc and arguments) should be tested.
For every boundary condition do one of the following:
You should test the interface of a method. If a method's interface (javadoc and arguments) does not prohibit a boundary value such as null, the method should be able to handle it.
If a method shouldn't be called with some boundary value: change the signature. That is, put the constraint as precondition in the javadoc of the method. This could be as easy as adding not null to the parameter declaration in the javadoc.
If the method allows boundary values: create test cases for these values. Following table shows the typical boundary values for which you should write a test.
| Argument | Boundary conditions |
| Object | null |
| boundary conditions of instance variables | |
| Numeric value | 0 |
| negative value | |
| minimum and maximum | |
| String | null |
| empty string | |
| length equal to maximum length (if maximum defined) | |
| length greater than maximum length (if maximum defined) | |
| Collections | null |
| empty collection | |
| collection with duplicate elements |
The conditions in bold are the minimal conditions for which a test needs to be written. Add extra boundary condition tests when necessary. For example, you should create tests for the length of a string argument when the method implementation has some kind of restriction on the length of a string.
Exception boundary conditions should also be tested. How to test such conditions is described in Test declared exceptions .
Suppose you have a class named UserAdmin that defines a method for adding a user. This method takes two arguments: a user and a role.
public class UserAdmin
{
/**
* Adds a user and links it to the given role.
*
* @param user the user to add
* @param role the user's role
*/
public void addUser(User user, Role role) {
users.add(user.getName());
roles.put(role.getName(), user)
}
}
After writing the tests for the typical behavior of this method, we should also add test for the boundary conditions of the arguments. Note that the method's javadoc doesn't mention that the method cannot be called with null arguments.
You should now either, add the proper preconditions to the javadoc
public class UserAdmin
{
/**
* Adds a user and links it to the given role.
*
* @param user the user to add, not null // Extra precondition
* @param role the user's role
*/
public void addUser(User user, Role role) {
users.add(user.getName());
roles.put(role.getName(), user)
}
}
or write a proper boundary condition test.
public class UserAdminTest extends TestCase {
/**
* Test for method for a null role argument.
*/
public void testAddUser_nullRole() {
userAdmin.addUser(user, null); // Boundary cond. test
//validate result
...
}
}
If you look at the implementation of the method, you see that this test will fail initially. You should fix the bug by implementing the proper null handling in the method and then rerun the test until it's successful.
Only test the behavior as is described in the public interface and javadoc. The internal implementation should not be tested. Do not test trivial methods such as getters and setters. Only methods that contain logic should be tested. Only test protected methods if they are meant to be called directly by subclasses, for example utility methods in abstract base classes.
The public interface of a class defines the behavior of the class. How a certain behavior is implemented is not important. Your tests should therefore not depend on how a class is implemented. Only testing the interface will make your tests less susceptible to change and refactorings. It will minimize the changes needed to your tests when changes to the implementation are made.
Tests should be written for methods that contain non-trivial logic. Trivial methods that contain no logic, such as getters and setters or methods that do pure delegation to objects of other classes, should not be tested. These tests don't provide added-value and only increase the effort needed to maintain all tests.
Suppose you have following class, containing a non-trivial public method, a non trivial getter, a trivial setter and an internal implementation method.
public class User {
/**
* Adds a new role, the guest role will be added if user is null.
*
* @param role the role to add, null for guest role
*/
public void addRole(Role role) {
if (role == null) {
role = createRole("guest");
roles.add(role);
}
}
/**
* Getter for the age field.
*
* @return the age in years
*/
public int getAge(){ //
// calculate age using birth date // Non trivial getter
return age; //
} //
/**
* Setter for birthDate field.
*
* @param birthDate the date, not null
*/
public void setBirthDate(Date birthDate){ //
this.birthDate = birthDate; // Trivial setter
} //
/**
* Factory method: creates a new role with the
* given name.
*
* @return the role
*/
protected Role createRole(String roleName) { //
return new Role(roleName); // Non-public method
} //
}
In this example you should only write tests for the addRole() and getAge() method since these contain real logic. Testing the trivial setBirthDate() method wouldn't give added-value. The createRole() method is used for the internal implementation and should therefore not be tested separately. It will implicitly be tested during the addUser() test.
Following code snippets show an example of a test that uses knowledge of a class's internal structure:
public class AuthenticatedUserRepository
{
private Map authenticatedUserMap = new HashMap();
/**
* Adds a new user.
*
* @param user the user to add, not null
*/
public void addAuthenticatedUser(User user) {
authenticatedUserMap.add(user.getSsn(), user);
}
/**
* Verifies if the user is authenticated
*
* @return true if the user is authenticated, false otherwise
*/
public boolean isAuthenticated(User user) {
return authenticatedUserMap.get(user.getSsn()) != null;
}
/**
* Returns all users that are authenticated as a Map.
*
* return the map
*/
public Map getAuthenticatedUserMap(){
return authenticatedUserMap;
}
}
Suppose you would create a test for addAuthenticatedUser() as follows:
public class UserAdminTest extends TestCase {
/**
* Tests the addAuthenticatedUser() method.
*/
public void testAddAuthenticatedUser() {
userAdmin.addAuthenticatedUser(user);
//validate result
Map authenticatedUserMap = userAdmin.getAuthenticatedUserMap();
User result = authenticatedUserMap.get(user.getSsn()); // Implementation detail
ReflectionAssert.assertLenEquals(user, result);
}
}
In this example, if you change the implementation of addAuthenticatedUser and isAuthenticated to use the username as key in the map instead of the SSN, the test will break. This can be avoided by using isAuthenticated in the unit test to verify that the addAuthenticatedUser behaves correctly.
Keep your test code as simple as possible. It should only contain:
Tests should only contain the minimal amount of code needed to perform the test. This will keep your tests simple and maintainable. Only write data setup and assertions that are really of value for your tests. There is no added-value in writing an assertion for a field value if the function of the tested method is to calculate and update the value of another field.
Unitils' ReflectionAssert can aid in only comparing values that are important for the test (See Use reflection for equality checking ). It can be configured to perform a lenient comparison: ignore fields for which the expected object contains java default (null/0) values and ignore the order of collections.
public class UserTest extends TestCase {
private User user;
/**
* Initializes the test fixture.
*/
protected void setUp() throws Exception {
super.setUp();
user = new User("john");
}
/**
* Test method for setting the age.
*/
public void testSetAge() {
user.setBirthDate(parseDate("20/05/1957");
assertEquals(50, user.getAge());
assertEquals("john", user.getName()); // Useless assertion
}
}
There is no reason for verifying the name of the User object in this test. The code in this test should be limited to testing the age only. On the one hand, this improves the readability of this test. On the other hand, it makes the test more maintainable because it is no longer dependent on the fact that the tested object was intialized with "john" as its name.
A unit test should not depend on the prior execution of another unit test. All tests should be independent of each other. One should be able to run each of the tests on its own.
The sequence in which test methods are called is not guaranteed. Unit tests should therefore never rely on any execution sequence. The test fixture must be completely (re-)initialized in the setup and test methods.
public class UserAdminTest extends TestCase {
/**
* Test method for adding a user.
*/
public void testAddUser() {
userAdmin.addUser(user);
assertEquals(user,userAdmin.getUser("user"));
}
/**
* Test method for removing a user.
*/
public void testRemoveUser() {
userAdmin.addUser(user); // Extra setup
userAdmin.removeUser(user);
assertNull(userAdmin.getUser("user"));
}
}
The testRemoveUser method may also pass if it wouldn't perform the addUser call first, since the method testAddUser is executed first by the test runner, Nevertheless, the method testRemoveUser() includes a call to addUser in it's setup, to make sure it is independent of any other tests.
Writing tests should be done while you are writing your code, not afterwards.
When you are writing code, you have to test it in one way or another. So why not write a unit test directly? Writing unit tests immediately makes sure all individual pieces work before trying out the whole. Bugs are more difficult to spot when testing your application through its end user interface, and the application has to be rebuilt and deployed each time you've done a fix. Also, the necessary test data is often not available or certain boundary conditions are difficult to simulate in integration testing mode.
Some people prefer writing tests before writing the actual code (test-first development). They start with writing the method's signature and write a test for the new functionality (which of course fails at that moment). Then they implement the code that makes this test succeed. Afterwards, tests are added for special cases and boundary conditions and the implementation is updated to make these tests pass. Although this approach is very viable, this way of coding doesn't work for everyone. We only advise you to write your test code at least while you are writing the code. In other words, the implementation is not completely finished if there are no unit tests.
Following approach is a mix between test-afterwards and test-first, that appears to work for many developers: You start with writing the signature and a first implementation of a method. Then you write a test for the main execution path and fix the implementation if necessary. After this, tests for extra boundary conditions and alternative flows are added before these alternative flows are implemented. For bug fixing the test-first approach also works very well: You start with writing a failing test that reveals the cause of the bug, then you make the change that makes this test pass.
Use reflection to check equality instead of the equals() method. As a consequence, you should avoid using assertEquals() for object types other that basic types or value objects such as String, Integer etc.
Equals methods for domain objects can contain business logic and should therefore not be used to test whether all inner values are equal.
Suppose for example that we want to verify that a User object returned by a DAO method matches an expected one. The equals() method of this user object may be implemented to return true when they have the same social security number. If you use assertEquals() on 2 user instances you therefore only test the equality of the social security number and not the equality of any of the other fields. Instead of using the equals() method, you should compare each of the field values one by one. This can be done by accessing the field values through their getter methods (if the field has one) and then comparing the values directly. Doing this manually however requires lots of coding and is error-prone. It's better to do this automatically by using reflection.
Unitils provides a utility class that does this for you: ReflectionAssert .
ReflectionAssert runs through the fields of the instances using reflection and compares each of the field values one by one. If a field value is again an object instance, all its fields are recursively compared the same way. The ReflectionAssert class also offers functionality for non-strict comparison of fields. You can for example ignore field values for which the expected object contains a Java default value (null or 0), or test whether a collection contains some expected values, but not necessarily in the same order. This makes it possible to only compare what is realy important for the test (See Keep test methods small ).
public class User {
/* The social security number */
private String ssn;
/* The last name of the user */
private String name;
/**
* Test whether the given object is the same user.
*/
public boolean equals(Object object) {
if (!(object instanceof User)) {
return false;
}
User user = (User) object; //
if (ssn == null && user.ssn == null) { //
return true; // Business logic
} //
return (ssn != null && ssn.equals(user.ssn)) //
}
user1.ssn = 111
user2.ssn = 111
user1.name = John
user2.name = Tom
assertEquals(user1, user2) true
ReflectionAssert.assertEquals(user1, user2) false
Whether or not to perform unit-testing is a decision that should be made at the level of a project.
The unit testing strategy that you adopt in your team has great impact. It should be fully adopted and supported by everyone; there is no use in some intermediate level. If you don't go for it all-the-way, you will eventually end up writing or running no unit tests at all.
Why doesn't it make sense when some of your team members are into unit testing and some don't? Because members that do write them will constantly be annoyed when others change their code and don't maintain the tests, or by having to write unit tests for untested code of somebody else when changing it. In the end, they will also give up their good efforts and stop writing unit tests.
The best way to get everyone to write unit-tests is to convince the project manager of the necessity of such tests and let him impose the unit test strategy on a project level.
When a test is broken, fix it as soon as possible.
The goal should be to keep the bar green as much as possible. This motivates people to write descent and working tests. If you don't start with a green bar, then it is less likely that you notice additional failing tests. If there are too many broken tests, the motivation to decreases to fix or to write any additional unit tests.
When you notice that you've broken a test, fix it at once. If this is not immediately possible, disable the test. This can be done by commenting out the test implementation and adding a logging statement for JUnit 3, by using the @Ignore annotation for JUnit 4, or by adding the test to an ignored group for TestNG.
Only run the tests of the code that you're working on. Rely on a continuous build system to perform the full regression suite.
Manually running all of the tests over after each code change is time-consuming. Only run the tests that could be impacted by the code that you are have changed and let an automatic build system run the other tests. Typically, you only run the tests that verify the behavior of the class that you are changing. E.g. if you changed something in the User class, perform all tests on the User*Test classes.
The continuous build server should be configured to run all unit-tests after every commit to the version control system and immediately give feedback when somebody broke a test. Developers should immediately react by fixing the broken test. (See Fix broken tests immediately )
Do not use a static definition of the test classes in a test suite, like for example JUnit's TestSuite class, for executing all the unit tests in you project. Tools exist that can dynamically generate a list of all unit tests that can be found in a source folder. You can use for example a build tool such as Ant or an IDE for this purpose.
JUnit offers the possibility to create a group of tests by adding them to a test suite. You could for example add every test in a package to a suite so that they can all be performed in a single run. Manually maintaining such suites however is asking for trouble. Each time you created a new test-class, you would have to manually add it to the test-suite. If you forget this, the tests will not be performed when the suite is run. After a while, a lot of 'hidden' tests will probably fail.
Build tools like ant and IDEs like Eclipse and IntelliJ can automatically create these test suites for you, so that you are sure that every test is being performed.
Ant build-script for running all your tests in your test source folder:
<junit printsummary="yes" haltonfailure="true">
<classpath>
<pathelement path="${project.classpath.junit}"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${project.build}/test">
<fileset dir="${project.source.test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
If you fix a bug, also write a test for the fix.
Although unit tests help improving the quality of your code dramatically, it is always possible that you code contains bugs. These bugs are typically intercepted in a later test-stage, such as during functional or acceptance testing.
When a bug is found and the code is modified to fix it, also write a unit test for this fix. You have to test it anyway, so why not simply writing a unit test. This also makes sure that the bug is not re-introduced at a later time (regression test). A typical way of handling bugs is to write a failing test that reveals the bug first, and then fix your code to make your test pass. (see Write tests while writing code )
Only test code that is being written or changed. Do not write unit tests for a project's legacy code unless you're modifying it. Do not write tests for libraries or non-project code.
Unit tests should test code that is written on the project. A common situation is that your project contains legacy code with little or no unit tests, and that you want to adopt a more agile approach for future development. A good practice is to write unit tests for new code and for existing code that you modify or rewrite. Writing tests for existing code is often difficult because such code is usually not written in a test-friendly way. Most likely, this software is used in production, so it has already been tested and debugged thoroughly in other ways.
Unit tests should not test code of libraries, external services or other projects that you depend on. These libraries and external projects should have their own test suites.
Write tests for exceptional conditions of functional nature, that are typically declared in the interface.
Functional exceptions are boundary conditions that should be tested in a unit-test. Such exceptions are typically declared in a method's javadoc. If you use JUnit 3, you can do this by triggering the exceptional condition in a try-catch block and calling the fail() method if the expected exception was not thrown (see below for an example). When using JUnit 4 or TestNG, you can specify the expected exception in the @Test annotation.
If a test method contains checked exceptions, and we're not dealing with a test for this specific exception, don't surround the method call with a try-catch but simply add a throws clause to the test method. Exceptional conditions that arise because of a bug in the code, e.g. NullPointerException s, should not be caught in a unit test. Let JUnit or TestNG handle these exceptions.
public class UserTest extends TestCase {
/**
* Test method for setting the age.
*/
public void testSetAge() throws Exception { // No try-catch for NegativeAgeException
user.setAge(50);
assertEquals(50, user.getAge());
}
/**
* Test method for setting the age.
*/
public void testSetAge_negative() {
try {
user.setAge(-1);
fail("Epected NegativeAgeException."); // No exception, so fail
} catch (NegativeAgeException e) {
//expected //
} // Expected flow
assertEquals(20, user.getAge()); //
}
}
The database layer is a very important though brittle layer. Since the database layer is implemented at two different levels (DAO layer + database itself), bugs occur frequently because of mismatches between these two. To spot problems in the persistence layer instantly, unit tests should be written for the persistence layer as a whole (DAO layer + database itself). DBUnit (http://dbunit.sourceforge.net/ ) is used as supporting technology.
Unit tests should not require a database filled with test data. If they depend on existing data, it becomes very difficult to make changes to this data without breaking other tests. Developers will therefore rather add new data than changing existing records. As more data is added to the database, this situation becomes more and more difficult to maintain.
We need to have complete control over the database structure and test data for each test individually. This can be realized by using a unit test database containing only the database structure and no data. Every unit test will then insert a small set of test data before running the test (See Keep your test data files simple ).
To avoid that different developers interfere with each other when executing unit tests, each developer should be given a separate database instance or schema (See Automate the maintenance of test databases ). These instances or schemas contain little or no data and can therefore easily be installed on a shared server or created as an extra schema on the existing development database.
The following schema represents a team with 3 developers. Every developer has it's own unit test database schema. The continuous build server, in this case Cruise Control, is also configured to run on a separate schema (UNIT_CRCO).
All the test databases are extra schemas that have been added to the development database.
The maintenance of test databases should be automated as much as possible. The recommended way is to use a pull strategy, where database updates are made available as scripts in the source control system and applied to the test database before running a test. Database updates can be performed incrementally or by recreating the database from scratch.
Manually maintaining a database instance or schema for every developer is quite a lot of work. Much time can be saved by automating the maintenance. This can be done:
The advantage of the from-scratch approach is the simplicity: You only have to maintain a script containing the complete database definition in the form of DDL statements. If the script contains an error, you simply fix it. The script can either be maintained manually or exported from some modeling tool. A major drawback is that, when the database size grows, recreating a database from scratch takes a considerable amount of time.
To keep your developers motivated to write and run tests, wait times should be avoided whenever possible. Unless your database is small, an incremental approach to database maintenance should be considered. Incremental updates to the database structure are performed almost without noticable delays. Also, since incremental updates are data-preserving, the same update scripts could be used to migrate test or production databases. A drawback of this approach is the increased complexity: If a DDL statement that contains an error was applied to a number of unit test database, you will have to make sure new script is executed that contains the fix for this error.
You can have the advantages of both approaches by combining an incremental and from-scratch approach as follows: Make sure the greater part of database updates is applied incrementally. To clean up your scripts from time to time or to fix an error, you can recreate the database schemas from scratch.
For distributing the changes to the unit test databases, you can use either a push or a pull strategy:
In absence of supporting tools, choosing a push strategy is the most obvious decision for a project, because it is simple to set up. You only have to write a simple tool or script that distributes the changes to all the databases. Nevertheless, a pull strategy offers nothing but advantages, especially when your scripts are maintained in a version control systems. This way, developers recieve update scripts along with the corresponding code changes. Before a test is run, these new scripts are be executed on the database, so code and database structure are always in synch. Another advantage is that maintenance is performed in a decentralized way. This allows developers to use a local test database, for example, when working on separate locations or at home.
Unitils' database maintainer can keep your test databases up-to-date using a pull strategy, supporting both the incremental and from-sratch ways. For more information, we refer to the tutorial .
Disable all foreign key and not null constraints in unit testing databases. This way, no more test data has to be inserted than required for the test. Automate the disabling process to reduce the burden of manual maintenance.
Database tests have to be written with as few test data as possible. In this context, database constraints do more harm than good. Disabling foreign key and not null constraints makes sure we only have to insert the data that we need for the test, nothing more.
To reduce the burden and avoid forgetting it, the disabling of constraints should be automated. Unitils' DBMaintainer will automatically disable all constraints after having performed an update to the database.
Suppose we have an application with users that have a role, are member of an organization and have multiple email addresses:
A test data set for a method that retrieves the email addresses for a user could be written as follows:
<?xml version='1.0' encoding='UTF-8'?>
<dataset>
<user id="1" username="jdoe" />
<email userid="1" emailaddress="jdoe@gmail.com" />
<email userid="1" emailaddress="jdoe@post.be" />
</dataset>
This data file contains all data we need for our unit test. If we would work with a unit test database with all constraints enabled, extra records and columns that are not relevant for the test need to be added to the data set:
<?xml version='1.0' encoding='UTF-8'?>
<dataset>
<user id="1" username="jdoe" name="Doe" roleid="1" companyid="1" />
<role id="1" rolename="admin" />
<company id="1" name="Ordina" />
<email userid="1" emailaddress="jdoe@gmail.com" />
<email userid="1" emailaddress="jdoe@post.be" />
</dataset>
The presence of constraints not only makes us spend a lot more time writing data, but also the maintenance cost increases dramatically. For example, if a new NOT NULL column is added to one of the tables, the data file must be changed, even though it has no impact on the actual test.
Unit tests for CRUD methods are very simple and all follow a similar structure. This guideline is intended for testing traditional DAO methods using a technology like JDBC. When using an O/R mapping tool like Hibernate, testing simple CRUD methods is not useful.
Unit tests for read, create, update and delete methods are very simple and share a similar structure.
Result-check data files (see Testing bulk data manipulation methods ) should not be used for checking if a create, update or delete method behaves as expected. This can be done very simply by using the interface of the DAO and the ReflectionAssert class. The examples show how these methods can be tested in a simple way.
When persistance is handled by a O/R mapping framework like Hibernate or JPA (EJB3 persistence), CRUD operations are implemented by the framework, and writing unit tests is not useful. Read Test mapping with database for more information on this topic.
Simple CRUD methods can be tested easily through the interface of the DAO. Here are for example the unit tests for the findById, create, update and delete methods of the UserDAO class:
@DataSet
public class UserDaoTest extends BaseDAOTest {
protected void setUp() throws Exception {
newUser = new User("john", "doe");
}
public void testCreate() {
userDao.create(newUser);
User toVerify = userDao.findById(user.getId());
ReflectionAssert.assertRefEqual(user, toVerify);
}
public void testUpdate() {
User user = userDao.getUser(1);
user.setName("testUpdated");
userDao.update(user);
User toVerify = userDao.findById(user.getId());
ReflectionAssert.assertRefEquals(user, toVerify);
}
public void testDelete() {
userDao.delete(existingUser);
User toVerify = userDao.findById(existingUser.getId());
assertNull(toVerify);
}
}
We have used following data file:
<?xml version='1.0' encoding='UTF-8'?>
<dataset>
<user id="1" first='jim' last="smith" />
</dataset>
A separate test for findById() is not included. The correct behavior of this method is implicitly tested by the other test methods.
SQL or HQL queries are often complex and difficult to maintain. Unit tests are therefore very important, especially for complex queries. Test data for testing queries should be very concise and only include columns on which query conditions are expressed.
Unit tests can be very effective for guaranteeing the correctness of a query, especially for complex queries. They are invaluable for making sure that changes to the database didn't break any queries.
Following steps can be performed when writing a unit test for a query:
Explanation of these steps : The complexity of a database query lies in the combination of join- and other conditions that make up the WHERE clause. To verify correctness of this WHERE clause, different test data records should be defined that are included in the result of the query, because of different reasons. A good test dataset also includes records that are not expected to appear in the result of the query. Make sure you only specify the data that is actually required for testing the query, i.e. the columns used in join and condition clauses. This will keep your datasets as small and maintainable as possible (See 3.7 Keep your test data files simple ).
The example shows a unit test for a query on a database that has following structure:
Every user belongs to exactly one company. Companies can have a number of partnerships with other companies. The DAO method under test, findPartnershipUsers , returns a List of all users that belong to a company that have some kind of partnership with the given company.
public List<User> findPartnershipUsers(Company comp) {
String query = "select u.id, u.username, u.name, u.fname, u.roleid, u.company "
"from user u, company c1, company c2, partnership p " +
"where u.companyid = c1.id " +
" and p.company1id = c1.id " +
" and p.company2id = c2.id " +
" and c2.name = ?";
return invokeQuery(query);
}
We've written following unit test:
public void testFindPartnershipUsers() {
List<User> foundUsers = userDao.findPartnershipUsers("MyCompany");
ReflectionAssert.assertPropertyLenEquals("id", Arrays.asList(2), foundUsers);
}
with this data file:
<?xml version='1.0' encoding='UTF-8'?>
<dataset>
<company id="1" name="MyCompany" />
<user id="1" companyid="1" />
<company id="2" />
<user id="2" companyid="2" />
<company id="3" />
<user id="3" companyid="3" />
<partnership company1id="1" company2id="2" />
<partnership company1id="2" company2id="3" />
</dataset>
The data file defines 3 users, the first one being member of MyCompany, the second is member of a second company, which has a partnership with MyCompany. The third user is member of company 3, which is a partner of company 2, but not a partner of MyCompany. Only the second user should be returned by the tested method, we therefore check whether a list is returned containing one user, with the id field equal to 2 .
For testing bulk insert, update or delete methods, which are cumbersome to test through the API of your DAOs, check the updated DB contents with a DBUnit result data file at the end of your test.
Database methods that perform bulk updates or deletes are often difficult to test. Verifying correctness of these methods through the interface of a DAO and with regular assert methods, is sometimes cumbersome.
Unitils' @ExpectedDataSet annotation can be used to check the result of a test by comparing the contents of the database with a result data file. Only the tables and columns that you actually want to check should be included in the data file.
Result data files are very powerful: you can check almost anything with a simple XML file. However, using them should be avoided whenever possible. You should first consider using methods of an existing DAO and assert statements to verify the results. When changing database structure, a number of data files will also have to be updated. The more data files you have, the more impact database refactorings will have.
Suppose you have a system in which users can place orders. A weekly batch job moves all orders that are older than a year to a history table. This history table cannot be queried by the application; we store the information for reporting purposes only. The batch job is implemented using a call to a stored procedure. We have following table structure:
In our application, nothing is foreseen to querying the archive tables. It is therefore a good idea to use a result data file to verify the correct behavior of this stored procedure. Our unit test looks like the following:
@ExpectedDataSet
public void testArchiveOrders() {
orderDao.archiveOrdersOlderThan(DateUtils.parseDate("21-05-2006");
}
And we have following data file:
<?xml version='1.0' encoding='UTF-8'?>
<dataset>
<order id="1" date="20-05-2006" />
<orderline id="11" orderid="1" />
<orderline id="12" orderid="1" />
<order id="2" date="21-05-2006" />
<orderline id="21" orderid="2" />
<orderline id="22" orderid="2" />
</dataset>
For verification following data set is used:
<?xml version='1.0' encoding='UTF-8'?>
<dataset>
<order id="2" />
<orderline id="21" />
<orderline id="22" />
<arch_order id= "1" date="20-05-2006" />
<arch_orderline id="11" arch_orderid="1" />
<arch_orderline id="12" arch_orderid="1" />
</dataset>
Putting test data in data files makes your unit test dependent on the database structure. Each time a change occurs to an existing database column, a number of files need to be changed. Your unit tests should encourage database refactoring, not discourage it. Therefore, you should use as few test data as possible, just enough to perform the test.
Data files should only fill columns that are involved in some kind of logic. This means filling up only the following columns:
The columns specified above are part of a query's logic. Other database fields used in the select queries are simply assigned to bean properties in the tested method. Adding them to your test datasets or assert statements provides little added value to your test, but it greatly increases the maintenance cost.
If you run into data files with too much data, try to simplify them.
Don't share data sets between different test classes. Doing so will make it very difficult to make changes to test data without breaking other tests. A good practice is to have a test data set per test class and in some cases a specific data set for a test method.
A common misconception is that time can be saved by creating one or a few general test data sets that can be reused by a large group of unit tests. This is in most cases not a good idea. The more unit tests reuse the same data set, the more brittle it becomes against changes.
Image the situation that you want to add a value for a column for testing a certain boundary condition. After doing the change, a number of existing tests appear to fail. As a reaction to this situation you will probably undo the change and try to find another way to test this boundary condition (or not test it at all). The simplest approach to avoid impacting other tests probably consists in adding new test records to the data set (although this may also break existing tests). After a while, this will result in very large and incoherent datasets.
This doesn't mean you cannot share data among tests at all, because having more test data sets also brings along more maintenance. A good practice is to use a test data set per test class and in some cases a specific data set for a method.
Refer to a DTD that represents your database in DBUnit test data files, to enable code completion while editing them.
Data files in DBUnit's FlatXMLDataSet format can be written more quickly when enabling code completion by using a DTD. If you use Unitils' database maintainer, a DTD is automatically created or updated on each developer's workstation when his unit test database is changed.
The test data file below shows how to use a DTD on the local file system.
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE dataset SYSTEM "../dtd/database.dtd">
<dataset>
<user id="1" username="jdoe" />
<email userid="1" emailaddress="jdoe@gmail.com" />
<email userid="1" emailaddress="jdoe@post.be" />
</dataset>
Don't test fields that are automatically filled up by the underlying framework or database, like generated keys or timestamps.
Databases tables generally contain columns like the id, created_by, created_on, last_updated_by, last_updated_on or version columns. They are used for identity, basic tracking (e.g. find the cause for data loss or corruption) and optimistic locking.
The implementation of tracking / optimistic locking should be handled application-wide, by the persistence mechanism and/or by DAO superclass methods. It is therefore not necessary to test it over and over again for every DAO method.
When using identity / auto-increment columns or sequences, make sure their values are high enough value before running tests. This way they don't interfere with the test data.
When a new sequence is created, it should be initialized to a sufficiently high value, to make sure auto generated ids and values don't clash with the ids and values of the test data files. When using Unitils' database maintainer for managing test database schema's, sequences and identity columns are automatically set to a configurable value.
The past few years, most projects use some form of Object-Relational mapping framework such as Hibernate or an implementation of JPA (EJB3) Persistence. The use of such a mapping tool greatly simplifies the persistence layer in an application. Errors can still exist in the mapping of the domain objects with the database or in queries written in HQL or EJB3 QL. Unit tests remain valuable to help finding these errors quickly and efficiently.
The guidelines described in the Database testing guidelines also apply to testing with Hibernate. This chapter explains the specifics when testing with Hibernate.
Unit tests for simple CRUD methods are overkill when using Hibernate. Instead, use a generic test that verifies the complete mapping with the database.
Testing simple CRUD methods explains how CRUD methods can be tested when using plain old JDBC. When using Hibernate, the correct functioning of simple CRUD depends mainly on the correct mapping of the domain model objects with the database. Here are some typical mapping problems:
Of course, these problems are usually easy to trace and to fix, but adding a unit test that verifies the complete mapping with the database at once, makes sure these problems are discovered very soon. This is a very cheap solution, since you only have to define this unit test once. Experience on projects has learnt us that this test ends up to be one of the most valuable tests in the test suite.
The Unitils project provides a test that checks the mapping of domain and database for Hibernate. The test works for both a configuration with Hibernate XML mapping files as when using annotations. For more information we refer to the tutorial .
public class HibernateMappingTest extends BaseHibernateDAOTest {
public void testMappingToDatabase() {
HibernateUnitils.assertMappingWithDatabaseConsistent();
}
}
Suppose the Person class is mapped with the PERSON column in the database using Hibernate. We add a phoneNumber property to the Person class, but forget to add this column in the database. The HibernateMappingTest now fails with the following message:
AssertionFailedError: Found mismatches between Java objects and database tables. Applying following DDL statements to the database should resolve the problem: alter table PERSON add column firstName varchar(255);
Add unit tests for methods that implement HQL queries. The same rules apply as for testing queries in plain old JDBC.
The guideline Testing queries explains how maintainable unit tests can be written for queries. The same rules apply for testing with HQL queries. The example included in this guideline is also applicable in an O/R mapping context.
Add unit tests for methods that implement bulk data manipulation. The same rules apply as for testing such methods written in plain old JDBC. Make sure the Hibernate Session is flushed before you access the test database trough plain old JDBC, without the intervenience of Hibernate.
The guideline Testing bulk data manipulation methods explains how unit tests can be written for bulk data manipulation methods. The same rules apply for testing with such methods written in HQL.
Make sure that, when comparing the database contents with a result data file, the Hibernate session is flushed. Otherwise, the update that you want to test will most probably not be executed on the database yet when you perform the check. When making use of Unitils, the Hibernate session is automatically flushed before doing the check.