TestNG for starters : running a same test method several times with different parameter values

When using TestNg to write unit tests, one of the most common mistake I see people make is writing a test method in which they make several assertions in order to test giving different values as input to the method to be tested.

To illustrate this common mistake, let's assume we have a class named StringConcatener, which provides a static method to concatenate 2 given String objects :

public class StringConcatener {
 
 public static String concatStrings(String s1, String s2){
  if(s1 == null){
   s1 = "null";
  }
  
  return s1 + s2;
 }
}

Now, we are going to write a test class in which we will implement a method that will check the behavior of our concatStrings( ) method against different input values. To do this, new TestNg users often tend to stuff lots of assertions inside a same test method, just like this : 

public class TestStringConcatener {

 @Test
 public void testConcatener() {
  Assert.assertEquals(StringConcatener.concatStrings("null", "null"), "nullnull");
  Assert.assertEquals(StringConcatener.concatStrings(null, null), "nullnull");
  Assert.assertEquals(StringConcatener.concatStrings("StringA", "StringB"), "StringAStringB");
  Assert.assertEquals(StringConcatener.concatStrings("StringA", null), "StringAnull");
  // and many other assertions...
 }
}

This test method works fine but only if every assertions get the expected result. Suppose the 1st assertion does not get the expected result. Then, none of the assertions that follows it will be tested. That, obviously, is not how you want your test class to behave.

Solution

1. Write a test method with 3 arguments : the 2 String objects to be concatenated + the expected result
2. Marked this test method with @Test(dataProvider="EqualsAssertionsProvider") : this method is a test method which expects input from the DataProvider named "EqualsAssertionsProvider"
3. Write the method that will provide the set of input values to the test method, and mark it with @DataProvider(name="EqualsAssertionsProvider"). The name attribute in the @DataProvider annotation specifies its name. 

public class TestStringConcatener {
 
 @DataProvider(name="EqualsAssertionsProvider")
 public Object[][] createConcatenationAssertionSet(){
  return new Object[][]{
    new Object []{"null", "null", "nullnull"},
    new Object []{null, null, "nullnull"},
    new Object []{"StringA", "StringB", "StringAStringB"},
    new Object []{"StringA", null, "StringAnull"},
  };
 }
 
 @Test(dataProvider="EqualsAssertionsProvider")
 public void testConcatener(String s1, String s2, String expectedResult){
  Assert.assertEquals(StringConcatener.concatStrings(s1, s2), expectedResult);
 }
}

As you can see, in the example above, the createConcatenationAssertionSet( ) method returns a 2 dimensional Object array :

• the first dimension determines how many input sets the DataProvider holds (that is, how many times our test method will run)
• the second dimension holds Object arrays that contains the values for each argument from our test method. As there are 3 arguments in our test method, each of these Object arrays contains 3 values.

Now that you're working with a DataProvider, should one of your assertions fail, the following ones will still be tested. To illustrate that, let's modify the expected result of the third input set from our DataProvider : 

new Object []{"StringA", "StringB", "StringAStringB"}

becomes

new Object []{"StringA", "StringB", "StringAStringX"}

Let's run our TestClass and see what are the results :

As you can see, the third assertions failed (the exact reason is shown on the right column), but the 4th assertions has still been checked. Voilà!

Look out!

As strange as it may look, DataProvider methods can only have 2 type of returns : Object[ ][ ] or Iterator<Object>[ ]. Maybe that would change in the future but at the time I wrote this post, I was using the latest available version of TestNg (6.3.1)