JUnit-4.x API

When a JUnit4-style test-class is annotated with ...

@RunWith(CallbackParamsRunner.class)
public class MyJUnit4ParameterizedTest {
  ...

... the CallbackParamsRunner instance will enforce the parameterization functionality through byte-code modifications and then use a callback-records factory to combine the callback-records.

Thereafter a suitable Runner class will be chosen to take care of the actual test-execution. For JUnit-4.5+ JUnit4 will be chosen. The chosen runner class gets to do its thing on the modified test-class once for each callback-record.

The chosen runner will of course know nothing about the parameterization, which has been enforced through byte-code modifications in this manner:

• @Before, @Test and @After methods that accept interface arguments have been substituted by equivalently annotated proxy-methods, which implementations invoke the original methods after first picking the proper callback arguments.
• Fields of interface-type that are annotated with @ParameterizedCallback will be initialized with the proper callbacks before(!) the test-class initializers and constructor are executed.
• Fields that are annotated with @ParameterizedValue will be initialized with a combined parameter-value.
  The field-type itself specifies the available parameter-values with a static method values() that is expected to return an array with the available parameter-values. This is why an enum-class would be convenient because it will have an implicit method values() that returns all of its enum-values. There will be an error if a non-enum field-type does not have a static values() method to provide the parameter-values! The exceptions are fields of types boolean or Boolean, which obviously have true and false as their available parameter-values.

For more educational examples of this functionality - please consider the tutorial articles.

JUnit-3.x API

CallbackParamsRunner can also be used for a JUnit-3.x style test-class:

/* TestCase subclass ... */

@RunWith(CallbackParamsRunner.class)
public class MyJUnit3ParameterizedTest extends TestCase {
  ...
}

/* ... or suite-style tests: */

@RunWith(CallbackParamsRunner.class)
public class MyJUnit3SuiteParameterizedTest {

  public static Test suite() {
    ...
  }
  ...
}

When CallbackParamsRunner encounters a JUnit3-style test-class it acts pretty much as for the JUnit4-style test but with a few differences:

• The modified test-class is handed to one of JUnit's JUnit-3.x runners, i.e. for JUnit-4.5+ JUnit38ClassRunner (for TestCase tests) or AllTests (for suite-style tests) will be used.
• The methods to be proxied are the public and void methods that take interface arguments and has names that starts with "test". (This means that the equivalents of @Before and @After methods, i.e. setUp() and tearDown() cannot take callback-arguments. - Use @ParameterizedCallback-annotated fields instead!)

Please note that it is the test-class that will have its byte-code modified. - I.e. for the suite-style test-example above it is the class MyJUnit3SuiteParameterizedTest that will have its byte-code modified, regardless of the nature of whatever is returned by its suite()-method. This will not make much sense when using this pattern to create a simple test-suite. However, it makes good sense when using PowerMockSuite from the Powermock framework or similarly implemented 3rd-party frameworks.

@WrappedRunner

By using the annotation @WrappedRunner it is possible to specify an arbitrary 3rd-party Runner, to which the modified test-class will be handed for test-execution:

@RunWith(CallbackParamsRunner.class)
@WrappedRunner(PowerMockRunner.class)
@PrepareForTest(ClassToBeTested.class)
public class MyParameterizedPowermockTest {
  ...
}

@Runwith(CallbackParamsRunner.class)
@WrappedRunner(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class MyParameterizedSpringContextTest {
  ...
}

Above are two examples of the power of @WrappedRunner. MyParameterizedPowermockTest works as a parameterized Powermock test and MyParameterizedSpringContextTest works as a parameterized test that also benenfits from all the Spring Framework features that SpringJUnit4ClassRunner can offer. - And that includes the annotation-based transaction support ...

CallbackTestSuite

Providing the JUnit-3.x style test-class with a suite-method that returns a CallbackTestSuite-instance of the test-class is the direct equivalent of annotating it with @RunWith(CallbackParams.class) ...

public class MyJUnit3ParameterizedTest extends TestCase {

  public static Test suite() {
    return new CallbackTestSuite(MyJUnit3ParameterizedTest.class);
  }
  ...
}

... with the previously mentioned difference that all (non-static, non-final) fields of interface-type will be initialized with a callback, regardless of whether annotated with @ParameterizedCallback or not.

CallbackTestCase

By having the test-class extend CallbackTestCase the test-class will make sure to create a byte-code modified version of itself as described above and then have the execution continue with instances of the modified class.

public class MyJUnit3ParameterizedTest extends CallbackTestCase {
  ...

Unlike CallbackTestSuite there are situations where CallbackTestCase provides an API that is desirable even if JUnit-4.x is used, since it does offer some special API that can be used to hamper with the combine machinary in a non-static context.

Having a test-class extend CallbackTestCase does not disqualify it from being annotated with @RunWith(CallbackParamsRunner.class):

@RunWith(CallbackParamsRunner.class)
public class MyJUnit3ParameterizedTest extends CallbackTestCase {
  ...

The only diffence from the non-annotated case is that only the @ParameterizedCallback-annotated fields will be initialized with callbacks.

Combine Strategy

Technically, a combine strategy is simply an implementation of the interface CombineStrategy. The resulting callback-records are returned by the method combine(...), which takes an argument of class ValuesCollection. A ValuesCollection instance is a list of object arrays. A typical array is the array that is retrieved by the static method values() (which is implicitly available on every enum-class).

Each one of the callback-records retrieved from combine(...) is expected (but not obliged) to contain exactly one element from each one of the arrays available in the argument ValuesCollection - but in reality a combine-strategy is free to produce callback-records any way it likes, possibly even ignoring whatever parameter-values are available in the argument ValuesCollection.

@CombineConfig

The recommended way to specify a non-default combine-strategy for a CallbackParams test-class is to annotate it with @CombineConfig:

@RunWith(CallbackParamsRunner.class)
@CombineConfig(strategy=MyCombineStrategy.class)

The mandatory annotation property strategy is set to the CombineStrategy implementation class to use. The annotation also makes it possible to set two additional properties maxCount and randomSeed, which will be passed to their respective setters on the combine-strategy instance.

@CallbackRecords

The annotation @CallbackRecords is to CallbackParams what @Parameterized.Parameters is to paremeterized tests in JUnit. I.e. it makes it possible to have a static method of the test-class compose callback-records for the testrun.

Please note that manual composing of callback-record is generally not recommended when using the CallbackParams framework. The main reason for offering this piece of API is to simplify for developers that wish to closely investigate the test execution for a certain callback-record.

@RunWith(CallbackParams.class)
// @CombineConfig(strategy=SomeCallbackStrategy.class)
public class MyTest {

  @CallbackRecords
  public static java.util.Collection tmpSingleRecordForDebugPurpose() {
    return java.util.Arrays.asList(new Object[][] {{
      MyEnum1.VALUE_A, MyEnum2.VALUE_B
    }};
  }
  ...

For the above test-class the developer has out-commented the @CombineConfig annotation in order to instead explicitly specify a certain callback-record by using a @CallbackRecords annotation.
The purpose is probably to debug the test execution for the specified callback-record, fix a specific problem and then remove the tmpSingleRecordForDebugPurpose() method and reactivate the @CombineConfig annotation.

JDK-1.4 (CallbackRecordsFactory)

When using JDK-1.4 it is not possible to benefit from any API that relies on annotations. I.e. the annotation @CombineConfig cannot be used for specifying the combine-strategy. Fortunatelly the JUnit-3.x API of CallbackParams offers other means for this.

The most flexible JDK-1.4 compatible alternative to the above annotations is to specify a CallbackRecordsFactory. When subclassing CallbackTestCase this is done by overriding the method getCallbackRecordsFactory ...

public class MyJUnit3ParameterizedTest extends CallbackTestCase {

  protected CallbackRecordsFactory getCallbackRecordsFactory() {

    return new CallbackRecordsFactory() {
  
      /**
       * This overridden method-implementation is the JDK-1.4 equivalent
       * for annotating the test-class with
       * @CombineConfig(strategy=CombineCompletely.class, maxCount=500)
       */
      public CombineStrategy retrieveCombineStrategy(Class testClass) {
        CombineStrategy strategy = new CombineCompletely();
        strategy.setMaxCount(500);
        return strategy;
      }
    };
  }
  ...
}

... and when using CallbackTestSuite it is possible to leverage from a certain constructor that takes your choice of CallbackRecordsFactory as the second argument:

public class MyJUnit3ParameterizedTest extends TestCase {

  public static Test suite() {
    return new CallbackTestSuite(MyJUnit3ParameterizedTest.class,
        new CallbackRecordsFactory() {
  
      /**
       * This overridden method-implementation is the JDK-1.4 equivalent
       * for annotating the test-class with
       * @CombineConfig(strategy=CombineCompletely.class, maxCount=500)
       */
      public CombineStrategy retrieveCombineStrategy(Class testClass) {
        CombineStrategy strategy = new CombineCompletely();
        strategy.setMaxCount(500);
        return strategy;
      }
    });
  }
  ...
}

Sandbox Example: Property

The sandbox-class Property and its nested resources have been developed as an example on how annotation-specified callbacks could be used. In this case it is the nested annotation @Property.Value that specifies the sibling class Property.BasicImpl as callback implementation.

Please note how Property.BasicImpl is an example of how both the callback interface (Property.Callback) and CallbackFactory is implemented. Whether a callback can be arranged depends on whether the method getCallback() decides to return itself or not!

Parameterized Rules

The folding mechanism will apply to the apply-methods of the JUnit Rules interfaces. This opens a door for callback parameters to specify their own set-up and tear-down functionality without involving any @Before- or @After-method in the test-class. It is done by declaring a public callback-parameterized field of an interface-type for JUnit rules and annotate it with the JUnit @Rule annotation.

JUnit-rules interfaces come in two different flavors ...

/**
 * The test-class uses this @Rule declaration
 * to properly fold all three TestRule callbacks.
 */
@Rule @ParameterizedCallback
public TestRule ruleCallbacks;

/**************************************************************************
 * Pattern I: Expose ExternalResource by implementing CallbackFactory
 *
 * The enum has a field beforeAndAfter for holding its ExternalResource
 * instance - and has the implementation of CallbackFactory#getCallback()
 * offer it as a callback.
 *
 * pro: Simple design that is easy to understand
 * con: Implementation of CallbackFactory#getCallback() is now occupied with this
 *      task and is not easily reused for exposing any other indirect callback.
 */
enum RuleFromCallbackFactory implements CallbackFactory {
  VALUE_1("value 1"),
  VALUE_2("value 2");

  String value;

  RuleFromCallbackFactory(String value) {this.value = value;}

  TestRule beforeAndAfter = new ExternalResource() {
    @Override protected void before() { /* Set-up code here */ }
    @Override protected void after() { /* Tear-down code here */ }
  };
  
  public Object getCallback() {
    return beforeAndAfter;
  }
}/* End of Pattern I ******************************************************/

/**************************************************************************
 * Pattern II: Apply ExternalResource by implementing TestRule directly
 *
 * This enum also has the beforeAndAfter field for holding its
 * ExternalResource instance but this time the implementation
 * of TestRule#apply(...) wraps the applying of its ExternalResource.
 *
 * pro: Does not occupy CallbackFactory#getCallback(), which is now freely
 *      available for other needs
 * con: Not as simple as "Pattern I" - developers, who are not necessarily
 *      familiar with the inner working of JUnit rules, are perhaps
 *      uncomfortable with this kind of code
 */
enum ImplementTestRule implements TestRule {
  VALUE_3("value 3"),
  VALUE_4("value 4");

  String value;

  ImplementTestRule(String value) {this.value = value;}

  TestRule beforeAndAfter = new ExternalResource() {
    @Override protected void before() { /* Set-up code here */ }
    @Override protected void after() { /* Tear-down code here */ }
  };

  public Statement apply(Statement base, Description description) {
    return beforeAndAfter.apply(base, description)
  }
}/* End of Pattern II *****************************************************/

/**************************************************************************
 * Pattern III: Abstract Wrapping extends ExternalResource
 * 
 * Fact is that the internals of CallbackParams do not actually know what
 * an enum is. What the internals actually do is to collect whichever
 * parameter values that are returned from a static values()-method, which
 * the Java-compiler implicitly adds to every enum-class!
 * The API interface Wrapping and WrappingSupport are designed to also
 * work as a powerful utility options for situations when the parameter
 * wraps primitive singleton data (in this case String-values).
 * By trusting the method WrappingSupport.html#wrap(...) to construct the
 * concrete Wrapping instances it is possible to have your parameter-type
 * extend another class (in this case ExternalResource) - whereas an enum-
 * class cannot extend any other class.
 *
 * pro: - by extending ExternalResource the before()- and after()-methods
 *        can be specified in the parameter-class directly - no need to wrap
 *        them inside any nested class
 *      - no constructor needed - the method WrappingSupport#wrap(...) makes
 *        sure the wrapped value is reached through the method wrappedValue()
 *      - less code
 * con: not an enum ...
 *      - the a static values()-method must be explicitly defined
 *      - a single parameter-value cannot easily override the default
 *        behaviour of a callback-method
 */
static abstract class WrappingExtendsExternalResource
extends ExternalResource implements Wrapping<String> {

  static WrappingExtendsExternalResource[] values() {
    return WrappingSupport.wrap(WrappingExtendsExternalResource.class,
        "value 5",
        "value 6");
  }

  String value = wrappedValue();

  @Override protected void before() { /* Set-up code here */ }
  @Override protected void after() { /* Tear-down code here */ }
}/* End of Pattern III ****************************************************/

public abstract class TargetAwareResource implements MethodRule {

  public Statement apply(final Statement base,
                         FrameworkMethod testMethodWhichIsIgnoredHere,
                         final Object target) {
    return new Statement() {
      @Override public void evaluate() throws Throwable {
        before(target);
        try {
          base.evaluate();
        } finally {
          after(target);
        }
      }
    };
  }

  /** Override to specify set-up code */
  protected void before(Object targetTest) throws Throwable {}

  /** Override to specify tear-down code */
  protected void after(Object targetTest) {}
}

/**
 * The test-class uses this @Rule declaration
 * to properly fold all three TestRule MethodRule callbacks.
 */
@Rule @ParameterizedCallback
public TestRule MethodRule ruleCallbacks;

/**************************************************************************
 * Pattern I: Expose TargetAwareResource by implementing CallbackFactory
 * ...
 */
enum RuleFromCallbackFactory implements CallbackFactory {
  VALUE_1("value 1"),
  VALUE_2("value 2");

  String value;

  RuleFromCallbackFactory(String value) {this.value = value;}

  TestRule beforeAndAfter = new ExternalResource() {
  MethodRule beforeAndAfter = new TargetAwareResource() {
    @Override protected void before(Object targetTest) { /* Set-up code here */ }
    @Override protected void after(Object targetTest) { /* Tear-down code here */ }
  };

  public Object getCallback() {
    return beforeAndAfter;
  }
}/* End of Pattern I ******************************************************/

/**************************************************************************
 * Pattern II: Apply TargetAwareResource by implementing TestRule MethodRule directly
 * ...
 */
enum ImplementTestRule implements TestRule MethodRule {
  VALUE_3("value 3"),
  VALUE_4("value 4");

  String value;

  ImplementTestRule(String value) {this.value = value;}

  TestRule beforeAndAfter = new ExternalResource() {
  MethodRule beforeAndAfter = new TargetAwareResource() {
    @Override protected void before(Object targetTest) { /* Set-up code here */ }
    @Override protected void after(Object targetTest) { /* Tear-down code here */ }
  };

  public Statement apply(Statement base, Description description) {
  public Statement apply(Statement base, FrameworkMethod method, Object target) {
    return beforeAndAfter.apply(base, description method, target)
  }
}/* End of Pattern II *****************************************************/

/**************************************************************************
 * Pattern III: Abstract Wrapping extends TargetAwareResource
 * ...
 */
static abstract class WrappingExtendsExternalResource
extends ExternalResource TargetAwareResource implements Wrapping<String> {

  static WrappingExtendsExternalResource[] values() {
    return WrappingSupport.wrap(WrappingExtendsExternalResource.class,
        "value 5",
        "value 6");
  }

  String value = wrappedValue();

  @Override protected void before(Object targetTest) { /* Set-up code here */ }
  @Override protected void after(Object targetTest) { /* Tear-down code here */ }
}/* End of Pattern III ****************************************************/

getCurrentCallbackRecord()

The method getCurrentCallbackRecord() offers immediate access to the current callback-record:

public class TestThatSortsTheValuesOfTheCallbackRecord {

  /**
   * The built-in combine-strategies do not sort the values of the
   * callback-records in any particular order. That is here taken care of by
   * having an initializer access the current callback-record through
   * CallbackControlPanel and sort the values in the callback-record.
   * Please notice how the use of an initializer can take care of this before
   * the constructor and "@Before"-methods get going ...
   */
  @ParameterizedCallback CallbackControlPanel panel;
  {
    Collections.sort(panel.getCurrentCallbackRecord(), new SomeComparator());
  }
...

If the test-developer needs to sort the values of the callback-record then the above pattern can be used

To use CallbackControlPanel as a mean to modify the callback-record is generally not recommended, however. It is very likely to lead to a number of undefined situations when future features are implemented.

getLatestCallbackResults()

CallbackControlPanel is currently the only way to access the return values in case there are some non-void callback-methods:

  @Test
  public void testMethod(MyCallback callback, CallbackControlPanel panel) {
    callback.isValidationFailureExpected();
    if (panel.getLatestCallbackResults().values().contains(Boolean.TRUE)) {
      testValidationFailure(callback);
    } else {
      testHappyPath(callback);
    }
  }

In the example above the boolean return-values of the callback-method "isValidationFailureExpected()" are investigated to determine whether any of them force the testrun to follow a validation-failure expecting path (instead of the happy path).
The above is a reasonable example on how to access return values of callback-methods but please note that the tuturial article Validation of Validation Failures shows how the framework offers much better patterns for testing validation failures.