Test runner

The test runner is responsible for running tests defined in a test class. Runner of a test class is defined by @RunWith(Runner.class) annotation, where Runner.class is a runner class. Runners provided by ecFeed are working with equivalence class models. The path to a model is provided by ‘@EcModel(“path/to/model/file.ect”)’ annotation. Both @RunWith and @EcModel annotations must be provided to a test class. The path may be either absolute (eg. "/home/user/workspace/project/test/package/model.ect") or local (e.g. "test/package/model.ect")

Custom runners provided by ecFeed are responsible for executing test cases defined by functions annotated with @Test that have parameters. Functions with no parameters are executed by standard jUnit runner.

Static runner

Static runner is responsible for running test cases provided in a model. It is defined in StaticRunner class.

  • By default static runner invokes all test cases that are defined for a test function in a model
  • Invoked test functions may be restricted by defining which test suites are executed
    • Required test suites are defined by a ‘@TestSuites` annotation
    • The annotation takes an argument which is array of strings defining names of test suites to execute, e.g.: @TestSuites({"Test suite 1", "Test suite 2"})
    • @TestSuites annotation may be used on a class or a test method. In the latter case the defined test suites overwrite globally defined test suites

Example:

package com.example.test;

import org.junit.Test;
import org.junit.runner.RunWith;

import com.testify.ecfeed.runner.StaticRunner;
import com.testify.ecfeed.runner.annotations.EcModel;
import com.testify.ecfeed.runner.annotations.TestSuites;

@RunWith(StaticRunner.class)
@EcModel("src/model.ect")
@TestSuites({"suite 1", "suite 2"})
public class TestClass {
  @Test 
  public void testFunction1(int argInt, String argString){
  // test cases belonging to "suite 1" and "suite 2" will be executed
  }

  @Test 
  @TestSuites({"suite 1"})
  public void testFunction2(int argInt, String argString){
  // only "suite 1" will be executed
  }

  @Test
  @TestSuites({"suite 1", "suite 3"})
  public void testFunction3(int arg1, int arg2, int arg3){
  // test cases belonging to "suite 1" and "suite 3" will be executed. 
  }
}

Online runner

The online runner is generating test cases on the fly, during execution. The advantage is that the test cases do not have to be stored in the model. On the other hand this makes them impossible to edit. The online runner is defined by the OnlineRunner class.

  • To invoke test functions with parameters generated on the fly, the runner must be provided with data generator
    • data generator is defined by @Generator(Generator.class) annotation, where Generator.class is a class defining data generator that implements IGenerator interface.
    • the @Generator annotation may be used on a class or on a method. Generator annotations used on methods overwrite global (class wide) generators.
    • Each Generator class requires different parameters set (to find out what are parameters required for each generator see generators documentation in Generators documentation).
      • generators parameters are provided either by @GeneratorParameter(name="name", value="value") annotation, or by pair of annotations @GeneratorParameterNames({"par1", "par2", "par3"}) and @GeneratorParameterValues({"val3", "val2", "val3"}).
      • @GeneratorParameter annotation may be used only if the generator requires exactly one parameter.
      • The size of arrays provided to @GeneratorParameterNames and @GeneratorParameterValues must be the same
      • Parameter value is provided as a string, but must be parseable to a type defined by parameter.
      • Generator parameters (provided in either way) defined for a method overwrite global parameters.
    • Generators may be configured to use constraints provided in the model.
      • Constraints are provided with @Constraints({“name1”, “name2”}) annotation for test method or globally test class
      • If no @Constraints annotation is used, no constraints will be used by default.
      • Locally defined constraints overwrite class wide definition
      • Two special values may be used in @Constraints annotation: @Constraints(Constraints.ALL) and @Constraints(Constraints.NONE), meaning respectively that all or none of constraints defined for test method will be used by generator

Example:

package com.example.test;

import org.junit.Test;
import org.junit.runner.RunWith;

import com.testify.ecfeed.runner.OnlineRunner;
import com.testify.ecfeed.runner.annotations.EcModel;
import com.testify.ecfeed.runner.annotations.Generator;
import com.testify.ecfeed.runner.annotations.GeneratorParameter;
import com.testify.ecfeed.runner.annotations.GeneratorParameterNames;
import com.testify.ecfeed.runner.annotations.GeneratorParameterValues;
import com.testify.generators.CartesianProductGenerator;
import com.testify.generators.NWiseGenerator;
import com.testify.generators.RandomGenerator;

@RunWith(OnlineRunner.class)
@EcModel("src/model.ect")
@Generator(RandomGenerator.class)
@GeneratorParameterNames({"Test suite size", "Duplicates"})
@GeneratorParameterValues({"2000", "true"})
@Constraints(Constraints.NONE)  //by default no constraints will be used by generators
public class SomeTestClass {

  @Test
  @Constraints(Constraints.ALL)
  public void testFunction1(int arg1, int arg2) {
  // this method will be executed with all test cases provided with generator
  // defined globally for the class and with globally defined parameters.
  // the generator will use all constraints defined in the model for this function
  }

  @Test
  @GeneratorParameterNames({"Test suite size", "Duplicates"})
  @GeneratorParameterValues({"10", "false"})
  public void testFunction2(int arg1, int arg2) {
  // this method will be executed with all test cases provided with generator
  // defined globally for the class, but using parameters defined explicitly for this 
  // function. Because no constraints are indicated for this function, the generator 
  // will use globally defined constraints (Constraints.NONE)
  }

  @Test
  @Generator(CartesianProductGenerator.class)
  @Constraints("constraint")
  public void testFunction3(int arg1, int arg2) {
  // this method will be executed with all test cases provided with generator
  // defined locally for the method. The generator will consider all constraints
  // defined for this function in the model that have name "constraint"
  }

  @Test
  @Generator(NWiseGenerator.class)
  @GeneratorParameter(name="N", value="2")
  public void testFunction4(int arg1, int arg2) {
  // this method will be executed with all test cases provided with generator
  // defined locally for the method. Parameters for this generator must be 
  // provided locally. Because generator needs only one parameter, it may be 
  // provided by @GeneratorParameter annotation. Global value for constraints is 
  // used by generator.
  }
}