Advanced features and data generators

Contents

Beyond basics

Creating test class and method

Test cases generation

Other generators

Test execution

Constraints

Expected parameters and values

Abstract choices

Labels

Global parameters

Exporting test cases to text file

Beyond basics

Let’s move to an example a little bit more resembling real life. Our example Flight Finder web application allows to query flight time table. The application can be accessed under this URL: http://ecfeed.com/samples/FlightFinder

tutorial-flight_finder_application

You can enter start and destination of the journey, fly out date and if this is a return journey also return date, and ticket class. After pressing Search flights button, a flight time table is presented. Also min and max prices of tickets are presented. Flight Finder checks also consistency of input data e.g. return date should be equal or later than Fly out date.

tutorial-flight_finder_application_results

In this tutorial we will prepare ecFeed model for the FlightFinder application. The complete running solution will be presented in
Web integration tutorial.

Creating test class and method

Create new Java project in Eclipse and name it FlightFinderTest. In the src folder create FlightFinderTest.ect file. Add new test class com.ecfeed.test.FlightFinderTest to the model. In the class add method findFlightsTest.

Goto parameters details page. We will add following parameters to our test method (creating of parameters was described in tutorial ‘create project and simple test model’):

  • airportFrom: String
  • airportTo: String
  • daysToFlyOut: String. We will not use direct value here, as it will become outdated every day. Instead we will just have a relative value that will be used to calculate the fly out date
  • isReturnFlight: boolean
  • daysBetweenFlights: int As in case of flyOutDate, the direct value will get outdated easily.
  • ticketClass: com.ecfeed.test.TicketClass This is a user type parameter.

Now we will define values for the parameters.

Add the following choices to the airportFrom parameter (e.g. Atlanta is choice name and ATL is value):

  • Atlanta[ATL]
  • New York JFK[JFK]
  • Paris Charles De Gaulle[CDG]
  • Frankfurt Am Main[FRA]
  • Delhi[DEL]
  • Tokyo Narita[NRT]

 

As parameter airportTo should have the same airports codes as airportFrom, the existing codes can be copied. To copy elements, just select them, press Ctrl+C/Ctrl+V or use right click menu. You can also drag and drop the elements with Ctrl key down (drag&drop with no key pressed will move elements).

For daysToFlyOut parameter we will have following choices:

  • today[TODAY]
  • tomorrow[TOMORROW]
  • next week[NEXT_WEEK]
  • next month[NEXT_MONTH]

 

The boolean parameter isReturnFlight will have two choices, as it is usually for booleans:

  • one way[false]
  • return[true]

 

For the journey duration we will check short, medium and long duration, so day, week and month. Apart from that we will create a special choice for one-way flight. Add the following choices to daysBetweenFlights parameter.

  • not relevant[0]
  • one day[1]
  • one week[7]
  • one month[31]

 

Finally, let’s define some ticket classes. Add the following choices to the ticketClass parameter:

  • economy[ECONOMY]
  • premium economy[PREMIUM_ECONOMY]
  • business[BUSINESS]
  • first[FIRST]

 

Click “implement” icon on methods details page to implement the class and the method.

tutorial-model-1

Note, that the enum TicketClass was implemented automatically as:

package com.ecfeed.test;

public enum TicketClass{
    ECONOMY, PREMIUM_ECONOMY, BUSINESS, FIRST
}

Test cases generation

Now we can generate test cases for the method. For the start we want to get all possible combinations of choices that we defined. Go to details page of our method and click Generate test suite button in the “Test cases” section. You will see a dialog where you can configure how the test suite will be generated. The top viewer can be used to select constraints for our generator. We do not have any constraint yet, so let’s skip it. Below constraints viewer we can choose which choices will be used by generator. By default all choices are considered and let’s keep it like this. The “Test suite” field is used to set a common name for all generated test cases. Please enter “Cartesian” in the text field. Now select “Cartesian product generator” from the Generator combo. Press OK.

tutorial-generator_dialog

The number of test cases will exceed 4000 and ecFeed will make sure if you want to add such large amount of data to the model. Big amount of elements in the model tend to slow down the application, so if it is not really necessary, it is usually better to avoid storing them. We will keep them however, for demonstration purposes.

Click OK. The generated test suite will be displayed in Test cases window.

tutorial-generate_large_test_suite

You have just generated test cases that cover all possible combinations of choices that were defined above.

The “Test cases” viewer now shows that all our test cases are executable. You can select the entire suite. Individual test cases are shown only if there are less than 500 test cases in the suite (not in this case). The Test online button in the top right corner of the method details page is enabled. Online test functionality is used if you do not need to store generated test cases, just need to execute them directly when they are generated. The dialog that appears when you click, is very similar to the one for test suite generation (one difference is that in test online dialog you do not set a name for the test suite).

Other generators

Cartesian product generator provides all combinations of the configured choices. This may result in huge number of generated test cases. One method to limit the number of test cases is to choose another generator. A generalization of cartesian product generator is “n-wise generator”. It generates such a suite that covers all possible n-tuples from our input domain.

Click Generate test suite button again. Set test suite name to “2-wise” and the select “N-wise generator” in the generator combo. The “N” parameter defines if the test suite will cover all pairs, all triples etc. If you set it to 6 (the number of parameters in findFlightTest function) you will get all combinations covered as for cartesian product generator. Set it to 2. The “coverage” parameter is used to determine how many percent of n-tuples we want to cover. For now 100 is what we want. Press OK.

The number of test cases we got is much lower than before, yet it covers all possible pairs of choices. You can check coverage for all 3-tuples and 4-tuples etc. by clicking Calculate coverage button and selecting test suites or individual test cases in which you are interested.

You can also use “Random generator” that will generate a set of random combinations. The two parameters are “length” (number of test cases we want to generate) and “duplicates”. If you leave the latter unchecked, the generator will not generate two identical test cases in one suite. Otherwise the generated test suite may contain two identical test cases. The “duplicates” parameter does not consider test cases that are already generated, so if you for example generate test cases to suite “default suite” and such suite already exists, you may get duplicates in the final suite.

The “Adaptive random generator” is generalization of random generator. For each step it generates a number of random test cases and then chooses from it the one that is furthest (in the Hamming distance sens) from the test cases generated so far. The parameter “Candidate set size” determines how many candidate test cases are generated each time. The “Depth” parameter is the number of recent test cases against which each candidate is compared in order to find the furthest one. If we set it to -1 the candidates will be compared to all test cases generated so far.

Test execution

In order to run the test, you can click method name in the left panel of model editor, select one or more of previously generated test suites and click Execute selected button. Or you can click Test online, choose a generator type and click OK button. EcFeed runs the test calling the test method with different sets of parameters.

For now the test method only prints out the values of its parameters, but in the real life it should exercise the flight search engine of the tested application.

Constraints

Constraints can be used to define dependencies between different input parameters or between input and output parameters. Precisely constraints are used to:

  • Specify what combinations of input values should be included in the generation process. In other words we can filter-out sets of input values which do not make sense for the tested method.
  • Assign a value to an expected parameter. Expected parameters are destribed in one of the following sections of this tutorial.

A constraint in ecFeed is expressed by two logical expressions: a premise and a consequence. For all test cases generated by a generator using given constraint, if the premise evaluates to logical true, the consequence must also evaluate to true, otherwise the test case is removed from the result suite.

Constraints can be used to restrict the generator not to add irrelevant test cases to generated suite. In our example when the isReturnFlight is set to false, the value of daysBetweenFlights is not relevant, so there is no sense in generating all bunch of the combinations of these two parameters. Instead we should restrict the generator in the following way.

  • The daysBetweenFlights should be not relevant in case of one way journey
  • The daysBetweenFlights should be different than not relevant in case of return journey

Click findFlightsTest method. Now click Add constraint in Constraints properties box. You just added neutral constraint “true=>true”, which evaluates to true for all possible test cases. Lets edit it to be a bit more useful.

Select the created constraint in the model viewer (or double click the constraint in the constraint viewer). This opens Constraint editor.
Set the constraint name to one way flight.
In constraint editor enter the premise: isReturnFlight = one way.
Click the consequence (the second true).
Enter the consequence: daysBetweenFlights = not relevant.

tutorial-constraint_one_way_journey

In the same way define the second constraint:

tutorial-return_flight_constraint_2

Then define the third constraint which excludes test cases where airport from is the same as airport to:

Now go back to the method details page and click Generate test suite button. In the constraint viewer leave the constraints marked. Name the suite Cartesian with constraints and generate Cartesian suite. You should get a test suite containing test cases that do not cover the restricted combinations.

Expected parameters and values

A parameter of a test function may be marked as expected. The expected parameter is provided to the test function as any other parameter, but instead stimulate the tested object it is rather used to check its output, typically using assert statements. We can define an expected value for each test case with the use of constraints, default value or assigning it manually to the test case. Expected parameters have no choices, (except of user defined types) so they will not affect number of test cases produced by generators.

We want to check if price of the single flight between airports in Europe is less than 300 Euro. To do this, let’s add a new float parameter to our test method and name it maxPrice. Go to it’s details page and check the Expected checkbox. By default our flights should not have max price, so we can define the default value to POSITIVE_INFINITY.

tutorial-model_2_expected_parameter

In case of flights inside Europe, we want however that the value of this parameter will be 300.00. We can ensure it by adding following constraint to the findFlightsTest method:

tutorial-constraint_max_price_in_europe

Abstract choices

The constraint above looks a bit complicated and would get much worse if there were more airports available in Europe (you would have to constrain each combination). Luckily choices can be combined into groups in order to simplify constraints. Let us group airports according to the continent where they a localized.

Add new choices to the airportFrom parameters: Europe[], America[], Asia[]. The actual values of these choices are not important in this case.
Select the two European airports and move them to the Europe (by drag&drop or by Cut&Paste with Ctrl-X, Ctrl-V or context menu). Do the same for other airports. The result should look like on the picture below.

tutorial-airports_abstract_choices

The same set of choices must be defined in the airportTo parameter. To do that let’s delete all its choices and copy&paste them from airportFrom
Once the choices of both parameters are organized in hierarchy, we can rewrite the maxPrice constraint as follows:

tutorial-constraint_max_price_in_europe_2

It is easy now to add European airports without the modification of constraint (although this will have impact on the number of generated test cases).

Labels

Labels have similar functions as abstract choices. You can add arbitrary amount of labels to choices and then use these labels in constraints.

Let’s assume that Premium Economy and First ticket classes are available only on flights to America and (why not?) to Paris. Generating test cases with these classes on other connections has no sense. Since these destinations are already in different branches and levels of choices hierarchy, it is not possible (or at least not easy) to reorganize the hierarchy in such a way that this constraint can be easily modeled. In such situations labels come in use.

Go to details page of America choice of airportTo parameter and click Add label button. The label has been added. Change its value of the label to premium. Note that it was inherited to all choices that are below America. Add identical label to the Paris – Charles De Gaule choice.

Similarly, for the ticket price we can add the label premium to choices premium economy and first of the parameter ticketClass. This will simplify the constraint:

tutorial-labeled_constraint

Generate test suite. The number of generated test cases should significantly decrease.

Global parameters

Global parameters are a cool feature very useful if you have many functions that depend on similar arguments. Parameters airportFrom and airportTo represent the same domain of values. You should already notice that it is annoying and cumbersome to maintain the parameters in sync, when we edited the hierarchy or set of labels. To facilitate that, we can create one common (global) parameter airport and link parameters the airportFrom and airportTo with it.

Global parameters may have entire model or only single class in scope. They can be created anew, but in our case it will be easier to modify the existing parameter to global. This way we will preserve the test cases and constraints that refer to the parameter we convert.

Select airportTo parameter and drag&drop it to the root node of the model with Ctrl and Shift keys pressed down (the actual keys may differ on other operating systems – we use Ubuntu). Rename the parameter to airport.

tutorial-global_parameter_airport

The airportFrom is now linking to global parameter definition. We now should link the airportTo parameter. The definition is already existing, so we don’t need to create it again. Instead we can just go to airportTo details page and check the Linked checkbox. If there were more than one global parameters of this type available, we would need to select one of them as the link target. Note that once we linked the airportTo parameter, we lost all test cases and those constraints that made use of choices from the original airportFrom parameter. That is because the airportTo now use completely distinct set of choices, although they have by coincidence the same names and values.

tutorial-linked_parameter

Now, if we want to add a new airport to our model, we do it in one place and both parameters that link to global definition will automatically use it.

Exporting test cases to text file

EcFeed allows to export test cases to text files in custom format. It is possible to export selected test cases that are added to the model, or export test cases directly from the generator, without storing them in the model (“online export”). Let’s check the latter method. Click Export online button on method property page. An Export online setup dialog will be displayed.

tutorial-export_online_dialog

Export online setup dialog contains fields which allow to select generator, method parameters and constraints. The dialog contains fields similar to the ones in the test cases generation dialog. Apart from these fields there is a field for selection of Export target file – the file which will be created during export.

Three export templates are available: CSV, XML and Gherkin.

As default ecFeed generates a csv file structure using all parameters of the test method. It places parameter names in the first row (header) and parameter values for generated test cases in successive rows. If there is a need to export data in a different way, click Advanced button. EcFeed will display template definition dialog.

The template consists of three sections: [Header], [TestCase] [Footer]. Section [TestCase] is obligatory, while sections [Header] and [Footer] can be removed.

In [Header] section you can define how the top of the exported file will look like. You can use the following expressions: %package – name of the package where method is located %class – name of the test class containing test method %method – name of the method containing exported test cases $N.name – name of the Nth parameter (counting from 1) in the test method. ecFeed will place the template in the beginning of the exported file, replacing the special expressions with specific values from the model.

In [TestCase] section you can define fields to export in each of the test cases. You can use all the expressions used for [Header] section, and also: $N.choice – short name of the choice that will be used in the test case as Nth parameter $N.full_choice – qualified name of the choice used in the test case as Nth parameter, containing all of it’s parent choices in the hierarchy. $N.value – value of the Nth parameter of this test case %suite – test suite name of the test case. %index – a number under which this test case was exported (starting from 0). Using notation $(expression) you can define simple arithmetical expressions that will be resolved during export. In the expression you can use operators like +, -, * and /. You can also group expressions using parentheses (). For example $((%index + 1)*2) will be evaluated for each test case with the incremented index of the test case multiplied by two.

Section [Footer] defines the bottom of the exported file. You can use all the expressions used for [Header] section.

Clicking Save As… you can save template on disk. Template can be read from disk after clicking Load… button.

When export target file is selected, click OK. EcFeed will generate test cases and export them to the text file without storing in the model.

It is also possible to export selected test cases from the model. To do so, select some test cases in the Test Cases section of the findFlightsTest and click Export selected button. EcFeed displays a dialog where you can define export template and select target file.

tutorial-export_selected_test_cases

After selecting the Target file and clicking OK button, the test cases are exported to this file.