elasticsearch/docs/reference/testing/testing-framework.asciidoc

[[testing-framework]]
== Java Testing Framework

added[1.0.0]

[[testing-intro]]

Testing is a crucial part of your application, and as information retrieval itself is already a complex topic, there should not be any additional complexity in setting up a testing infrastructure, which uses elasticsearch. This is the main reason why we decided to release an additional file to the release, which allows you to use the same testing infrastructure we do in the elasticsearch core. The testing framework allows you to setup clusters with multiple nodes in order to check if your code covers everything needed to run in a cluster. The framework prevents you from writing complex code yourself to start, stop or manage several test nodes in a cluster. In addition there is another very important feature called randomized testing, which you are getting for free as it is part of the elasticsearch infrastructure.



[[why-randomized-testing]]
=== why randomized testing?

The key concept of randomized testing is not to use the same input values for every testcase, but still be able to reproduce it in case of a failure. This allows to test with vastly different input variables in order to make sure, that your implementation is actually independent from your provided test data.

If you are interested in the implementation being used, check out the http://labs.carrotsearch.com/randomizedtesting.html[RandomizedTesting webpage].


[[using-elasticsearch-test-classes]]
=== Using the elasticsearch test classes

First, you need to include the testing dependency in your project. If you use maven and its `pom.xml` file, it looks like this

[[source,xml]]
--------------------------------------------------
<dependencies>
  <dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-test-framework</artifactId>
    <version>${lucene.version}</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>${elasticsearch.version}</version>
    <scope>test</scope>
    <type>test-jar</type>
  </dependency>
  <dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>${elasticsearch.version}</version>
    <scope>test</scope>
  </dependency>
</dependencies>
--------------------------------------------------

Replace the elasticsearch version and the lucene versions with the current elasticsearch version and its accompanying lucene release. 

There are already have a couple of classes, you can inherit from in your own test classes. The advantages of doing so is having already defined loggers, the whole randomized infrastructure is set up already.


[[unit-tests]]
=== unit tests

In case you only need to execute a unit test, because your implementation can be isolated that good and does not require an up and running elasticsearch cluster, you can use the `ElasticsearchTestCase`. If you are testing lucene features, use `ElasticsearchLuceneTestCase` and if you are testing concrete token streams, use the `ElasticsearchTokenStreamTestCase` class. Those specific classes execute additional checks, which ensure that no resources leaks are happening, after the test has run.


[[integration-tests]]
=== integration tests

These kind of tests require firing up a whole cluster of nodes, before the tests can actually be run. Compared to unit tests they are obviously way more time consuming, but the test infrastructure tries to minimize the time cost by only restarting the whole cluster, if this is configured explicitely.

The class your tests have to inherit from is `ElasticsearchIntegrationTest`. As soon as you inherit, there is no need for you to start any elasticsearch nodes manually in your test anymore, though you might need to ensure that at least a certain amount of nodes is up running.

[[helper-methods]]
==== generic helper methods

There are a couple of helper methods in `ElasticsearchIntegrationTest`, which will make your tests shorter and more concise.

[horizontal]
`refresh()`::           Refreshes all indices in a cluster
`ensureGreen()`::       Ensures a green health cluster state, waiting for relocations. Waits the default timeout of 30 seconds before failing.
`ensureYellow()`::      Ensures a yellow health cluster state, also waits for 30 seconds before failing.
`createIndex(name)`::   Creates an index with the specified name
`flush()`::             Flushes all indices in a cluster
`flushAndRefresh()`::   Combines `flush()` and `refresh()` calls
`optimize()`::          Waits for all relocations and optimized all indices in the cluster to one segment.
`indexExists(name)`::   Checks if given index exists
`admin()`::             Returns an `AdminClient` for administrative tasks
`clusterService()`::    Returns the cluster service java class
`cluster()`::           Returns the test cluster class, which is explained in the next paragraphs


[[test-cluster-methods]]
==== test cluster methods

The `TestCluster` class is the heart of the cluster functionality in a randomized test and allows you to configure a specific setting or replay certain types of outages to check, how your custom code reacts.

[horizontal]
`ensureAtLeastNumNodes(n)`::        Ensure at least the specified number of nodes is running in the cluster
`ensureAtMostNumNodes(n)`::         Ensure at most the specified number of nodes is running in the cluster
`getInstance()`::                   Get a guice instantiated instance of a class from a random node
`getInstanceFromNode()`::           Get a guice instantiated instance of a class from a specified node
`stopRandomNode()`::                Stop a random node in your cluster to mimic an outage
`stopCurrentMasterNode()`::         Stop the current master node to force a new election
`stopRandomNonMaster()`::           Stop a random non master node to mimic an outage
`buildNode()`::                     Create a new elasticsearch node
`startNode(settings)`::             Create and start a new elasticsearch node


[[accessing-clients]]
==== Accessing clients

In order to execute any actions, you have to use a client. You can use the `ElasticsearchIntegrationTest.client()` method to get back a random client. This client can be a `TransportClient` or a `NodeClient` - and usually you do not need to care as long as the action gets executed. There are several more methods for client selection inside of the `TestCluster` class, which can be accessed using the `ElasticsearchIntegrationTest.cluster()` method.

[horizontal]
`iterator()`::                  An iterator over all available clients
`masterClient()`::              Returns a client which is connected to the master node
`nonMasterClient()`::           Returns a client which is not connected to the master node
`clientNodeClient()`::          Returns a client, which is running on a client node
`client(String nodeName)`::     Returns a client to a given node
`smartClient()`::               Returns a smart client


[[scoping]]
==== Scoping

By default the tests are run without restarting the cluster between tests or test classes in order to be as fast as possible. Of course all indices and templates are deleted between each test. However, sometimes you need to start a new cluster for each test or for a whole test suite - for example, if you load a certain plugin, but you do not want to load it for every test.

You can use the `@ClusterScope` annotation at class level to configure this behaviour

[source,java]
-----------------------------------------
@ClusterScope(scope=SUITE, numNodes=1)
public class CustomSuggesterSearchTests extends ElasticsearchIntegrationTest {
  // ... tests go here
}
-----------------------------------------

The above sample configures an own cluster for this test suite, which is the class. Other values could be `GLOBAL` (the default) or `TEST` in order to spawn a new cluster for each test. The `numNodes` settings allows you to only start a certain number of nodes, which can speed up test execution, as starting a new node is a costly and time consuming operation and might not be needed for this test.


[[changing-node-configuration]]
==== Changing node configuration

As elasticsearch is using JUnit 4, using the `@Before` and `@After` annotations is not a problem. However you should keep in mind, that this does not have any effect in your cluster setup, as the cluster is already up and running when those methods are run. So in case you want to configure settings - like loading a plugin on node startup - before the node is actually running, you should overwrite the `nodeSettings()` method from the `ElasticsearchIntegrationTest` class and change the cluster scope to `SUITE`.

[source,java]
-----------------------------------------
@Override
protected Settings nodeSettings(int nodeOrdinal) {
  return ImmutableSettings.settingsBuilder()
           .put("plugin.types", CustomSuggesterPlugin.class.getName())
           .put(super.nodeSettings(nodeOrdinal)).build();
}
-----------------------------------------



[[randomized-testing]]
=== Randomized testing

The code snippets you saw so far did not show any trace of randomized testing features, as they are carefully hidden under the hood. However when you are writing your own tests, you should make use of these features as well. Before starting with that, you should know, how to repeat a failed test with the same setup, how it failed. Luckily this is quite easy, as the whole mvn call is logged together with failed tests, which means you can simply copy and paste that line and run the test.

[[generating-random-data]]
==== Generating random data

The next step is to convert your test using static test data into a test using randomized test data. The kind of data you could randomize varies a lot with the functionality you are testing against. Take a look at the following examples (note, that this list could go on for pages, as a distributed system has many, many moving parts):

* Searching for data using arbitrary UTF8 signs
* Changing your mapping configuration, index and field names with each run
* Changing your response sizes/configurable limits with each run
* Changing the number of shards/replicas when creating an index

So, how can you create random data. The most important thing to know is, that you never should instantiate your own `Random` instance, but use the one provided in the `RandomizedTest`, from which all elasticsearch dependent test classes inherit from.

[horizontal]
`getRandom()`::         Returns the random instance, which can recreated when calling the test with specific parameters
`randomBoolean()`::     Returns a random boolean
`randomByte()`::        Returns a random byte
`randomShort()`::       Returns a random short
`randomInt()`::         Returns a random integer
`randomLong()`::        Returns a random long
`randomFloat()`::       Returns a random float
`randomDouble()`::      Returns a random double

`randomInt(max)`::      Returns a random integer between 0 and max
`between()`::           Returns a random between the supplied range
`atLeast()`::           Returns a random integer of at least the specified integer
`atMost()`::            Returns a random integer of at most the specified integer

`randomLocale()`::      Returns a random locale
`randomTimeZone()`::    Returns a random timezone

In addition, there are a couple of helper methods, allowing you to create random ASCII and Unicode strings, see methods beginning with `randomAscii`, `randomUnicode`, and `randomRealisticUnicode` in the random test class. The latter one tries to create more realistic unicode string by not being arbitrary random.

If you want to debug a specific problem with a specific random seed, you can use the `@Seed` annotation to configure a specific seed for a test. If you want to run a test more than once, instead of starting the whole test suite over and over again, you can use the `@Repeat` annotation with an arbitrary value. Each iteration than gets run with a different seed.


[[assertions]]
=== Assertions

As many elasticsearch tests are checking for a similar output, like the amount of hits or the first hit or special highlighting, a couple of predefined assertions have been created. Those have been put into the `ElasticsearchAssertions` class.

[horizontal]
`assertHitCount()`::        Checks hit count of a search or count request
`assertAcked()`::           Ensure the a request has been ackknowledged by the master
`assertSearchHits()`::      Asserts a search response contains specific ids
`assertMatchCount()`::      Asserts a matching count from a percolation response
`assertFirstHit()`::        Asserts the first hit hits the specified matcher
`assertSecondHit()`::       Asserts the second hit hits the specified matcher
`assertThirdHit()`::        Asserts the third hits hits the specified matcher
`assertSearchHit()`::       Assert a certain element in a search response hits the specified matcher
`assertNoFailures()`::      Asserts that no shard failures have occured in the response
`assertHighlight()`::       Assert specific highlights matched
`assertSuggestion()`::      Assert for specific suggestions
`assertSuggestionSize()`::  Assert for specific suggestion count
`assertThrows()`::          Assert a specific exception has been thrown

Common matchers

[horizontal]
`hasId()`::     Matcher to check for a search hit id
`hasType()`::   Matcher to check for a search hit type
`hasIndex()`::  Matcher to check for a search hit index

Usually, you would combine assertions and matchers in your test like this

[source,java]
----------------------------
SearchResponse seearchResponse = client().prepareSearch() ...;
assertHitCount(searchResponse, 4);
assertFirstHit(searchResponse, hasId("4"));
assertSearchHits(searchResponse, "1", "2", "3", "4");
----------------------------