[TEST] Replaced RestTestSuiteRunner with parametrized test that uses RandomizedRunner directly

ElasticsearchRestTests extends now ElasticsearchIntegrationTest and makes use of our ordinary test infrastructure, in particular all randomized aspects now come for free instead of having to maintain a separate (custom) tests runner

We previously parsed only the tests that needed to be run given the version of the cluster the tests are running against. This doesn't happen anymore as it didn't buy much and it would be harder to support as the tests get now parsed before the test cluster gets started. Thus all the tests are now parsed regardless of their skip sections, afterwards the ones that don't need to be run will be skipped through assume directives.

Fixed REST tests that rely on a specific number of shards as this change introduces also random number of shards and replicas (through randomIndexTemplate)

Closes #5654

TESTING.asciidoc

@@ -186,22 +186,18 @@ mvn test -Dtests.class=org.elasticsearch.test.rest.ElasticsearchRestTests
 `ElasticsearchRestTests` is the executable test class that runs all the
 yaml suites available within the `rest-api-spec` folder.
 
-The following are the options supported by the REST tests runner:
+The REST tests support all the options provided by the randomized runner, plus the following:
 
-* `tests.rest[true|false|host:port]`: determines whether the REST tests need
-to be run and if so whether to rely on an external cluster (providing host
-and port) or fire a test cluster (default). It's possible to provide a
-comma separated list of addresses to send requests in a round-robin fashion.
+* `tests.rest[true|false]`: determines whether the REST tests need to be run (default) or not.
 * `tests.rest.suite`: comma separated paths of the test suites to be run
 (by default loaded from /rest-api-spec/test). It is possible to run only a subset
 of the tests providing a sub-folder or even a single yaml file (the default
 /rest-api-spec/test prefix is optional when files are loaded from classpath)
 e.g. -Dtests.rest.suite=index,get,create/10_with_id
-* `tests.rest.section`: regex that allows to filter the test sections that
-are going to be run. If provided, only the section names that match (case
-insensitive) against it will be executed
 * `tests.rest.spec`: REST spec path (default /rest-api-spec/api)
-* `tests.iters`: runs multiple iterations
-* `tests.seed`: seed to base the random behaviours on
-* `tests.appendseed[true|false]`: enables adding the seed to each test
-section's description (default false)
+
+Note that the REST tests, like all the integration tests, can be run against an external
+cluster by specifying the `tests.cluster` property, which if present needs to contain a
+comma separated list of nodes to connect to (e.g. localhost:9300). A transport client will
+be created based on that and used for all the before|after test operations, and to extract
+the http addresses of the nodes so that REST requests can be sent to them.

dev-tools/build_release.py

@@ -388,7 +388,7 @@ def smoke_test_release(release, files, expected_hash, plugins):
           if version['build_hash'].strip() !=  expected_hash:
             raise RuntimeError('HEAD hash does not match expected [%s] but got [%s]' % (expected_hash, version['build_hash']))
           print('  Running REST Spec tests against package [%s]' % release_file)
-          run_mvn('test -Dtests.rest=%s -Dtests.class=*.*RestTests' % ("127.0.0.1:9200"))
+          run_mvn('test -Dtests.cluster=%s -Dtests.class=*.*RestTests' % ("127.0.0.1:9300"))
           print('  Verify if plugins are listed in _nodes')
           conn.request('GET', '/_nodes?plugin=true&pretty=true')
           res = conn.getresponse()

pom.xml

@@ -52,7 +52,7 @@
         <dependency>
             <groupId>com.carrotsearch.randomizedtesting</groupId>
             <artifactId>randomizedtesting-runner</artifactId>
-            <version>2.1.1</version>
+            <version>2.1.2</version>
             <scope>test</scope>
         </dependency>
         <dependency>
@@ -387,7 +387,7 @@
             <plugin>
                 <groupId>com.carrotsearch.randomizedtesting</groupId>
                 <artifactId>junit4-maven-plugin</artifactId>
-                <version>2.1.1</version>
+                <version>2.1.2</version>
                 <executions>
                     <execution>
                         <id>tests</id>
@@ -445,6 +445,7 @@
                             <sysouts>${tests.verbose}</sysouts>
                             <seed>${tests.seed}</seed>
                             <haltOnFailure>${tests.failfast}</haltOnFailure>
+                            <uniqueSuiteNames>false</uniqueSuiteNames>
                             <systemProperties>
                                 <java.io.tmpdir>.</java.io.tmpdir> <!-- we use '.' since this is different per JVM-->
                                 <!-- RandomizedTesting library system properties -->
@@ -1138,12 +1139,8 @@
                                 <include>org/elasticsearch/cache/recycler/MockPageCacheRecycler.class</include>
                                 <include>org/apache/lucene/util/AbstractRandomizedTest.class</include>
                                 <include>org/apache/lucene/util/AbstractRandomizedTest$*.class</include>
-                                <!-- inner classes -->
-                                <include>com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.class</include>
                             </includes>
                             <excludes>
-                                <!-- we only distribute the REST tests runner, not the executable test  -->
-                                <exclude>org/elasticsearch/test/rest/ElasticsearchRestTests.class</exclude>
                                 <!-- unit tests for yaml suite parser & rest spec parser need to be excluded -->
                                 <exclude>org/elasticsearch/test/rest/test/**/*</exclude>
                             </excludes>

rest-api-spec/test/cat.allocation/10_basic.yaml

@@ -55,7 +55,7 @@
   - match:
       $body: >
             /^
-              ( [1-5]               \s+
+              ( \d+                 \s+
                 \d+(\.\d+)?[kmgt]b  \s+
                 \d+(\.\d+)?[kmgt]b  \s+
                 \d+(\.\d+)?[kmgt]b  \s+

rest-api-spec/test/cat.shards/10_basic.yaml

@@ -9,12 +9,12 @@
                /^$/
 
   - do:
-      index:
-        index:  index1
-        type:   type1
-        id:     1
-        body:   { foo: bar }
-        refresh: true
+      indices.create:
+        index: index1
+        body:
+          settings:
+            number_of_shards: "5"
+            number_of_replicas: "1"
   - do:
        cluster.health:
          wait_for_status: yellow
@@ -30,6 +30,7 @@
         index: index2
         body:
           settings:
+            number_of_shards: "5"
             number_of_replicas: "0"
   - do:
        cluster.health:

rest-api-spec/test/cluster.put_settings/10_basic.yaml

@@ -1,4 +1,9 @@
 ---
+setup:
+  - skip:
+      version: 0 - 999
+      reason:  leaves transient metadata behind, need to fix it
+---
 "Test put settings":
   - do:
       cluster.put_settings:

rest-api-spec/test/create/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/create/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/delete/30_routing.yaml

@@ -1,6 +1,12 @@
 ---
 "Routing":
 
+ - do:
+      indices.create:
+          index: test_1
+          body:
+            settings:
+              number_of_shards: 5
  - do:
       index:
           index:   test_1

rest-api-spec/test/delete/40_parent.yaml

@@ -4,6 +4,8 @@
       indices.create:
           index: test_1
           body:
+            settings:
+              number_of_shards: 5
             mappings:
               test:
                 _parent: { type: "foo" }

rest-api-spec/test/delete/45_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/exists/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/exists/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/get/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/get/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/get_source/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/get_source/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/index/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/index/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/indices.get_settings/10_basic.yaml

@@ -3,9 +3,17 @@ setup:
   - do:
         indices.create:
           index: test_1
+          body:
+            settings:
+              number_of_shards: 5
+              number_of_replicas: 1
   - do:
         indices.create:
           index: test_2
+          body:
+            settings:
+              number_of_shards: 3
+              number_of_replicas: 0
 
 ---
 "Get /_settings":
@@ -15,8 +23,8 @@ setup:
 
  - match: { test_1.settings.index.number_of_shards: "5"}
  - match: { test_1.settings.index.number_of_replicas: "1"}
- - match: { test_2.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_replicas: "1"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
+ - match: { test_2.settings.index.number_of_replicas: "0"}
 
 ---
 "Get /{index}/_settings":
@@ -98,7 +106,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -111,7 +119,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -125,7 +133,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -138,7 +146,7 @@ setup:
         name: index.number_of_shards
 
  - match: { test_1.settings.index.number_of_shards: "5"}
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1.settings.index.number_of_replicas
  - is_false: test_2.settings.index.number_of_replicas
 
@@ -150,7 +158,7 @@ setup:
         index: '*2'
         name: index.number_of_shards
 
- - match: { test_2.settings.index.number_of_shards: "5"}
+ - match: { test_2.settings.index.number_of_shards: "3"}
  - is_false: test_1
  - is_false: test_2.settings.index.number_of_replicas
 

rest-api-spec/test/mget/30_parent.yaml

@@ -8,6 +8,8 @@
             mappings:
               test:
                 _parent: { type: "foo" }
+            settings:
+              number_of_shards: 5
  - do:
       cluster.health:
           wait_for_status: yellow

rest-api-spec/test/mget/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/mget/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

rest-api-spec/test/search/20_default_values.yaml

@@ -34,10 +34,31 @@
               foo: bar
 
   - match: {hits.total: 2}
+
+  - do:
+      search:
+        index: test_1
+        type:  test
+        body:
+          query:
+            match:
+              foo: bar
+
+  - match: {hits.total: 1}
   - match: {hits.hits.0._index: test_1 }
   - match: {hits.hits.0._type:  test   }
   - match: {hits.hits.0._id:    "1"    }
 
-  - match: {hits.hits.1._index: test_2 }
-  - match: {hits.hits.1._type:  test   }
-  - match: {hits.hits.1._id:    "42"   }
+  - do:
+      search:
+        index: test_2
+        type:  test
+        body:
+          query:
+            match:
+              foo: bar
+
+  - match: {hits.total: 1}
+  - match: {hits.hits.0._index: test_2 }
+  - match: {hits.hits.0._type:  test   }
+  - match: {hits.hits.0._id:    "42"   }

rest-api-spec/test/update/40_routing.yaml

@@ -7,6 +7,7 @@
        body:
          settings:
            index:
+             number_of_shards: 5
              number_of_replicas: 0
 
  - do:

rest-api-spec/test/update/55_parent_with_routing.yaml

@@ -9,6 +9,7 @@
               test:
                 _parent: { type: "foo" }
             settings:
+              number_of_shards: 5
               number_of_replicas: 0
 
  - do:

src/test/java/com/carrotsearch/randomizedtesting/StandaloneRandomizedContext.java

@@ -1,68 +0,0 @@
-/*
- * Licensed to Elasticsearch under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. Elasticsearch licenses this
- * file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations under
- * the License.
- */
-
-package com.carrotsearch.randomizedtesting;
-
-/**
- * Exposes methods that allow to use a {@link RandomizedContext} without using a {@link RandomizedRunner}
- * This was specifically needed by the REST tests since they run with a custom junit runner ({@link org.elasticsearch.test.rest.junit.RestTestSuiteRunner})
- */
-public final class StandaloneRandomizedContext {
-
-    private StandaloneRandomizedContext() {
-
-    }
-
-    /**
-     * Creates a new {@link RandomizedContext} associated to the current thread
-     */
-    public static void createRandomizedContext(Class<?> testClass, Randomness runnerRandomness) {
-        //the randomized runner is passed in as null, which is fine as long as we don't try to access it afterwards
-        RandomizedContext randomizedContext = RandomizedContext.create(Thread.currentThread().getThreadGroup(), testClass, null);
-        randomizedContext.push(runnerRandomness.clone(Thread.currentThread()));
-    }
-
-    /**
-     * Destroys the {@link RandomizedContext} associated to the current thread
-     */
-    public static void disposeRandomizedContext() {
-        RandomizedContext.current().dispose();
-    }
-
-    public static void pushRandomness(Randomness randomness) {
-        RandomizedContext.current().push(randomness);
-    }
-
-    public static void popAndDestroy() {
-        RandomizedContext.current().popAndDestroy();
-    }
-
-    /**
-     * Returns the string formatted seed associated to the current thread's randomized context
-     */
-    public static String getSeedAsString() {
-        return SeedUtils.formatSeed(RandomizedContext.current().getRandomness().getSeed());
-    }
-
-    /**
-     * Util method to extract the seed out of a {@link Randomness} instance
-     */
-    public static long getSeed(Randomness randomness) {
-        return randomness.getSeed();
-    }
-}

src/test/java/org/elasticsearch/test/ElasticsearchIntegrationTest.java

@@ -1112,7 +1112,7 @@ public abstract class ElasticsearchIntegrationTest extends ElasticsearchTestCase
 
     }
 
-    private final static void initializeSuiteScope() throws Exception {
+    private static void initializeSuiteScope() throws Exception {
         Class<?> targetClass = getContext().getTargetClass();
         assert INSTANCE == null;
         if (isSuiteScope(targetClass)) {

src/test/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java

@@ -26,13 +26,18 @@ import org.elasticsearch.common.logging.ESLogger;
 import org.elasticsearch.common.logging.Loggers;
 import org.elasticsearch.test.ElasticsearchTestCase;
 import org.elasticsearch.test.TestCluster;
+import org.elasticsearch.test.rest.ElasticsearchRestTests;
 import org.junit.internal.AssumptionViolatedException;
 import org.junit.runner.Description;
 import org.junit.runner.notification.Failure;
 import org.junit.runner.notification.RunListener;
 
 import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_ITERATIONS;
+import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_TESTMETHOD;
 import static org.elasticsearch.test.ElasticsearchIntegrationTest.TESTS_CLUSTER;
+import static org.elasticsearch.test.rest.ElasticsearchRestTests.REST_TESTS_SPEC;
+import static org.elasticsearch.test.rest.ElasticsearchRestTests.REST_TESTS_SUITE;
+
 
 /**
  * A {@link RunListener} that emits to {@link System#err} a string with command
@@ -63,7 +68,13 @@ public class ReproduceInfoPrinter extends RunListener {
         final StringBuilder b = new StringBuilder();
         b.append("FAILURE  : ").append(d.getDisplayName()).append("\n");
         b.append("REPRODUCE WITH  : mvn test");
-        reproduceErrorMessageBuilder(b).appendAllOpts(failure.getDescription());
+        MavenMessageBuilder mavenMessageBuilder = new MavenMessageBuilder(b);
+        mavenMessageBuilder.appendAllOpts(failure.getDescription());
+
+        //ElasticsearchRestTests is a special case as it allows for additional parameters
+        if (ElasticsearchRestTests.class.isAssignableFrom(failure.getDescription().getTestClass())) {
+            mavenMessageBuilder.appendRestTestsProperties();
+        }
 
         b.append("\n");
         b.append("Throwable:\n");
@@ -74,10 +85,6 @@ public class ReproduceInfoPrinter extends RunListener {
         logger.error(b.toString());
     }
 
-    protected ReproduceErrorMessageBuilder reproduceErrorMessageBuilder(StringBuilder b) {
-        return new MavenMessageBuilder(b);
-    }
-
     protected TraceFormatting traces() {
         TraceFormatting traces = new TraceFormatting();
         try {
@@ -97,6 +104,12 @@ public class ReproduceInfoPrinter extends RunListener {
         @Override
         public ReproduceErrorMessageBuilder appendAllOpts(Description description) {
             super.appendAllOpts(description);
+
+            if (description.getMethodName() != null) {
+                //prints out the raw method description instead of methodName(description) which filters out the parameters
+                super.appendOpt(SYSPROP_TESTMETHOD(), "\"" + description.getMethodName() + "\"");
+            }
+
             return appendESProperties();
         }
 
@@ -108,9 +121,14 @@ public class ReproduceInfoPrinter extends RunListener {
             if (sysPropName.equals(SYSPROP_ITERATIONS())) { // we don't want the iters to be in there!
                 return this;
             }
+            if (sysPropName.equals(SYSPROP_TESTMETHOD())) {
+                //don't print out the test method, we print it ourselves in appendAllOpts
+                //without filtering out the parameters (needed for REST tests)
+                return this;
+            }
             if (Strings.hasLength(value)) {
                 return super.appendOpt(sysPropName, value);
-            } 
+            }
             return this;
         }
 
@@ -123,6 +141,10 @@ public class ReproduceInfoPrinter extends RunListener {
             return this;
         }
 
+        public ReproduceErrorMessageBuilder appendRestTestsProperties() {
+            return appendProperties(REST_TESTS_SUITE, REST_TESTS_SPEC);
+        }
+
         protected ReproduceErrorMessageBuilder appendProperties(String... properties) {
             for (String sysPropName : properties) {
                 if (Strings.hasLength(System.getProperty(sysPropName))) {

src/test/java/org/elasticsearch/test/rest/ElasticsearchRestTests.java

@@ -16,22 +16,174 @@
  * specific language governing permissions and limitations
  * under the License.
  */
+
 package org.elasticsearch.test.rest;
 
-import org.elasticsearch.test.rest.junit.RestTestSuiteRunner;
-import org.junit.runner.RunWith;
+import com.carrotsearch.randomizedtesting.SysGlobals;
+import com.carrotsearch.randomizedtesting.annotations.Name;
+import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
+import com.carrotsearch.randomizedtesting.annotations.ReplicateOnEachVm;
+import com.google.common.collect.Lists;
+import org.elasticsearch.cluster.routing.operation.hash.djb.DjbHashFunction;
+import org.elasticsearch.common.Strings;
+import org.elasticsearch.common.math.MathUtils;
+import org.elasticsearch.test.ElasticsearchIntegrationTest;
+import org.elasticsearch.test.rest.client.RestException;
+import org.elasticsearch.test.rest.parser.RestTestParseException;
+import org.elasticsearch.test.rest.parser.RestTestSuiteParser;
+import org.elasticsearch.test.rest.section.*;
+import org.elasticsearch.test.rest.spec.RestSpec;
+import org.elasticsearch.test.rest.support.FileUtils;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
 
-import static org.apache.lucene.util.LuceneTestCase.Slow;
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
 
 /**
- * Runs the clients test suite against an elasticsearch node, which can be an external node or an automatically created cluster.
- * Communicates with elasticsearch exclusively via REST layer.
- *
- * @see RestTestSuiteRunner for extensive documentation and all the supported options
+ * Runs the clients test suite against an elasticsearch cluster.
  */
-@Slow
-@RunWith(RestTestSuiteRunner.class)
-public class ElasticsearchRestTests {
+//tests distribution disabled for now since it causes reporting problems,
+// due to the non unique suite name
+//@ReplicateOnEachVm
+public class ElasticsearchRestTests extends ElasticsearchIntegrationTest {
+
+    public static final String REST_TESTS = "tests.rest";
+    public static final String REST_TESTS_SUITE = "tests.rest.suite";
+    public static final String REST_TESTS_SPEC = "tests.rest.spec";
+
+    private static final String DEFAULT_TESTS_PATH = "/rest-api-spec/test";
+    private static final String DEFAULT_SPEC_PATH = "/rest-api-spec/api";
+
+    private static final String PATHS_SEPARATOR = ",";
+
+    private static RestTestExecutionContext restTestExecutionContext;
+
+    //private static final int JVM_COUNT = systemPropertyAsInt(SysGlobals.CHILDVM_SYSPROP_JVM_COUNT, 1);
+    //private static final int CURRENT_JVM_ID = systemPropertyAsInt(SysGlobals.CHILDVM_SYSPROP_JVM_ID, 0);
+
+    private final RestTestCandidate testCandidate;
+
+    public ElasticsearchRestTests(@Name("yaml") RestTestCandidate testCandidate) {
+        this.testCandidate = testCandidate;
+    }
+
+    @ParametersFactory
+    public static Iterable<Object[]> parameters() throws IOException, RestTestParseException {
+        List<RestTestCandidate> restTestCandidates = collectTestCandidates();
+        List<Object[]> objects = Lists.newArrayList();
+        for (RestTestCandidate restTestCandidate : restTestCandidates) {
+            objects.add(new Object[]{restTestCandidate});
+        }
+        return objects;
+    }
+
+    private static List<RestTestCandidate> collectTestCandidates() throws RestTestParseException, IOException {
+        String[] paths = resolvePathsProperty(REST_TESTS_SUITE, DEFAULT_TESTS_PATH);
+        Map<String, Set<File>> yamlSuites = FileUtils.findYamlSuites(DEFAULT_TESTS_PATH, paths);
+
+        //yaml suites are grouped by directory (effectively by api)
+        List<String> apis = Lists.newArrayList(yamlSuites.keySet());
+
+        List<RestTestCandidate> testCandidates = Lists.newArrayList();
+        RestTestSuiteParser restTestSuiteParser = new RestTestSuiteParser();
+        for (String api : apis) {
+            List<File> yamlFiles = Lists.newArrayList(yamlSuites.get(api));
+            for (File yamlFile : yamlFiles) {
+                //tests distribution disabled for now since it causes reporting problems,
+                // due to the non unique suite name
+                //if (mustExecute(yamlFile.getAbsolutePath())) {
+                    RestTestSuite restTestSuite = restTestSuiteParser.parse(api, yamlFile);
+                    for (TestSection testSection : restTestSuite.getTestSections()) {
+                        testCandidates.add(new RestTestCandidate(restTestSuite, testSection));
+                    }
+                //}
+            }
+        }
+        return testCandidates;
+    }
+
+    /*private static boolean mustExecute(String test) {
+        //we distribute the tests across the forked jvms if > 1
+        if (JVM_COUNT > 1) {
+            int jvmId = MathUtils.mod(DjbHashFunction.DJB_HASH(test), JVM_COUNT);
+            if (jvmId != CURRENT_JVM_ID) {
+                return false;
+            }
+        }
+        return true;
+    }*/
+
+    private static String[] resolvePathsProperty(String propertyName, String defaultValue) {
+        String property = System.getProperty(propertyName);
+        if (!Strings.hasLength(property)) {
+            return new String[]{defaultValue};
+        } else {
+            return property.split(PATHS_SEPARATOR);
+        }
+    }
+
+    @BeforeClass
+    public static void initExecutionContext() throws IOException, RestException {
+        //skip REST tests if disabled through -Dtests.rest=false
+        assumeTrue(systemPropertyAsBoolean(REST_TESTS, true));
+
+        String[] specPaths = resolvePathsProperty(REST_TESTS_SPEC, DEFAULT_SPEC_PATH);
+        RestSpec restSpec = RestSpec.parseFrom(DEFAULT_SPEC_PATH, specPaths);
+        restTestExecutionContext = new RestTestExecutionContext(restSpec);
+    }
+
+    @AfterClass
+    public static void close() {
+        restTestExecutionContext.close();
+    }
+
+    @Before
+    public void reset() throws IOException, RestException {
+        restTestExecutionContext.resetClient(immutableCluster().httpAddresses());
+        restTestExecutionContext.clear();
+
+        assumeFalse(buildSkipMessage(testCandidate.getSuiteDescription(), testCandidate.getSetupSection().getSkipSection()),
+                testCandidate.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion()));
+        assumeFalse(buildSkipMessage(testCandidate.getDescription(), testCandidate.getTestSection().getSkipSection()),
+                testCandidate.getTestSection().getSkipSection().skip(restTestExecutionContext.esVersion()));
+    }
+
+    private static String buildSkipMessage(String description, SkipSection skipSection) {
+        StringBuilder messageBuilder = new StringBuilder();
+        if (skipSection.isVersionCheck()) {
+            messageBuilder.append("[").append(description).append("] skipped, reason: [").append(skipSection.getReason()).append("] ");
+        } else {
+            messageBuilder.append("[").append(description).append("] skipped, reason: features ").append(skipSection.getFeatures()).append(" not supported");
+        }
+        return messageBuilder.toString();
+    }
+
+    @Test
+    public void test() throws IOException {
+        //let's check that there is something to run, otherwise there might be a problem with the test section
+        if (testCandidate.getTestSection().getExecutableSections().size() == 0) {
+            throw new IllegalArgumentException("No executable sections loaded for ["
+                    + testCandidate.getSuiteDescription() + "/" + testCandidate.getTestSection().getName() + "]");
+        }
+
+        if (!testCandidate.getSetupSection().isEmpty()) {
+            logger.info("start setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
+            for (DoSection doSection : testCandidate.getSetupSection().getDoSections()) {
+                doSection.execute(restTestExecutionContext);
+            }
+            logger.info("end setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
+        }
 
+        restTestExecutionContext.clear();
 
-}
+        for (ExecutableSection executableSection : testCandidate.getTestSection().getExecutableSections()) {
+            executableSection.execute(restTestExecutionContext);
+        }
+    }
+}

src/test/java/org/elasticsearch/test/rest/junit/RestTestCandidate.java → src/test/java/org/elasticsearch/test/rest/RestTestCandidate.java

@@ -16,37 +16,24 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.elasticsearch.test.rest.junit;
+package org.elasticsearch.test.rest;
 
 import org.elasticsearch.test.rest.section.RestTestSuite;
 import org.elasticsearch.test.rest.section.SetupSection;
 import org.elasticsearch.test.rest.section.TestSection;
-import org.junit.runner.Description;
 
 /**
- * Wraps {@link org.elasticsearch.test.rest.section.TestSection}s ready to be run,
- * properly enriched with the needed execution information.
- * The tests tree structure gets flattened to the leaves (test sections)
+ * Wraps {@link org.elasticsearch.test.rest.section.TestSection}s ready to be run.
+ * Each test section is associated to its {@link org.elasticsearch.test.rest.section.RestTestSuite}.
  */
 public class RestTestCandidate {
 
     private final RestTestSuite restTestSuite;
-    private final Description suiteDescription;
     private final TestSection testSection;
-    private final Description testDescription;
-    private final long seed;
 
-    static RestTestCandidate empty(RestTestSuite restTestSuite, Description suiteDescription) {
-        return new RestTestCandidate(restTestSuite, suiteDescription, null, null, -1);
-    }
-
-    RestTestCandidate(RestTestSuite restTestSuite, Description suiteDescription,
-                      TestSection testSection, Description testDescription, long seed) {
+    public RestTestCandidate(RestTestSuite restTestSuite, TestSection testSection) {
         this.restTestSuite = restTestSuite;
-        this.suiteDescription = suiteDescription;
         this.testSection = testSection;
-        this.testDescription = testDescription;
-        this.seed = seed;
     }
 
     public String getApi() {
@@ -61,12 +48,8 @@ public class RestTestCandidate {
         return restTestSuite.getDescription();
     }
 
-    public Description describeSuite() {
-        return suiteDescription;
-    }
-
-    public Description describeTest() {
-        return testDescription;
+    public String getDescription() {
+        return getSuiteDescription() + "/" + testSection.getName();
     }
 
     public SetupSection getSetupSection() {
@@ -77,7 +60,8 @@ public class RestTestCandidate {
         return testSection;
     }
 
-    public long getSeed() {
-        return seed;
+    @Override
+    public String toString() {
+        return getSuiteDescription() + "/" + testSection.getName();
     }
 }

src/test/java/org/elasticsearch/test/rest/RestTestExecutionContext.java

@@ -44,17 +44,16 @@ public class RestTestExecutionContext implements Closeable {
 
     private static final ESLogger logger = Loggers.getLogger(RestTestExecutionContext.class);
 
-    private final RestClient restClient;
+    private final Stash stash = new Stash();
 
-    private final String esVersion;
+    private final RestSpec restSpec;
 
-    private final Stash stash = new Stash();
+    private RestClient restClient;
 
     private RestResponse response;
 
-    public RestTestExecutionContext(InetSocketAddress[] addresses, RestSpec restSpec) throws RestException, IOException {
-        this.restClient = new RestClient(addresses, restSpec);
-        this.esVersion = restClient.getEsVersion();
+    public RestTestExecutionContext(RestSpec restSpec) throws RestException, IOException {
+        this.restSpec = restSpec;
     }
 
     /**
@@ -104,15 +103,6 @@ public class RestTestExecutionContext implements Closeable {
         return XContentFactory.jsonBuilder().map(body).string();
     }
 
-    /**
-     * Calls an elasticsearch api internally without saving the obtained response in the context.
-     * Useful for internal calls (e.g. delete index during teardown)
-     * @throws RestException if the returned status code is non ok
-     */
-    public RestResponse callApiInternal(String apiName, String... params) throws IOException, RestException {
-        return restClient.callApi(apiName, params);
-    }
-
     private RestResponse callApiInternal(String apiName, Map<String, String> params, String body) throws IOException, RestException  {
         return restClient.callApi(apiName, params, body);
     }
@@ -124,11 +114,22 @@ public class RestTestExecutionContext implements Closeable {
         return response.evaluate(path);
     }
 
+    /**
+     * Recreates the embedded REST client which will point to the given addresses
+     */
+    public void resetClient(InetSocketAddress[] addresses) throws IOException, RestException {
+        if (restClient == null) {
+            restClient = new RestClient(addresses, restSpec);
+        } else {
+            restClient.updateAddresses(addresses);
+        }
+    }
+
     /**
      * Clears the last obtained response and the stashed fields
      */
     public void clear() {
-        logger.debug("resetting response and stash");
+        logger.debug("resetting client, response and stash");
         response = null;
         stash.clear();
     }
@@ -141,7 +142,7 @@ public class RestTestExecutionContext implements Closeable {
      * Returns the current es version as a string
      */
     public String esVersion() {
-        return esVersion;
+        return restClient.getEsVersion();
     }
 
     /**

src/test/java/org/elasticsearch/test/rest/client/RestClient.java

@@ -48,7 +48,7 @@ public class RestClient implements Closeable {
     private final RestSpec restSpec;
     private final CloseableHttpClient httpClient;
 
-    private final InetSocketAddress[] addresses;
+    private InetSocketAddress[] addresses;
 
     private final String esVersion;
 
@@ -96,21 +96,10 @@ public class RestClient implements Closeable {
     }
 
     /**
-     * Calls an api with the provided parameters
-     * @throws RestException if the obtained status code is non ok, unless the specific error code needs to be ignored
-     * according to the ignore parameter received as input (which won't get sent to elasticsearch)
+     * Allows to update the addresses the client needs to connect to
      */
-    public RestResponse callApi(String apiName, String... params) throws IOException, RestException {
-        if (params.length % 2 != 0) {
-            throw new IllegalArgumentException("The number of params passed must be even but was [" + params.length + "]");
-        }
-
-        Map<String, String> paramsMap = Maps.newHashMap();
-        for (int i = 0; i < params.length; i++) {
-            paramsMap.put(params[i++], params[i]);
-        }
-
-        return callApi(apiName, paramsMap, null);
+    public void updateAddresses(InetSocketAddress[] addresses) {
+        this.addresses = addresses;
     }
 
     /**

src/test/java/org/elasticsearch/test/rest/junit/DescriptionHelper.java

@@ -1,79 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.google.common.base.Joiner;
-import org.elasticsearch.test.rest.section.RestTestSuite;
-import org.elasticsearch.test.rest.section.TestSection;
-import org.junit.runner.Description;
-
-import java.util.Map;
-
-/**
- * Helper that knows how to assign proper junit {@link Description}s to each of the node in the tests tree
- */
-public final class DescriptionHelper {
-
-    private DescriptionHelper() {
-
-    }
-
-    /*
-    The following generated ids need to be unique throughout a tests run.
-    Ids are also shown by IDEs (with junit 4.11 unique ids can be different from what gets shown, not yet in 4.10).
-    Some tricks are applied to control what gets shown in IDEs in order to keep the ids unique and nice to see at the same time.
-     */
-
-    static Description createRootDescription(String name) {
-        return Description.createSuiteDescription(name);
-    }
-
-    static Description createApiDescription(String api) {
-        return Description.createSuiteDescription(api);
-    }
-
-    static Description createTestSuiteDescription(RestTestSuite restTestSuite) {
-        //e.g. "indices.open (10_basic)", which leads to 10_basic being shown by IDEs
-        String name = restTestSuite.getApi() + " (" + restTestSuite.getName() + ")";
-        return Description.createSuiteDescription(name);
-    }
-
-    static Description createTestSectionWithRepetitionsDescription(RestTestSuite restTestSuite, TestSection testSection) {
-        //e.g. "indices.open/10_basic (Basic test for index open/close)", which leads to
-        //"Basic test for index open/close" being shown by IDEs
-        String name = restTestSuite.getDescription() + " (" + testSection.getName() + ")";
-        return Description.createSuiteDescription(name);
-    }
-
-    static Description createTestSectionIterationDescription(RestTestSuite restTestSuite, TestSection testSection, Map<String, Object> args) {
-        //e.g. "Basic test for index open/close {#0} (indices.open/10_basic)" some IDEs might strip out the part between parentheses
-        String name = testSection.getName() + formatMethodArgs(args) + " ("  + restTestSuite.getDescription() + ")";
-        return Description.createSuiteDescription(name);
-    }
-
-    private static String formatMethodArgs(Map<String, Object> args) {
-        if (args == null || args.isEmpty()) return "";
-
-        StringBuilder b = new StringBuilder(" {");
-        Joiner.on(" ").withKeyValueSeparator("").appendTo(b, args);
-        b.append("}");
-
-        return b.toString();
-    }
-}

src/test/java/org/elasticsearch/test/rest/junit/RestReproduceInfoPrinter.java

@@ -1,107 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.carrotsearch.randomizedtesting.ReproduceErrorMessageBuilder;
-import com.carrotsearch.randomizedtesting.StandaloneRandomizedContext;
-import com.carrotsearch.randomizedtesting.TraceFormatting;
-import org.elasticsearch.common.logging.ESLogger;
-import org.elasticsearch.common.logging.Loggers;
-import org.elasticsearch.test.junit.listeners.ReproduceInfoPrinter;
-import org.elasticsearch.test.rest.ElasticsearchRestTests;
-import org.elasticsearch.test.rest.junit.RestTestSuiteRunner.RunMode;
-import org.junit.runner.Description;
-
-import java.util.Arrays;
-
-import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_RANDOM_SEED;
-import static com.carrotsearch.randomizedtesting.SysGlobals.SYSPROP_TESTCLASS;
-import static org.elasticsearch.test.rest.junit.RestTestSuiteRunner.*;
-
-/**
- * A {@link org.junit.runner.notification.RunListener} that emits to {@link System#err} a string with command
- * line parameters allowing quick REST test re-run under MVN command line.
- */
-class RestReproduceInfoPrinter extends ReproduceInfoPrinter {
-
-    protected static final ESLogger logger = Loggers.getLogger(RestReproduceInfoPrinter.class);
-
-    private static boolean isTestCluster() {
-        return runMode() == RunMode.TEST_CLUSTER;
-    }
-
-    @Override
-    protected TraceFormatting traces() {
-        return new TraceFormatting(
-                Arrays.asList(
-                    "org.junit.",
-                    "junit.framework.",
-                    "sun.",
-                    "java.lang.reflect.",
-                    "com.carrotsearch.randomizedtesting.",
-                    "org.elasticsearch.test.rest.junit."
-                ));
-    }
-
-    @Override
-    protected ReproduceErrorMessageBuilder reproduceErrorMessageBuilder(StringBuilder b) {
-        return new MavenMessageBuilder(b);
-    }
-
-    private static class MavenMessageBuilder extends ReproduceInfoPrinter.MavenMessageBuilder {
-
-        public MavenMessageBuilder(StringBuilder b) {
-            super(b);
-        }
-
-        @Override
-        public ReproduceErrorMessageBuilder appendAllOpts(Description description) {
-
-            try {
-                appendOpt(SYSPROP_RANDOM_SEED(), StandaloneRandomizedContext.getSeedAsString());
-            } catch (IllegalStateException e) {
-                logger.warn("No context available when dumping reproduce options?");
-            }
-
-            //we know that ElasticsearchRestTests is the only one that runs with RestTestSuiteRunner
-            appendOpt(SYSPROP_TESTCLASS(), ElasticsearchRestTests.class.getName());
-
-            if (description.getClassName() != null) {
-                appendOpt(REST_TESTS_SUITE, description.getClassName());
-            }
-
-            appendRunnerProperties();
-            appendEnvironmentSettings();
-
-            appendProperties("es.logger.level");
-
-            if (isTestCluster()) {
-                appendProperties("es.node.mode", "es.node.local");
-            }
-
-            appendRestTestsProperties();
-
-            return this;
-        }
-
-        public ReproduceErrorMessageBuilder appendRestTestsProperties() {
-            return appendProperties(REST_TESTS_MODE, REST_TESTS_SPEC);
-        }
-    }
-}

src/test/java/org/elasticsearch/test/rest/junit/RestTestSuiteRunner.java

@@ -1,598 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.carrotsearch.hppc.hash.MurmurHash3;
-import com.carrotsearch.randomizedtesting.RandomizedTest;
-import com.carrotsearch.randomizedtesting.Randomness;
-import com.carrotsearch.randomizedtesting.SeedUtils;
-import com.google.common.collect.Lists;
-import org.elasticsearch.common.Strings;
-import org.elasticsearch.common.logging.ESLogger;
-import org.elasticsearch.common.logging.Loggers;
-import org.elasticsearch.common.transport.InetSocketTransportAddress;
-import org.elasticsearch.http.HttpServerTransport;
-import org.elasticsearch.test.ElasticsearchTestCase;
-import org.elasticsearch.test.TestCluster;
-import org.elasticsearch.test.rest.RestTestExecutionContext;
-import org.elasticsearch.test.rest.client.RestException;
-import org.elasticsearch.test.rest.client.RestResponse;
-import org.elasticsearch.test.rest.parser.RestTestParseException;
-import org.elasticsearch.test.rest.parser.RestTestSuiteParser;
-import org.elasticsearch.test.rest.section.DoSection;
-import org.elasticsearch.test.rest.section.ExecutableSection;
-import org.elasticsearch.test.rest.section.RestTestSuite;
-import org.elasticsearch.test.rest.section.TestSection;
-import org.elasticsearch.test.rest.spec.RestSpec;
-import org.elasticsearch.test.rest.support.Features;
-import org.elasticsearch.test.rest.support.FileUtils;
-import org.junit.runner.Description;
-import org.junit.runner.notification.Failure;
-import org.junit.runner.notification.RunNotifier;
-import org.junit.runners.ParentRunner;
-import org.junit.runners.model.InitializationError;
-import org.junit.runners.model.Statement;
-
-import java.io.File;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.*;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.regex.Pattern;
-
-import static com.carrotsearch.randomizedtesting.SeedUtils.parseSeedChain;
-import static com.carrotsearch.randomizedtesting.StandaloneRandomizedContext.*;
-import static com.carrotsearch.randomizedtesting.SysGlobals.*;
-import static org.elasticsearch.test.TestCluster.clusterName;
-import static org.elasticsearch.test.rest.junit.DescriptionHelper.*;
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-
-/**
- * JUnit runner for elasticsearch REST tests
- *
- * Supports the following options provided as system properties:
- * - tests.rest[true|false|host:port]: determines whether the REST tests need to be run and if so
- *                                     whether to rely on an external cluster (providing host and port) or fire a test cluster (default)
- * - tests.rest.suite: comma separated paths of the test suites to be run (by default loaded from /rest-api-spec/test)
- *                     it is possible to run only a subset of the tests providing a directory or a single yaml file
- *                     (the default /rest-api-spec/test prefix is optional when files are loaded from classpath)
- * - tests.rest.section: regex that allows to filter the test sections that are going to be run. If provided, only the
- *                       section names that match (case insensitive) against it will be executed
- * - tests.rest.spec: REST spec path (default /rest-api-spec/api)
- * - tests.iters: runs multiple iterations
- * - tests.seed: seed to base the random behaviours on
- * - tests.appendseed[true|false]: enables adding the seed to each test section's description (default false)
- *
- */
-public class RestTestSuiteRunner extends ParentRunner<RestTestCandidate> {
-
-    private static final ESLogger logger = Loggers.getLogger(RestTestSuiteRunner.class);
-
-    public static final String REST_TESTS_MODE = "tests.rest";
-    public static final String REST_TESTS_SUITE = "tests.rest.suite";
-    public static final String REST_TESTS_SECTION = "tests.rest.section";
-    public static final String REST_TESTS_SPEC = "tests.rest.spec";
-
-    private static final String DEFAULT_TESTS_PATH = "/rest-api-spec/test";
-    private static final String DEFAULT_SPEC_PATH = "/rest-api-spec/api";
-    private static final int DEFAULT_ITERATIONS = 1;
-
-    private static final String PATHS_SEPARATOR = ",";
-
-    private final RestTestExecutionContext restTestExecutionContext;
-    private final List<RestTestCandidate> restTestCandidates;
-    private final Description rootDescription;
-
-    private final RunMode runMode;
-
-    private final TestCluster testCluster;
-
-    private static final AtomicInteger sequencer = new AtomicInteger();
-
-    /** The runner's seed (master). */
-    private final Randomness runnerRandomness;
-
-    /**
-     * If {@link com.carrotsearch.randomizedtesting.SysGlobals#SYSPROP_RANDOM_SEED} property is used with two arguments
-     * (master:test_section) then this field contains test section level override.
-     */
-    private final Randomness testSectionRandomnessOverride;
-
-    enum RunMode {
-        NO, TEST_CLUSTER, EXTERNAL_CLUSTER
-    }
-
-    static RunMode runMode() {
-        String mode = System.getProperty(REST_TESTS_MODE);
-        if (!Strings.hasLength(mode)) {
-            //default true: we run the tests starting our own test cluster
-            mode = Boolean.TRUE.toString();
-        }
-
-        if (Boolean.FALSE.toString().equalsIgnoreCase(mode)) {
-            return RunMode.NO;
-        }
-        if (Boolean.TRUE.toString().equalsIgnoreCase(mode)) {
-            return RunMode.TEST_CLUSTER;
-        }
-        return RunMode.EXTERNAL_CLUSTER;
-    }
-
-    public RestTestSuiteRunner(Class<?> testClass) throws InitializationError {
-        super(testClass);
-
-        this.runMode = runMode();
-
-        if (runMode == RunMode.NO) {
-            //the tests won't be run. the run method will be called anyway but we'll just mark the whole suite as ignored
-            //no need to go ahead and parse the test suites then
-            this.runnerRandomness = null;
-            this.testSectionRandomnessOverride = null;
-            this.restTestExecutionContext = null;
-            this.restTestCandidates = null;
-            this.rootDescription = createRootDescription(getRootSuiteTitle());
-            this.rootDescription.addChild(createApiDescription("empty suite"));
-            this.testCluster = null;
-            return;
-        }
-
-        //the REST test suite is supposed to be run only once per jvm against either an external es node or a self started one
-        if (sequencer.getAndIncrement() > 0) {
-            throw new InitializationError("only one instance of RestTestSuiteRunner can be created per jvm");
-        }
-
-        //either read the seed from system properties (first one in the chain) or generate a new one
-        final String globalSeed = System.getProperty(SYSPROP_RANDOM_SEED());
-        final long initialSeed;
-        Randomness randomnessOverride = null;
-        if (Strings.hasLength(globalSeed)) {
-            final long[] seedChain = parseSeedChain(globalSeed);
-            if (seedChain.length == 0 || seedChain.length > 2) {
-                throw new IllegalArgumentException("Invalid system property "
-                        + SYSPROP_RANDOM_SEED() + " specification: " + globalSeed);
-            }
-            if (seedChain.length > 1) {
-                //read the test section level seed if present
-                randomnessOverride = new Randomness(seedChain[1]);
-            }
-            initialSeed = seedChain[0];
-        } else {
-            initialSeed = MurmurHash3.hash(System.nanoTime());
-        }
-        this.runnerRandomness = new Randomness(initialSeed);
-        this.testSectionRandomnessOverride = randomnessOverride;
-        logger.info("Master seed: {}", SeedUtils.formatSeed(initialSeed));
-
-        List<InetSocketAddress> addresses = Lists.newArrayList();
-        if (runMode == RunMode.TEST_CLUSTER) {
-            this.testCluster = new TestCluster(initialSeed, 1, 3,
-                    clusterName("REST-tests", ElasticsearchTestCase.CHILD_VM_ID, initialSeed));
-            this.testCluster.beforeTest(runnerRandomness.getRandom(), 0.0f);
-            for (HttpServerTransport httpServerTransport : testCluster.getInstances(HttpServerTransport.class)) {
-                addresses.add(((InetSocketTransportAddress) httpServerTransport.boundAddress().publishAddress()).address());
-            }
-        } else {
-            this.testCluster = null;
-            String testsMode = System.getProperty(REST_TESTS_MODE);
-            String[] stringAddresses = testsMode.split(",");
-            for (String stringAddress : stringAddresses) {
-                String[] split = stringAddress.split(":");
-                if (split.length < 2) {
-                    throw new InitializationError("address [" + testsMode + "] not valid");
-                }
-                try {
-                    addresses.add(new InetSocketAddress(split[0], Integer.valueOf(split[1])));
-                } catch(NumberFormatException e) {
-                    throw new InitializationError("port is not valid, expected number but was [" + split[1] + "]");
-                }
-            }
-        }
-
-        try {
-            String[] specPaths = resolvePathsProperty(REST_TESTS_SPEC, DEFAULT_SPEC_PATH);
-            RestSpec restSpec = RestSpec.parseFrom(DEFAULT_SPEC_PATH, specPaths);
-            this.restTestExecutionContext = new RestTestExecutionContext(addresses.toArray(new InetSocketAddress[addresses.size()]), restSpec);
-            this.rootDescription = createRootDescription(getRootSuiteTitle());
-            this.restTestCandidates = collectTestCandidates(rootDescription);
-        } catch (InitializationError e) {
-          stopTestCluster();
-            throw e;
-        } catch (Throwable e) {
-            stopTestCluster();
-            throw new InitializationError(e);
-        }
-    }
-
-    /**
-     * Parse the test suites and creates the test candidates to be run, together with their junit descriptions.
-     * The descriptions will be part of a tree containing api/yaml file/test section/eventual multiple iterations.
-     * The test candidates will be instead flattened out to the leaves level (iterations), the part that needs to be run.
-     */
-    protected List<RestTestCandidate> collectTestCandidates(Description rootDescription) throws InitializationError, IOException {
-
-        String[] paths = resolvePathsProperty(REST_TESTS_SUITE, DEFAULT_TESTS_PATH);
-        Map<String, Set<File>> yamlSuites = FileUtils.findYamlSuites(DEFAULT_TESTS_PATH, paths);
-
-        String sectionFilter = System.getProperty(REST_TESTS_SECTION);
-        Pattern sectionFilterPattern = null;
-        if (Strings.hasLength(sectionFilter)) {
-            sectionFilterPattern = Pattern.compile(sectionFilter, Pattern.CASE_INSENSITIVE);
-        }
-
-        int iterations = determineTestSectionIterationCount();
-        boolean appendSeedParameter = RandomizedTest.systemPropertyAsBoolean(SYSPROP_APPEND_SEED(), false);
-
-        //we iterate over the files and we shuffle them (grouped by api, and by yaml file)
-        //meanwhile we create the junit descriptions and test candidates (one per iteration)
-
-        //yaml suites are grouped by directory (effectively by api)
-        List<String> apis = Lists.newArrayList(yamlSuites.keySet());
-        Collections.shuffle(apis, runnerRandomness.getRandom());
-
-        final boolean fixedSeed = testSectionRandomnessOverride != null;
-        final boolean hasRepetitions = iterations > 1;
-
-        List<Throwable> parseExceptions = Lists.newArrayList();
-        List<RestTestCandidate> testCandidates = Lists.newArrayList();
-        RestTestSuiteParser restTestSuiteParser = new RestTestSuiteParser();
-        for (String api : apis) {
-
-            Description apiDescription = createApiDescription(api);
-
-            List<File> yamlFiles = Lists.newArrayList(yamlSuites.get(api));
-            Collections.shuffle(yamlFiles, runnerRandomness.getRandom());
-
-            for (File yamlFile : yamlFiles) {
-                RestTestSuite restTestSuite;
-                try {
-                    restTestSuite = restTestSuiteParser.parse(restTestExecutionContext.esVersion(), api, yamlFile);
-                } catch (RestTestParseException e) {
-                    parseExceptions.add(e);
-                    //we continue so that we collect all parse errors and show them all at once
-                    continue;
-                }
-
-                Description testSuiteDescription = createTestSuiteDescription(restTestSuite);
-
-                if (restTestSuite.getTestSections().size() == 0) {
-                    assert restTestSuite.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion());
-                    testCandidates.add(RestTestCandidate.empty(restTestSuite, testSuiteDescription));
-                    continue;
-                }
-
-                Collections.shuffle(restTestSuite.getTestSections(), runnerRandomness.getRandom());
-
-                for (TestSection testSection : restTestSuite.getTestSections()) {
-
-                    if (sectionFilterPattern != null) {
-                        if (!sectionFilterPattern.matcher(testSection.getName()).find()) {
-                            continue;
-                        }
-                    }
-
-                    //no need to generate seed if we are going to skip the test section
-                    if (testSection.getSkipSection().skip(restTestExecutionContext.esVersion())) {
-                        Description testSectionDescription = createTestSectionIterationDescription(restTestSuite, testSection, null);
-                        testSuiteDescription.addChild(testSectionDescription);
-                        testCandidates.add(new RestTestCandidate(restTestSuite, testSuiteDescription, testSection, testSectionDescription, -1));
-                        continue;
-                    }
-
-                    Description parentDescription;
-                    if (hasRepetitions) {
-                        //additional level to group multiple iterations under the same test section's node
-                        parentDescription = createTestSectionWithRepetitionsDescription(restTestSuite, testSection);
-                        testSuiteDescription.addChild(parentDescription);
-                    } else {
-                        parentDescription = testSuiteDescription;
-                    }
-
-                    final long testSectionSeed = determineTestSectionSeed(restTestSuite.getDescription() + "/" + testSection.getName());
-                    for (int i = 0; i < iterations; i++) {
-                        //test section name argument needs to be unique here
-                        long thisSeed = (fixedSeed ? testSectionSeed : testSectionSeed ^ MurmurHash3.hash((long) i));
-
-                        final LinkedHashMap<String, Object> args = new LinkedHashMap<>();
-                        if (hasRepetitions) {
-                            args.put("#", i);
-                        }
-                        if (hasRepetitions || appendSeedParameter) {
-                            args.put("seed=", SeedUtils.formatSeedChain(runnerRandomness, new Randomness(thisSeed)));
-                        }
-
-                        Description testSectionDescription = createTestSectionIterationDescription(restTestSuite, testSection, args);
-                        parentDescription.addChild(testSectionDescription);
-                        testCandidates.add(new RestTestCandidate(restTestSuite, testSuiteDescription, testSection, testSectionDescription, thisSeed));
-                    }
-                }
-
-                //we add the suite only if it has at least a section left
-                if (testSuiteDescription.getChildren().size() > 0) {
-                    apiDescription.addChild(testSuiteDescription);
-                }
-            }
-
-            //we add the api only if it has at least a suite left
-            if (apiDescription.getChildren().size() > 0) {
-                rootDescription.addChild(apiDescription);
-            }
-        }
-
-        if (!parseExceptions.isEmpty()) {
-            throw new InitializationError(parseExceptions);
-        }
-
-        if (rootDescription.getChildren().size() == 0) {
-            throw new InitializationError("No tests to run");
-        }
-
-        return testCandidates;
-    }
-
-    protected String getRootSuiteTitle() {
-        if (runMode == RunMode.NO) {
-            return "elasticsearch REST Tests - not run";
-        }
-        if (runMode == RunMode.TEST_CLUSTER) {
-            return String.format(Locale.ROOT, "elasticsearch REST Tests - test cluster");
-        }
-        if (runMode == RunMode.EXTERNAL_CLUSTER) {
-            return String.format(Locale.ROOT, "elasticsearch REST Tests - external cluster %s", System.getProperty(REST_TESTS_MODE));
-        }
-        throw new UnsupportedOperationException("runMode [" + runMode + "] not supported");
-    }
-
-    private int determineTestSectionIterationCount() {
-        int iterations = RandomizedTest.systemPropertyAsInt(SYSPROP_ITERATIONS(), DEFAULT_ITERATIONS);
-        if (iterations < 1) {
-            throw new IllegalArgumentException("System property " + SYSPROP_ITERATIONS() + " must be >= 1 but was [" + iterations + "]");
-        }
-        return iterations;
-    }
-
-    protected static String[] resolvePathsProperty(String propertyName, String defaultValue) {
-        String property = System.getProperty(propertyName);
-        if (!Strings.hasLength(property)) {
-            return new String[]{defaultValue};
-        } else {
-            return property.split(PATHS_SEPARATOR);
-        }
-    }
-
-    /**
-     * Determine a given test section's initial random seed
-     */
-    private long determineTestSectionSeed(String testSectionName) {
-        if (testSectionRandomnessOverride != null) {
-            return getSeed(testSectionRandomnessOverride);
-        }
-
-        // We assign each test section a different starting hash based on the global seed
-        // and a hash of their name (so that the order of sections does not matter, only their names)
-        return getSeed(runnerRandomness) ^ MurmurHash3.hash((long) testSectionName.hashCode());
-    }
-
-    @Override
-    protected List<RestTestCandidate> getChildren() {
-        return restTestCandidates;
-    }
-
-    @Override
-    public Description getDescription() {
-        return rootDescription;
-    }
-
-    @Override
-    protected Description describeChild(RestTestCandidate child) {
-        return child.describeTest();
-    }
-
-    @Override
-    protected Statement classBlock(RunNotifier notifier) {
-        //we remove support for @BeforeClass & @AfterClass and JUnit Rules (as we don't call super)
-        Statement statement = childrenInvoker(notifier);
-        statement = withExecutionContextClose(statement);
-        if (testCluster != null) {
-            return withTestClusterClose(statement);
-        }
-        return statement;
-    }
-
-    protected Statement withExecutionContextClose(Statement statement) {
-        return new RunAfter(statement, new Statement() {
-            @Override
-            public void evaluate() throws Throwable {
-                restTestExecutionContext.close();
-            }
-        });
-    }
-
-    protected Statement withTestClusterClose(Statement statement) {
-        return new RunAfter(statement, new Statement() {
-            @Override
-            public void evaluate() throws Throwable {
-                stopTestCluster();
-            }
-        });
-    }
-
-    @Override
-    public void run(final RunNotifier notifier) {
-
-        if (runMode == RunMode.NO) {
-            notifier.fireTestIgnored(rootDescription.getChildren().get(0));
-            return;
-        }
-        final RestReproduceInfoPrinter restReproduceInfoPrinter = new RestReproduceInfoPrinter();
-        notifier.addListener(restReproduceInfoPrinter);
-        try {
-            //the test suite gets run on a separate thread as the randomized context is per thread
-            //once the randomized context is disposed it's not possible to create it again on the same thread
-            final Thread thread = new Thread() {
-                @Override
-                public void run() {
-                    try {
-                        createRandomizedContext(getTestClass().getJavaClass(), runnerRandomness);
-                        RestTestSuiteRunner.super.run(notifier);
-                    } finally {
-                        disposeRandomizedContext();
-                    }
-                }
-            };
-
-            thread.start();
-            try {
-                thread.join();
-            } catch (InterruptedException e) {
-                notifier.fireTestFailure(new Failure(getDescription(),
-                        new RuntimeException("Interrupted while waiting for the suite runner? Weird.", e)));
-            }
-        } finally {
-            // remove the listener once the suite is done otherwise it will print
-            // a bogus line if a subsequent test fails that is not a
-            // REST test. The RunNotifier is used across suites!
-            notifier.removeListener(restReproduceInfoPrinter);
-        }
-    }
-
-    @Override
-    protected void runChild(RestTestCandidate testCandidate, RunNotifier notifier) {
-
-        //if the while suite needs to be skipped, no test sections were loaded, only an empty one that we need to mark as ignored
-        if (testCandidate.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion())) {
-            if (logger.isInfoEnabled()) {
-                if (testCandidate.getSetupSection().getSkipSection().isVersionCheck()) {
-                    logger.info("skipped test suite [{}]\nreason: {}\nskip versions: {} (current version: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getSetupSection().getSkipSection().getReason(),
-                            testCandidate.getSetupSection().getSkipSection().getVersion(), restTestExecutionContext.esVersion());
-                } else {
-                    logger.info("skipped test suite [{}]\nreason: feature not supported\nrequired features: {} (supported features: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getSetupSection().getSkipSection().getFeatures(), Features.getSupported());
-                }
-            }
-            notifier.fireTestIgnored(testCandidate.describeSuite());
-            return;
-        }
-
-        //from now on no more empty test candidates are expected
-        assert testCandidate.getTestSection() != null;
-
-        if (testCandidate.getTestSection().getSkipSection().skip(restTestExecutionContext.esVersion())) {
-            if (logger.isInfoEnabled()) {
-                if (testCandidate.getTestSection().getSkipSection().isVersionCheck()) {
-                    logger.info("skipped test [{}/{}]\nreason: {}\nskip versions: {} (current version: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName(),
-                            testCandidate.getTestSection().getSkipSection().getReason(),
-                            testCandidate.getTestSection().getSkipSection().getVersion(), restTestExecutionContext.esVersion());
-                } else {
-                    logger.info("skipped test [{}/{}]\nreason: feature not supported\nrequired features: {} (supported features: {})",
-                            testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName(),
-                            testCandidate.getTestSection().getSkipSection().getFeatures(), Features.getSupported());
-                }
-            }
-
-            notifier.fireTestIgnored(testCandidate.describeTest());
-            return;
-        }
-
-        runLeaf(methodBlock(testCandidate), testCandidate.describeTest(), notifier);
-    }
-
-    protected Statement methodBlock(final RestTestCandidate testCandidate) {
-        return new Statement() {
-            @Override
-            public void evaluate() throws Throwable {
-                final String testThreadName = "TEST-" + testCandidate.getSuiteDescription() +
-                        "." + testCandidate.getTestSection().getName() + "-seed#" + SeedUtils.formatSeedChain(runnerRandomness);
-                // This has a side effect of setting up a nested context for the test thread.
-                final String restoreName = Thread.currentThread().getName();
-                try {
-                    Thread.currentThread().setName(testThreadName);
-                    pushRandomness(new Randomness(testCandidate.getSeed()));
-                    runTestSection(testCandidate);
-                } finally {
-                    Thread.currentThread().setName(restoreName);
-                    popAndDestroy();
-                }
-            }
-        };
-    }
-
-    protected void runTestSection(RestTestCandidate testCandidate)
-            throws IOException, RestException {
-
-        //let's check that there is something to run, otherwise there might be a problem with the test section
-        if (testCandidate.getTestSection().getExecutableSections().size() == 0) {
-            throw new IllegalArgumentException("No executable sections loaded for ["
-                    + testCandidate.getSuiteDescription() + "/" + testCandidate.getTestSection().getName() + "]");
-        }
-
-        logger.info("cleaning up before test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-        tearDown();
-
-        logger.info("start test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-
-        if (!testCandidate.getSetupSection().isEmpty()) {
-            logger.info("start setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-            for (DoSection doSection : testCandidate.getSetupSection().getDoSections()) {
-                doSection.execute(restTestExecutionContext);
-            }
-            logger.info("end setup test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-        }
-
-        restTestExecutionContext.clear();
-
-        for (ExecutableSection executableSection : testCandidate.getTestSection().getExecutableSections()) {
-            executableSection.execute(restTestExecutionContext);
-        }
-
-        logger.info("end test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-
-        logger.info("cleaning up after test [{}: {}]", testCandidate.getSuiteDescription(), testCandidate.getTestSection().getName());
-        tearDown();
-    }
-
-    private void tearDown() throws IOException, RestException {
-        wipeIndices();
-        wipeTemplates();
-        restTestExecutionContext.clear();
-    }
-
-    private void wipeIndices() throws IOException, RestException {
-        logger.debug("deleting all indices");
-        RestResponse restResponse = restTestExecutionContext.callApiInternal("indices.delete", "index", "_all");
-        assertThat(restResponse.getStatusCode(), equalTo(200));
-    }
-
-    @SuppressWarnings("unchecked")
-    public void wipeTemplates() throws IOException, RestException {
-        logger.debug("deleting all templates");
-        RestResponse restResponse = restTestExecutionContext.callApiInternal("indices.delete_template", "name", "*");
-        assertThat(restResponse.getStatusCode(), equalTo(200));
-    }
-
-    private void stopTestCluster() {
-        if (runMode == RunMode.TEST_CLUSTER) {
-            assert testCluster != null;
-            testCluster.afterTest();
-            testCluster.close();
-        }
-    }
-}

src/test/java/org/elasticsearch/test/rest/junit/RunAfter.java

@@ -1,56 +0,0 @@
-/*
- * Licensed to Elasticsearch under one or more contributor
- * license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright
- * ownership. Elasticsearch licenses this file to you under
- * the Apache License, Version 2.0 (the "License"); you may
- * not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.elasticsearch.test.rest.junit;
-
-import com.google.common.collect.Lists;
-import org.junit.runners.model.MultipleFailureException;
-import org.junit.runners.model.Statement;
-
-import java.util.List;
-
-/**
- * {@link Statement} that allows to run a specific statement after another one
- */
-public class RunAfter extends Statement {
-
-    private final Statement next;
-    private final Statement after;
-
-    public RunAfter(Statement next, Statement after) {
-        this.next = next;
-        this.after = after;
-    }
-
-    @Override
-    public void evaluate() throws Throwable {
-        List<Throwable> errors = Lists.newArrayList();
-        try {
-            next.evaluate();
-        } catch (Throwable e) {
-            errors.add(e);
-        } finally {
-            try {
-                after.evaluate();
-            } catch (Throwable e) {
-                errors.add(e);
-            }
-        }
-        MultipleFailureException.assertEmpty(errors);
-    }
-}

src/test/java/org/elasticsearch/test/rest/parser/RestTestSectionParser.java

@@ -25,9 +25,6 @@ import java.io.IOException;
 
 /**
  * Parser for a complete test section
- *
- * Depending on the elasticsearch version the tests are going to run against, test sections might need to get skipped
- * In that case the relevant test sections parsing is entirely skipped
  */
 public class RestTestSectionParser implements RestTestFragmentParser<TestSection> {
 
@@ -39,21 +36,9 @@ public class RestTestSectionParser implements RestTestFragmentParser<TestSection
         parser.nextToken();
         testSection.setSkipSection(parseContext.parseSkipSection());
 
-        boolean skip = testSection.getSkipSection().skip(parseContext.getCurrentVersion());
-
         while ( parser.currentToken() != XContentParser.Token.END_ARRAY) {
-            if (skip) {
-                //if there was a skip section, there was a setup section as well, which means that we are sure
-                // the current token is at the beginning of a new object
-                assert parser.currentToken() == XContentParser.Token.START_OBJECT;
-                //we need to be at the beginning of an object to be able to skip children
-                parser.skipChildren();
-                //after skipChildren we are at the end of the skipped object, need to move on
-                parser.nextToken();
-            } else {
-                parseContext.advanceToFieldName();
-                testSection.addExecutableSection(parseContext.parseExecutableSection());
-            }
+            parseContext.advanceToFieldName();
+            testSection.addExecutableSection(parseContext.parseExecutableSection());
         }
 
         parser.nextToken();

src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParseContext.java

@@ -53,13 +53,11 @@ public class RestTestSuiteParseContext {
     private final String api;
     private final String suiteName;
     private final XContentParser parser;
-    private final String currentVersion;
 
-    public RestTestSuiteParseContext(String api, String suiteName, XContentParser parser, String currentVersion) {
+    public RestTestSuiteParseContext(String api, String suiteName, XContentParser parser) {
         this.api = api;
         this.suiteName = suiteName;
         this.parser = parser;
-        this.currentVersion = currentVersion;
     }
 
     public String getApi() {
@@ -74,10 +72,6 @@ public class RestTestSuiteParseContext {
         return parser;
     }
 
-    public String getCurrentVersion() {
-        return currentVersion;
-    }
-
     public SetupSection parseSetupSection() throws IOException, RestTestParseException {
 
         advanceToFieldName();

src/test/java/org/elasticsearch/test/rest/parser/RestTestSuiteParser.java

@@ -31,13 +31,10 @@ import java.io.RandomAccessFile;
 
 /**
  * Parser for a complete test suite (yaml file)
- *
- * Depending on the elasticsearch version the tests are going to run against, a whole test suite might need to get skipped
- * In that case the relevant test sections parsing is entirely skipped
  */
 public class RestTestSuiteParser implements RestTestFragmentParser<RestTestSuite> {
 
-    public RestTestSuite parse(String currentVersion, String api, File file) throws IOException, RestTestParseException {
+    public RestTestSuite parse(String api, File file) throws IOException, RestTestParseException {
 
         if (!file.isFile()) {
             throw new IllegalArgumentException(file.getAbsolutePath() + " is not a file");
@@ -62,9 +59,14 @@ public class RestTestSuiteParser implements RestTestFragmentParser<RestTestSuite
             IOUtils.close(randomAccessFile);
         }
 
-        try (XContentParser parser = YamlXContent.yamlXContent.createParser(new FileInputStream(file))) {
-            RestTestSuiteParseContext testParseContext = new RestTestSuiteParseContext(api, filename, parser, currentVersion);
+        XContentParser parser = YamlXContent.yamlXContent.createParser(new FileInputStream(file));
+        try {
+            RestTestSuiteParseContext testParseContext = new RestTestSuiteParseContext(api, filename, parser);
             return parse(testParseContext);
+        } catch(Exception e) {
+            throw new RestTestParseException("Error parsing " + api + "/" + filename, e);
+        } finally {
+            parser.close();
         }
     }
 
@@ -79,8 +81,6 @@ public class RestTestSuiteParser implements RestTestFragmentParser<RestTestSuite
 
         restTestSuite.setSetupSection(parseContext.parseSetupSection());
 
-        boolean skip = restTestSuite.getSetupSection().getSkipSection().skip(parseContext.getCurrentVersion());
-
         while(true) {
             //the "---" section separator is not understood by the yaml parser. null is returned, same as when the parser is closed
             //we need to somehow distinguish between a null in the middle of a test ("---")
@@ -91,19 +91,9 @@ public class RestTestSuiteParser implements RestTestFragmentParser<RestTestSuite
                 }
             }
 
-            if (skip) {
-                //if there was a skip section, there was a setup section as well, which means that we are sure
-                // the current token is at the beginning of a new object
-                assert parser.currentToken() == XContentParser.Token.START_OBJECT;
-                //we need to be at the beginning of an object to be able to skip children
-                parser.skipChildren();
-                //after skipChildren we are at the end of the skipped object, need to move on
-                parser.nextToken();
-            } else {
-                TestSection testSection = parseContext.parseTestSection();
-                if (!restTestSuite.addTestSection(testSection)) {
-                    throw new RestTestParseException("duplicate test section [" + testSection.getName() + "] found in [" + restTestSuite.getDescription() + "]");
-                }
+            TestSection testSection = parseContext.parseTestSection();
+            if (!restTestSuite.addTestSection(testSection)) {
+                throw new RestTestParseException("duplicate test section [" + testSection.getName() + "] found in [" + restTestSuite.getDescription() + "]");
             }
         }
 

src/test/java/org/elasticsearch/test/rest/parser/SetupSectionParser.java

@@ -36,27 +36,15 @@ public class SetupSectionParser implements RestTestFragmentParser<SetupSection>
         SetupSection setupSection = new SetupSection();
         setupSection.setSkipSection(parseContext.parseSkipSection());
 
-        boolean skip = setupSection.getSkipSection().skip(parseContext.getCurrentVersion());
-
         while (parser.currentToken() != XContentParser.Token.END_ARRAY) {
-            if (skip) {
-                //if there was a skip section, there was a setup section as well, which means that we are sure
-                // the current token is at the beginning of a new object
-                assert parser.currentToken() == XContentParser.Token.START_OBJECT;
-                //we need to be at the beginning of an object to be able to skip children
-                parser.skipChildren();
-                //after skipChildren we are at the end of the skipped object, need to move on
-                parser.nextToken();
-            } else {
-                parseContext.advanceToFieldName();
-                if (!"do".equals(parser.currentName())) {
-                    throw new RestTestParseException("section [" + parser.currentName() + "] not supported within setup section");
-                }
-
-                parser.nextToken();
-                setupSection.addDoSection(parseContext.parseDoSection());
-                parser.nextToken();
+            parseContext.advanceToFieldName();
+            if (!"do".equals(parser.currentName())) {
+                throw new RestTestParseException("section [" + parser.currentName() + "] not supported within setup section");
             }
+
+            parser.nextToken();
+            setupSection.addDoSection(parseContext.parseDoSection());
+            parser.nextToken();
         }
 
         parser.nextToken();

src/test/java/org/elasticsearch/test/rest/section/RestTestSuite.java

@@ -51,7 +51,7 @@ public class RestTestSuite {
     }
 
     //describes the rest test suite (e.g. index/10_with_id)
-    //useful also to reproduce failures (RestReproduceInfoPrinter)
+    //useful also to reproduce failures
     public String getDescription() {
         return api + "/" + name;
     }

src/test/java/org/elasticsearch/test/rest/test/AssertionParsersTests.java

@@ -37,7 +37,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         IsTrueParser isTrueParser = new IsTrueParser();
-        IsTrueAssertion trueAssertion = isTrueParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        IsTrueAssertion trueAssertion = isTrueParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(trueAssertion, notNullValue());
         assertThat(trueAssertion.getField(), equalTo("get.fields._timestamp"));
@@ -50,7 +50,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         IsFalseParser isFalseParser = new IsFalseParser();
-        IsFalseAssertion falseAssertion = isFalseParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        IsFalseAssertion falseAssertion = isFalseParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(falseAssertion, notNullValue());
         assertThat(falseAssertion.getField(), equalTo("docs.1._source"));
@@ -63,7 +63,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         GreaterThanParser greaterThanParser = new GreaterThanParser();
-        GreaterThanAssertion greaterThanAssertion = greaterThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        GreaterThanAssertion greaterThanAssertion = greaterThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(greaterThanAssertion, notNullValue());
         assertThat(greaterThanAssertion.getField(), equalTo("field"));
         assertThat(greaterThanAssertion.getExpectedValue(), instanceOf(Integer.class));
@@ -77,7 +77,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         LessThanParser lessThanParser = new LessThanParser();
-        LessThanAssertion lessThanAssertion = lessThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        LessThanAssertion lessThanAssertion = lessThanParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(lessThanAssertion, notNullValue());
         assertThat(lessThanAssertion.getField(), equalTo("field"));
         assertThat(lessThanAssertion.getExpectedValue(), instanceOf(Integer.class));
@@ -91,7 +91,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         LengthParser lengthParser = new LengthParser();
-        LengthAssertion lengthAssertion = lengthParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        LengthAssertion lengthAssertion = lengthParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(lengthAssertion, notNullValue());
         assertThat(lengthAssertion.getField(), equalTo("_id"));
         assertThat(lengthAssertion.getExpectedValue(), instanceOf(Integer.class));
@@ -106,7 +106,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("field"));
@@ -122,7 +122,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("foo"));
@@ -138,7 +138,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("matches"));
@@ -157,7 +157,7 @@ public class AssertionParsersTests extends AbstractParserTests {
         );
 
         MatchParser matchParser = new MatchParser();
-        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        MatchAssertion matchAssertion = matchParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(matchAssertion, notNullValue());
         assertThat(matchAssertion.getField(), equalTo("_source"));

src/test/java/org/elasticsearch/test/rest/test/DoSectionParserTests.java

@@ -47,7 +47,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -66,7 +66,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -87,7 +87,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -120,7 +120,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -147,7 +147,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -172,7 +172,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         String body = "{ \"_source\": [ \"include.field1\", \"include.field2\" ], \"query\": { \"match_all\": {} }}";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -209,7 +209,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         bodies[3] = "{ \"f1\":\"v2\", \"f2\": 47 }";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -243,7 +243,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         bodies[1] = "{ \"f1\":\"v1\", \"f2\": 42 }";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -273,7 +273,7 @@ public class DoSectionParserTests extends AbstractParserTests {
                 "]}";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -295,7 +295,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection, notNullValue());
@@ -322,7 +322,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         String body = "{ \"size\": 100, \"query\": { \"match_all\": {} } }";
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         ApiCallSection apiCallSection = doSection.getApiCallSection();
 
         assertThat(apiCallSection.getApi(), equalTo("index"));
@@ -344,7 +344,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(doSection.getCatch(), equalTo("missing"));
         assertThat(doSection.getApiCallSection(), notNullValue());
@@ -360,7 +360,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 
     @Test
@@ -373,7 +373,7 @@ public class DoSectionParserTests extends AbstractParserTests {
         );
 
         DoSectionParser doSectionParser = new DoSectionParser();
-        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        DoSection doSection = doSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(doSection.getCatch(), nullValue());
         assertThat(doSection.getApiCallSection(), notNullValue());

src/test/java/org/elasticsearch/test/rest/test/RestTestParserTests.java

@@ -84,7 +84,7 @@ public class RestTestParserTests extends ElasticsearchTestCase {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(restTestSuite, notNullValue());
         assertThat(restTestSuite.getName(), equalTo("suite"));
@@ -136,179 +136,6 @@ public class RestTestParserTests extends ElasticsearchTestCase {
         assertThat(matchAssertion.getExpectedValue().toString(), equalTo("whitespace"));
     }
 
-    @Test
-    public void testParseTestSetupAndSectionsSkipLastSection() throws Exception {
-        parser = YamlXContent.yamlXContent.createParser(
-                "setup:\n" +
-                        "  - do:\n" +
-                        "        indices.create:\n" +
-                        "          index: test_index\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get index mapping\":\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "\n" +
-                        "  - match: {test_index.test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_index.test_type.properties.text.analyzer: whitespace}\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get type mapping - pre 1.0\":\n" +
-                        "\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.9 - 999\"\n" +
-                        "      reason:      \"for newer versions the index name is always returned\"\n" +
-                        "\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "        type: test_type\n" +
-                        "\n" +
-                        "  - match: {test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_type.properties.text.analyzer: whitespace}\n"
-        );
-
-        RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "1.0.0"));
-
-        assertThat(restTestSuite, notNullValue());
-        assertThat(restTestSuite.getName(), equalTo("suite"));
-        assertThat(restTestSuite.getSetupSection(), notNullValue());
-        assertThat(restTestSuite.getSetupSection().getSkipSection().isEmpty(), equalTo(true));
-
-        assertThat(restTestSuite.getSetupSection().getDoSections().size(), equalTo(1));
-        assertThat(restTestSuite.getSetupSection().getDoSections().get(0).getApiCallSection().getApi(), equalTo("indices.create"));
-        assertThat(restTestSuite.getSetupSection().getDoSections().get(0).getApiCallSection().getParams().size(), equalTo(1));
-        assertThat(restTestSuite.getSetupSection().getDoSections().get(0).getApiCallSection().getParams().get("index"), equalTo("test_index"));
-
-        assertThat(restTestSuite.getTestSections().size(), equalTo(2));
-
-        assertThat(restTestSuite.getTestSections().get(0).getName(), equalTo("Get index mapping"));
-        assertThat(restTestSuite.getTestSections().get(0).getSkipSection().isEmpty(), equalTo(true));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().size(), equalTo(3));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().get(0), instanceOf(DoSection.class));
-        DoSection doSection = (DoSection) restTestSuite.getTestSections().get(0).getExecutableSections().get(0);
-        assertThat(doSection.getApiCallSection().getApi(), equalTo("indices.get_mapping"));
-        assertThat(doSection.getApiCallSection().getParams().size(), equalTo(1));
-        assertThat(doSection.getApiCallSection().getParams().get("index"), equalTo("test_index"));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().get(1), instanceOf(MatchAssertion.class));
-        MatchAssertion matchAssertion = (MatchAssertion) restTestSuite.getTestSections().get(0).getExecutableSections().get(1);
-        assertThat(matchAssertion.getField(), equalTo("test_index.test_type.properties.text.type"));
-        assertThat(matchAssertion.getExpectedValue().toString(), equalTo("string"));
-        assertThat(restTestSuite.getTestSections().get(0).getExecutableSections().get(2), instanceOf(MatchAssertion.class));
-        matchAssertion = (MatchAssertion) restTestSuite.getTestSections().get(0).getExecutableSections().get(2);
-        assertThat(matchAssertion.getField(), equalTo("test_index.test_type.properties.text.analyzer"));
-        assertThat(matchAssertion.getExpectedValue().toString(), equalTo("whitespace"));
-
-        assertThat(restTestSuite.getTestSections().get(1).getName(), equalTo("Get type mapping - pre 1.0"));
-        assertThat(restTestSuite.getTestSections().get(1).getSkipSection().isEmpty(), equalTo(false));
-        assertThat(restTestSuite.getTestSections().get(1).getSkipSection().getReason(), equalTo("for newer versions the index name is always returned"));
-        assertThat(restTestSuite.getTestSections().get(1).getSkipSection().getVersion(), equalTo("0.90.9 - 999"));
-        assertThat(restTestSuite.getTestSections().get(1).getExecutableSections().size(), equalTo(0));
-    }
-
-    @Test
-    public void testParseTestSetupAndSectionsSkipEntireFile() throws Exception {
-        parser = YamlXContent.yamlXContent.createParser(
-                "setup:\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.3 - 0.90.6\"\n" +
-                        "      reason:      \"test skip entire file\"\n" +
-                        "  - do:\n" +
-                        "        indices.create:\n" +
-                        "          index: test_index\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get index mapping\":\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "\n" +
-                        "  - match: {test_index.test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_index.test_type.properties.text.analyzer: whitespace}\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get type mapping - pre 1.0\":\n" +
-                        "\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.9 - 999\"\n" +
-                        "      reason:      \"for newer versions the index name is always returned\"\n" +
-                        "\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "        type: test_type\n" +
-                        "\n" +
-                        "  - match: {test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_type.properties.text.analyzer: whitespace}\n"
-        );
-
-        RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
-
-        assertThat(restTestSuite, notNullValue());
-        assertThat(restTestSuite.getName(), equalTo("suite"));
-        assertThat(restTestSuite.getSetupSection(), notNullValue());
-
-        assertThat(restTestSuite.getSetupSection().getSkipSection().isEmpty(), equalTo(false));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getVersion(), equalTo("0.90.3 - 0.90.6"));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getReason(), equalTo("test skip entire file"));
-
-        assertThat(restTestSuite.getSetupSection().getDoSections().size(), equalTo(0));
-
-        assertThat(restTestSuite.getTestSections().size(), equalTo(0));
-    }
-
-    @Test
-    public void testParseTestSetupAndSectionsSkipEntireFileNoDo() throws Exception {
-        parser = YamlXContent.yamlXContent.createParser(
-                "setup:\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.3 - 0.90.6\"\n" +
-                        "      reason:      \"test skip entire file\"\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get index mapping\":\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "\n" +
-                        "  - match: {test_index.test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_index.test_type.properties.text.analyzer: whitespace}\n" +
-                        "\n" +
-                        "---\n" +
-                        "\"Get type mapping - pre 1.0\":\n" +
-                        "\n" +
-                        "  - skip:\n" +
-                        "      version:     \"0.90.9 - 999\"\n" +
-                        "      reason:      \"for newer versions the index name is always returned\"\n" +
-                        "\n" +
-                        "  - do:\n" +
-                        "      indices.get_mapping:\n" +
-                        "        index: test_index\n" +
-                        "        type: test_type\n" +
-                        "\n" +
-                        "  - match: {test_type.properties.text.type:     string}\n" +
-                        "  - match: {test_type.properties.text.analyzer: whitespace}\n"
-        );
-
-        RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
-
-        assertThat(restTestSuite, notNullValue());
-        assertThat(restTestSuite.getName(), equalTo("suite"));
-        assertThat(restTestSuite.getSetupSection(), notNullValue());
-
-        assertThat(restTestSuite.getSetupSection().getSkipSection().isEmpty(), equalTo(false));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getVersion(), equalTo("0.90.3 - 0.90.6"));
-        assertThat(restTestSuite.getSetupSection().getSkipSection().getReason(), equalTo("test skip entire file"));
-
-        assertThat(restTestSuite.getSetupSection().getDoSections().size(), equalTo(0));
-
-        assertThat(restTestSuite.getTestSections().size(), equalTo(0));
-    }
-
     @Test
     public void testParseTestSingleTestSection() throws Exception {
         parser = YamlXContent.yamlXContent.createParser(
@@ -342,7 +169,7 @@ public class RestTestParserTests extends ElasticsearchTestCase {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(restTestSuite, notNullValue());
         assertThat(restTestSuite.getName(), equalTo("suite"));
@@ -456,7 +283,7 @@ public class RestTestParserTests extends ElasticsearchTestCase {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        RestTestSuite restTestSuite = testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(restTestSuite, notNullValue());
         assertThat(restTestSuite.getName(), equalTo("suite"));
@@ -531,6 +358,6 @@ public class RestTestParserTests extends ElasticsearchTestCase {
         );
 
         RestTestSuiteParser testParser = new RestTestSuiteParser();
-        testParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        testParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 }

src/test/java/org/elasticsearch/test/rest/test/SetSectionParserTests.java

@@ -38,7 +38,7 @@ public class SetSectionParserTests extends AbstractParserTests {
 
         SetSectionParser setSectionParser = new SetSectionParser();
 
-        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setSection, notNullValue());
         assertThat(setSection.getStash(), notNullValue());
@@ -54,7 +54,7 @@ public class SetSectionParserTests extends AbstractParserTests {
 
         SetSectionParser setSectionParser = new SetSectionParser();
 
-        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SetSection setSection = setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setSection, notNullValue());
         assertThat(setSection.getStash(), notNullValue());
@@ -72,6 +72,6 @@ public class SetSectionParserTests extends AbstractParserTests {
 
         SetSectionParser setSectionParser = new SetSectionParser();
 
-        setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        setSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 }

src/test/java/org/elasticsearch/test/rest/test/SetupSectionParserTests.java

@@ -48,7 +48,7 @@ public class SetupSectionParserTests extends AbstractParserTests {
         );
 
         SetupSectionParser setupSectionParser = new SetupSectionParser();
-        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setupSection, notNullValue());
         assertThat(setupSection.getSkipSection().isEmpty(), equalTo(true));
@@ -57,38 +57,6 @@ public class SetupSectionParserTests extends AbstractParserTests {
         assertThat(setupSection.getDoSections().get(1).getApiCallSection().getApi(), equalTo("index2"));
     }
 
-    @Test
-    public void testParseSetupAndSkipSectionSkip() throws Exception {
-
-        parser = YamlXContent.yamlXContent.createParser(
-                "  - skip:\n" +
-                "      version:  \"0.90.0 - 0.90.7\"\n" +
-                "      reason:   \"Update doesn't return metadata fields, waiting for #3259\"\n" +
-                "  - do:\n" +
-                "      index1:\n" +
-                "        index:  test_1\n" +
-                "        type:   test\n" +
-                "        id:     1\n" +
-                "        body:   { \"include\": { \"field1\": \"v1\", \"field2\": \"v2\" }, \"count\": 1 }\n" +
-                "  - do:\n" +
-                "      index2:\n" +
-                "        index:  test_1\n" +
-                "        type:   test\n" +
-                "        id:     2\n" +
-                "        body:   { \"include\": { \"field1\": \"v1\", \"field2\": \"v2\" }, \"count\": 1 }\n"
-        );
-
-        SetupSectionParser setupSectionParser = new SetupSectionParser();
-        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
-
-        assertThat(setupSection, notNullValue());
-        assertThat(setupSection.getSkipSection().isEmpty(), equalTo(false));
-        assertThat(setupSection.getSkipSection(), notNullValue());
-        assertThat(setupSection.getSkipSection().getVersion(), equalTo("0.90.0 - 0.90.7"));
-        assertThat(setupSection.getSkipSection().getReason(), equalTo("Update doesn't return metadata fields, waiting for #3259"));
-        assertThat(setupSection.getDoSections().size(), equalTo(0));
-    }
-
     @Test
     public void testParseSetupAndSkipSectionNoSkip() throws Exception {
 
@@ -111,7 +79,7 @@ public class SetupSectionParserTests extends AbstractParserTests {
         );
 
         SetupSectionParser setupSectionParser = new SetupSectionParser();
-        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.8"));
+        SetupSection setupSection = setupSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(setupSection, notNullValue());
         assertThat(setupSection.getSkipSection().isEmpty(), equalTo(false));

src/test/java/org/elasticsearch/test/rest/test/SkipSectionParserTests.java

@@ -38,7 +38,7 @@ public class SkipSectionParserTests extends AbstractParserTests {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(skipSection, notNullValue());
         assertThat(skipSection.getVersion(), equalTo("0 - 0.90.2"));
@@ -54,7 +54,7 @@ public class SkipSectionParserTests extends AbstractParserTests {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(skipSection, notNullValue());
         assertThat(skipSection.getVersion(), nullValue());
@@ -71,7 +71,7 @@ public class SkipSectionParserTests extends AbstractParserTests {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        SkipSection skipSection = skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(skipSection, notNullValue());
         assertThat(skipSection.getVersion(), nullValue());
@@ -92,7 +92,7 @@ public class SkipSectionParserTests extends AbstractParserTests {
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
 
-        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 
     @Test(expected = RestTestParseException.class)
@@ -102,7 +102,7 @@ public class SkipSectionParserTests extends AbstractParserTests {
         );
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
-        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 
     @Test(expected = RestTestParseException.class)
@@ -112,6 +112,6 @@ public class SkipSectionParserTests extends AbstractParserTests {
         );
 
         SkipSectionParser skipSectionParser = new SkipSectionParser();
-        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        skipSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
     }
 }

src/test/java/org/elasticsearch/test/rest/test/TestSectionParserTests.java

@@ -42,7 +42,7 @@ public class TestSectionParserTests extends AbstractParserTests {
         );
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("First test section"));
@@ -56,34 +56,6 @@ public class TestSectionParserTests extends AbstractParserTests {
         assertThat(doSection.getApiCallSection().hasBody(), equalTo(false));
     }
 
-    @Test
-    public void testParseTestSectionWithDoSetAndSkipSectionsSkip() throws Exception {
-        String yaml =
-                "\"First test section\": \n" +
-                "  - skip:\n" +
-                "      version:  \"0.90.0 - 0.90.7\"\n" +
-                "      reason:   \"Update doesn't return metadata fields, waiting for #3259\"\n" +
-                "  - do :\n" +
-                "      catch: missing\n" +
-                "      indices.get_warmer:\n" +
-                "          index: test_index\n" +
-                "          name: test_warmer\n" +
-                "  - set: {_scroll_id: scroll_id}";
-
-
-        RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        parser = YamlXContent.yamlXContent.createParser(yaml);
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.7"));
-
-        assertThat(testSection, notNullValue());
-        assertThat(testSection.getName(), equalTo("First test section"));
-        assertThat(testSection.getSkipSection(), notNullValue());
-        assertThat(testSection.getSkipSection().getVersion(), equalTo("0.90.0 - 0.90.7"));
-        assertThat(testSection.getSkipSection().getReason(), equalTo("Update doesn't return metadata fields, waiting for #3259"));
-        //skip parsing when needed
-        assertThat(testSection.getExecutableSections().size(), equalTo(0));
-    }
-
     @Test
     public void testParseTestSectionWithDoSetAndSkipSectionsNoSkip() throws Exception {
         String yaml =
@@ -101,7 +73,7 @@ public class TestSectionParserTests extends AbstractParserTests {
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
         parser = YamlXContent.yamlXContent.createParser(yaml);
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.8"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("First test section"));
@@ -139,7 +111,7 @@ public class TestSectionParserTests extends AbstractParserTests {
         );
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("Basic"));
@@ -193,7 +165,7 @@ public class TestSectionParserTests extends AbstractParserTests {
         );
 
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
 
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("Basic"));
@@ -265,7 +237,7 @@ public class TestSectionParserTests extends AbstractParserTests {
                 "  - is_true: nodes\n" +
                 "  - is_true: cluster_name\n");
         RestTestSectionParser testSectionParser = new RestTestSectionParser();
-        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser, "0.90.5"));
+        TestSection testSection = testSectionParser.parse(new RestTestSuiteParseContext("api", "suite", parser));
         assertThat(testSection, notNullValue());
         assertThat(testSection.getName(), equalTo("node_info test"));
         assertThat(testSection.getExecutableSections().size(), equalTo(3));