Update README for YAML Rest API specs (#107837)

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/README.asciidoc

@@ -87,7 +87,108 @@ in the `indices.get_settings` API.
 
 == Skipping tests:
 
-=== Skip for Elasticsearch versions
+If a test section should only be run for certain releases of Elasticsearch,
+then the first entry in the section (after the title) should be called
+`requires` and / or `skip` depending on the use case.
+
+A `requires` section defines requirements that have to be met in order for tests to run, such as:
+
+- `cluster_features` to <<cluster_features, require cluster features>>.
+- `test_runner_features` to <<requires_test_runner_features, require test runner features>>.
+
+A `requires` section must specify at least one of the two options above.
+If `cluster_features` are required, a `reason` must be provided in addition.
+
+A `skip` section, on the other hand, defines certain conditions that, if met, will skip the test, such as:
+
+- `cluster_features` to <<cluster_features, skip if cluster features are present>>.
+- `known_issues` to <<skip_known_issues, skip on known issues (based on cluster features)>>.
+- `awaits_fix` to <<skip_awaits_fix, always skip / mute a test due to a pending fix>>.
+- `os` to <<skip_os, skip on operating systems>>.
+- `features`: Only kept for a transition period, please use <<requires_test_runner_features, `test_runner_features`>>
+  in the `requires` section instead.
+
+A `skip` section must specify at least one of the options above.
+Unless only legacy test runner `features` are required, a `reason` must also be provided.
+
+`requires` and / or `skip` can also be used at the top level of the file in the `setup` and `teardown` blocks,
+so all the tests in a file will be skipped if either any requirement fails or any skip condition applies regardless
+if defined in `setup` and `teardown`.
+
+[[cluster_features]]
+=== Require or skip cluster features
+
+In the past, Elasticsearch has been over-reliant on release version checks all across its code base.
+This has become a limiting factor when deploying Elasticsearch in a more continuous fashion.
+
+Moving forward, the goal is to base any such check on the availability of a cluster feature.
+Rather than <<skip_version, skipping tests based on various release version ranges>>, you can
+require `cluster_features` to be either present (`requires`) and / or absent (`skip`) in order to run a test.
+For instance:
+
+....
+    "Parent":
+     - requires:
+          cluster_features: feature_x
+          reason:           Feature X was introduced
+     - skip:
+          cluster_features: feature_x_changed
+          reason:           Change to feature X breaks this test
+
+     - do:
+       ... test definitions ...
+....
+
+The `cluster_features` field can either be a string or an array of strings.
+
+[[synthetic_cluster_features]]
+Note: In order to smoothen the transition from version checks to cluster feature checks, a REST-test specific
+synthetic cluster feature named `gte_v{VERSION}` is available for all release versions until including 8.14.0.
+For instance, `gte_v8.12.2` would be available for all release versions greater than or equal to 8.12.2.
+
+[[skip_known_issues]]
+=== Skip on known issues
+
+Previously, it was possible to skip ranges of broken release versions using <<skip_version,`version`>>.
+`known_issues` provides a more explicit way to express and skip a certain range of buggy releases based on cluster features.
+Each of possibly multiple issues is a pair of `cluster_feature` and `fixed_by`, where an issue was
+introduced by the former feature and eventually fixed by the latter one. For instance:
+
+....
+    "Parent":
+     - skip:
+          known_issues:
+            - cluster_feature: feature_y
+              fixed_by:        feature_y_fix
+            - cluster_feature: feature_z
+              fixed_by:        feature_z_fix
+          reason: Skipped for buggy feature_y until fixed by feature_y_fix and feature_z until fixed by feature_z_fix
+
+     - do:
+       ... test definitions ...
+....
+
+Note: If a known issue cannot be defined in terms of existing cluster features, the previously described
+<<synthetic_cluster_features,synthetic version based cluster features>> can be used.
+
+[[skip_awaits_fix]]
+=== Skip while awaiting fix
+
+In certain cases there's no fix available yet. In order to mute a test, use `awaits_fix` with the corresponding ticket / issue.
+
+For instance:
+....
+    "Parent":
+     - skip:
+          awaits_fix: https://github.com/elastic/elasticsearch/issues/xyz
+          reason:     Muted due to #xyz
+
+     - do:
+       ... test definitions ...
+....
+
+[[skip_version]]
+=== Skip for Elasticsearch versions (deprecated)
 
 If a test section should only be run on certain versions of Elasticsearch,
 then the first entry in the section (after the title) should be called
@@ -135,48 +236,11 @@ The value for version can also be `all`, to skip in any version of
 Elasticsearch. This can be used for example when a feature is being implemented
 or awaiting a fix.
 
-`skip` can also be used at the top level of the file in the `setup` and `teardown` blocks,
-so all the tests in a file will be skipped if the condition applies.
-A particular test is skipped if any of the skip conditions for the test,
-the setup or the teardown apply.
-This can have a similar effect to the multi-range support described above
-in that we can specify tests that only run within a specific range.
-For example, if a new feature was introduced in 8.1.0, we could create a test file
-with the `setup` block containing a `skip.version` of `" - 8.0.99"`, causing all tests
-to be skipped for earlier versions. Then specific tests that are added later could
-add to this by either:
-
-* increasing the upper bound for positive tests (test new enhancement works):
-`skip.version: " - 8.6.99"`
-* or creating an additional lower bound for negative tests
-(test that exception is thrown for older versions, as in multi-range example above):
-`skip.version: "8.8.0 - "`
-
-=== Skip on missing runner features
-
-The skip section can also be used to list new features that need to be
-supported in order to run a test. This way the up-to-date runners will
-run the test, while the ones that don't support the feature yet can
-temporarily skip it, and avoid having lots of test failures in the meantime.
-Once all runners have implemented the feature, it can be declared supported
-by default, thus the related skip sections can be removed from the tests.
-
-The skip section can also be used to selectively mute tests in certain
-cases where they would otherwise fail, see `default_shards` and `fips_140`.
+[[skip_os]]
+=== Skip on certain operating systems
 
-....
-    "Parent":
-     - skip:
-          features:    regex
-
-     - do:
-       ... test definitions ...
-....
-
-The `features` field can either be a string or an array of strings.
-
-The skip section can also be used to mute tests for certain operating systems.
-This way it is not necessary to mute the whole test if a operating system
+The `skip` section can also be used to mute tests for certain operating systems.
+This way it is not necessary to mute the whole test if an operating system
 specific problem appears.
 
 The operating system is taken from the pretty name that elasticsearch reports
@@ -185,13 +249,14 @@ for:
 
 `initializing client, minimum es version`
 
-When muting by operating system, a reason is mandatory and features must contain
-skip_os:
+When muting by operating system, a `reason` is mandatory and `skip_os` must be defined as requirement in
+`test_runner_features` (see below).
 
 ....
     "Parent":
+     - requires:
+          test_runner_features: skip_os
      - skip:
-          features: skip_os
           os:       debian-8
           reason:   memory accounting problems on debian 8, see gh#xyz
 
@@ -201,37 +266,63 @@ skip_os:
 
 The `os` field can either be a string or an array of strings.
 
-The skip section requires to specify either a `version`, `features` or `os` list.
+[[requires_test_runner_features]]
+=== Require specific test runner features
+
+The `requires` section can also be used to list test runner features that need to be
+supported by the runner in order to execute a test. This way the up-to-date runners will
+run the test, while the ones that don't support the feature yet can
+temporarily skip it, and avoid having lots of test failures in the meantime.
+Once all runners have implemented the feature, it can be declared supported
+by default, thus the related `requires` sections can be removed from the tests.
+
+The `requires` section can also be used to selectively mute tests in certain
+cases where they would otherwise fail, see `default_shards` and `fips_140`.
+
+....
+    "Parent":
+     - requires:
+          test_runner_features:    regex
+
+     - do:
+       ... test definitions ...
+....
+
+The `test_runner_features` field can either be a string or an array of strings.
+
+Note:
+Tests that are still using `features` in the `skip` sections should be migrated to
+`test_runner_features` to avoid confusion with recently added cluster features.
 
-=== Available Features
+==== Available test runner features
 
-==== `xpack`
+===== `xpack`
 Requires x-pack to be enabled on the `Elasticsearch` instance the rest test is running against
 
-==== `no_xpack`
+===== `no_xpack`
 Requires the test to run against an oss distribution of `Elasticsearch`
 
-==== `catch_unauthorized`
+===== `catch_unauthorized`
 
 Runner supports `catch: unauthorized` on a `do` operator.
 
-==== `default_shards`
+===== `default_shards`
 
 This test can only run if the cluster is running with the distributions default number of shards.
 
 The Java test runner introduces randomness and sometimes overrides the default number of shards to `2`.
 If the default number of shards is changed, test marked with this feature should *not* run
 
-==== `headers`
+===== `headers`
 
 The runner is able to set per request headers on the `do` operation
 
-==== `node_selector`
+===== `node_selector`
 
 Indicates the runner can parse `node_selector` under the `do` operator and use its metadata to select the node to
 perform the `do` operation on.
 
-==== `stash_in_key`
+===== `stash_in_key`
 
 Allows you to use a stashed value in any key of an object during a `match` assertion
 
@@ -248,7 +339,7 @@ Allows you to use a stashed value in any key of an object during a `match` asser
      }
 ....
 
-==== `stash_in_path`
+===== `stash_in_path`
 
 Allows a stashed value to be referenced in path lookups as a single token. E.g:
 
@@ -256,7 +347,7 @@ Allows a stashed value to be referenced in path lookups as a single token. E.g:
 path.$stash.value
 ....
 
-==== `embedded_stash_key`
+===== `embedded_stash_key`
 
 Allows a stashed key to appear anywhere in the path (note the placeholder needs to be within curly brackets too in this case):
 
@@ -264,7 +355,7 @@ Allows a stashed key to appear anywhere in the path (note the placeholder needs
 field1.e${placeholder}ments.element1
 ....
 
-==== `stash_path_replace`
+===== `stash_path_replace`
 Used only in the doc snippet tests. Allow you to do ease replacements using a special `$_path` marker.
 
 ....
@@ -272,30 +363,30 @@ Used only in the doc snippet tests. Allow you to do ease replacements using a sp
 somevalue with whatever is the response in the same position."
 ....
 
-==== `warnings`
+===== `warnings`
 
 The runner can assert specific warnings headers are returned by Elasticsearch through the `warning:` assertations
 under `do:`  operations. The test will fail if the warning is not found.
 
-==== `warnings_regex`
+===== `warnings_regex`
 
 The same as `warnings`, but matches warning headers with the given regular expression.
 
 
-==== `allowed_warnings`
+===== `allowed_warnings`
 
 The runner will allow specific warnings headers to be returned by Elasticsearch through the `allowed_warning:` assertations
 under `do:`  operations. The test will not fail if the warning is not found.
 
-==== `allowed_warnings_regex`
+===== `allowed_warnings_regex`
 
 The same as `allowed_warnings`, but matches warning headers with the given regular expression.
 
-==== `yaml`
+===== `yaml`
 
 The runner is able to send and receive `application/yaml` and perform all assertions on the returned data.
 
-==== `contains`
+===== `contains`
 
 Asserts an array of object contains an object with a property set to a certain value. e.g:
 
@@ -310,11 +401,11 @@ Alternatively, this can be used to assert that a string response contains a cert
 ...
 contains: { items.0.index.error.reason: "must be mapped" }
 
-==== `transform_and_set`
+===== `transform_and_set`
 
 Supports the `transform_and_set` operator as described in this document.
 
-==== `arbitrary_key`
+===== `arbitrary_key`
 
 Allows you to stash an arbitrary key from a returned map e.g:
 
@@ -325,7 +416,7 @@ Allows you to stash an arbitrary key from a returned map e.g:
 
 This means: Stash any of the keys returned under `nodes` as `$node_id`
 
-==== `fips_140`
+===== `fips_140`
 
 This test should not be run when the test cluster is set in FIPS 140 mode.