Introduce repository test kit/analyser (#67247)

Today we rely on blob stores behaving in a certain way so that they can be used
as a snapshot repository. There are an increasing number of third-party blob
stores that claim to be S3-compatible, but which may not offer a suitably
correct or performant implementation of the S3 API. We rely on somesubtle
semantics with concurrent readers and writers, but some blob stores may not
implement it correctly. Hitting a corner case in the implementation may be rare
in normal use, and may be hard to reproduce or to distinguish from an
Elasticsearch bug.

This commit introduces a new `POST /_snapshot/.../_analyse` API which exercises
the more problematic corners of the repository implementation looking for
correctness bugs and measures the details of the performance of the repository
under concurrent load.

docs/reference/snapshot-restore/apis/repo-analysis-api.asciidoc

@@ -0,0 +1,479 @@
+[role="xpack"]
+[[repo-analysis-api]]
+=== Repository analysis API
+++++
+<titleabbrev>Repository analysis</titleabbrev>
+++++
+
+Analyzes a repository, reporting its performance characteristics and any
+incorrect behaviour found.
+
+////
+[source,console]
+----
+PUT /_snapshot/my_repository
+{
+  "type": "fs",
+  "settings": {
+    "location": "my_backup_location"
+  }
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+POST /_snapshot/my_repository/_analyze?blob_count=10&max_blob_size=1mb&timeout=120s
+----
+
+[[repo-analysis-api-request]]
+==== {api-request-title}
+
+`POST /_snapshot/<repository>/_analyze`
+
+[[repo-analysis-api-desc]]
+==== {api-description-title}
+
+There are a large number of third-party storage systems available, not all of
+which are suitable for use as a snapshot repository by {es}. Some storage
+systems behave incorrectly, or perform poorly, especially when accessed
+concurrently by multiple clients as the nodes of an {es} cluster do.
+
+The Repository analysis API performs a collection of read and write operations
+on your repository which are designed to detect incorrect behaviour and to
+measure the performance characteristics of your storage system.
+
+The default values for the parameters to this API are deliberately low to
+reduce the impact of running an analysis. A realistic experiment should set
+`blob_count` to at least `2000`, `max_blob_size` to at least `2gb`, and
+`max_total_data_size` to at least `1tb`, and will almost certainly need to
+increase the `timeout` to allow time for the process to complete successfully.
+You should run the analysis on a multi-node cluster of a similar size to your
+production cluster so that it can detect any problems that only arise when the
+repository is accessed by many nodes at once.
+
+If the analysis is successful this API returns details of the testing process,
+optionally including how long each operation took. You can use this information
+to determine the performance of your storage system. If any operation fails or
+returns an incorrect result, this API returns an error. If the API returns an
+error then it may not have removed all the data it wrote to the repository. The
+error will indicate the location of any leftover data, and this path is also
+recorded in the {es} logs. You should verify yourself that this location has
+been cleaned up correctly. If there is still leftover data at the specified
+location then you should manually remove it.
+
+If the connection from your client to {es} is closed while the client is
+waiting for the result of the analysis then the test is cancelled. Some clients
+are configured to close their connection if no response is received within a
+certain timeout. An analysis takes a long time to complete so you may need to
+relax any such client-side timeouts. On cancellation the analysis attempts to
+clean up the data it was writing, but it may not be able to remove it all. The
+path to the leftover data is recorded in the {es} logs. You should verify
+yourself that this location has been cleaned up correctly. If there is still
+leftover data at the specified location then you should manually remove it.
+
+If the analysis is successful then it detected no incorrect behaviour, but this
+does not mean that correct behaviour is guaranteed. The analysis attempts to
+detect common bugs but it certainly does not offer 100% coverage. Additionally,
+it does not test the following:
+
+- Your repository must perform durable writes. Once a blob has been written it
+  must remain in place until it is deleted, even after a power loss or similar
+  disaster.
+
+- Your repository must not suffer from silent data corruption. Once a blob has
+  been written its contents must remain unchanged until it is deliberately
+  modified or deleted.
+
+- Your repository must behave correctly even if connectivity from the cluster
+  is disrupted. Reads and writes may fail in this case, but they must not return
+  incorrect results.
+
+IMPORTANT: An analysis writes a substantial amount of data to your repository
+and then reads it back again. This consumes bandwidth on the network between
+the cluster and the repository, and storage space and IO bandwidth on the
+repository itself. You must ensure this load does not affect other users of
+these systems. Analyses respect the repository settings
+`max_snapshot_bytes_per_sec` and `max_restore_bytes_per_sec` if available, and
+the cluster setting `indices.recovery.max_bytes_per_sec` which you can use to
+limit the bandwidth they consume.
+
+NOTE: This API is intended for exploratory use by humans. You should expect the
+request parameters and the response format to vary in future versions.
+
+NOTE: This API may not work correctly in a mixed-version cluster.
+
+==== Implementation details
+
+NOTE: This section of documentation describes how the Repository analysis API
+works in this version of {es}, but you should expect the implementation to vary
+between versions. The request parameters and response format depend on details
+of the implementation so may also be different in newer versions.
+
+The analysis comprises a number of blob-level tasks, as set by the `blob_count`
+parameter. The blob-level tasks are distributed over the data and
+master-eligible nodes in the cluster for execution.
+
+For most blob-level tasks, the executing node first writes a blob to the
+repository, and then instructs some of the other nodes in the cluster to
+attempt to read the data it just wrote. The size of the blob is chosen
+randomly, according to the `max_blob_size` and `max_total_data_size`
+parameters. If any of these reads fails then the repository does not implement
+the necessary read-after-write semantics that {es} requires.
+
+For some blob-level tasks, the executing node will instruct some of its peers
+to attempt to read the data before the writing process completes. These reads
+are permitted to fail, but must not return partial data. If any read returns
+partial data then the repository does not implement the necessary atomicity
+semantics that {es} requires.
+
+For some blob-level tasks, the executing node will overwrite the blob while its
+peers are reading it. In this case the data read may come from either the
+original or the overwritten blob, but must not return partial data or a mix of
+data from the two blobs. If any of these reads returns partial data or a mix of
+the two blobs then the repository does not implement the necessary atomicity
+semantics that {es} requires for overwrites.
+
+The executing node will use a variety of different methods to write the blob.
+For instance, where applicable, it will use both single-part and multi-part
+uploads. Similarly, the reading nodes will use a variety of different methods
+to read the data back again. For instance they may read the entire blob from
+start to end, or may read only a subset of the data.
+
+[[repo-analysis-api-path-params]]
+==== {api-path-parms-title}
+
+`<repository>`::
+(Required, string)
+Name of the snapshot repository to test.
+
+[[repo-analysis-api-query-params]]
+==== {api-query-parms-title}
+
+`blob_count`::
+(Optional, integer) The total number of blobs to write to the repository during
+the test. Defaults to `100`. For realistic experiments you should set this to
+at least `2000`.
+
+`max_blob_size`::
+(Optional, <<size-units, size units>>) The maximum size of a blob to be written
+during the test. Defaults to `10mb`. For realistic experiments you should set
+this to at least `2gb`.
+
+`max_total_data_size`::
+(Optional, <<size-units, size units>>) An upper limit on the total size of all
+the blobs written during the test. Defaults to `1gb`. For realistic experiments
+you should set this to at least `1tb`.
+
+`timeout`::
+(Optional, <<time-units, time units>>) Specifies the period of time to wait for
+the test to complete. If no response is received before the timeout expires,
+the test is cancelled and returns an error. Defaults to `30s`.
+
+===== Advanced query parameters
+
+The following parameters allow additional control over the analysis, but you
+will usually not need to adjust them.
+
+`concurrency`::
+(Optional, integer) The number of write operations to perform concurrently.
+Defaults to `10`.
+
+`read_node_count`::
+(Optional, integer) The number of nodes on which to perform a read operation
+after writing each blob.  Defaults to `10`.
+
+`early_read_node_count`::
+(Optional, integer) The number of nodes on which to perform an early read
+operation while writing each blob. Defaults to `2`. Early read operations are
+only rarely performed.
+
+`rare_action_probability`::
+(Optional, double) The probability of performing a rare action (an early read
+or an overwrite) on each blob. Defaults to `0.02`.
+
+`seed`::
+(Optional, integer) The seed for the pseudo-random number generator used to
+generate the list of operations performed during the test. To repeat the same
+set of operations in multiple experiments, use the same seed in each
+experiment. Note that the operations are performed concurrently so may not
+always happen in the same order on each run.
+
+`detailed`::
+(Optional, boolean) Whether to return detailed results, including timing
+information for every operation performed during the analysis. Defaults to
+`false`, meaning to return only a summary of the analysis.
+
+[role="child_attributes"]
+[[repo-analysis-api-response-body]]
+==== {api-response-body-title}
+
+The response exposes implementation details of the analysis which may change
+from version to version. The response body format is therefore not considered
+stable and may be different in newer versions.
+
+`coordinating_node`::
+(object)
+Identifies the node which coordinated the analysis and performed the final cleanup.
++
+.Properties of `coordinating_node`
+[%collapsible%open]
+====
+`id`::
+(string)
+The id of the coordinating node.
+
+`name`::
+(string)
+The name of the coordinating node
+====
+
+`repository`::
+(string)
+The name of the repository that was the subject of the analysis.
+
+`blob_count`::
+(integer)
+The number of blobs written to the repository during the test, equal to the
+`?blob_count` request parameter.
+
+`concurrency`::
+(integer)
+The number of write operations performed concurrently during the test, equal to
+the `?concurrency` request parameter.
+
+`read_node_count`::
+(integer)
+The limit on the number of nodes on which read operations were performed after
+writing each blob, equal to the `?read_node_count` request parameter.
+
+`early_read_node_count`::
+(integer)
+The limit on the number of nodes on which early read operations were performed
+after writing each blob, equal to the `?early_read_node_count` request
+parameter.
+
+`max_blob_size`::
+(string)
+The limit on the size of a blob written during the test, equal to the
+`?max_blob_size` parameter.
+
+`max_total_data_size`::
+(string)
+The limit on the total size of all blob written during the test, equal to the
+`?max_total_data_size` parameter.
+
+`seed`::
+(long)
+The seed for the pseudo-random number generator used to generate the operations
+used during the test. Equal to the `?seed` request parameter if set.
+
+`rare_action_probability`::
+(double)
+The probability of performing rare actions during the test. Equal to the
+`?rare_action_probability` request parameter.
+
+`blob_path`::
+(string)
+The path in the repository under which all the blobs were written during the
+test.
+
+`issues_detected`::
+(list)
+A list of correctness issues detected, which will be empty if the API
+succeeded. Included to emphasize that a successful response does not guarantee
+correct behaviour in future.
+
+`summary`::
+(object)
+A collection of statistics that summarise the results of the test.
++
+.Properties of `summary`
+[%collapsible%open]
+====
+`write`::
+(object)
+A collection of statistics that summarise the results of the write operations
+in the test.
++
+.Properties of `write`
+[%collapsible%open]
+=====
+`count`::
+(integer)
+The numer of write operations performed in the test.
+
+`total_bytes`::
+(integer)
+The total size of all the blobs written in the test, in bytes.
+
+`total_throttled_nanos`::
+(integer)
+The total time spent waiting due to the `max_snapshot_bytes_per_sec` throttle,
+in nanoseconds.
+
+`total_elapsed_nanos`::
+(integer)
+The total elapsed time spent on writing blobs in the test, in nanoseconds.
+=====
+
+`read`::
+(object)
+A collection of statistics that summarise the results of the read operations in
+the test.
++
+.Properties of `read`
+[%collapsible%open]
+=====
+`count`::
+(integer)
+The numer of read operations performed in the test.
+
+`total_bytes`::
+(integer)
+The total size of all the blobs or partial blobs read in the test, in bytes.
+
+`total_throttled_nanos`::
+(integer)
+The total time spent waiting due to the `max_restore_bytes_per_sec` or
+`indices.recovery.max_bytes_per_sec` throttles, in nanoseconds.
+
+`total_wait_nanos`::
+(integer)
+The total time spent waiting for the first byte of each read request to be
+received, in nanoseconds.
+
+`max_wait_nanos`::
+(integer)
+The maximum time spent waiting for the first byte of any read request to be
+received, in nanoseconds.
+
+`total_elapsed_nanos`::
+(integer)
+The total elapsed time spent on reading blobs in the test, in nanoseconds.
+=====
+====
+
+`details`::
+(array)
+A description of every read and write operation performed during the test. This is
+only returned if the `?detailed` request parameter is set to `true`.
++
+.Properties of items within `details`
+[%collapsible]
+====
+`blob`::
+(object)
+A description of the blob that was written and read.
++
+.Properties of `blob`
+[%collapsible%open]
+=====
+`name`::
+(string)
+The name of the blob.
+
+`size`::
+(long)
+The size of the blob in bytes.
+
+`read_start`::
+(long)
+The position, in bytes, at which read operations started.
+
+`read_end`::
+(long)
+The position, in bytes, at which read operations completed.
+
+`read_early`::
+(boolean)
+Whether any read operations were started before the write operation completed.
+
+`overwritten`::
+(boolean)
+Whether the blob was overwritten while the read operations were ongoing.
+=====
+
+`writer_node`::
+(object)
+Identifies the node which wrote this blob and coordinated the read operations.
++
+.Properties of `writer_node`
+[%collapsible%open]
+=====
+`id`::
+(string)
+The id of the writer node.
+
+`name`::
+(string)
+The name of the writer node
+=====
+
+`write_elapsed_nanos`::
+(object)
+The elapsed time spent writing this blob, in nanoseconds.
+
+`overwrite_elapsed_nanos`::
+(object)
+The elapsed time spent overwriting this blob, in nanoseconds. Omitted if the
+blob was not overwritten.
+
+`write_throttled_nanos`::
+(object)
+The length of time spent waiting for the `max_snapshot_bytes_per_sec` throttle
+while writing this blob, in nanoseconds.
+
+`reads`::
+(array)
+A description of every read operation performed on this blob.
++
+.Properties of items within `reads`
+[%collapsible%open]
+=====
+`node`::
+(object)
+Identifies the node which performed the read operation.
++
+.Properties of `node`
+[%collapsible%open]
+======
+`id`::
+(string)
+The id of the reader node.
+
+`name`::
+(string)
+The name of the reader node
+======
+
+`before_write_complete`::
+(boolean)
+Whether the read operation may have started before the write operation was
+complete. Omitted if `false`.
+
+`found`::
+(boolean)
+Whether the blob was found by this read operation or not. May be `false` if the
+read was started before the write completed.
+
+`first_byte_nanos`::
+(boolean)
+The length of time waiting for the first byte of the read operation to be
+received, in nanoseconds. Omitted if the blob was not found.
+
+`elapsed_nanos`::
+(boolean)
+The length of time spent reading this blob, in nanoseconds. Omitted if the blob
+was not found.
+
+`throttled_nanos`::
+(boolean)
+The length of time time spent waiting due to the `max_restore_bytes_per_sec` or
+`indices.recovery.max_bytes_per_sec` throttles during the read of this blob, in
+nanoseconds. Omitted if the blob was not found.
+
+=====
+
+====

docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc

@@ -27,6 +27,7 @@ For more information, see <<snapshot-restore>>.
 
 include::put-repo-api.asciidoc[]
 include::verify-repo-api.asciidoc[]
+include::repo-analysis-api.asciidoc[]
 include::get-repo-api.asciidoc[]
 include::delete-repo-api.asciidoc[]
 include::clean-up-repo-api.asciidoc[]

docs/reference/snapshot-restore/register-repository.asciidoc

@@ -260,6 +260,9 @@ POST /_snapshot/my_unverified_backup/_verify
 
 It returns a list of nodes where repository was successfully verified or an error message if verification process failed.
 
+If desired, you can also test a repository more thoroughly using the
+<<repo-analysis-api,Repository analysis API>>.
+
 [discrete]
 [[snapshots-repository-cleanup]]
 === Repository cleanup

server/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java

@@ -37,6 +37,7 @@ import java.nio.file.StandardOpenOption;
 import java.nio.file.attribute.BasicFileAttributes;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.atomic.AtomicLong;
@@ -88,7 +89,7 @@ public class FsBlobContainer extends AbstractBlobContainer {
         Map<String, BlobMetadata> builder = new HashMap<>();
 
         blobNamePrefix = blobNamePrefix == null ? "" : blobNamePrefix;
-        try (DirectoryStream<Path> stream = Files.newDirectoryStream(path, blobNamePrefix + "*")) {
+        try (DirectoryStream<Path> stream = newDirectoryStreamIfFound(blobNamePrefix)) {
             for (Path file : stream) {
                 final BasicFileAttributes attrs;
                 try {
@@ -105,6 +106,34 @@ public class FsBlobContainer extends AbstractBlobContainer {
         return unmodifiableMap(builder);
     }
 
+    private DirectoryStream<Path> newDirectoryStreamIfFound(String blobNamePrefix) throws IOException {
+        try {
+            return Files.newDirectoryStream(path, blobNamePrefix + "*");
+        } catch (FileNotFoundException | NoSuchFileException e) {
+            // a nonexistent directory contains no blobs
+            return new DirectoryStream<>() {
+                @Override
+                public Iterator<Path> iterator() {
+                    return new Iterator<>() {
+                        @Override
+                        public boolean hasNext() {
+                            return false;
+                        }
+
+                        @Override
+                        public Path next() {
+                            return null;
+                        }
+                    };
+                }
+
+                @Override
+                public void close() {
+                }
+            };
+        }
+    }
+
     @Override
     public DeleteResult delete() throws IOException {
         final AtomicLong filesDeleted = new AtomicLong(0L);

server/src/main/java/org/elasticsearch/repositories/RepositoriesService.java

@@ -380,7 +380,7 @@ public class RepositoriesService extends AbstractLifecycleComponent implements C
         });
     }
 
-    static boolean isDedicatedVotingOnlyNode(Set<DiscoveryNodeRole> roles) {
+    public static boolean isDedicatedVotingOnlyNode(Set<DiscoveryNodeRole> roles) {
         return roles.contains(DiscoveryNodeRole.MASTER_ROLE) && roles.contains(DiscoveryNodeRole.DATA_ROLE) == false &&
             roles.stream().anyMatch(role -> role.roleName().equals("voting_only"));
     }

server/src/main/java/org/elasticsearch/repositories/RepositoryVerificationException.java

@@ -35,5 +35,11 @@ public class RepositoryVerificationException extends RepositoryException {
     public RepositoryVerificationException(StreamInput in) throws IOException{
         super(in);
     }
+
+    @Override
+    public synchronized Throwable fillInStackTrace() {
+        // stack trace for a verification failure is uninteresting, the message has all the information we need
+        return this;
+    }
 }
 

server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

@@ -2471,17 +2471,47 @@ public abstract class BlobStoreRepository extends AbstractLifecycleComponent imp
         });
     }
 
-    private static InputStream maybeRateLimit(InputStream stream, Supplier<RateLimiter> rateLimiterSupplier, CounterMetric metric) {
-        return new RateLimitingInputStream(stream, rateLimiterSupplier, metric::inc);
+    private static InputStream maybeRateLimit(
+            InputStream stream,
+            Supplier<RateLimiter> rateLimiterSupplier,
+            RateLimitingInputStream.Listener throttleListener) {
+        return new RateLimitingInputStream(stream, rateLimiterSupplier, throttleListener);
     }
 
+    /**
+     * Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting
+     * `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is reported to the given listener and not otherwise
+     * recorded in the value returned by {@link BlobStoreRepository#getRestoreThrottleTimeInNanos}.
+     */
     public InputStream maybeRateLimitRestores(InputStream stream) {
-        return maybeRateLimit(maybeRateLimit(stream, () -> restoreRateLimiter, restoreRateLimitingTimeInNanos),
-            recoverySettings::rateLimiter, restoreRateLimitingTimeInNanos);
+        return maybeRateLimitRestores(stream, restoreRateLimitingTimeInNanos::inc);
     }
 
+    /**
+     * Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting
+     * `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is recorded in the value returned by {@link
+     * BlobStoreRepository#getRestoreThrottleTimeInNanos}.
+     */
+    public InputStream maybeRateLimitRestores(InputStream stream, RateLimitingInputStream.Listener throttleListener) {
+        return maybeRateLimit(maybeRateLimit(stream, () -> restoreRateLimiter, throttleListener),
+            recoverySettings::rateLimiter, throttleListener);
+    }
+
+    /**
+     * Wrap the snapshot rate limiter (controlled by the repository setting `max_snapshot_bytes_per_sec`) around the given stream. Any
+     * throttling is recorded in the value returned by {@link BlobStoreRepository#getSnapshotThrottleTimeInNanos()}.
+     */
     public InputStream maybeRateLimitSnapshots(InputStream stream) {
-        return maybeRateLimit(stream, () -> snapshotRateLimiter, snapshotRateLimitingTimeInNanos);
+        return maybeRateLimitSnapshots(stream, snapshotRateLimitingTimeInNanos::inc);
+    }
+
+    /**
+     * Wrap the snapshot rate limiter (controlled by the repository setting `max_snapshot_bytes_per_sec`) around the given stream. Any
+     * throttling is reported to the given listener and not otherwise recorded in the value returned by {@link
+     * BlobStoreRepository#getSnapshotThrottleTimeInNanos()}.
+     */
+    public InputStream maybeRateLimitSnapshots(InputStream stream, RateLimitingInputStream.Listener throttleListener) {
+        return maybeRateLimit(stream, () -> snapshotRateLimiter, throttleListener);
     }
 
     @Override
@@ -2721,6 +2751,10 @@ public abstract class BlobStoreRepository extends AbstractLifecycleComponent imp
         }
     }
 
+    public boolean supportURLRepo() {
+        return supportURLRepo;
+    }
+
     /**
      * The result of removing a snapshot from a shard folder in the repository.
      */

server/src/main/java/org/elasticsearch/rest/RestRequest.java

@@ -360,6 +360,18 @@ public class RestRequest implements ToXContent.Params {
         }
     }
 
+    public double paramAsDouble(String key, double defaultValue) {
+        String sValue = param(key);
+        if (sValue == null) {
+            return defaultValue;
+        }
+        try {
+            return Double.parseDouble(sValue);
+        } catch (NumberFormatException e) {
+            throw new IllegalArgumentException("Failed to parse double parameter [" + key + "] with value [" + sValue + "]", e);
+        }
+    }
+
     public int paramAsInt(String key, int defaultValue) {
         String sValue = param(key);
         if (sValue == null) {

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/operator/OperatorOnlyRegistry.java

@@ -33,7 +33,12 @@ public class OperatorOnlyRegistry {
         "cluster:admin/autoscaling/put_autoscaling_policy",
         "cluster:admin/autoscaling/delete_autoscaling_policy",
         "cluster:admin/autoscaling/get_autoscaling_policy",
-        "cluster:admin/autoscaling/get_autoscaling_capacity");
+        "cluster:admin/autoscaling/get_autoscaling_capacity",
+        // Repository analysis actions are not mentioned in core, literal strings are needed.
+        "cluster:admin/repository/analyze",
+        "cluster:admin/repository/analyze/blob",
+        "cluster:admin/repository/analyze/blob/read"
+        );
 
     private final ClusterSettings clusterSettings;
 

x-pack/plugin/snapshot-repo-test-kit/build.gradle

@@ -0,0 +1,30 @@
+apply plugin: 'elasticsearch.internal-cluster-test'
+apply plugin: 'elasticsearch.esplugin'
+esplugin {
+  name 'snapshot-repo-test-kit'
+  description 'A plugin for a test kit for snapshot repositories'
+  classname 'org.elasticsearch.repositories.blobstore.testkit.SnapshotRepositoryTestKit'
+  extendedPlugins = ['x-pack-core']
+}
+archivesBaseName = 'x-pack-snapshot-repo-test-kit'
+
+dependencies {
+  compileOnly project(path: xpackModule('core'), configuration: 'default')
+  internalClusterTestImplementation project(path: xpackModule('core'), configuration: 'testArtifacts')
+}
+
+addQaCheckDependencies()
+
+configurations {
+  testArtifacts.extendsFrom testRuntime
+  testArtifacts.extendsFrom testImplementation
+}
+
+def testJar = tasks.register("testJar", Jar) {
+  appendix 'test'
+  from sourceSets.test.output
+}
+
+artifacts {
+  testArtifacts testJar
+}

x-pack/plugin/snapshot-repo-test-kit/qa/build.gradle

@@ -0,0 +1,6 @@
+apply plugin: 'elasticsearch.build'
+tasks.named("test").configure { enabled = false }
+
+dependencies {
+  api project(':test:framework')
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/build.gradle

@@ -0,0 +1,27 @@
+apply plugin: 'elasticsearch.testclusters'
+apply plugin: 'elasticsearch.standalone-rest-test'
+apply plugin: 'elasticsearch.rest-test'
+apply plugin: 'elasticsearch.rest-resources'
+
+dependencies {
+  testImplementation project(path: xpackModule('snapshot-repo-test-kit'), configuration: 'testArtifacts')
+}
+
+// TODO we want 3rd-party tests for other repository types too
+final File repoDir = file("$buildDir/testclusters/repo")
+
+tasks.named("integTest").configure {
+  systemProperty 'tests.path.repo', repoDir
+}
+
+testClusters.matching { it.name == "integTest" }.configureEach {
+  testDistribution = 'DEFAULT'
+  setting 'path.repo', repoDir.absolutePath
+}
+
+restResources {
+  restApi {
+    includeCore 'indices', 'search', 'bulk', 'snapshot', 'nodes', '_common'
+    includeXpack 'snapshot_repo_test_kit'
+  }
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/test/java/org/elasticsearch/repositories/blobstore/testkit/rest/FsSnapshotRepoTestKitIT.java

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit.rest;
+
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.repositories.blobstore.testkit.AbstractSnapshotRepoTestKitRestTestCase;
+import org.elasticsearch.repositories.fs.FsRepository;
+
+public class FsSnapshotRepoTestKitIT extends AbstractSnapshotRepoTestKitRestTestCase {
+
+    @Override
+    protected String repositoryType() {
+        return FsRepository.TYPE;
+    }
+
+    @Override
+    protected Settings repositorySettings() {
+        return Settings.builder().put("location", System.getProperty("tests.path.repo")).build();
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/test/java/org/elasticsearch/repositories/blobstore/testkit/rest/SnapshotRepoTestKitClientYamlTestSuiteIT.java

@@ -0,0 +1,24 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit.rest;
+
+import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
+import org.elasticsearch.test.rest.yaml.ClientYamlTestCandidate;
+import org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase;
+
+public class SnapshotRepoTestKitClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase {
+
+    public SnapshotRepoTestKitClientYamlTestSuiteIT(final ClientYamlTestCandidate testCandidate) {
+        super(testCandidate);
+    }
+
+    @ParametersFactory
+    public static Iterable<Object[]> parameters() throws Exception {
+        return ESClientYamlSuiteTestCase.createParameters();
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/test/resources/rest-api-spec/api/snapshot.repository_analyze.json

@@ -0,0 +1,68 @@
+{
+  "snapshot.repository_analyze":{
+    "documentation":{
+      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-snapshots.html",
+      "description":"Analyzes a repository for correctness and performance"
+    },
+    "stability":"stable",
+    "visibility":"public",
+    "url":{
+      "paths":[
+        {
+          "path":"/_snapshot/{repository}/_analyze",
+          "methods":[
+            "POST"
+          ],
+          "parts":{
+            "repository":{
+              "type":"string",
+              "description":"A repository name"
+            }
+          }
+        }
+      ]
+    },
+    "params":{
+      "blob_count":{
+        "type":"number",
+        "description":"Number of blobs to create during the test. Defaults to 100."
+      },
+      "concurrency":{
+        "type":"number",
+        "description":"Number of operations to run concurrently during the test. Defaults to 10."
+      },
+      "read_node_count":{
+        "type":"number",
+        "description":"Number of nodes on which to read a blob after writing. Defaults to 10."
+      },
+      "early_read_node_count":{
+        "type":"number",
+        "description":"Number of nodes on which to perform an early read on a blob, i.e. before writing has completed. Early reads are rare actions so the 'rare_action_probability' parameter is also relevant. Defaults to 2."
+      },
+      "seed":{
+        "type":"number",
+        "description":"Seed for the random number generator used to create the test workload. Defaults to a random value."
+      },
+      "rare_action_probability":{
+        "type":"number",
+        "description":"Probability of taking a rare action such as an early read or an overwrite. Defaults to 0.02."
+      },
+      "max_blob_size":{
+        "type":"string",
+        "description":"Maximum size of a blob to create during the test, e.g '1gb' or '100mb'. Defaults to '10mb'."
+      },
+      "max_total_data_size":{
+        "type":"string",
+        "description":"Maximum total size of all blobs to create during the test, e.g '1tb' or '100gb'. Defaults to '1gb'."
+      },
+      "timeout":{
+        "type":"time",
+        "description":"Explicit operation timeout. Defaults to '30s'."
+      },
+      "detailed":{
+        "type":"boolean",
+        "description":"Whether to return detailed results or a summary. Defaults to 'false' so that only the summary is returned."
+      }
+    }
+  }
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/test/resources/rest-api-spec/test/10_analyze.yml

@@ -0,0 +1,147 @@
+---
+setup:
+
+  - do:
+      snapshot.create_repository:
+        repository: test_repo
+        body:
+          type: fs
+          settings:
+            location: "test_repo_loc"
+
+  - do:
+      snapshot.create_repository:
+        repository: test_repo_readonly
+        body:
+          type: fs
+          settings:
+            readonly: true
+            location: "test_repo_loc"
+
+---
+"Analysis fails on readonly repositories":
+  - skip:
+      version: "- 7.99.99"
+      reason: "introduced in 8.0"
+
+  - do:
+      catch: bad_request
+      snapshot.repository_analyze:
+        repository: test_repo_readonly
+
+  - match: { status: 400 }
+  - match: { error.type: illegal_argument_exception }
+  - match: { error.reason: "repository [test_repo_readonly] is read-only" }
+
+
+---
+"Analysis without details":
+  - skip:
+      version: "- 7.99.99"
+      reason: "introduced in 8.0"
+
+  - do:
+      snapshot.repository_analyze:
+        repository: test_repo
+        blob_count: 10
+        concurrency: 5
+        max_blob_size: 1mb
+        read_node_count: 2
+        early_read_node_count: 1
+        rare_action_probability: 0.01
+        max_total_data_size: 5mb
+
+  - is_true: coordinating_node.id
+  - is_true: coordinating_node.name
+  - match: { repository:    test_repo }
+  - match: { blob_count:    10 }
+  - match: { concurrency:   5 }
+  - match: { read_node_count: 2 }
+  - match: { early_read_node_count: 1 }
+  - match: { rare_action_probability: 0.01 }
+  - match: { max_blob_size: 1mb }
+  - match: { max_blob_size_bytes: 1048576 }
+  - match: { max_total_data_size: 5mb }
+  - match: { max_total_data_size_bytes: 5242880 }
+  - is_true: seed
+  - is_true: blob_path
+  - is_false: details
+  - is_true: listing_elapsed
+  - is_true: delete_elapsed
+  - gte: { listing_elapsed_nanos: 0}
+  - gte: { delete_elapsed_nanos:  0}
+  - match: { summary.write.count: 10}
+  - gte: { summary.write.total_size_bytes: 0}
+  - is_true: summary.write.total_size
+  - gte: { summary.write.total_throttled_nanos: 0}
+  - is_true: summary.write.total_throttled
+  - gte: { summary.write.total_elapsed_nanos: 0}
+  - is_true: summary.write.total_elapsed
+  - gte: { summary.read.count: 10}
+  - gte: { summary.read.total_size_bytes: 0}
+  - is_true: summary.read.total_size
+  - gte: { summary.read.total_wait_nanos: 0}
+  - is_true: summary.read.total_wait
+  - gte: { summary.read.max_wait_nanos: 0}
+  - is_true: summary.read.max_wait
+  - gte: { summary.read.total_throttled_nanos: 0}
+  - is_true: summary.read.total_throttled
+  - gte: { summary.read.total_elapsed_nanos: 0}
+  - is_true: summary.read.total_elapsed
+
+---
+"Analysis with details":
+  - skip:
+      version: "- 7.99.99"
+      reason: "introduced in 8.0"
+
+  - do:
+      snapshot.repository_analyze:
+        repository: test_repo
+        blob_count: 10
+        concurrency: 5
+        max_blob_size: 1mb
+        detailed: true
+
+  - is_true: coordinating_node.id
+  - is_true: coordinating_node.name
+  - match: { repository:    test_repo }
+  - match: { blob_count:    10 }
+  - match: { concurrency:   5 }
+  - match: { read_node_count: 10 }
+  - match: { early_read_node_count: 2 }
+  - match: { rare_action_probability: 0.02 }
+  - match: { max_blob_size: 1mb }
+  - is_true: seed
+  - is_true: blob_path
+  - is_true: details
+  - is_true: listing_elapsed
+  - is_true: delete_elapsed
+  - gte: { listing_elapsed_nanos: 0}
+  - gte: { delete_elapsed_nanos:  0}
+
+---
+"Analysis with ?human=false":
+  - skip:
+      version: "- 7.99.99"
+      reason: "introduced in 8.0"
+
+  - do:
+      snapshot.repository_analyze:
+        repository: test_repo
+        blob_count: 10
+        concurrency: 5
+        max_blob_size: 1mb
+        detailed: false
+        human: false
+
+  - is_false: listing_elapsed
+  - is_false: delete_elapsed
+  - is_false: summary.write.total_size
+  - is_false: summary.write.total_throttled
+  - is_false: summary.write.total_elapsed
+  - is_false: summary.read.total_size
+  - is_false: summary.read.total_wait
+  - is_false: summary.read.max_wait
+  - is_false: summary.read.total_throttled
+  - is_false: summary.read.total_elapsed

x-pack/plugin/snapshot-repo-test-kit/src/internalClusterTest/java/org/elasticsearch/repositories/blobstore/testkit/RepositoryAnalysisFailureIT.java

@@ -0,0 +1,470 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.ExceptionsHelper;
+import org.elasticsearch.cluster.metadata.RepositoryMetadata;
+import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.blobstore.BlobContainer;
+import org.elasticsearch.common.blobstore.BlobMetadata;
+import org.elasticsearch.common.blobstore.BlobPath;
+import org.elasticsearch.common.blobstore.BlobStore;
+import org.elasticsearch.common.blobstore.DeleteResult;
+import org.elasticsearch.common.blobstore.support.PlainBlobMetadata;
+import org.elasticsearch.common.bytes.BytesReference;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.util.BigArrays;
+import org.elasticsearch.common.util.concurrent.ConcurrentCollections;
+import org.elasticsearch.common.util.concurrent.CountDown;
+import org.elasticsearch.common.xcontent.NamedXContentRegistry;
+import org.elasticsearch.env.Environment;
+import org.elasticsearch.indices.recovery.RecoverySettings;
+import org.elasticsearch.plugins.Plugin;
+import org.elasticsearch.plugins.RepositoryPlugin;
+import org.elasticsearch.repositories.RepositoriesService;
+import org.elasticsearch.repositories.Repository;
+import org.elasticsearch.repositories.RepositoryMissingException;
+import org.elasticsearch.repositories.RepositoryVerificationException;
+import org.elasticsearch.repositories.blobstore.BlobStoreRepository;
+import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.snapshots.AbstractSnapshotIntegTestCase;
+import org.elasticsearch.xpack.core.LocalStateCompositeXPackPlugin;
+import org.junit.Before;
+
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.FileAlreadyExistsException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.Matchers.anEmptyMap;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.nullValue;
+
+public class RepositoryAnalysisFailureIT extends AbstractSnapshotIntegTestCase {
+
+    private DisruptableBlobStore blobStore;
+
+    @Before
+    public void suppressConsistencyChecks() {
+        disableRepoConsistencyCheck("repository is not used for snapshots");
+    }
+
+    @Override
+    protected Collection<Class<? extends Plugin>> nodePlugins() {
+        return List.of(TestPlugin.class, LocalStateCompositeXPackPlugin.class, SnapshotRepositoryTestKit.class);
+    }
+
+    @Before
+    public void createBlobStore() {
+        createRepositoryNoVerify("test-repo", TestPlugin.DISRUPTABLE_REPO_TYPE);
+
+        blobStore = new DisruptableBlobStore();
+        for (final RepositoriesService repositoriesService : internalCluster().getInstances(RepositoriesService.class)) {
+            try {
+                ((DisruptableRepository) repositoriesService.repository("test-repo")).setBlobStore(blobStore);
+            } catch (RepositoryMissingException e) {
+                // it's only present on voting masters and data nodes
+            }
+        }
+    }
+
+    public void testSuccess() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.blobCount(1);
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        final RepositoryAnalyzeAction.Response response = analyseRepository(request);
+        assertThat(response.status(), equalTo(RestStatus.OK));
+    }
+
+    public void testFailsOnReadError() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        final CountDown countDown = new CountDown(between(1, request.getBlobCount()));
+        blobStore.setDisruption(new Disruption() {
+            @Override
+            public byte[] onRead(byte[] actualContents, long position, long length) throws IOException {
+                if (countDown.countDown()) {
+                    throw new IOException("simulated");
+                }
+                return actualContents;
+            }
+        });
+
+        final Exception exception = expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+        final IOException ioException = (IOException) ExceptionsHelper.unwrap(exception, IOException.class);
+        assert ioException != null : exception;
+        assertThat(ioException.getMessage(), equalTo("simulated"));
+    }
+
+    public void testFailsOnNotFoundAfterWrite() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+        request.earlyReadNodeCount(0); // not found on early read is ok
+
+        final CountDown countDown = new CountDown(between(1, request.getBlobCount()));
+
+        blobStore.setDisruption(new Disruption() {
+            @Override
+            public byte[] onRead(byte[] actualContents, long position, long length) {
+                if (countDown.countDown()) {
+                    return null;
+                }
+                return actualContents;
+            }
+        });
+
+        expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+    }
+
+    public void testFailsOnChecksumMismatch() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        final CountDown countDown = new CountDown(between(1, request.getBlobCount()));
+
+        blobStore.setDisruption(new Disruption() {
+            @Override
+            public byte[] onRead(byte[] actualContents, long position, long length) {
+                final byte[] disruptedContents = actualContents == null ? null : Arrays.copyOf(actualContents, actualContents.length);
+                if (actualContents != null && countDown.countDown()) {
+                    // CRC32 should always detect a single bit flip
+                    disruptedContents[Math.toIntExact(position + randomLongBetween(0, length - 1))] ^= 1 << between(0, 7);
+                }
+                return disruptedContents;
+            }
+        });
+
+        expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+    }
+
+    public void testFailsOnWriteException() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        final CountDown countDown = new CountDown(between(1, request.getBlobCount()));
+
+        blobStore.setDisruption(new Disruption() {
+
+            @Override
+            public void onWrite() throws IOException {
+                if (countDown.countDown()) {
+                    throw new IOException("simulated");
+                }
+            }
+
+        });
+
+        final Exception exception = expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+        final IOException ioException = (IOException) ExceptionsHelper.unwrap(exception, IOException.class);
+        assert ioException != null : exception;
+        assertThat(ioException.getMessage(), equalTo("simulated"));
+    }
+
+    public void testFailsOnIncompleteListing() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        blobStore.setDisruption(new Disruption() {
+
+            @Override
+            public Map<String, BlobMetadata> onList(Map<String, BlobMetadata> actualListing) {
+                final HashMap<String, BlobMetadata> listing = new HashMap<>(actualListing);
+                listing.keySet().iterator().remove();
+                return listing;
+            }
+
+        });
+
+        expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+    }
+
+    public void testFailsOnListingException() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        final CountDown countDown = new CountDown(1);
+        blobStore.setDisruption(new Disruption() {
+
+            @Override
+            public Map<String, BlobMetadata> onList(Map<String, BlobMetadata> actualListing) throws IOException {
+                if (countDown.countDown()) {
+                    throw new IOException("simulated");
+                }
+                return actualListing;
+            }
+        });
+
+        expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+    }
+
+    public void testFailsOnDeleteException() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        blobStore.setDisruption(new Disruption() {
+            @Override
+            public void onDelete() throws IOException {
+                throw new IOException("simulated");
+            }
+        });
+
+        expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+    }
+
+    public void testFailsOnIncompleteDelete() {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+        request.maxBlobSize(new ByteSizeValue(10L));
+
+        blobStore.setDisruption(new Disruption() {
+
+            volatile boolean isDeleted;
+
+            @Override
+            public void onDelete() {
+                isDeleted = true;
+            }
+
+            @Override
+            public Map<String, BlobMetadata> onList(Map<String, BlobMetadata> actualListing) {
+                if (isDeleted) {
+                    assertThat(actualListing, anEmptyMap());
+                    return Collections.singletonMap("leftover", new PlainBlobMetadata("leftover", 1));
+                } else {
+                    return actualListing;
+                }
+            }
+        });
+
+        expectThrows(RepositoryVerificationException.class, () -> analyseRepository(request));
+    }
+
+    private RepositoryAnalyzeAction.Response analyseRepository(RepositoryAnalyzeAction.Request request) {
+        return client().execute(RepositoryAnalyzeAction.INSTANCE, request).actionGet(30L, TimeUnit.SECONDS);
+    }
+
+    public static class TestPlugin extends Plugin implements RepositoryPlugin {
+
+        static final String DISRUPTABLE_REPO_TYPE = "disruptable";
+
+        @Override
+        public Map<String, Repository.Factory> getRepositories(
+            Environment env,
+            NamedXContentRegistry namedXContentRegistry,
+            ClusterService clusterService,
+            BigArrays bigArrays,
+            RecoverySettings recoverySettings
+        ) {
+            return Map.of(
+                DISRUPTABLE_REPO_TYPE,
+                metadata -> new DisruptableRepository(
+                    metadata,
+                    namedXContentRegistry,
+                    clusterService,
+                    bigArrays,
+                    recoverySettings,
+                    new BlobPath()
+                )
+            );
+        }
+    }
+
+    static class DisruptableRepository extends BlobStoreRepository {
+
+        private final AtomicReference<BlobStore> blobStoreRef = new AtomicReference<>();
+
+        DisruptableRepository(
+            RepositoryMetadata metadata,
+            NamedXContentRegistry namedXContentRegistry,
+            ClusterService clusterService,
+            BigArrays bigArrays,
+            RecoverySettings recoverySettings,
+            BlobPath basePath
+        ) {
+            super(metadata, namedXContentRegistry, clusterService, bigArrays, recoverySettings, basePath);
+        }
+
+        void setBlobStore(BlobStore blobStore) {
+            assertTrue(blobStoreRef.compareAndSet(null, blobStore));
+        }
+
+        @Override
+        protected BlobStore createBlobStore() {
+            final BlobStore blobStore = blobStoreRef.get();
+            assertNotNull(blobStore);
+            return blobStore;
+        }
+    }
+
+    static class DisruptableBlobStore implements BlobStore {
+
+        @Nullable // if deleted
+        private DisruptableBlobContainer blobContainer;
+
+        private Disruption disruption = Disruption.NONE;
+
+        @Override
+        public BlobContainer blobContainer(BlobPath path) {
+            synchronized (this) {
+                if (blobContainer == null) {
+                    blobContainer = new DisruptableBlobContainer(path, this::deleteContainer, disruption);
+                }
+                return blobContainer;
+            }
+        }
+
+        private void deleteContainer(DisruptableBlobContainer container) {
+            blobContainer = null;
+        }
+
+        @Override
+        public void close() {}
+
+        public void setDisruption(Disruption disruption) {
+            assertThat("cannot change disruption while blob container exists", blobContainer, nullValue());
+            this.disruption = disruption;
+        }
+    }
+
+    interface Disruption {
+
+        Disruption NONE = new Disruption() {
+        };
+
+        default byte[] onRead(byte[] actualContents, long position, long length) throws IOException {
+            return actualContents;
+        }
+
+        default void onWrite() throws IOException {}
+
+        default Map<String, BlobMetadata> onList(Map<String, BlobMetadata> actualListing) throws IOException {
+            return actualListing;
+        }
+
+        default void onDelete() throws IOException {}
+    }
+
+    static class DisruptableBlobContainer implements BlobContainer {
+
+        private final BlobPath path;
+        private final Consumer<DisruptableBlobContainer> deleteContainer;
+        private final Disruption disruption;
+        private final Map<String, byte[]> blobs = ConcurrentCollections.newConcurrentMap();
+
+        DisruptableBlobContainer(BlobPath path, Consumer<DisruptableBlobContainer> deleteContainer, Disruption disruption) {
+            this.path = path;
+            this.deleteContainer = deleteContainer;
+            this.disruption = disruption;
+        }
+
+        @Override
+        public BlobPath path() {
+            return path;
+        }
+
+        @Override
+        public boolean blobExists(String blobName) {
+            return blobs.containsKey(blobName);
+        }
+
+        @Override
+        public InputStream readBlob(String blobName) throws IOException {
+            final byte[] actualContents = blobs.get(blobName);
+            final byte[] disruptedContents = disruption.onRead(actualContents, 0L, actualContents == null ? 0L : actualContents.length);
+            if (disruptedContents == null) {
+                throw new FileNotFoundException(blobName + " not found");
+            }
+            return new ByteArrayInputStream(disruptedContents);
+        }
+
+        @Override
+        public InputStream readBlob(String blobName, long position, long length) throws IOException {
+            final byte[] actualContents = blobs.get(blobName);
+            final byte[] disruptedContents = disruption.onRead(actualContents, position, length);
+            if (disruptedContents == null) {
+                throw new FileNotFoundException(blobName + " not found");
+            }
+            final int truncatedLength = Math.toIntExact(Math.min(length, disruptedContents.length - position));
+            return new ByteArrayInputStream(disruptedContents, Math.toIntExact(position), truncatedLength);
+        }
+
+        @Override
+        public void writeBlob(String blobName, InputStream inputStream, long blobSize, boolean failIfAlreadyExists) throws IOException {
+            writeBlobAtomic(blobName, inputStream, failIfAlreadyExists);
+        }
+
+        @Override
+        public void writeBlob(String blobName, BytesReference bytes, boolean failIfAlreadyExists) throws IOException {
+            writeBlob(blobName, bytes.streamInput(), bytes.length(), failIfAlreadyExists);
+        }
+
+        @Override
+        public void writeBlobAtomic(String blobName, BytesReference bytes, boolean failIfAlreadyExists) throws IOException {
+            writeBlobAtomic(blobName, bytes.streamInput(), failIfAlreadyExists);
+        }
+
+        private void writeBlobAtomic(String blobName, InputStream inputStream, boolean failIfAlreadyExists) throws IOException {
+            if (failIfAlreadyExists && blobs.get(blobName) != null) {
+                throw new FileAlreadyExistsException(blobName);
+            }
+
+            final byte[] contents = inputStream.readAllBytes();
+            disruption.onWrite();
+            blobs.put(blobName, contents);
+        }
+
+        @Override
+        public DeleteResult delete() throws IOException {
+            disruption.onDelete();
+            deleteContainer.accept(this);
+            final DeleteResult deleteResult = new DeleteResult(blobs.size(), blobs.values().stream().mapToLong(b -> b.length).sum());
+            blobs.clear();
+            return deleteResult;
+        }
+
+        @Override
+        public void deleteBlobsIgnoringIfNotExists(List<String> blobNames) {
+            blobs.keySet().removeAll(blobNames);
+        }
+
+        @Override
+        public Map<String, BlobMetadata> listBlobs() throws IOException {
+            return disruption.onList(
+                blobs.entrySet()
+                    .stream()
+                    .collect(Collectors.toMap(Map.Entry::getKey, e -> new PlainBlobMetadata(e.getKey(), e.getValue().length)))
+            );
+        }
+
+        @Override
+        public Map<String, BlobContainer> children() {
+            return Map.of();
+        }
+
+        @Override
+        public Map<String, BlobMetadata> listBlobsByPrefix(String blobNamePrefix) throws IOException {
+            final Map<String, BlobMetadata> blobMetadataByName = listBlobs();
+            blobMetadataByName.keySet().removeIf(s -> s.startsWith(blobNamePrefix) == false);
+            return blobMetadataByName;
+        }
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/internalClusterTest/java/org/elasticsearch/repositories/blobstore/testkit/RepositoryAnalysisSuccessIT.java

@@ -0,0 +1,363 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.cluster.metadata.RepositoryMetadata;
+import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.blobstore.BlobContainer;
+import org.elasticsearch.common.blobstore.BlobMetadata;
+import org.elasticsearch.common.blobstore.BlobPath;
+import org.elasticsearch.common.blobstore.BlobStore;
+import org.elasticsearch.common.blobstore.DeleteResult;
+import org.elasticsearch.common.blobstore.support.PlainBlobMetadata;
+import org.elasticsearch.common.bytes.BytesReference;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.util.BigArrays;
+import org.elasticsearch.common.util.concurrent.ConcurrentCollections;
+import org.elasticsearch.common.xcontent.NamedXContentRegistry;
+import org.elasticsearch.env.Environment;
+import org.elasticsearch.indices.recovery.RecoverySettings;
+import org.elasticsearch.plugins.Plugin;
+import org.elasticsearch.plugins.RepositoryPlugin;
+import org.elasticsearch.repositories.RepositoriesService;
+import org.elasticsearch.repositories.Repository;
+import org.elasticsearch.repositories.RepositoryMissingException;
+import org.elasticsearch.repositories.blobstore.BlobStoreRepository;
+import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.snapshots.AbstractSnapshotIntegTestCase;
+import org.elasticsearch.xpack.core.LocalStateCompositeXPackPlugin;
+import org.junit.Before;
+
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.hamcrest.Matchers.nullValue;
+import static org.hamcrest.Matchers.startsWith;
+
+public class RepositoryAnalysisSuccessIT extends AbstractSnapshotIntegTestCase {
+
+    @Before
+    public void suppressConsistencyChecks() {
+        disableRepoConsistencyCheck("repository is not used for snapshots");
+    }
+
+    @Override
+    protected Collection<Class<? extends Plugin>> nodePlugins() {
+        return List.of(TestPlugin.class, LocalStateCompositeXPackPlugin.class, SnapshotRepositoryTestKit.class);
+    }
+
+    public void testRepositoryAnalysis() {
+
+        createRepositoryNoVerify("test-repo", TestPlugin.ASSERTING_REPO_TYPE);
+
+        final AssertingBlobStore blobStore = new AssertingBlobStore();
+        for (final RepositoriesService repositoriesService : internalCluster().getInstances(RepositoriesService.class)) {
+            try {
+                ((AssertingRepository) repositoriesService.repository("test-repo")).setBlobStore(blobStore);
+            } catch (RepositoryMissingException e) {
+                // nbd, it's only present on voting masters and data nodes
+            }
+        }
+
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("test-repo");
+
+        if (randomBoolean()) {
+            request.concurrency(between(1, 5));
+            blobStore.ensureMaxWriteConcurrency(request.getConcurrency());
+        }
+
+        if (randomBoolean()) {
+            request.blobCount(between(1, 100));
+            blobStore.ensureMaxBlobCount(request.getBlobCount());
+        }
+
+        if (request.getBlobCount() > 3 || randomBoolean()) {
+            // only use the default blob size of 10MB if writing a small number of blobs, since this is all in-memory
+            request.maxBlobSize(new ByteSizeValue(between(1, 2048)));
+            blobStore.ensureMaxBlobSize(request.getMaxBlobSize().getBytes());
+        }
+
+        if (usually()) {
+            request.maxTotalDataSize(new ByteSizeValue(between(1, 1 << 20)));
+            blobStore.ensureMaxTotalBlobSize(request.getMaxTotalDataSize().getBytes());
+        }
+
+        request.timeout(TimeValue.timeValueSeconds(5));
+
+        final RepositoryAnalyzeAction.Response response = client().execute(RepositoryAnalyzeAction.INSTANCE, request)
+            .actionGet(30L, TimeUnit.SECONDS);
+
+        assertThat(response.status(), equalTo(RestStatus.OK));
+        assertThat(blobStore.currentPath, nullValue());
+    }
+
+    public static class TestPlugin extends Plugin implements RepositoryPlugin {
+
+        static final String ASSERTING_REPO_TYPE = "asserting";
+
+        @Override
+        public Map<String, Repository.Factory> getRepositories(
+            Environment env,
+            NamedXContentRegistry namedXContentRegistry,
+            ClusterService clusterService,
+            BigArrays bigArrays,
+            RecoverySettings recoverySettings
+        ) {
+            return Map.of(
+                ASSERTING_REPO_TYPE,
+                metadata -> new AssertingRepository(
+                    metadata,
+                    namedXContentRegistry,
+                    clusterService,
+                    bigArrays,
+                    recoverySettings,
+                    new BlobPath()
+                )
+            );
+        }
+    }
+
+    static class AssertingRepository extends BlobStoreRepository {
+
+        private final AtomicReference<BlobStore> blobStoreRef = new AtomicReference<>();
+
+        AssertingRepository(
+            RepositoryMetadata metadata,
+            NamedXContentRegistry namedXContentRegistry,
+            ClusterService clusterService,
+            BigArrays bigArrays,
+            RecoverySettings recoverySettings,
+            BlobPath basePath
+        ) {
+            super(metadata, namedXContentRegistry, clusterService, bigArrays, recoverySettings, basePath);
+        }
+
+        void setBlobStore(BlobStore blobStore) {
+            assertTrue(blobStoreRef.compareAndSet(null, blobStore));
+        }
+
+        @Override
+        protected BlobStore createBlobStore() {
+            final BlobStore blobStore = blobStoreRef.get();
+            assertNotNull(blobStore);
+            return blobStore;
+        }
+    }
+
+    static class AssertingBlobStore implements BlobStore {
+
+        @Nullable // if no current blob container
+        private String currentPath;
+
+        @Nullable // if no current blob container
+        private AssertingBlobContainer currentBlobContainer;
+
+        private Semaphore writeSemaphore = new Semaphore(new RepositoryAnalyzeAction.Request("dummy").getConcurrency());
+        private int maxBlobCount = new RepositoryAnalyzeAction.Request("dummy").getBlobCount();
+        private long maxBlobSize = new RepositoryAnalyzeAction.Request("dummy").getMaxBlobSize().getBytes();
+        private long maxTotalBlobSize = new RepositoryAnalyzeAction.Request("dummy").getMaxTotalDataSize().getBytes();
+
+        @Override
+        public BlobContainer blobContainer(BlobPath path) {
+            assertThat(path.buildAsString(), startsWith("temp-analysis-"));
+
+            synchronized (this) {
+                if (currentPath == null) {
+                    currentPath = path.buildAsString();
+                    currentBlobContainer = new AssertingBlobContainer(
+                        path,
+                        this::deleteContainer,
+                        writeSemaphore,
+                        maxBlobCount,
+                        maxBlobSize,
+                        maxTotalBlobSize
+                    );
+                }
+                assertThat(path.buildAsString(), equalTo(currentPath));
+                return currentBlobContainer;
+            }
+        }
+
+        private void deleteContainer(AssertingBlobContainer container) {
+            synchronized (this) {
+                assertThat(currentPath, equalTo(container.path.buildAsString()));
+                currentPath = null;
+                currentBlobContainer = null;
+            }
+        }
+
+        @Override
+        public void close() {}
+
+        public void ensureMaxWriteConcurrency(int concurrency) {
+            this.writeSemaphore = new Semaphore(concurrency);
+        }
+
+        public void ensureMaxBlobCount(int maxBlobCount) {
+            this.maxBlobCount = maxBlobCount;
+        }
+
+        public void ensureMaxBlobSize(long maxBlobSize) {
+            this.maxBlobSize = maxBlobSize;
+        }
+
+        public void ensureMaxTotalBlobSize(long maxTotalBlobSize) {
+            this.maxTotalBlobSize = maxTotalBlobSize;
+        }
+    }
+
+    static class AssertingBlobContainer implements BlobContainer {
+
+        private static final byte[] EMPTY = new byte[0];
+
+        private final BlobPath path;
+        private final Consumer<AssertingBlobContainer> deleteContainer;
+        private final Semaphore writeSemaphore;
+        private final int maxBlobCount;
+        private final long maxBlobSize;
+        private final long maxTotalBlobSize;
+        private final Map<String, byte[]> blobs = ConcurrentCollections.newConcurrentMap();
+        private final AtomicLong totalBytesWritten = new AtomicLong();
+
+        AssertingBlobContainer(
+            BlobPath path,
+            Consumer<AssertingBlobContainer> deleteContainer,
+            Semaphore writeSemaphore,
+            int maxBlobCount,
+            long maxBlobSize,
+            long maxTotalBlobSize
+        ) {
+            this.path = path;
+            this.deleteContainer = deleteContainer;
+            this.writeSemaphore = writeSemaphore;
+            this.maxBlobCount = maxBlobCount;
+            this.maxBlobSize = maxBlobSize;
+            this.maxTotalBlobSize = maxTotalBlobSize;
+        }
+
+        @Override
+        public BlobPath path() {
+            return path;
+        }
+
+        @Override
+        public boolean blobExists(String blobName) {
+            return blobs.containsKey(blobName);
+        }
+
+        @Override
+        public InputStream readBlob(String blobName) throws IOException {
+            final byte[] contents = blobs.get(blobName);
+            if (contents == null) {
+                throw new FileNotFoundException(blobName + " not found");
+            }
+            return new ByteArrayInputStream(contents);
+        }
+
+        @Override
+        public InputStream readBlob(String blobName, long position, long length) throws IOException {
+            final byte[] contents = blobs.get(blobName);
+            if (contents == null) {
+                throw new FileNotFoundException(blobName + " not found");
+            }
+            final int truncatedLength = Math.toIntExact(Math.min(length, contents.length - position));
+            return new ByteArrayInputStream(contents, Math.toIntExact(position), truncatedLength);
+        }
+
+        @Override
+        public void writeBlob(String blobName, InputStream inputStream, long blobSize, boolean failIfAlreadyExists) throws IOException {
+            assertTrue("must only write blob [" + blobName + "] non-atomically if it doesn't already exist", failIfAlreadyExists);
+            assertNull("blob [" + blobName + "] must not exist", blobs.get(blobName));
+
+            blobs.put(blobName, EMPTY);
+            writeBlobAtomic(blobName, inputStream, blobSize, false);
+        }
+
+        @Override
+        public void writeBlob(String blobName, BytesReference bytes, boolean failIfAlreadyExists) throws IOException {
+            writeBlob(blobName, bytes.streamInput(), bytes.length(), failIfAlreadyExists);
+        }
+
+        @Override
+        public void writeBlobAtomic(String blobName, BytesReference bytes, boolean failIfAlreadyExists) throws IOException {
+            writeBlobAtomic(blobName, bytes.streamInput(), bytes.length(), failIfAlreadyExists);
+        }
+
+        private void writeBlobAtomic(String blobName, InputStream inputStream, long blobSize, boolean failIfAlreadyExists)
+            throws IOException {
+
+            final byte[] existingBlob = blobs.get(blobName);
+            if (failIfAlreadyExists) {
+                assertNull("blob [" + blobName + "] must not exist", existingBlob);
+            }
+            final int existingSize = existingBlob == null ? 0 : existingBlob.length;
+
+            assertThat(blobSize, lessThanOrEqualTo(maxBlobSize));
+
+            assertTrue(writeSemaphore.tryAcquire());
+            try {
+                final byte[] contents = inputStream.readAllBytes();
+                assertThat((long) contents.length, equalTo(blobSize));
+                blobs.put(blobName, contents);
+                assertThat(blobs.size(), lessThanOrEqualTo(maxBlobCount));
+                final long currentTotal = totalBytesWritten.addAndGet(blobSize - existingSize);
+                assertThat(currentTotal, lessThanOrEqualTo(maxTotalBlobSize));
+            } finally {
+                writeSemaphore.release();
+            }
+        }
+
+        @Override
+        public DeleteResult delete() {
+            deleteContainer.accept(this);
+            final DeleteResult deleteResult = new DeleteResult(blobs.size(), blobs.values().stream().mapToLong(b -> b.length).sum());
+            blobs.clear();
+            return deleteResult;
+        }
+
+        @Override
+        public void deleteBlobsIgnoringIfNotExists(List<String> blobNames) {
+            blobs.keySet().removeAll(blobNames);
+        }
+
+        @Override
+        public Map<String, BlobMetadata> listBlobs() {
+            return blobs.entrySet()
+                .stream()
+                .collect(Collectors.toMap(Map.Entry::getKey, e -> new PlainBlobMetadata(e.getKey(), e.getValue().length)));
+        }
+
+        @Override
+        public Map<String, BlobContainer> children() {
+            return Map.of();
+        }
+
+        @Override
+        public Map<String, BlobMetadata> listBlobsByPrefix(String blobNamePrefix) {
+            final Map<String, BlobMetadata> blobMetadataByName = listBlobs();
+            blobMetadataByName.keySet().removeIf(s -> s.startsWith(blobNamePrefix) == false);
+            return blobMetadataByName;
+        }
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/BlobAnalyzeAction.java

@@ -0,0 +1,950 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.message.ParameterizedMessage;
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.action.ActionListenerResponseHandler;
+import org.elasticsearch.action.ActionRequest;
+import org.elasticsearch.action.ActionRequestValidationException;
+import org.elasticsearch.action.ActionResponse;
+import org.elasticsearch.action.ActionType;
+import org.elasticsearch.action.StepListener;
+import org.elasticsearch.action.support.ActionFilters;
+import org.elasticsearch.action.support.GroupedActionListener;
+import org.elasticsearch.action.support.HandledTransportAction;
+import org.elasticsearch.action.support.ThreadedActionListener;
+import org.elasticsearch.cluster.node.DiscoveryNode;
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.blobstore.BlobContainer;
+import org.elasticsearch.common.blobstore.BlobPath;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.io.stream.InputStreamStreamInput;
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.io.stream.Writeable;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.xcontent.ToXContentFragment;
+import org.elasticsearch.common.xcontent.ToXContentObject;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.repositories.RepositoriesService;
+import org.elasticsearch.repositories.Repository;
+import org.elasticsearch.repositories.RepositoryVerificationException;
+import org.elasticsearch.repositories.blobstore.BlobStoreRepository;
+import org.elasticsearch.tasks.CancellableTask;
+import org.elasticsearch.tasks.Task;
+import org.elasticsearch.tasks.TaskAwareRequest;
+import org.elasticsearch.tasks.TaskId;
+import org.elasticsearch.threadpool.ThreadPool;
+import org.elasticsearch.transport.TransportRequestOptions;
+import org.elasticsearch.transport.TransportService;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.LongPredicate;
+import java.util.stream.Collectors;
+
+import static org.elasticsearch.repositories.blobstore.testkit.SnapshotRepositoryTestKit.humanReadableNanos;
+
+/**
+ * Action which instructs a node to write a blob to the blob store and verify that it can be read correctly by other nodes. The other nodes
+ * may read the whole blob or just a range; we verify the data that is read by checksum using {@link GetBlobChecksumAction}.
+ *
+ * The other nodes may optionally be instructed to attempt their read just before the write completes (which may indicate that the blob is
+ * not found but must not yield partial data), and may optionally overwrite the blob while the reads are ongoing (which may yield either
+ * version of the blob, but again must not yield partial data). Usually, however, we write once and only read after the write completes, and
+ * in this case we insist that the read succeeds.
+ *
+ *
+ * <pre>
+ *
+ * +---------+                           +-------+                               +---------+
+ * | Writer  |                           | Repo  |                               | Readers |
+ * +---------+                           +-------+                               +---------+
+ *      | --------------\                    |                                        |
+ *      |-| Write phase |                    |                                        |
+ *      | |-------------|                    |                                        |
+ *      |                                    |                                        |
+ *      | Write blob with random content     |                                        |
+ *      |-----------------------------------→|                                        |
+ *      |                                    |                                        |
+ *      | Read range during write (rarely)   |                                        |
+ *      |----------------------------------------------------------------------------→|
+ *      |                                    |                                        |
+ *      |                                    |                             Read range |
+ *      |                                    |←---------------------------------------|
+ *      |                                    |                                        |
+ *      |                                    | Contents of range, or "not found"      |
+ *      |                                    |---------------------------------------→|
+ *      |                                    |                                        |
+ *      |                               Acknowledge read, including checksum if found |
+ *      |←----------------------------------------------------------------------------|
+ *      |                                    |                                        |
+ *      |                     Write complete |                                        |
+ *      |←-----------------------------------|                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      | -------------\                     |                                        |
+ *      |-| Read phase |                     |                                        |
+ *      | |------------|                     |                                        |
+ *      |                                    |                                        |
+ *      | Read range [a,b)                   |                                        |
+ *      |----------------------------------------------------------------------------→|
+ *      |                                    |                                        |
+ *      |                                    |                             Read range |
+ *      |                                    |←---------------------------------------|
+ *      |                                    |                                        |
+ *      | Overwrite blob (rarely)            |                                        |
+ *      |-----------------------------------→|                                        |
+ *      |                                    |                                        |
+ *      |                                    | Contents of range                      |
+ *      |                                    |---------------------------------------→|
+ *      |                                    |                                        |
+ *      |                 Overwrite complete |                                        |
+ *      |←-----------------------------------|                                        |
+ *      |                                    |                                        |
+ *      |                                    |               Ack read (with checksum) |
+ *      |←----------------------------------------------------------------------------|
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      |                                    |                                        |
+ *      | ---------------\                   |                                        |
+ *      |-| Verify phase |                   |                                        |
+ *      | |--------------|                   |                                        |
+ *      |                                    |                                        |
+ *      | Confirm checksums                  |                                        |
+ *      |------------------                  |                                        |
+ *      |                 |                  |                                        |
+ *      |←-----------------                  |                                        |
+ *      |                                    |                                        |
+ *
+ * </pre>
+ *
+ *
+ * On success, details of how long everything took are returned. On failure, cancels the remote read tasks to try and avoid consuming
+ * unnecessary resources.
+ *
+ */
+public class BlobAnalyzeAction extends ActionType<BlobAnalyzeAction.Response> {
+
+    private static final Logger logger = LogManager.getLogger(BlobAnalyzeAction.class);
+
+    public static final BlobAnalyzeAction INSTANCE = new BlobAnalyzeAction();
+    public static final String NAME = "cluster:admin/repository/analyze/blob";
+
+    private BlobAnalyzeAction() {
+        super(NAME, Response::new);
+    }
+
+    public static class TransportAction extends HandledTransportAction<Request, Response> {
+
+        private static final Logger logger = BlobAnalyzeAction.logger;
+
+        private final RepositoriesService repositoriesService;
+        private final TransportService transportService;
+
+        @Inject
+        public TransportAction(TransportService transportService, ActionFilters actionFilters, RepositoriesService repositoriesService) {
+            super(NAME, transportService, actionFilters, Request::new, ThreadPool.Names.SNAPSHOT);
+            this.repositoriesService = repositoriesService;
+            this.transportService = transportService;
+        }
+
+        @Override
+        protected void doExecute(Task task, Request request, ActionListener<Response> listener) {
+            final Repository repository = repositoriesService.repository(request.getRepositoryName());
+            if (repository instanceof BlobStoreRepository == false) {
+                throw new IllegalArgumentException("repository [" + request.getRepositoryName() + "] is not a blob-store repository");
+            }
+            if (repository.isReadOnly()) {
+                throw new IllegalArgumentException("repository [" + request.getRepositoryName() + "] is read-only");
+            }
+            final BlobStoreRepository blobStoreRepository = (BlobStoreRepository) repository;
+            final BlobContainer blobContainer = blobStoreRepository.blobStore().blobContainer(new BlobPath().add(request.blobPath));
+
+            logger.trace("handling [{}]", request);
+
+            assert task instanceof CancellableTask;
+            new BlobAnalysis(transportService, (CancellableTask) task, request, blobStoreRepository, blobContainer, listener).run();
+        }
+    }
+
+    /**
+     * Analysis on a single blob, performing the write(s) and orchestrating the read(s).
+     */
+    static class BlobAnalysis {
+        private final TransportService transportService;
+        private final CancellableTask task;
+        private final BlobAnalyzeAction.Request request;
+        private final BlobStoreRepository repository;
+        private final BlobContainer blobContainer;
+        private final ActionListener<BlobAnalyzeAction.Response> listener;
+        private final Random random;
+        private final boolean checksumWholeBlob;
+        private final long checksumStart;
+        private final long checksumEnd;
+        private final List<DiscoveryNode> earlyReadNodes;
+        private final List<DiscoveryNode> readNodes;
+        private final GroupedActionListener<NodeResponse> readNodesListener;
+        private final StepListener<WriteDetails> write1Step = new StepListener<>();
+        private final StepListener<WriteDetails> write2Step = new StepListener<>();
+
+        BlobAnalysis(
+            TransportService transportService,
+            CancellableTask task,
+            BlobAnalyzeAction.Request request,
+            BlobStoreRepository repository,
+            BlobContainer blobContainer,
+            ActionListener<BlobAnalyzeAction.Response> listener
+        ) {
+            this.transportService = transportService;
+            this.task = task;
+            this.request = request;
+            this.repository = repository;
+            this.blobContainer = blobContainer;
+            this.listener = listener;
+            this.random = new Random(this.request.seed);
+
+            checksumWholeBlob = random.nextBoolean();
+            if (checksumWholeBlob) {
+                checksumStart = 0L;
+                checksumEnd = request.targetLength;
+            } else {
+                checksumStart = randomLongBetween(0L, request.targetLength);
+                checksumEnd = randomLongBetween(checksumStart + 1, request.targetLength + 1);
+            }
+
+            final ArrayList<DiscoveryNode> nodes = new ArrayList<>(request.nodes); // copy for shuffling purposes
+            if (request.readEarly) {
+                Collections.shuffle(nodes, random);
+                earlyReadNodes = nodes.stream().limit(request.earlyReadNodeCount).collect(Collectors.toList());
+            } else {
+                earlyReadNodes = List.of();
+            }
+            Collections.shuffle(nodes, random);
+            readNodes = nodes.stream().limit(request.readNodeCount).collect(Collectors.toList());
+
+            final StepListener<Collection<NodeResponse>> readsCompleteStep = new StepListener<>();
+            readNodesListener = new GroupedActionListener<>(
+                new ThreadedActionListener<>(logger, transportService.getThreadPool(), ThreadPool.Names.SNAPSHOT, readsCompleteStep, false),
+                earlyReadNodes.size() + readNodes.size()
+            );
+
+            // The order is important in this chain: if writing fails then we may never even start all the reads, and we want to cancel
+            // any read tasks that were started, but the reads step only fails after all the reads have completed so there's no need to
+            // cancel anything.
+            write1Step.whenComplete(
+                write1Details -> write2Step.whenComplete(
+                    write2Details -> readsCompleteStep.whenComplete(
+                        responses -> onReadsComplete(responses, write1Details, write2Details),
+                        this::cleanUpAndReturnFailure
+                    ),
+                    this::cancelReadsCleanUpAndReturnFailure
+                ),
+                this::cancelReadsCleanUpAndReturnFailure
+            );
+        }
+
+        void run() {
+            writeRandomBlob(request.readEarly || random.nextBoolean(), true, this::doReadBeforeWriteComplete, write1Step);
+
+            if (request.writeAndOverwrite) {
+                assert request.targetLength <= Integer.MAX_VALUE : "oversized atomic write";
+                write1Step.whenComplete(ignored -> writeRandomBlob(true, false, this::doReadAfterWrite, write2Step), ignored -> {});
+            } else {
+                write2Step.onResponse(null);
+                doReadAfterWrite();
+            }
+        }
+
+        private void writeRandomBlob(boolean atomic, boolean failIfExists, Runnable onLastRead, StepListener<WriteDetails> stepListener) {
+            assert atomic == false || request.targetLength <= Integer.MAX_VALUE : "oversized atomic write";
+            final RandomBlobContent content = new RandomBlobContent(
+                request.getRepositoryName(),
+                random.nextLong(),
+                task::isCancelled,
+                onLastRead
+            );
+            final AtomicLong throttledNanos = new AtomicLong();
+
+            if (logger.isTraceEnabled()) {
+                logger.trace("writing blob [atomic={}, failIfExists={}] for [{}]", atomic, failIfExists, request.getDescription());
+            }
+            final long startNanos = System.nanoTime();
+            ActionListener.completeWith(stepListener, () -> {
+
+                // TODO push some of this writing logic down into the blob container implementation.
+                // E.g. for S3 blob containers we would like to choose somewhat more randomly between single-part and multi-part uploads,
+                // rather than relying on the usual distinction based on the size of the blob.
+
+                if (atomic || (request.targetLength <= Integer.MAX_VALUE && random.nextBoolean())) {
+                    final RandomBlobContentBytesReference bytesReference = new RandomBlobContentBytesReference(
+                        content,
+                        Math.toIntExact(request.getTargetLength())
+                    ) {
+                        @Override
+                        public StreamInput streamInput() throws IOException {
+                            return new InputStreamStreamInput(
+                                repository.maybeRateLimitSnapshots(super.streamInput(), throttledNanos::addAndGet)
+                            );
+                        }
+                    };
+                    if (atomic) {
+                        blobContainer.writeBlobAtomic(request.blobName, bytesReference, failIfExists);
+                    } else {
+                        blobContainer.writeBlob(request.blobName, bytesReference, failIfExists);
+                    }
+                } else {
+                    blobContainer.writeBlob(
+                        request.blobName,
+                        repository.maybeRateLimitSnapshots(
+                            new RandomBlobContentStream(content, request.getTargetLength()),
+                            throttledNanos::addAndGet
+                        ),
+                        request.targetLength,
+                        failIfExists
+                    );
+                }
+                final long elapsedNanos = System.nanoTime() - startNanos;
+                final long checksum = content.getChecksum(checksumStart, checksumEnd);
+                if (logger.isTraceEnabled()) {
+                    logger.trace("finished writing blob for [{}], got checksum [{}]", request.getDescription(), checksum);
+                }
+                return new WriteDetails(request.targetLength, elapsedNanos, throttledNanos.get(), checksum);
+            });
+        }
+
+        private void doReadBeforeWriteComplete() {
+            if (earlyReadNodes.isEmpty() == false) {
+                if (logger.isTraceEnabled()) {
+                    logger.trace("sending read request to [{}] for [{}] before write complete", earlyReadNodes, request.getDescription());
+                }
+                readOnNodes(earlyReadNodes, true);
+            }
+        }
+
+        private void doReadAfterWrite() {
+            if (logger.isTraceEnabled()) {
+                logger.trace("sending read request to [{}] for [{}] after write complete", readNodes, request.getDescription());
+            }
+            readOnNodes(readNodes, false);
+        }
+
+        private void readOnNodes(List<DiscoveryNode> nodes, boolean beforeWriteComplete) {
+            for (DiscoveryNode node : nodes) {
+                if (task.isCancelled()) {
+                    // record dummy response since we're already on the path to failure
+                    readNodesListener.onResponse(
+                        new NodeResponse(node, beforeWriteComplete, GetBlobChecksumAction.Response.BLOB_NOT_FOUND)
+                    );
+                } else {
+                    // no need for extra synchronization after checking if we were cancelled a couple of lines ago -- we haven't notified
+                    // the outer listener yet so any bans on the children are still in place
+                    final GetBlobChecksumAction.Request blobChecksumRequest = getBlobChecksumRequest();
+                    transportService.sendChildRequest(
+                        node,
+                        GetBlobChecksumAction.NAME,
+                        blobChecksumRequest,
+                        task,
+                        TransportRequestOptions.EMPTY,
+                        new ActionListenerResponseHandler<>(new ActionListener<>() {
+                            @Override
+                            public void onResponse(GetBlobChecksumAction.Response response) {
+                                if (beforeWriteComplete == false && response.isNotFound()) {
+                                    readNodesListener.onFailure(
+                                        new RepositoryVerificationException(
+                                            request.getRepositoryName(),
+                                            "["
+                                                + blobChecksumRequest
+                                                + "] (after write complete) failed on node ["
+                                                + node
+                                                + "]: blob not found"
+                                        )
+                                    );
+                                } else {
+                                    readNodesListener.onResponse(makeNodeResponse(node, beforeWriteComplete, response));
+                                }
+                            }
+
+                            @Override
+                            public void onFailure(Exception e) {
+                                readNodesListener.onFailure(
+                                    new RepositoryVerificationException(
+                                        request.getRepositoryName(),
+                                        "["
+                                            + blobChecksumRequest
+                                            + "] ("
+                                            + (beforeWriteComplete ? "before" : "after")
+                                            + " write complete) failed on node ["
+                                            + node
+                                            + "]",
+                                        e
+                                    )
+                                );
+
+                            }
+                        }, GetBlobChecksumAction.Response::new)
+                    );
+                }
+            }
+        }
+
+        private GetBlobChecksumAction.Request getBlobChecksumRequest() {
+            return new GetBlobChecksumAction.Request(
+                request.getRepositoryName(),
+                request.getBlobPath(),
+                request.getBlobName(),
+                checksumStart,
+                checksumWholeBlob ? 0L : checksumEnd
+            );
+        }
+
+        private NodeResponse makeNodeResponse(DiscoveryNode node, boolean beforeWriteComplete, GetBlobChecksumAction.Response response) {
+            logger.trace(
+                "received read response [{}] from [{}] for [{}] [beforeWriteComplete={}]",
+                response,
+                node,
+                request.getDescription(),
+                beforeWriteComplete
+            );
+            return new NodeResponse(node, beforeWriteComplete, response);
+        }
+
+        private void cancelReadsCleanUpAndReturnFailure(Exception exception) {
+            transportService.getTaskManager().cancelTaskAndDescendants(task, "task failed", false, ActionListener.wrap(() -> {}));
+            cleanUpAndReturnFailure(exception);
+        }
+
+        private void cleanUpAndReturnFailure(Exception exception) {
+            if (logger.isTraceEnabled()) {
+                logger.trace(new ParameterizedMessage("analysis failed [{}] cleaning up", request.getDescription()), exception);
+            }
+            try {
+                blobContainer.deleteBlobsIgnoringIfNotExists(List.of(request.blobName));
+            } catch (IOException ioException) {
+                exception.addSuppressed(ioException);
+                logger.warn(
+                    new ParameterizedMessage(
+                        "failure during post-failure cleanup while analysing repository [{}], you may need to manually remove [{}/{}]",
+                        request.getRepositoryName(),
+                        request.getBlobPath(),
+                        request.getBlobName()
+                    ),
+                    exception
+                );
+            }
+            listener.onFailure(
+                new RepositoryVerificationException(
+                    request.getRepositoryName(),
+                    "failure processing [" + request.getDescription() + "]",
+                    exception
+                )
+            );
+        }
+
+        private void onReadsComplete(Collection<NodeResponse> responses, WriteDetails write1Details, @Nullable WriteDetails write2Details) {
+            if (task.isCancelled()) {
+                cleanUpAndReturnFailure(
+                    new RepositoryVerificationException(request.getRepositoryName(), "cancelled during checksum verification")
+                );
+                return;
+            }
+
+            final long checksumLength = checksumEnd - checksumStart;
+            final String expectedChecksumDescription;
+            final LongPredicate checksumPredicate;
+            if (write2Details == null) {
+                checksumPredicate = l -> l == write1Details.checksum;
+                expectedChecksumDescription = Long.toString(write1Details.checksum);
+            } else {
+                checksumPredicate = l -> l == write1Details.checksum || l == write2Details.checksum;
+                expectedChecksumDescription = write1Details.checksum + " or " + write2Details.checksum;
+            }
+
+            RepositoryVerificationException failure = null;
+            for (final NodeResponse nodeResponse : responses) {
+                final GetBlobChecksumAction.Response response = nodeResponse.response;
+                final RepositoryVerificationException nodeFailure;
+                if (response.isNotFound()) {
+                    if (request.readEarly) {
+                        nodeFailure = null; // "not found" is legitimate iff we tried to read it before the write completed
+                    } else if (request.writeAndOverwrite) {
+                        nodeFailure = null; // overwrites surprisingly not necessarily atomic, e.g. in a FsBlobContainer
+                    } else {
+                        nodeFailure = new RepositoryVerificationException(
+                            request.getRepositoryName(),
+                            "node [" + nodeResponse.node + "] reported blob not found after it was written"
+                        );
+                    }
+                } else {
+                    final long actualChecksum = response.getChecksum();
+                    if (response.getBytesRead() == checksumLength && checksumPredicate.test(actualChecksum)) {
+                        nodeFailure = null; // checksum ok
+                    } else {
+                        nodeFailure = new RepositoryVerificationException(
+                            request.getRepositoryName(),
+                            "node ["
+                                + nodeResponse.node
+                                + "] failed during analysis: expected to read ["
+                                + checksumStart
+                                + "-"
+                                + checksumEnd
+                                + "], ["
+                                + checksumLength
+                                + "] bytes, with checksum ["
+                                + expectedChecksumDescription
+                                + "] but read ["
+                                + response
+                                + "]"
+                        );
+                    }
+                }
+
+                if (nodeFailure != null) {
+                    if (failure == null) {
+                        failure = nodeFailure;
+                    } else {
+                        failure.addSuppressed(nodeFailure);
+                    }
+                }
+            }
+            if (failure != null) {
+                cleanUpAndReturnFailure(failure);
+                return;
+            }
+
+            final long overwriteElapsedNanos = write2Details == null ? 0L : write2Details.elapsedNanos;
+            final long overwriteThrottledNanos = write2Details == null ? 0L : write2Details.throttledNanos;
+            listener.onResponse(
+                new Response(
+                    transportService.getLocalNode().getId(),
+                    transportService.getLocalNode().getName(),
+                    request.blobName,
+                    request.targetLength,
+                    request.readEarly,
+                    request.writeAndOverwrite,
+                    checksumStart,
+                    checksumEnd,
+                    write1Details.elapsedNanos,
+                    overwriteElapsedNanos,
+                    write1Details.throttledNanos + overwriteThrottledNanos,
+                    responses.stream()
+                        .map(
+                            nr -> new ReadDetail(
+                                nr.node.getId(),
+                                nr.node.getName(),
+                                nr.beforeWriteComplete,
+                                nr.response.isNotFound(),
+                                nr.response.getFirstByteNanos(),
+                                nr.response.getElapsedNanos(),
+                                nr.response.getThrottleNanos()
+                            )
+                        )
+                        .collect(Collectors.toList())
+                )
+            );
+        }
+
+        /**
+         * @return random non-negative long in [min, max)
+         */
+        private long randomLongBetween(long min, long max) {
+            assert 0 <= min && min <= max;
+            final long range = max - min;
+            return range == 0 ? min : min + (random.nextLong() & Long.MAX_VALUE) % range;
+        }
+    }
+
+    private static class NodeResponse {
+        final DiscoveryNode node;
+        final boolean beforeWriteComplete;
+        final GetBlobChecksumAction.Response response;
+
+        NodeResponse(DiscoveryNode node, boolean beforeWriteComplete, GetBlobChecksumAction.Response response) {
+            this.node = node;
+            this.beforeWriteComplete = beforeWriteComplete;
+            this.response = response;
+        }
+    }
+
+    private static class WriteDetails {
+        final long bytesWritten;
+        final long elapsedNanos;
+        final long throttledNanos;
+        final long checksum;
+
+        private WriteDetails(long bytesWritten, long elapsedNanos, long throttledNanos, long checksum) {
+            this.bytesWritten = bytesWritten;
+            this.elapsedNanos = elapsedNanos;
+            this.throttledNanos = throttledNanos;
+            this.checksum = checksum;
+        }
+    }
+
+    public static class Request extends ActionRequest implements TaskAwareRequest {
+        private final String repositoryName;
+        private final String blobPath;
+        private final String blobName;
+
+        private final long targetLength;
+        private final long seed;
+        private final List<DiscoveryNode> nodes;
+        private final int readNodeCount;
+        private final int earlyReadNodeCount;
+        private final boolean readEarly;
+        private final boolean writeAndOverwrite;
+
+        Request(
+            String repositoryName,
+            String blobPath,
+            String blobName,
+            long targetLength,
+            long seed,
+            List<DiscoveryNode> nodes,
+            int readNodeCount,
+            int earlyReadNodeCount,
+            boolean readEarly,
+            boolean writeAndOverwrite
+        ) {
+            assert 0 < targetLength;
+            assert targetLength <= Integer.MAX_VALUE || (readEarly == false && writeAndOverwrite == false) : "oversized atomic write";
+            this.repositoryName = repositoryName;
+            this.blobPath = blobPath;
+            this.blobName = blobName;
+            this.targetLength = targetLength;
+            this.seed = seed;
+            this.nodes = nodes;
+            this.readNodeCount = readNodeCount;
+            this.earlyReadNodeCount = earlyReadNodeCount;
+            this.readEarly = readEarly;
+            this.writeAndOverwrite = writeAndOverwrite;
+        }
+
+        Request(StreamInput in) throws IOException {
+            super(in);
+            repositoryName = in.readString();
+            blobPath = in.readString();
+            blobName = in.readString();
+            targetLength = in.readVLong();
+            seed = in.readLong();
+            nodes = in.readList(DiscoveryNode::new);
+            readNodeCount = in.readVInt();
+            earlyReadNodeCount = in.readVInt();
+            readEarly = in.readBoolean();
+            writeAndOverwrite = in.readBoolean();
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            super.writeTo(out);
+            out.writeString(repositoryName);
+            out.writeString(blobPath);
+            out.writeString(blobName);
+            out.writeVLong(targetLength);
+            out.writeLong(seed);
+            out.writeList(nodes);
+            out.writeVInt(readNodeCount);
+            out.writeVInt(earlyReadNodeCount);
+            out.writeBoolean(readEarly);
+            out.writeBoolean(writeAndOverwrite);
+        }
+
+        @Override
+        public ActionRequestValidationException validate() {
+            return null;
+        }
+
+        @Override
+        public String getDescription() {
+            return "blob analysis ["
+                + repositoryName
+                + ":"
+                + blobPath
+                + "/"
+                + blobName
+                + ", length="
+                + targetLength
+                + ", seed="
+                + seed
+                + ", readEarly="
+                + readEarly
+                + ", writeAndOverwrite="
+                + writeAndOverwrite
+                + "]";
+        }
+
+        @Override
+        public String toString() {
+            return "BlobAnalyzeAction.Request{" + getDescription() + "}";
+        }
+
+        @Override
+        public Task createTask(long id, String type, String action, TaskId parentTaskId, Map<String, String> headers) {
+            return new CancellableTask(id, type, action, getDescription(), parentTaskId, headers) {
+                @Override
+                public boolean shouldCancelChildrenOnCancellation() {
+                    return true;
+                }
+            };
+        }
+
+        public String getRepositoryName() {
+            return repositoryName;
+        }
+
+        public String getBlobPath() {
+            return blobPath;
+        }
+
+        public String getBlobName() {
+            return blobName;
+        }
+
+        public long getTargetLength() {
+            return targetLength;
+        }
+
+        public long getSeed() {
+            return seed;
+        }
+
+    }
+
+    public static class Response extends ActionResponse implements ToXContentObject {
+
+        private final String nodeId;
+        private final String nodeName;
+        private final String blobName;
+        private final long blobLength;
+        private final boolean readEarly;
+        private final boolean overwrite;
+        private final long checksumStart;
+        private final long checksumEnd;
+
+        private final long writeElapsedNanos;
+        private final long overwriteElapsedNanos;
+        private final long writeThrottledNanos;
+        private final List<ReadDetail> readDetails;
+
+        public Response(
+            String nodeId,
+            String nodeName,
+            String blobName,
+            long blobLength,
+            boolean readEarly,
+            boolean overwrite,
+            long checksumStart,
+            long checksumEnd,
+            long writeElapsedNanos,
+            long overwriteElapsedNanos,
+            long writeThrottledNanos,
+            List<ReadDetail> readDetails
+        ) {
+            this.nodeId = nodeId;
+            this.nodeName = nodeName;
+            this.blobName = blobName;
+            this.blobLength = blobLength;
+            this.readEarly = readEarly;
+            this.overwrite = overwrite;
+            this.checksumStart = checksumStart;
+            this.checksumEnd = checksumEnd;
+            this.writeElapsedNanos = writeElapsedNanos;
+            this.overwriteElapsedNanos = overwriteElapsedNanos;
+            this.writeThrottledNanos = writeThrottledNanos;
+            this.readDetails = readDetails;
+        }
+
+        public Response(StreamInput in) throws IOException {
+            super(in);
+            nodeId = in.readString();
+            nodeName = in.readString();
+            blobName = in.readString();
+            blobLength = in.readVLong();
+            readEarly = in.readBoolean();
+            overwrite = in.readBoolean();
+            checksumStart = in.readVLong();
+            checksumEnd = in.readVLong();
+            writeElapsedNanos = in.readVLong();
+            overwriteElapsedNanos = in.readVLong();
+            writeThrottledNanos = in.readVLong();
+            readDetails = in.readList(ReadDetail::new);
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            out.writeString(nodeId);
+            out.writeString(nodeName);
+            out.writeString(blobName);
+            out.writeVLong(blobLength);
+            out.writeBoolean(readEarly);
+            out.writeBoolean(overwrite);
+            out.writeVLong(checksumStart);
+            out.writeVLong(checksumEnd);
+            out.writeVLong(writeElapsedNanos);
+            out.writeVLong(overwriteElapsedNanos);
+            out.writeVLong(writeThrottledNanos);
+            out.writeList(readDetails);
+        }
+
+        @Override
+        public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+            builder.startObject();
+
+            builder.startObject("blob");
+            builder.field("name", blobName);
+            builder.humanReadableField("size_bytes", "size", new ByteSizeValue(blobLength));
+            builder.field("read_start", checksumStart);
+            builder.field("read_end", checksumEnd);
+            builder.field("read_early", readEarly);
+            builder.field("overwritten", overwrite);
+            builder.endObject();
+
+            builder.startObject("writer_node");
+            builder.field("id", nodeId);
+            builder.field("name", nodeName);
+            builder.endObject();
+
+            humanReadableNanos(builder, "write_elapsed_nanos", "write_elapsed", writeElapsedNanos);
+            if (overwrite) {
+                humanReadableNanos(builder, "overwrite_elapsed_nanos", "overwrite_elapsed", overwriteElapsedNanos);
+            }
+            humanReadableNanos(builder, "write_throttled_nanos", "write_throttled", writeThrottledNanos);
+
+            builder.startArray("reads");
+            for (ReadDetail readDetail : readDetails) {
+                readDetail.toXContent(builder, params);
+            }
+            builder.endArray();
+
+            builder.endObject();
+            return builder;
+        }
+
+        long getWriteBytes() {
+            return blobLength + (overwrite ? blobLength : 0L);
+        }
+
+        long getWriteThrottledNanos() {
+            return writeThrottledNanos;
+        }
+
+        long getWriteElapsedNanos() {
+            return writeElapsedNanos + overwriteElapsedNanos;
+        }
+
+        List<ReadDetail> getReadDetails() {
+            return readDetails;
+        }
+
+        long getChecksumBytes() {
+            return checksumEnd - checksumStart;
+        }
+    }
+
+    public static class ReadDetail implements Writeable, ToXContentFragment {
+
+        private final String nodeId;
+        private final String nodeName;
+        private final boolean beforeWriteComplete;
+        private final boolean isNotFound;
+        private final long firstByteNanos;
+        private final long throttleNanos;
+        private final long elapsedNanos;
+
+        public ReadDetail(
+            String nodeId,
+            String nodeName,
+            boolean beforeWriteComplete,
+            boolean isNotFound,
+            long firstByteNanos,
+            long elapsedNanos,
+            long throttleNanos
+        ) {
+            this.nodeId = nodeId;
+            this.nodeName = nodeName;
+            this.beforeWriteComplete = beforeWriteComplete;
+            this.isNotFound = isNotFound;
+            this.firstByteNanos = firstByteNanos;
+            this.throttleNanos = throttleNanos;
+            this.elapsedNanos = elapsedNanos;
+        }
+
+        public ReadDetail(StreamInput in) throws IOException {
+            nodeId = in.readString();
+            nodeName = in.readString();
+            beforeWriteComplete = in.readBoolean();
+            isNotFound = in.readBoolean();
+            firstByteNanos = in.readVLong();
+            throttleNanos = in.readVLong();
+            elapsedNanos = in.readVLong();
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            out.writeString(nodeId);
+            out.writeString(nodeName);
+            out.writeBoolean(beforeWriteComplete);
+            out.writeBoolean(isNotFound);
+            out.writeVLong(firstByteNanos);
+            out.writeVLong(throttleNanos);
+            out.writeVLong(elapsedNanos);
+        }
+
+        @Override
+        public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+            builder.startObject();
+
+            builder.startObject("node");
+            builder.field("id", nodeId);
+            builder.field("name", nodeName);
+            builder.endObject();
+
+            if (beforeWriteComplete) {
+                builder.field("before_write_complete", true);
+            }
+
+            if (isNotFound) {
+                builder.field("found", false);
+            } else {
+                builder.field("found", true);
+                humanReadableNanos(builder, "first_byte_time_nanos", "first_byte_time", firstByteNanos);
+                humanReadableNanos(builder, "elapsed_nanos", "elapsed", elapsedNanos);
+                humanReadableNanos(builder, "throttled_nanos", "throttled", throttleNanos);
+            }
+
+            builder.endObject();
+            return builder;
+        }
+
+        long getFirstByteNanos() {
+            return firstByteNanos;
+        }
+
+        long getThrottledNanos() {
+            return throttleNanos;
+        }
+
+        long getElapsedNanos() {
+            return elapsedNanos;
+        }
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/GetBlobChecksumAction.java

@@ -0,0 +1,372 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.action.ActionRequest;
+import org.elasticsearch.action.ActionRequestValidationException;
+import org.elasticsearch.action.ActionResponse;
+import org.elasticsearch.action.ActionType;
+import org.elasticsearch.action.support.ActionFilters;
+import org.elasticsearch.action.support.HandledTransportAction;
+import org.elasticsearch.common.blobstore.BlobContainer;
+import org.elasticsearch.common.blobstore.BlobPath;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.unit.ByteSizeUnit;
+import org.elasticsearch.core.internal.io.IOUtils;
+import org.elasticsearch.repositories.RepositoriesService;
+import org.elasticsearch.repositories.Repository;
+import org.elasticsearch.repositories.RepositoryVerificationException;
+import org.elasticsearch.repositories.blobstore.BlobStoreRepository;
+import org.elasticsearch.tasks.CancellableTask;
+import org.elasticsearch.tasks.Task;
+import org.elasticsearch.tasks.TaskAwareRequest;
+import org.elasticsearch.tasks.TaskId;
+import org.elasticsearch.threadpool.ThreadPool;
+import org.elasticsearch.transport.TransportService;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.NoSuchFileException;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.zip.CRC32;
+
+/**
+ * Action which instructs a node to read a range of a blob from a {@link org.elasticsearch.repositories.blobstore.BlobStoreRepository}
+ * (possibly the entire blob) and compute its checksum. It is acceptable if the blob is not found but we do not accept the blob being
+ * otherwise unreadable.
+ */
+public class GetBlobChecksumAction extends ActionType<GetBlobChecksumAction.Response> {
+
+    private static final Logger logger = LogManager.getLogger(GetBlobChecksumAction.class);
+
+    public static final GetBlobChecksumAction INSTANCE = new GetBlobChecksumAction();
+
+    public static final String NAME = "cluster:admin/repository/analyze/blob/read";
+
+    private GetBlobChecksumAction() {
+        super(NAME, Response::new);
+    }
+
+    public static class TransportAction extends HandledTransportAction<Request, Response> {
+
+        private static final Logger logger = GetBlobChecksumAction.logger;
+
+        private static final int BUFFER_SIZE = ByteSizeUnit.KB.toIntBytes(8);
+
+        private final RepositoriesService repositoriesService;
+
+        @Inject
+        public TransportAction(TransportService transportService, ActionFilters actionFilters, RepositoriesService repositoriesService) {
+            super(NAME, transportService, actionFilters, Request::new, ThreadPool.Names.SNAPSHOT);
+            this.repositoriesService = repositoriesService;
+        }
+
+        @Override
+        protected void doExecute(Task task, Request request, ActionListener<Response> listener) {
+
+            assert task instanceof CancellableTask;
+            CancellableTask cancellableTask = (CancellableTask) task;
+
+            final Repository repository = repositoriesService.repository(request.getRepositoryName());
+            if (repository instanceof BlobStoreRepository == false) {
+                throw new IllegalArgumentException("repository [" + request.getRepositoryName() + "] is not a blob store repository");
+            }
+
+            final BlobStoreRepository blobStoreRepository = (BlobStoreRepository) repository;
+            final BlobContainer blobContainer = blobStoreRepository.blobStore().blobContainer(new BlobPath().add(request.getBlobPath()));
+
+            logger.trace("handling [{}]", request);
+
+            final InputStream rawInputStream;
+            try {
+                if (request.isWholeBlob()) {
+                    rawInputStream = blobContainer.readBlob(request.getBlobName());
+                } else {
+                    rawInputStream = blobContainer.readBlob(request.getBlobName(), request.getRangeStart(), request.getRangeLength());
+                }
+            } catch (FileNotFoundException | NoSuchFileException e) {
+                logger.trace("blob not found for [{}]", request);
+                listener.onResponse(Response.BLOB_NOT_FOUND);
+                return;
+            } catch (IOException e) {
+                logger.warn("failed to read blob for [{}]", request);
+                listener.onFailure(e);
+                return;
+            }
+
+            logger.trace("reading blob for [{}]", request);
+
+            final AtomicLong throttleNanos = new AtomicLong();
+            final InputStream throttledInputStream = blobStoreRepository.maybeRateLimitRestores(rawInputStream, throttleNanos::addAndGet);
+            final CRC32 crc32 = new CRC32();
+            final byte[] buffer = new byte[BUFFER_SIZE];
+            long bytesRead = 0L;
+            final long startTimeNanos = System.nanoTime();
+            long firstByteNanos = startTimeNanos;
+
+            boolean success = false;
+            try {
+                while (true) {
+                    final int readSize;
+                    try {
+                        readSize = throttledInputStream.read(buffer, 0, buffer.length);
+                    } catch (IOException e) {
+                        logger.warn("exception while read blob for [{}]", request);
+                        listener.onFailure(e);
+                        return;
+                    }
+
+                    if (readSize == -1) {
+                        break;
+                    }
+
+                    if (readSize > 0) {
+                        if (bytesRead == 0L) {
+                            firstByteNanos = System.nanoTime();
+                        }
+
+                        crc32.update(buffer, 0, readSize);
+                        bytesRead += readSize;
+                    }
+
+                    if (cancellableTask.isCancelled()) {
+                        throw new RepositoryVerificationException(
+                            request.repositoryName,
+                            "cancelled [" + request.getDescription() + "] after reading [" + bytesRead + "] bytes"
+                        );
+                    }
+                }
+                success = true;
+            } finally {
+                if (success == false) {
+                    IOUtils.closeWhileHandlingException(throttledInputStream);
+                }
+            }
+            try {
+                throttledInputStream.close();
+            } catch (IOException e) {
+                throw new RepositoryVerificationException(
+                    request.repositoryName,
+                    "failed to close input stream when handling [" + request.getDescription() + "]",
+                    e
+                );
+            }
+
+            final long endTimeNanos = System.nanoTime();
+
+            if (request.isWholeBlob() == false && bytesRead != request.getRangeLength()) {
+                throw new RepositoryVerificationException(
+                    request.repositoryName,
+                    "unexpectedly read [" + bytesRead + "] bytes when handling [" + request.getDescription() + "]"
+                );
+            }
+
+            final Response response = new Response(
+                bytesRead,
+                crc32.getValue(),
+                firstByteNanos - startTimeNanos,
+                endTimeNanos - startTimeNanos,
+                throttleNanos.get()
+            );
+            logger.trace("responding to [{}] with [{}]", request, response);
+            listener.onResponse(response);
+        }
+
+    }
+
+    public static class Request extends ActionRequest implements TaskAwareRequest {
+
+        private final String repositoryName;
+        private final String blobPath;
+        private final String blobName;
+
+        // Requesting the range [rangeStart, rangeEnd), but we treat the range [0, 0) as a special case indicating to read the whole blob.
+        private final long rangeStart;
+        private final long rangeEnd;
+
+        Request(StreamInput in) throws IOException {
+            super(in);
+            repositoryName = in.readString();
+            blobPath = in.readString();
+            blobName = in.readString();
+            rangeStart = in.readVLong();
+            rangeEnd = in.readVLong();
+        }
+
+        @Override
+        public ActionRequestValidationException validate() {
+            return null;
+        }
+
+        Request(String repositoryName, String blobPath, String blobName, long rangeStart, long rangeEnd) {
+            assert rangeStart == 0L && rangeEnd == 0L || 0L <= rangeStart && rangeStart < rangeEnd : rangeStart + "-" + rangeEnd;
+            this.repositoryName = repositoryName;
+            this.blobPath = blobPath;
+            this.blobName = blobName;
+            this.rangeStart = rangeStart;
+            this.rangeEnd = rangeEnd;
+        }
+
+        public String getRepositoryName() {
+            return repositoryName;
+        }
+
+        public String getBlobPath() {
+            return blobPath;
+        }
+
+        public String getBlobName() {
+            return blobName;
+        }
+
+        public long getRangeStart() {
+            assert isWholeBlob() == false;
+            return rangeStart;
+        }
+
+        public long getRangeEnd() {
+            assert isWholeBlob() == false;
+            return rangeEnd;
+        }
+
+        public long getRangeLength() {
+            assert isWholeBlob() == false;
+            return rangeEnd - rangeStart;
+        }
+
+        /**
+         * @return whether we should read the whole blob or a range.
+         */
+        boolean isWholeBlob() {
+            return rangeEnd == 0L;
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            super.writeTo(out);
+            out.writeString(repositoryName);
+            out.writeString(blobPath);
+            out.writeString(blobName);
+            out.writeVLong(rangeStart);
+            out.writeVLong(rangeEnd);
+        }
+
+        @Override
+        public String getDescription() {
+            return "retrieve ["
+                + (isWholeBlob() ? "whole blob" : (getRangeStart() + "-" + getRangeEnd()))
+                + "] from ["
+                + getRepositoryName()
+                + ":"
+                + getBlobPath()
+                + "/"
+                + getBlobName()
+                + "]";
+        }
+
+        @Override
+        public String toString() {
+            return "GetRepositoryBlobChecksumRequest{" + getDescription() + "}";
+        }
+
+        @Override
+        public Task createTask(long id, String type, String action, TaskId parentTaskId, Map<String, String> headers) {
+            return new CancellableTask(id, type, action, getDescription(), parentTaskId, headers) {
+                @Override
+                public boolean shouldCancelChildrenOnCancellation() {
+                    return false; // no children
+                }
+            };
+        }
+    }
+
+    public static class Response extends ActionResponse {
+
+        static Response BLOB_NOT_FOUND = new Response(0L, 0L, 0L, 0L, 0L);
+
+        private final long bytesRead; // 0 if not found
+        private final long checksum; // 0 if not found
+        private final long firstByteNanos; // 0 if not found
+        private final long elapsedNanos; // 0 if not found
+        private final long throttleNanos; // 0 if not found
+
+        Response(long bytesRead, long checksum, long firstByteNanos, long elapsedNanos, long throttleNanos) {
+            this.bytesRead = bytesRead;
+            this.checksum = checksum;
+            this.firstByteNanos = firstByteNanos;
+            this.elapsedNanos = elapsedNanos;
+            this.throttleNanos = throttleNanos;
+        }
+
+        Response(StreamInput in) throws IOException {
+            super(in);
+            this.bytesRead = in.readVLong();
+            this.checksum = in.readLong();
+            this.firstByteNanos = in.readVLong();
+            this.elapsedNanos = in.readVLong();
+            this.throttleNanos = in.readVLong();
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            out.writeVLong(bytesRead);
+            out.writeLong(checksum);
+            out.writeVLong(firstByteNanos);
+            out.writeVLong(elapsedNanos);
+            out.writeVLong(throttleNanos);
+        }
+
+        @Override
+        public String toString() {
+            return "GetRepositoryBlobChecksumResponse{"
+                + "bytesRead="
+                + bytesRead
+                + ", checksum="
+                + checksum
+                + ", firstByteNanos="
+                + firstByteNanos
+                + ", elapsedNanos="
+                + elapsedNanos
+                + ", throttleNanos="
+                + throttleNanos
+                + '}';
+        }
+
+        public long getBytesRead() {
+            return bytesRead;
+        }
+
+        public long getChecksum() {
+            return checksum;
+        }
+
+        public long getFirstByteNanos() {
+            return firstByteNanos;
+        }
+
+        public long getElapsedNanos() {
+            return elapsedNanos;
+        }
+
+        public long getThrottleNanos() {
+            return throttleNanos;
+        }
+
+        public boolean isNotFound() {
+            return bytesRead == 0L && checksum == 0L && firstByteNanos == 0L && elapsedNanos == 0L && throttleNanos == 0L;
+        }
+
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RandomBlobContent.java

@@ -0,0 +1,82 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.repositories.RepositoryVerificationException;
+
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BooleanSupplier;
+import java.util.zip.CRC32;
+
+/**
+ * A random ~8kB block of data that is arbitrarily repeated, for use as a dummy payload for testing a blob store repo.
+ */
+class RandomBlobContent {
+
+    static final int BUFFER_SIZE = 8191; // nearly 8kB, but prime, so unlikely to be aligned with any other block sizes
+
+    final byte[] buffer = new byte[BUFFER_SIZE];
+    private final BooleanSupplier isCancelledSupplier;
+    private final AtomicReference<Runnable> onLastRead;
+    private final String repositoryName;
+
+    /**
+     * @param repositoryName The name of the repository being tested, for use in exception messages.
+     * @param seed RNG seed to use for its contents.
+     * @param isCancelledSupplier Predicate that causes reads to throw a {@link RepositoryVerificationException}, allowing for fast failure
+     *                            on cancellation.
+     * @param onLastRead Runs when a {@code read()} call returns the last byte of the file, or on {@code close()} if the file was not fully
+     *                   read. Only runs once even if the last byte is read multiple times using {@code mark()} and {@code reset()}.
+     */
+    RandomBlobContent(String repositoryName, long seed, BooleanSupplier isCancelledSupplier, Runnable onLastRead) {
+        this.repositoryName = repositoryName;
+        this.isCancelledSupplier = isCancelledSupplier;
+        this.onLastRead = new AtomicReference<>(onLastRead);
+        new Random(seed).nextBytes(buffer);
+    }
+
+    long getChecksum(long checksumRangeStart, long checksumRangeEnd) {
+        assert 0 <= checksumRangeStart && checksumRangeStart <= checksumRangeEnd;
+        final CRC32 crc32 = new CRC32();
+
+        final long startBlock = checksumRangeStart / buffer.length;
+        final long endBlock = (checksumRangeEnd - 1) / buffer.length;
+
+        if (startBlock == endBlock) {
+            crc32.update(
+                buffer,
+                Math.toIntExact(checksumRangeStart % buffer.length),
+                Math.toIntExact(checksumRangeEnd - checksumRangeStart)
+            );
+        } else {
+            final int bufferStart = Math.toIntExact(checksumRangeStart % buffer.length);
+            crc32.update(buffer, bufferStart, buffer.length - bufferStart);
+            for (long block = startBlock + 1; block < endBlock; block++) {
+                crc32.update(buffer);
+            }
+            crc32.update(buffer, 0, Math.toIntExact((checksumRangeEnd - 1) % buffer.length) + 1);
+        }
+
+        return crc32.getValue();
+    }
+
+    void ensureNotCancelled(final String position) {
+        if (isCancelledSupplier.getAsBoolean()) {
+            throw new RepositoryVerificationException(repositoryName, "blob upload cancelled at position [" + position + "]");
+        }
+    }
+
+    void onLastRead() {
+        final Runnable runnable = onLastRead.getAndSet(null);
+        if (runnable != null) {
+            runnable.run();
+        }
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RandomBlobContentBytesReference.java

@@ -0,0 +1,90 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefIterator;
+import org.elasticsearch.common.bytes.AbstractBytesReference;
+import org.elasticsearch.common.bytes.BytesReference;
+
+/**
+ * A {@link BytesReference} that's the same random ~8kB of data repeatedly, for use as a dummy payload for testing a blob store repo.
+ */
+class RandomBlobContentBytesReference extends AbstractBytesReference {
+
+    private final int length;
+    private final RandomBlobContent randomBlobContent;
+
+    /**
+     * @param randomBlobContent The (simulated) content of the blob
+     * @param length            The length of this blob.
+     */
+    RandomBlobContentBytesReference(RandomBlobContent randomBlobContent, int length) {
+        assert 0 < length;
+        this.randomBlobContent = randomBlobContent;
+        this.length = length;
+    }
+
+    @Override
+    public byte get(int index) {
+        return randomBlobContent.buffer[index % randomBlobContent.buffer.length];
+    }
+
+    @Override
+    public int length() {
+        return length;
+    }
+
+    @Override
+    public BytesReference slice(int from, int length) {
+        assert false : "must not slice a RandomBlobContentBytesReference";
+        throw new UnsupportedOperationException("RandomBlobContentBytesReference#slice(int, int) is unsupported");
+    }
+
+    @Override
+    public long ramBytesUsed() {
+        // no need for accurate accounting of the overhead since we don't really account for these things anyway
+        return randomBlobContent.buffer.length;
+    }
+
+    @Override
+    public BytesRef toBytesRef() {
+        assert false : "must not materialize a RandomBlobContentBytesReference";
+        throw new UnsupportedOperationException("RandomBlobContentBytesReference#toBytesRef() is unsupported");
+    }
+
+    @Override
+    public BytesRefIterator iterator() {
+        final byte[] buffer = randomBlobContent.buffer;
+        final int lastBlock = (length - 1) / buffer.length;
+
+        return new BytesRefIterator() {
+
+            int nextBlock = 0;
+
+            @Override
+            public BytesRef next() {
+                final int block = nextBlock++;
+                if (block > lastBlock) {
+                    return null;
+                }
+
+                randomBlobContent.ensureNotCancelled(block * buffer.length + "/" + length);
+                final int end;
+                if (block == lastBlock) {
+                    randomBlobContent.onLastRead();
+                    end = (length - 1) % buffer.length + 1;
+                } else {
+                    end = buffer.length;
+                }
+
+                return new BytesRef(buffer, 0, end);
+            }
+        };
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RandomBlobContentStream.java

@@ -0,0 +1,115 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import java.io.InputStream;
+
+/**
+ * An {@link InputStream} that's the same random ~8kB of data repeatedly, for use as a dummy payload for testing a blob store repo.
+ */
+class RandomBlobContentStream extends InputStream {
+
+    private final long length;
+    private final RandomBlobContent randomBlobContent;
+    private long position;
+    private long markPosition;
+
+    /**
+     * @param randomBlobContent The (simulated) content of the blob.
+     * @param length The length of this stream.
+     */
+    RandomBlobContentStream(RandomBlobContent randomBlobContent, long length) {
+        assert 0 < length;
+        this.randomBlobContent = randomBlobContent;
+        this.length = length;
+    }
+
+    private int bufferPosition() {
+        return Math.toIntExact(position % randomBlobContent.buffer.length);
+    }
+
+    @Override
+    public int read() {
+        randomBlobContent.ensureNotCancelled(position + "/" + length);
+
+        if (length <= position) {
+            return -1;
+        }
+
+        final int b = Byte.toUnsignedInt(randomBlobContent.buffer[bufferPosition()]);
+        position += 1;
+
+        if (position == length) {
+            randomBlobContent.onLastRead();
+        }
+
+        return b;
+    }
+
+    @Override
+    public int read(byte[] b, int off, int len) {
+        randomBlobContent.ensureNotCancelled(position + "+" + len + "/" + length);
+
+        if (length <= position) {
+            return -1;
+        }
+
+        len = Math.toIntExact(Math.min(len, length - position));
+        int remaining = len;
+        while (0 < remaining) {
+            assert position < length : position + " vs " + length;
+
+            final int bufferPosition = bufferPosition();
+            final int copied = Math.min(randomBlobContent.buffer.length - bufferPosition, remaining);
+            System.arraycopy(randomBlobContent.buffer, bufferPosition, b, off, copied);
+            off += copied;
+            remaining -= copied;
+            position += copied;
+        }
+
+        assert position <= length : position + " vs " + length;
+
+        if (position == length) {
+            randomBlobContent.onLastRead();
+        }
+
+        return len;
+    }
+
+    @Override
+    public void close() {
+        randomBlobContent.onLastRead();
+    }
+
+    @Override
+    public boolean markSupported() {
+        return true;
+    }
+
+    @Override
+    public synchronized void mark(int readlimit) {
+        markPosition = position;
+    }
+
+    @Override
+    public synchronized void reset() {
+        position = markPosition;
+    }
+
+    @Override
+    public long skip(long n) {
+        final long oldPosition = position;
+        position = Math.min(length, position + n);
+        return position - oldPosition;
+    }
+
+    @Override
+    public int available() {
+        return Math.toIntExact(Math.min(length - position, Integer.MAX_VALUE));
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RepositoryAnalyzeAction.java

@@ -0,0 +1,1022 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import com.carrotsearch.hppc.ObjectContainer;
+import com.carrotsearch.hppc.cursors.ObjectCursor;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.message.ParameterizedMessage;
+import org.elasticsearch.ExceptionsHelper;
+import org.elasticsearch.action.ActionListener;
+import org.elasticsearch.action.ActionListenerResponseHandler;
+import org.elasticsearch.action.ActionRequest;
+import org.elasticsearch.action.ActionRequestValidationException;
+import org.elasticsearch.action.ActionResponse;
+import org.elasticsearch.action.ActionRunnable;
+import org.elasticsearch.action.ActionType;
+import org.elasticsearch.action.support.ActionFilters;
+import org.elasticsearch.action.support.HandledTransportAction;
+import org.elasticsearch.cluster.ClusterState;
+import org.elasticsearch.cluster.node.DiscoveryNode;
+import org.elasticsearch.cluster.node.DiscoveryNodes;
+import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.UUIDs;
+import org.elasticsearch.common.blobstore.BlobContainer;
+import org.elasticsearch.common.blobstore.BlobMetadata;
+import org.elasticsearch.common.blobstore.BlobPath;
+import org.elasticsearch.common.inject.Inject;
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.util.concurrent.ConcurrentCollections;
+import org.elasticsearch.common.util.concurrent.CountDown;
+import org.elasticsearch.common.xcontent.StatusToXContentObject;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.repositories.RepositoriesService;
+import org.elasticsearch.repositories.Repository;
+import org.elasticsearch.repositories.RepositoryVerificationException;
+import org.elasticsearch.repositories.blobstore.BlobStoreRepository;
+import org.elasticsearch.rest.RestStatus;
+import org.elasticsearch.tasks.CancellableTask;
+import org.elasticsearch.tasks.Task;
+import org.elasticsearch.tasks.TaskCancelledException;
+import org.elasticsearch.tasks.TaskId;
+import org.elasticsearch.threadpool.ThreadPool;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+import org.elasticsearch.transport.TransportException;
+import org.elasticsearch.transport.TransportRequestOptions;
+import org.elasticsearch.transport.TransportResponseHandler;
+import org.elasticsearch.transport.TransportService;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.LongSupplier;
+import java.util.stream.IntStream;
+
+import static org.elasticsearch.repositories.blobstore.testkit.SnapshotRepositoryTestKit.humanReadableNanos;
+
+/**
+ * Action which distributes a bunch of {@link BlobAnalyzeAction}s over the nodes in the cluster, with limited concurrency, and collects
+ * the results. Tries to fail fast by cancelling everything if any child task fails, or the timeout is reached, to avoid consuming
+ * unnecessary resources. On completion, does a best-effort wait until the blob list contains all the expected blobs, then deletes them all.
+ */
+public class RepositoryAnalyzeAction extends ActionType<RepositoryAnalyzeAction.Response> {
+
+    private static final Logger logger = LogManager.getLogger(RepositoryAnalyzeAction.class);
+
+    public static final RepositoryAnalyzeAction INSTANCE = new RepositoryAnalyzeAction();
+    public static final String NAME = "cluster:admin/repository/analyze";
+
+    private RepositoryAnalyzeAction() {
+        super(NAME, Response::new);
+    }
+
+    public static class TransportAction extends HandledTransportAction<Request, Response> {
+
+        private final TransportService transportService;
+        private final ClusterService clusterService;
+        private final RepositoriesService repositoriesService;
+
+        @Inject
+        public TransportAction(
+            TransportService transportService,
+            ActionFilters actionFilters,
+            ClusterService clusterService,
+            RepositoriesService repositoriesService
+        ) {
+            super(NAME, transportService, actionFilters, RepositoryAnalyzeAction.Request::new, ThreadPool.Names.SAME);
+            this.transportService = transportService;
+            this.clusterService = clusterService;
+            this.repositoriesService = repositoriesService;
+        }
+
+        @Override
+        protected void doExecute(Task task, Request request, ActionListener<Response> listener) {
+            final ClusterState state = clusterService.state();
+
+            final ThreadPool threadPool = transportService.getThreadPool();
+            request.reseed(threadPool.relativeTimeInMillis());
+
+            final DiscoveryNode localNode = transportService.getLocalNode();
+            if (isSnapshotNode(localNode)) {
+                final Repository repository = repositoriesService.repository(request.getRepositoryName());
+                if (repository instanceof BlobStoreRepository == false) {
+                    throw new IllegalArgumentException("repository [" + request.getRepositoryName() + "] is not a blob-store repository");
+                }
+                if (repository.isReadOnly()) {
+                    throw new IllegalArgumentException("repository [" + request.getRepositoryName() + "] is read-only");
+                }
+
+                assert task instanceof CancellableTask;
+                new AsyncAction(
+                    transportService,
+                    (BlobStoreRepository) repository,
+                    (CancellableTask) task,
+                    request,
+                    state.nodes(),
+                    threadPool::relativeTimeInMillis,
+                    listener
+                ).run();
+                return;
+            }
+
+            if (request.getReroutedFrom() != null) {
+                assert false : request.getReroutedFrom();
+                throw new IllegalArgumentException(
+                    "analysis of repository ["
+                        + request.getRepositoryName()
+                        + "] rerouted from ["
+                        + request.getReroutedFrom()
+                        + "] to non-snapshot node"
+                );
+            }
+
+            request.reroutedFrom(localNode);
+            final List<DiscoveryNode> snapshotNodes = getSnapshotNodes(state.nodes());
+            if (snapshotNodes.isEmpty()) {
+                listener.onFailure(
+                    new IllegalArgumentException("no snapshot nodes found for analysis of repository [" + request.getRepositoryName() + "]")
+                );
+            } else {
+                if (snapshotNodes.size() > 1) {
+                    snapshotNodes.remove(state.nodes().getMasterNode());
+                }
+                final DiscoveryNode targetNode = snapshotNodes.get(new Random(request.getSeed()).nextInt(snapshotNodes.size()));
+                RepositoryAnalyzeAction.logger.trace("rerouting analysis [{}] to [{}]", request.getDescription(), targetNode);
+                transportService.sendChildRequest(
+                    targetNode,
+                    NAME,
+                    request,
+                    task,
+                    TransportRequestOptions.EMPTY,
+                    new ActionListenerResponseHandler<>(listener, Response::new)
+                );
+            }
+        }
+    }
+
+    private static boolean isSnapshotNode(DiscoveryNode discoveryNode) {
+        return (discoveryNode.isDataNode() || discoveryNode.isMasterNode())
+            && RepositoriesService.isDedicatedVotingOnlyNode(discoveryNode.getRoles()) == false;
+    }
+
+    private static List<DiscoveryNode> getSnapshotNodes(DiscoveryNodes discoveryNodes) {
+        final ObjectContainer<DiscoveryNode> nodesContainer = discoveryNodes.getMasterAndDataNodes().values();
+        final List<DiscoveryNode> nodes = new ArrayList<>(nodesContainer.size());
+        for (ObjectCursor<DiscoveryNode> cursor : nodesContainer) {
+            if (isSnapshotNode(cursor.value)) {
+                nodes.add(cursor.value);
+            }
+        }
+        return nodes;
+    }
+
+    /**
+     * Compute a collection of blob sizes which respects the blob count and the limits on the size of each blob and the total size to write.
+     * Tries its best to achieve an even spread of different-sized blobs to try and exercise the various size-based code paths well. See
+     * the corresponding unit tests for examples of its output.
+     */
+    // Exposed for tests
+    static List<Long> getBlobSizes(Request request) {
+
+        int blobCount = request.getBlobCount();
+        long maxTotalBytes = request.getMaxTotalDataSize().getBytes();
+        long maxBlobSize = request.getMaxBlobSize().getBytes();
+
+        if (maxTotalBytes - maxBlobSize < blobCount - 1) {
+            throw new IllegalArgumentException(
+                "cannot satisfy max total bytes ["
+                    + maxTotalBytes
+                    + "B/"
+                    + request.getMaxTotalDataSize()
+                    + "]: must write at least one byte per blob and at least one max-sized blob which is ["
+                    + (blobCount + maxBlobSize - 1)
+                    + "B] in total"
+            );
+        }
+
+        final List<Long> blobSizes = new ArrayList<>();
+        for (long s = 1; 0 < s && s < maxBlobSize; s <<= 1) {
+            blobSizes.add(s);
+        }
+        blobSizes.add(maxBlobSize);
+        // TODO also use sizes that aren't a power of two
+
+        // Try and form an even spread of blob sizes by accounting for as many evenly spreads repeats as possible up-front.
+        final long evenSpreadSize = blobSizes.stream().mapToLong(l -> l).sum();
+        int evenSpreadCount = 0;
+        while (blobSizes.size() <= blobCount && blobCount - blobSizes.size() <= maxTotalBytes - evenSpreadSize) {
+            evenSpreadCount += 1;
+            maxTotalBytes -= evenSpreadSize;
+            blobCount -= blobSizes.size();
+        }
+
+        if (evenSpreadCount == 0) {
+            // Not enough bytes for even a single even spread, account for the one max-sized blob anyway.
+            blobCount -= 1;
+            maxTotalBytes -= maxBlobSize;
+        }
+
+        final List<Long> perBlobSizes = new BlobCountCalculator(blobCount, maxTotalBytes, blobSizes).calculate();
+
+        if (evenSpreadCount == 0) {
+            perBlobSizes.add(maxBlobSize);
+        } else {
+            for (final long blobSize : blobSizes) {
+                for (int i = 0; i < evenSpreadCount; i++) {
+                    perBlobSizes.add(blobSize);
+                }
+            }
+        }
+
+        assert perBlobSizes.size() == request.getBlobCount();
+        assert perBlobSizes.stream().mapToLong(l -> l).sum() <= request.getMaxTotalDataSize().getBytes();
+        assert perBlobSizes.stream().allMatch(l -> 1 <= l && l <= request.getMaxBlobSize().getBytes());
+        assert perBlobSizes.stream().anyMatch(l -> l == request.getMaxBlobSize().getBytes());
+        return perBlobSizes;
+    }
+
+    /**
+     * Calculates a reasonable set of blob sizes, with the correct number of blobs and a total size that respects the max in the request.
+     */
+    private static class BlobCountCalculator {
+
+        private final int blobCount;
+        private final long maxTotalBytes;
+        private final List<Long> blobSizes;
+        private final int sizeCount;
+
+        private final int[] blobsBySize;
+        private long totalBytes;
+        private int totalBlobs;
+
+        BlobCountCalculator(int blobCount, long maxTotalBytes, List<Long> blobSizes) {
+            this.blobCount = blobCount;
+            this.maxTotalBytes = maxTotalBytes;
+            assert blobCount <= maxTotalBytes;
+
+            this.blobSizes = blobSizes;
+            sizeCount = blobSizes.size();
+            assert sizeCount > 0;
+
+            blobsBySize = new int[sizeCount];
+            assert invariant();
+        }
+
+        List<Long> calculate() {
+            // add blobs roughly evenly while keeping the total size under maxTotalBytes
+            addBlobsRoughlyEvenly(sizeCount - 1);
+
+            // repeatedly remove a blob and replace it with some number of smaller blobs, until there are enough blobs
+            while (totalBlobs < blobCount) {
+                assert totalBytes <= maxTotalBytes;
+
+                final int minSplitCount = Arrays.stream(blobsBySize).skip(1).allMatch(i -> i <= 1) ? 1 : 2;
+                final int splitIndex = IntStream.range(1, sizeCount).filter(i -> blobsBySize[i] >= minSplitCount).reduce(-1, (i, j) -> j);
+                assert splitIndex > 0 : "split at " + splitIndex;
+                assert blobsBySize[splitIndex] >= minSplitCount;
+
+                final long splitSize = blobSizes.get(splitIndex);
+                blobsBySize[splitIndex] -= 1;
+                totalBytes -= splitSize;
+                totalBlobs -= 1;
+
+                addBlobsRoughlyEvenly(splitIndex - 1);
+                assert invariant();
+            }
+
+            return getPerBlobSizes();
+        }
+
+        private List<Long> getPerBlobSizes() {
+            assert invariant();
+
+            final List<Long> perBlobSizes = new ArrayList<>(blobCount);
+            for (int sizeIndex = 0; sizeIndex < sizeCount; sizeIndex++) {
+                final long size = blobSizes.get(sizeIndex);
+                for (int i = 0; i < blobsBySize[sizeIndex]; i++) {
+                    perBlobSizes.add(size);
+                }
+            }
+
+            return perBlobSizes;
+        }
+
+        private void addBlobsRoughlyEvenly(int startingIndex) {
+            while (totalBlobs < blobCount && totalBytes < maxTotalBytes) {
+                boolean progress = false;
+                for (int sizeIndex = startingIndex; 0 <= sizeIndex && totalBlobs < blobCount && totalBytes < maxTotalBytes; sizeIndex--) {
+                    final long size = blobSizes.get(sizeIndex);
+                    if (totalBytes + size <= maxTotalBytes) {
+                        progress = true;
+                        blobsBySize[sizeIndex] += 1;
+                        totalBlobs += 1;
+                        totalBytes += size;
+                    }
+                }
+                assert progress;
+                assert invariant();
+            }
+        }
+
+        private boolean invariant() {
+            assert IntStream.of(blobsBySize).sum() == totalBlobs : this;
+            assert IntStream.range(0, sizeCount).mapToLong(i -> blobSizes.get(i) * blobsBySize[i]).sum() == totalBytes : this;
+            assert totalBlobs <= blobCount : this;
+            assert totalBytes <= maxTotalBytes : this;
+            return true;
+        }
+    }
+
+    public static class AsyncAction {
+
+        private final TransportService transportService;
+        private final BlobStoreRepository repository;
+        private final CancellableTask task;
+        private final Request request;
+        private final DiscoveryNodes discoveryNodes;
+        private final LongSupplier currentTimeMillisSupplier;
+        private final ActionListener<Response> listener;
+        private final long timeoutTimeMillis;
+
+        // choose the blob path nondeterministically to avoid clashes, assuming that the actual path doesn't matter for reproduction
+        private final String blobPath = "temp-analysis-" + UUIDs.randomBase64UUID();
+
+        private final Queue<VerifyBlobTask> queue = ConcurrentCollections.newQueue();
+        private final AtomicReference<Exception> failure = new AtomicReference<>();
+        private final Semaphore innerFailures = new Semaphore(5); // limit the number of suppressed failures
+        private final int workerCount;
+        private final CountDown workerCountdown;
+        private final Set<String> expectedBlobs = ConcurrentCollections.newConcurrentSet();
+        private final List<BlobAnalyzeAction.Response> responses;
+        private final RepositoryPerformanceSummary.Builder summary = new RepositoryPerformanceSummary.Builder();
+
+        public AsyncAction(
+            TransportService transportService,
+            BlobStoreRepository repository,
+            CancellableTask task,
+            Request request,
+            DiscoveryNodes discoveryNodes,
+            LongSupplier currentTimeMillisSupplier,
+            ActionListener<Response> listener
+        ) {
+            this.transportService = transportService;
+            this.repository = repository;
+            this.task = task;
+            this.request = request;
+            this.discoveryNodes = discoveryNodes;
+            this.currentTimeMillisSupplier = currentTimeMillisSupplier;
+            this.timeoutTimeMillis = currentTimeMillisSupplier.getAsLong() + request.getTimeout().millis();
+            this.listener = listener;
+
+            this.workerCount = request.getConcurrency();
+            this.workerCountdown = new CountDown(workerCount);
+
+            responses = new ArrayList<>(request.blobCount);
+        }
+
+        private void fail(Exception e) {
+            if (failure.compareAndSet(null, e)) {
+                transportService.getTaskManager().cancelTaskAndDescendants(task, "task failed", false, ActionListener.wrap(() -> {}));
+            } else {
+                if (innerFailures.tryAcquire()) {
+                    final Throwable cause = ExceptionsHelper.unwrapCause(e);
+                    if (cause instanceof TaskCancelledException || cause instanceof ReceiveTimeoutTransportException) {
+                        innerFailures.release();
+                    } else {
+                        failure.get().addSuppressed(e);
+                    }
+                }
+            }
+        }
+
+        /**
+         * Check that we haven't already failed or been cancelled or timed out; if newly cancelled or timed out then record this as the root
+         * cause of failure.
+         */
+        private boolean isRunning() {
+            if (failure.get() != null) {
+                return false;
+            }
+
+            if (task.isCancelled()) {
+                failure.compareAndSet(null, new RepositoryVerificationException(request.repositoryName, "verification cancelled"));
+                // if this CAS failed then we're failing for some other reason, nbd; also if the task is cancelled then its descendants are
+                // also cancelled, so no further action is needed either way.
+                return false;
+            }
+
+            if (timeoutTimeMillis < currentTimeMillisSupplier.getAsLong()) {
+                if (failure.compareAndSet(
+                    null,
+                    new RepositoryVerificationException(request.repositoryName, "analysis timed out after [" + request.getTimeout() + "]")
+                )) {
+                    transportService.getTaskManager().cancelTaskAndDescendants(task, "timed out", false, ActionListener.wrap(() -> {}));
+                }
+                // if this CAS failed then we're already failing for some other reason, nbd
+                return false;
+            }
+
+            return true;
+        }
+
+        public void run() {
+            assert queue.isEmpty() : "must only run action once";
+            assert failure.get() == null : "must only run action once";
+            assert workerCountdown.isCountedDown() == false : "must only run action once";
+
+            logger.info("running analysis of repository [{}] using path [{}]", request.getRepositoryName(), blobPath);
+
+            final Random random = new Random(request.getSeed());
+
+            final List<DiscoveryNode> nodes = getSnapshotNodes(discoveryNodes);
+            final List<Long> blobSizes = getBlobSizes(request);
+            Collections.shuffle(blobSizes, random);
+
+            for (int i = 0; i < request.getBlobCount(); i++) {
+                final long targetLength = blobSizes.get(i);
+                final boolean smallBlob = targetLength <= Integer.MAX_VALUE; // avoid the atomic API for larger blobs
+                final VerifyBlobTask verifyBlobTask = new VerifyBlobTask(
+                    nodes.get(random.nextInt(nodes.size())),
+                    new BlobAnalyzeAction.Request(
+                        request.getRepositoryName(),
+                        blobPath,
+                        "test-blob-" + i + "-" + UUIDs.randomBase64UUID(random),
+                        targetLength,
+                        random.nextLong(),
+                        nodes,
+                        request.getReadNodeCount(),
+                        request.getEarlyReadNodeCount(),
+                        smallBlob && random.nextDouble() < request.getRareActionProbability(),
+                        repository.supportURLRepo() && smallBlob && random.nextDouble() < request.getRareActionProbability()
+                    )
+                );
+                queue.add(verifyBlobTask);
+            }
+
+            for (int i = 0; i < workerCount; i++) {
+                processNextTask();
+            }
+        }
+
+        private void processNextTask() {
+            final VerifyBlobTask thisTask = queue.poll();
+            if (isRunning() == false || thisTask == null) {
+                onWorkerCompletion();
+            } else {
+                logger.trace("processing [{}]", thisTask);
+                // NB although all this is on the SAME thread, the per-blob verification runs on a SNAPSHOT thread so we don't have to worry
+                // about local requests resulting in a stack overflow here
+                final TransportRequestOptions transportRequestOptions = TransportRequestOptions.timeout(
+                    TimeValue.timeValueMillis(timeoutTimeMillis - currentTimeMillisSupplier.getAsLong())
+                );
+                transportService.sendChildRequest(
+                    thisTask.node,
+                    BlobAnalyzeAction.NAME,
+                    thisTask.request,
+                    task,
+                    transportRequestOptions,
+                    new TransportResponseHandler<BlobAnalyzeAction.Response>() {
+                        @Override
+                        public void handleResponse(BlobAnalyzeAction.Response response) {
+                            logger.trace("finished [{}]", thisTask);
+                            expectedBlobs.add(thisTask.request.getBlobName()); // each task cleans up its own mess on failure
+                            if (request.detailed) {
+                                synchronized (responses) {
+                                    responses.add(response);
+                                }
+                            }
+                            summary.add(response);
+                            processNextTask();
+                        }
+
+                        @Override
+                        public void handleException(TransportException exp) {
+                            logger.debug(new ParameterizedMessage("failed [{}]", thisTask), exp);
+                            fail(exp);
+                            onWorkerCompletion();
+                        }
+
+                        @Override
+                        public BlobAnalyzeAction.Response read(StreamInput in) throws IOException {
+                            return new BlobAnalyzeAction.Response(in);
+                        }
+                    }
+                );
+            }
+
+        }
+
+        private BlobContainer getBlobContainer() {
+            return repository.blobStore().blobContainer(new BlobPath().add(blobPath));
+        }
+
+        private void onWorkerCompletion() {
+            if (workerCountdown.countDown()) {
+                transportService.getThreadPool().executor(ThreadPool.Names.SNAPSHOT).execute(ActionRunnable.run(listener, () -> {
+                    final long listingStartTimeNanos = System.nanoTime();
+                    ensureConsistentListing();
+                    final long deleteStartTimeNanos = System.nanoTime();
+                    deleteContainer();
+                    sendResponse(listingStartTimeNanos, deleteStartTimeNanos);
+                }));
+            }
+        }
+
+        private void ensureConsistentListing() {
+            if (timeoutTimeMillis < currentTimeMillisSupplier.getAsLong() || task.isCancelled()) {
+                logger.warn(
+                    "analysis of repository [{}] failed before cleanup phase, attempting best-effort cleanup "
+                        + "but you may need to manually remove [{}]",
+                    request.getRepositoryName(),
+                    blobPath
+                );
+                isRunning(); // set failure if not already set
+            } else {
+                logger.trace(
+                    "all tasks completed, checking expected blobs exist in [{}:{}] before cleanup",
+                    request.repositoryName,
+                    blobPath
+                );
+                try {
+                    final BlobContainer blobContainer = getBlobContainer();
+                    final Set<String> missingBlobs = new HashSet<>(expectedBlobs);
+                    final Map<String, BlobMetadata> blobsMap = blobContainer.listBlobs();
+                    missingBlobs.removeAll(blobsMap.keySet());
+
+                    if (missingBlobs.isEmpty()) {
+                        logger.trace("all expected blobs found, cleaning up [{}:{}]", request.getRepositoryName(), blobPath);
+                    } else {
+                        final RepositoryVerificationException repositoryVerificationException = new RepositoryVerificationException(
+                            request.repositoryName,
+                            "expected blobs " + missingBlobs + " missing in [" + request.repositoryName + ":" + blobPath + "]"
+                        );
+                        logger.debug("failing due to missing blobs", repositoryVerificationException);
+                        fail(repositoryVerificationException);
+                    }
+                } catch (Exception e) {
+                    logger.debug(new ParameterizedMessage("failure during cleanup of [{}:{}]", request.getRepositoryName(), blobPath), e);
+                    fail(e);
+                }
+            }
+        }
+
+        private void deleteContainer() {
+            try {
+                final BlobContainer blobContainer = getBlobContainer();
+                blobContainer.delete();
+                if (failure.get() != null) {
+                    return;
+                }
+                final Map<String, BlobMetadata> blobsMap = blobContainer.listBlobs();
+                if (blobsMap.isEmpty() == false) {
+                    final RepositoryVerificationException repositoryVerificationException = new RepositoryVerificationException(
+                        request.repositoryName,
+                        "failed to clean up blobs " + blobsMap.keySet()
+                    );
+                    logger.debug("failing due to leftover blobs", repositoryVerificationException);
+                    fail(repositoryVerificationException);
+                }
+            } catch (Exception e) {
+                fail(e);
+            }
+        }
+
+        private void sendResponse(final long listingStartTimeNanos, final long deleteStartTimeNanos) {
+            final Exception exception = failure.get();
+            if (exception == null) {
+                final long completionTimeNanos = System.nanoTime();
+
+                logger.trace("[{}] completed successfully", request.getDescription());
+
+                listener.onResponse(
+                    new Response(
+                        transportService.getLocalNode().getId(),
+                        transportService.getLocalNode().getName(),
+                        request.getRepositoryName(),
+                        request.blobCount,
+                        request.concurrency,
+                        request.readNodeCount,
+                        request.earlyReadNodeCount,
+                        request.maxBlobSize,
+                        request.maxTotalDataSize,
+                        request.seed,
+                        request.rareActionProbability,
+                        blobPath,
+                        summary.build(),
+                        responses,
+                        deleteStartTimeNanos - listingStartTimeNanos,
+                        completionTimeNanos - deleteStartTimeNanos
+                    )
+                );
+            } else {
+                logger.debug(new ParameterizedMessage("analysis of repository [{}] failed", request.repositoryName), exception);
+                listener.onFailure(
+                    new RepositoryVerificationException(
+                        request.getRepositoryName(),
+                        "analysis failed, you may need to manually remove [" + blobPath + "]",
+                        exception
+                    )
+                );
+            }
+        }
+
+        private static class VerifyBlobTask {
+            final DiscoveryNode node;
+            final BlobAnalyzeAction.Request request;
+
+            VerifyBlobTask(DiscoveryNode node, BlobAnalyzeAction.Request request) {
+                this.node = node;
+                this.request = request;
+            }
+
+            @Override
+            public String toString() {
+                return "VerifyBlobTask{" + "node=" + node + ", request=" + request + '}';
+            }
+        }
+    }
+
+    public static class Request extends ActionRequest {
+
+        private final String repositoryName;
+
+        private int blobCount = 100;
+        private int concurrency = 10;
+        private int readNodeCount = 10;
+        private int earlyReadNodeCount = 2;
+        private long seed = 0L;
+        private double rareActionProbability = 0.02;
+        private TimeValue timeout = TimeValue.timeValueSeconds(30);
+        private ByteSizeValue maxBlobSize = ByteSizeValue.ofMb(10);
+        private ByteSizeValue maxTotalDataSize = ByteSizeValue.ofGb(1);
+        private boolean detailed = false;
+        private DiscoveryNode reroutedFrom = null;
+
+        public Request(String repositoryName) {
+            this.repositoryName = repositoryName;
+        }
+
+        public Request(StreamInput in) throws IOException {
+            super(in);
+            repositoryName = in.readString();
+            seed = in.readLong();
+            rareActionProbability = in.readDouble();
+            blobCount = in.readVInt();
+            concurrency = in.readVInt();
+            readNodeCount = in.readVInt();
+            earlyReadNodeCount = in.readVInt();
+            timeout = in.readTimeValue();
+            maxBlobSize = new ByteSizeValue(in);
+            maxTotalDataSize = new ByteSizeValue(in);
+            detailed = in.readBoolean();
+            reroutedFrom = in.readOptionalWriteable(DiscoveryNode::new);
+        }
+
+        @Override
+        public ActionRequestValidationException validate() {
+            return null;
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            super.writeTo(out);
+            out.writeString(repositoryName);
+            out.writeLong(seed);
+            out.writeDouble(rareActionProbability);
+            out.writeVInt(blobCount);
+            out.writeVInt(concurrency);
+            out.writeVInt(readNodeCount);
+            out.writeVInt(earlyReadNodeCount);
+            out.writeTimeValue(timeout);
+            maxBlobSize.writeTo(out);
+            maxTotalDataSize.writeTo(out);
+            out.writeBoolean(detailed);
+            out.writeOptionalWriteable(reroutedFrom);
+        }
+
+        @Override
+        public Task createTask(long id, String type, String action, TaskId parentTaskId, Map<String, String> headers) {
+            return new CancellableTask(id, type, action, getDescription(), parentTaskId, headers);
+        }
+
+        public void blobCount(int blobCount) {
+            if (blobCount <= 0) {
+                throw new IllegalArgumentException("blobCount must be >0, but was [" + blobCount + "]");
+            }
+            if (blobCount > 100000) {
+                // Coordination work is O(blobCount) but is supposed to be lightweight, so limit the blob count.
+                throw new IllegalArgumentException("blobCount must be <= 100000, but was [" + blobCount + "]");
+            }
+            this.blobCount = blobCount;
+        }
+
+        public void concurrency(int concurrency) {
+            if (concurrency <= 0) {
+                throw new IllegalArgumentException("concurrency must be >0, but was [" + concurrency + "]");
+            }
+            this.concurrency = concurrency;
+        }
+
+        public void seed(long seed) {
+            this.seed = seed;
+        }
+
+        public void timeout(TimeValue timeout) {
+            this.timeout = timeout;
+        }
+
+        public void maxBlobSize(ByteSizeValue maxBlobSize) {
+            if (maxBlobSize.getBytes() <= 0) {
+                throw new IllegalArgumentException("maxBlobSize must be >0, but was [" + maxBlobSize + "]");
+            }
+            this.maxBlobSize = maxBlobSize;
+        }
+
+        public void maxTotalDataSize(ByteSizeValue maxTotalDataSize) {
+            if (maxTotalDataSize.getBytes() <= 0) {
+                throw new IllegalArgumentException("maxTotalDataSize must be >0, but was [" + maxTotalDataSize + "]");
+            }
+            this.maxTotalDataSize = maxTotalDataSize;
+        }
+
+        public void detailed(boolean detailed) {
+            this.detailed = detailed;
+        }
+
+        public int getBlobCount() {
+            return blobCount;
+        }
+
+        public int getConcurrency() {
+            return concurrency;
+        }
+
+        public String getRepositoryName() {
+            return repositoryName;
+        }
+
+        public TimeValue getTimeout() {
+            return timeout;
+        }
+
+        public long getSeed() {
+            return seed;
+        }
+
+        public ByteSizeValue getMaxBlobSize() {
+            return maxBlobSize;
+        }
+
+        public ByteSizeValue getMaxTotalDataSize() {
+            return maxTotalDataSize;
+        }
+
+        public boolean getDetailed() {
+            return detailed;
+        }
+
+        public DiscoveryNode getReroutedFrom() {
+            return reroutedFrom;
+        }
+
+        public void reroutedFrom(DiscoveryNode discoveryNode) {
+            reroutedFrom = discoveryNode;
+        }
+
+        public void readNodeCount(int readNodeCount) {
+            if (readNodeCount <= 0) {
+                throw new IllegalArgumentException("readNodeCount must be >0, but was [" + readNodeCount + "]");
+            }
+            this.readNodeCount = readNodeCount;
+        }
+
+        public int getReadNodeCount() {
+            return readNodeCount;
+        }
+
+        public void earlyReadNodeCount(int earlyReadNodeCount) {
+            if (earlyReadNodeCount < 0) {
+                throw new IllegalArgumentException("earlyReadNodeCount must be >=0, but was [" + earlyReadNodeCount + "]");
+            }
+            this.earlyReadNodeCount = earlyReadNodeCount;
+        }
+
+        public int getEarlyReadNodeCount() {
+            return earlyReadNodeCount;
+        }
+
+        public void rareActionProbability(double rareActionProbability) {
+            if (rareActionProbability < 0. || rareActionProbability > 1.) {
+                throw new IllegalArgumentException(
+                    "rareActionProbability must be between 0 and 1, but was [" + rareActionProbability + "]"
+                );
+            }
+            this.rareActionProbability = rareActionProbability;
+        }
+
+        public double getRareActionProbability() {
+            return rareActionProbability;
+        }
+
+        @Override
+        public String toString() {
+            return "Request{" + getDescription() + '}';
+        }
+
+        @Override
+        public String getDescription() {
+            return "analysis [repository="
+                + repositoryName
+                + ", blobCount="
+                + blobCount
+                + ", concurrency="
+                + concurrency
+                + ", readNodeCount="
+                + readNodeCount
+                + ", earlyReadNodeCount="
+                + earlyReadNodeCount
+                + ", seed="
+                + seed
+                + ", rareActionProbability="
+                + rareActionProbability
+                + ", timeout="
+                + timeout
+                + ", maxBlobSize="
+                + maxBlobSize
+                + ", maxTotalDataSize="
+                + maxTotalDataSize
+                + ", detailed="
+                + detailed
+                + "]";
+        }
+
+        public void reseed(long newSeed) {
+            if (seed == 0L) {
+                seed = newSeed;
+            }
+        }
+
+    }
+
+    public static class Response extends ActionResponse implements StatusToXContentObject {
+
+        private final String coordinatingNodeId;
+        private final String coordinatingNodeName;
+        private final String repositoryName;
+        private final int blobCount;
+        private final int concurrency;
+        private final int readNodeCount;
+        private final int earlyReadNodeCount;
+        private final ByteSizeValue maxBlobSize;
+        private final ByteSizeValue maxTotalDataSize;
+        private final long seed;
+        private final double rareActionProbability;
+        private final String blobPath;
+        private final RepositoryPerformanceSummary summary;
+        private final List<BlobAnalyzeAction.Response> blobResponses;
+        private final long listingTimeNanos;
+        private final long deleteTimeNanos;
+
+        public Response(
+            String coordinatingNodeId,
+            String coordinatingNodeName,
+            String repositoryName,
+            int blobCount,
+            int concurrency,
+            int readNodeCount,
+            int earlyReadNodeCount,
+            ByteSizeValue maxBlobSize,
+            ByteSizeValue maxTotalDataSize,
+            long seed,
+            double rareActionProbability,
+            String blobPath,
+            RepositoryPerformanceSummary summary,
+            List<BlobAnalyzeAction.Response> blobResponses,
+            long listingTimeNanos,
+            long deleteTimeNanos
+        ) {
+            this.coordinatingNodeId = coordinatingNodeId;
+            this.coordinatingNodeName = coordinatingNodeName;
+            this.repositoryName = repositoryName;
+            this.blobCount = blobCount;
+            this.concurrency = concurrency;
+            this.readNodeCount = readNodeCount;
+            this.earlyReadNodeCount = earlyReadNodeCount;
+            this.maxBlobSize = maxBlobSize;
+            this.maxTotalDataSize = maxTotalDataSize;
+            this.seed = seed;
+            this.rareActionProbability = rareActionProbability;
+            this.blobPath = blobPath;
+            this.summary = summary;
+            this.blobResponses = blobResponses;
+            this.listingTimeNanos = listingTimeNanos;
+            this.deleteTimeNanos = deleteTimeNanos;
+        }
+
+        public Response(StreamInput in) throws IOException {
+            super(in);
+            coordinatingNodeId = in.readString();
+            coordinatingNodeName = in.readString();
+            repositoryName = in.readString();
+            blobCount = in.readVInt();
+            concurrency = in.readVInt();
+            readNodeCount = in.readVInt();
+            earlyReadNodeCount = in.readVInt();
+            maxBlobSize = new ByteSizeValue(in);
+            maxTotalDataSize = new ByteSizeValue(in);
+            seed = in.readLong();
+            rareActionProbability = in.readDouble();
+            blobPath = in.readString();
+            summary = new RepositoryPerformanceSummary(in);
+            blobResponses = in.readList(BlobAnalyzeAction.Response::new);
+            listingTimeNanos = in.readVLong();
+            deleteTimeNanos = in.readVLong();
+        }
+
+        @Override
+        public void writeTo(StreamOutput out) throws IOException {
+            out.writeString(coordinatingNodeId);
+            out.writeString(coordinatingNodeName);
+            out.writeString(repositoryName);
+            out.writeVInt(blobCount);
+            out.writeVInt(concurrency);
+            out.writeVInt(readNodeCount);
+            out.writeVInt(earlyReadNodeCount);
+            maxBlobSize.writeTo(out);
+            maxTotalDataSize.writeTo(out);
+            out.writeLong(seed);
+            out.writeDouble(rareActionProbability);
+            out.writeString(blobPath);
+            summary.writeTo(out);
+            out.writeList(blobResponses);
+            out.writeVLong(listingTimeNanos);
+            out.writeVLong(deleteTimeNanos);
+        }
+
+        @Override
+        public RestStatus status() {
+            return RestStatus.OK;
+        }
+
+        @Override
+        public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+            builder.startObject();
+
+            builder.startObject("coordinating_node");
+            builder.field("id", coordinatingNodeId);
+            builder.field("name", coordinatingNodeName);
+            builder.endObject();
+
+            builder.field("repository", repositoryName);
+            builder.field("blob_count", blobCount);
+            builder.field("concurrency", concurrency);
+            builder.field("read_node_count", readNodeCount);
+            builder.field("early_read_node_count", earlyReadNodeCount);
+            builder.humanReadableField("max_blob_size_bytes", "max_blob_size", maxBlobSize);
+            builder.humanReadableField("max_total_data_size_bytes", "max_total_data_size", maxTotalDataSize);
+            builder.field("seed", seed);
+            builder.field("rare_action_probability", rareActionProbability);
+            builder.field("blob_path", blobPath);
+
+            builder.startArray("issues_detected");
+            // nothing to report here, if we detected an issue then we would have thrown an exception, but we include this to emphasise
+            // that we are only detecting issues, not guaranteeing their absence
+            builder.endArray();
+
+            builder.field("summary", summary);
+
+            if (blobResponses.size() > 0) {
+                builder.startArray("details");
+                for (BlobAnalyzeAction.Response blobResponse : blobResponses) {
+                    blobResponse.toXContent(builder, params);
+                }
+                builder.endArray();
+            }
+
+            humanReadableNanos(builder, "listing_elapsed_nanos", "listing_elapsed", listingTimeNanos);
+            humanReadableNanos(builder, "delete_elapsed_nanos", "delete_elapsed", deleteTimeNanos);
+
+            builder.endObject();
+            return builder;
+        }
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RepositoryPerformanceSummary.java

@@ -0,0 +1,158 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.common.io.stream.StreamOutput;
+import org.elasticsearch.common.io.stream.Writeable;
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.common.xcontent.ToXContentFragment;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.LongAccumulator;
+import java.util.concurrent.atomic.LongAdder;
+
+import static org.elasticsearch.repositories.blobstore.testkit.SnapshotRepositoryTestKit.humanReadableNanos;
+
+public class RepositoryPerformanceSummary implements Writeable, ToXContentFragment {
+
+    private final long writeCount;
+    private final long writeBytes;
+    private final long writeThrottledNanos;
+    private final long writeElapsedNanos;
+    private final long readCount;
+    private final long readBytes;
+    private final long readWaitNanos;
+    private final long maxReadWaitNanos;
+    private final long readThrottledNanos;
+    private final long readElapsedNanos;
+
+    public RepositoryPerformanceSummary(
+        long writeCount,
+        long writeBytes,
+        long writeThrottledNanos,
+        long writeElapsedNanos,
+        long readCount,
+        long readBytes,
+        long readWaitNanos,
+        long maxReadWaitNanos,
+        long readThrottledNanos,
+        long readElapsedNanos
+    ) {
+        this.writeCount = writeCount;
+        this.writeBytes = writeBytes;
+        this.writeThrottledNanos = writeThrottledNanos;
+        this.writeElapsedNanos = writeElapsedNanos;
+        this.readCount = readCount;
+        this.readBytes = readBytes;
+        this.readWaitNanos = readWaitNanos;
+        this.maxReadWaitNanos = maxReadWaitNanos;
+        this.readThrottledNanos = readThrottledNanos;
+        this.readElapsedNanos = readElapsedNanos;
+    }
+
+    public RepositoryPerformanceSummary(StreamInput in) throws IOException {
+        writeCount = in.readVLong();
+        writeBytes = in.readVLong();
+        writeThrottledNanos = in.readVLong();
+        writeElapsedNanos = in.readVLong();
+        readCount = in.readVLong();
+        readBytes = in.readVLong();
+        readWaitNanos = in.readVLong();
+        maxReadWaitNanos = in.readVLong();
+        readThrottledNanos = in.readVLong();
+        readElapsedNanos = in.readVLong();
+    }
+
+    @Override
+    public void writeTo(StreamOutput out) throws IOException {
+        out.writeVLong(writeCount);
+        out.writeVLong(writeBytes);
+        out.writeVLong(writeThrottledNanos);
+        out.writeVLong(writeElapsedNanos);
+        out.writeVLong(readCount);
+        out.writeVLong(readBytes);
+        out.writeVLong(readWaitNanos);
+        out.writeVLong(maxReadWaitNanos);
+        out.writeVLong(readThrottledNanos);
+        out.writeVLong(readElapsedNanos);
+    }
+
+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        builder.startObject();
+
+        builder.startObject("write");
+        builder.field("count", writeCount);
+        builder.humanReadableField("total_size_bytes", "total_size", new ByteSizeValue(writeBytes));
+        humanReadableNanos(builder, "total_throttled_nanos", "total_throttled", writeThrottledNanos);
+        humanReadableNanos(builder, "total_elapsed_nanos", "total_elapsed", writeElapsedNanos);
+        builder.endObject();
+
+        builder.startObject("read");
+        builder.field("count", readCount);
+        builder.humanReadableField("total_size_bytes", "total_size", new ByteSizeValue(readBytes));
+        humanReadableNanos(builder, "total_wait_nanos", "total_wait", readWaitNanos);
+        humanReadableNanos(builder, "max_wait_nanos", "max_wait", maxReadWaitNanos);
+        humanReadableNanos(builder, "total_throttled_nanos", "total_throttled", readThrottledNanos);
+        humanReadableNanos(builder, "total_elapsed_nanos", "total_elapsed", readElapsedNanos);
+        builder.endObject();
+
+        builder.endObject();
+        return builder;
+    }
+
+    static class Builder {
+
+        private final LongAdder writeCount = new LongAdder();
+        private final LongAdder writeBytes = new LongAdder();
+        private final LongAdder writeThrottledNanos = new LongAdder();
+        private final LongAdder writeElapsedNanos = new LongAdder();
+
+        private final LongAdder readCount = new LongAdder();
+        private final LongAdder readBytes = new LongAdder();
+        private final LongAdder readWaitNanos = new LongAdder();
+        private final LongAccumulator maxReadWaitNanos = new LongAccumulator(Long::max, Long.MIN_VALUE);
+        private final LongAdder readThrottledNanos = new LongAdder();
+        private final LongAdder readElapsedNanos = new LongAdder();
+
+        public RepositoryPerformanceSummary build() {
+            return new RepositoryPerformanceSummary(
+                writeCount.longValue(),
+                writeBytes.longValue(),
+                writeThrottledNanos.longValue(),
+                writeElapsedNanos.longValue(),
+                readCount.longValue(),
+                readBytes.longValue(),
+                readWaitNanos.longValue(),
+                Long.max(0L, maxReadWaitNanos.longValue()),
+                readThrottledNanos.longValue(),
+                readElapsedNanos.longValue()
+            );
+        }
+
+        public void add(BlobAnalyzeAction.Response response) {
+            writeCount.add(1L);
+            writeBytes.add(response.getWriteBytes());
+            writeThrottledNanos.add(response.getWriteThrottledNanos());
+            writeElapsedNanos.add(response.getWriteElapsedNanos());
+
+            final long checksumBytes = response.getChecksumBytes();
+
+            for (final BlobAnalyzeAction.ReadDetail readDetail : response.getReadDetails()) {
+                readCount.add(1L);
+                readBytes.add(checksumBytes);
+                readWaitNanos.add(readDetail.getFirstByteNanos());
+                maxReadWaitNanos.accumulate(readDetail.getFirstByteNanos());
+                readThrottledNanos.add(readDetail.getThrottledNanos());
+                readElapsedNanos.add(readDetail.getElapsedNanos());
+            }
+        }
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RestRepositoryAnalyzeAction.java

@@ -0,0 +1,68 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.client.node.NodeClient;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.rest.BaseRestHandler;
+import org.elasticsearch.rest.RestRequest;
+import org.elasticsearch.rest.RestResponse;
+import org.elasticsearch.rest.action.RestCancellableNodeClient;
+import org.elasticsearch.rest.action.RestStatusToXContentListener;
+
+import java.util.List;
+
+import static org.elasticsearch.rest.RestRequest.Method.POST;
+
+public class RestRepositoryAnalyzeAction extends BaseRestHandler {
+
+    @Override
+    public List<Route> routes() {
+        return List.of(new Route(POST, "/_snapshot/{repository}/_analyze"));
+    }
+
+    @Override
+    public String getName() {
+        return "repository_analyze";
+    }
+
+    @Override
+    public RestChannelConsumer prepareRequest(final RestRequest request, final NodeClient client) {
+        RepositoryAnalyzeAction.Request analyzeRepositoryRequest = new RepositoryAnalyzeAction.Request(request.param("repository"));
+
+        analyzeRepositoryRequest.blobCount(request.paramAsInt("blob_count", analyzeRepositoryRequest.getBlobCount()));
+        analyzeRepositoryRequest.concurrency(request.paramAsInt("concurrency", analyzeRepositoryRequest.getConcurrency()));
+        analyzeRepositoryRequest.readNodeCount(request.paramAsInt("read_node_count", analyzeRepositoryRequest.getReadNodeCount()));
+        analyzeRepositoryRequest.earlyReadNodeCount(
+            request.paramAsInt("early_read_node_count", analyzeRepositoryRequest.getEarlyReadNodeCount())
+        );
+        analyzeRepositoryRequest.seed(request.paramAsLong("seed", analyzeRepositoryRequest.getSeed()));
+        analyzeRepositoryRequest.rareActionProbability(
+            request.paramAsDouble("rare_action_probability", analyzeRepositoryRequest.getRareActionProbability())
+        );
+        analyzeRepositoryRequest.maxBlobSize(request.paramAsSize("max_blob_size", analyzeRepositoryRequest.getMaxBlobSize()));
+        analyzeRepositoryRequest.maxTotalDataSize(
+            request.paramAsSize("max_total_data_size", analyzeRepositoryRequest.getMaxTotalDataSize())
+        );
+        analyzeRepositoryRequest.timeout(request.paramAsTime("timeout", analyzeRepositoryRequest.getTimeout()));
+        analyzeRepositoryRequest.detailed(request.paramAsBoolean("detailed", analyzeRepositoryRequest.getDetailed()));
+
+        RestCancellableNodeClient cancelClient = new RestCancellableNodeClient(client, request.getHttpChannel());
+        return channel -> cancelClient.execute(
+            RepositoryAnalyzeAction.INSTANCE,
+            analyzeRepositoryRequest,
+            new RestStatusToXContentListener<>(channel) {
+                @Override
+                public RestResponse buildResponse(RepositoryAnalyzeAction.Response response, XContentBuilder builder) throws Exception {
+                    builder.humanReadable(request.paramAsBoolean("human", true));
+                    return super.buildResponse(response, builder);
+                }
+            }
+        );
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/SnapshotRepositoryTestKit.java

@@ -0,0 +1,62 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.action.ActionRequest;
+import org.elasticsearch.action.ActionResponse;
+import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
+import org.elasticsearch.cluster.node.DiscoveryNodes;
+import org.elasticsearch.common.settings.ClusterSettings;
+import org.elasticsearch.common.settings.IndexScopedSettings;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.settings.SettingsFilter;
+import org.elasticsearch.common.unit.TimeValue;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.plugins.ActionPlugin;
+import org.elasticsearch.plugins.Plugin;
+import org.elasticsearch.rest.RestController;
+import org.elasticsearch.rest.RestHandler;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class SnapshotRepositoryTestKit extends Plugin implements ActionPlugin {
+
+    @Override
+    public List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> getActions() {
+        return List.of(
+            new ActionHandler<>(RepositoryAnalyzeAction.INSTANCE, RepositoryAnalyzeAction.TransportAction.class),
+            new ActionHandler<>(BlobAnalyzeAction.INSTANCE, BlobAnalyzeAction.TransportAction.class),
+            new ActionHandler<>(GetBlobChecksumAction.INSTANCE, GetBlobChecksumAction.TransportAction.class)
+        );
+    }
+
+    @Override
+    public List<RestHandler> getRestHandlers(
+        Settings settings,
+        RestController restController,
+        ClusterSettings clusterSettings,
+        IndexScopedSettings indexScopedSettings,
+        SettingsFilter settingsFilter,
+        IndexNameExpressionResolver indexNameExpressionResolver,
+        Supplier<DiscoveryNodes> nodesInCluster
+    ) {
+        return List.of(new RestRepositoryAnalyzeAction());
+    }
+
+    static void humanReadableNanos(XContentBuilder builder, String rawFieldName, String readableFieldName, long nanos) throws IOException {
+        assert rawFieldName.equals(readableFieldName) == false : rawFieldName + " vs " + readableFieldName;
+
+        if (builder.humanReadable()) {
+            builder.field(readableFieldName, TimeValue.timeValueNanos(nanos).toHumanReadableString(2));
+        }
+
+        builder.field(rawFieldName, nanos);
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/src/test/java/org/elasticsearch/repositories/blobstore/testkit/AbstractSnapshotRepoTestKitRestTestCase.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.apache.http.client.methods.HttpPost;
+import org.elasticsearch.client.Request;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.test.rest.ESRestTestCase;
+
+public abstract class AbstractSnapshotRepoTestKitRestTestCase extends ESRestTestCase {
+
+    protected abstract String repositoryType();
+
+    protected abstract Settings repositorySettings();
+
+    public void testRepositoryAnalysis() throws Exception {
+        final String repositoryType = repositoryType();
+        final Settings repositorySettings = repositorySettings();
+
+        final String repository = "repository";
+        logger.info("creating repository [{}] of type [{}]", repository, repositoryType);
+        registerRepository(repository, repositoryType, true, repositorySettings);
+
+        final Request request = new Request(HttpPost.METHOD_NAME, "/_snapshot/" + repository + "/_analyze");
+        request.addParameter("blob_count", "10");
+        request.addParameter("concurrency", "4");
+        request.addParameter("max_blob_size", "1mb");
+        request.addParameter("timeout", "120s");
+        assertOK(client().performRequest(request));
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/test/java/org/elasticsearch/repositories/blobstore/testkit/RandomBlobContentBytesReferenceTests.java

@@ -0,0 +1,52 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.test.ESTestCase;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.zip.CRC32;
+
+import static org.elasticsearch.repositories.blobstore.testkit.RandomBlobContent.BUFFER_SIZE;
+import static org.hamcrest.Matchers.equalTo;
+
+public class RandomBlobContentBytesReferenceTests extends ESTestCase {
+
+    public void testStreamInput() throws IOException {
+        final AtomicBoolean readComplete = new AtomicBoolean();
+        final RandomBlobContent randomBlobContent = new RandomBlobContent(
+            "repo",
+            randomLong(),
+            () -> false,
+            () -> assertTrue("multiple notifications", readComplete.compareAndSet(false, true))
+        );
+
+        final int length = randomSize();
+        final int checksumStart = between(0, length - 1);
+        final int checksumEnd = between(checksumStart, length);
+
+        final byte[] output = new RandomBlobContentBytesReference(randomBlobContent, length).streamInput().readAllBytes();
+
+        assertThat(output.length, equalTo(length));
+        assertTrue(readComplete.get());
+
+        for (int i = BUFFER_SIZE; i < output.length; i++) {
+            assertThat("output repeats at position " + i, output[i], equalTo(output[i % BUFFER_SIZE]));
+        }
+
+        final CRC32 crc32 = new CRC32();
+        crc32.update(output, checksumStart, checksumEnd - checksumStart);
+        assertThat("checksum computed correctly", randomBlobContent.getChecksum(checksumStart, checksumEnd), equalTo(crc32.getValue()));
+    }
+
+    private static int randomSize() {
+        return between(0, 30000);
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/test/java/org/elasticsearch/repositories/blobstore/testkit/RandomBlobContentStreamTests.java

@@ -0,0 +1,136 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.repositories.RepositoryVerificationException;
+import org.elasticsearch.test.ESTestCase;
+
+import java.io.IOException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.zip.CRC32;
+
+import static org.elasticsearch.repositories.blobstore.testkit.RandomBlobContent.BUFFER_SIZE;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.not;
+
+public class RandomBlobContentStreamTests extends ESTestCase {
+
+    public void testReadAndResetAndSkip() {
+
+        final byte[] output = new byte[randomSize()];
+        final AtomicBoolean readComplete = new AtomicBoolean();
+        int position = 0;
+        int mark = 0;
+        int resetCount = 0;
+
+        final int checksumStart = between(0, output.length - 1);
+        final int checksumEnd = between(checksumStart, output.length);
+        final long checksum;
+
+        final RandomBlobContent randomBlobContent = new RandomBlobContent(
+            "repo",
+            randomLong(),
+            () -> false,
+            () -> assertTrue("multiple notifications", readComplete.compareAndSet(false, true))
+        );
+
+        try (RandomBlobContentStream randomBlobContentStream = new RandomBlobContentStream(randomBlobContent, output.length)) {
+
+            assertTrue(randomBlobContentStream.markSupported());
+
+            checksum = randomBlobContent.getChecksum(checksumStart, checksumEnd);
+
+            while (readComplete.get() == false) {
+                switch (between(1, resetCount < 10 ? 4 : 3)) {
+                    case 1:
+                        randomBlobContentStream.mark(between(0, Integer.MAX_VALUE));
+                        mark = position;
+                        break;
+                    case 2:
+                        final int nextByte = randomBlobContentStream.read();
+                        assertThat(nextByte, not(equalTo(-1)));
+                        output[position++] = (byte) nextByte;
+                        break;
+                    case 3:
+                        final int len = between(0, output.length - position);
+                        assertThat(randomBlobContentStream.read(output, position, len), equalTo(len));
+                        position += len;
+                        break;
+                    case 4:
+                        randomBlobContentStream.reset();
+                        resetCount += 1;
+                        if (randomBoolean()) {
+                            final int skipBytes = between(0, position - mark);
+                            assertThat(randomBlobContentStream.skip(skipBytes), equalTo((long) skipBytes));
+                            position = mark + skipBytes;
+                        } else {
+                            position = mark;
+                        }
+                        break;
+                }
+            }
+
+            if (randomBoolean()) {
+                assertThat(randomBlobContentStream.read(), equalTo(-1));
+            } else {
+                assertThat(randomBlobContentStream.read(output, 0, between(0, output.length)), equalTo(-1));
+            }
+        }
+
+        for (int i = BUFFER_SIZE; i < output.length; i++) {
+            assertThat("output repeats at position " + i, output[i], equalTo(output[i % BUFFER_SIZE]));
+        }
+
+        final CRC32 crc32 = new CRC32();
+        crc32.update(output, checksumStart, checksumEnd - checksumStart);
+        assertThat("checksum computed correctly", checksum, equalTo(crc32.getValue()));
+    }
+
+    public void testNotifiesOfCompletionOnce() throws IOException {
+
+        final AtomicBoolean readComplete = new AtomicBoolean();
+        final RandomBlobContent randomBlobContent = new RandomBlobContent(
+            "repo",
+            randomLong(),
+            () -> false,
+            () -> assertTrue("multiple notifications", readComplete.compareAndSet(false, true))
+        );
+
+        try (RandomBlobContentStream randomBlobContentStream = new RandomBlobContentStream(randomBlobContent, randomSize())) {
+            for (int i = between(0, 4); i > 0; i--) {
+                randomBlobContentStream.reset();
+                randomBlobContentStream.readAllBytes();
+                assertTrue(readComplete.get());
+            }
+        }
+
+        assertTrue(readComplete.get()); // even if not completely read
+    }
+
+    public void testReadingOneByteThrowsExceptionAfterCancellation() {
+        final RandomBlobContent randomBlobContent = new RandomBlobContent("repo", randomLong(), () -> true, () -> {});
+
+        try (RandomBlobContentStream randomBlobContentStream = new RandomBlobContentStream(randomBlobContent, randomSize())) {
+            // noinspection ResultOfMethodCallIgnored
+            expectThrows(RepositoryVerificationException.class, randomBlobContentStream::read);
+        }
+    }
+
+    public void testReadingBytesThrowsExceptionAfterCancellation() {
+        final RandomBlobContent randomBlobContent = new RandomBlobContent("repo", randomLong(), () -> true, () -> {});
+
+        try (RandomBlobContentStream randomBlobContentStream = new RandomBlobContentStream(randomBlobContent, randomSize())) {
+            expectThrows(RepositoryVerificationException.class, randomBlobContentStream::readAllBytes);
+        }
+    }
+
+    private static int randomSize() {
+        return between(0, 30000);
+    }
+
+}

x-pack/plugin/snapshot-repo-test-kit/src/test/java/org/elasticsearch/repositories/blobstore/testkit/RepositoryAnalyzeActionTests.java

@@ -0,0 +1,111 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit;
+
+import org.elasticsearch.common.unit.ByteSizeValue;
+import org.elasticsearch.test.ESTestCase;
+
+import java.util.List;
+import java.util.stream.IntStream;
+import java.util.stream.LongStream;
+
+import static org.hamcrest.Matchers.equalTo;
+
+public class RepositoryAnalyzeActionTests extends ESTestCase {
+
+    public void testGetBlobSizesAssertions() {
+        // Just checking that no assertions are thrown
+        getBlobSizesOrException(randomLongBetween(1L, Long.MAX_VALUE), randomLongBetween(1L, Long.MAX_VALUE), between(100, 10000));
+    }
+
+    public void testGetBlobSizesAssertionsSmall() {
+        // Just checking that no assertions are thrown for an exhaustive set of small examples
+        for (long maxBlobSize = 1L; maxBlobSize < 50L; maxBlobSize++) {
+            for (long maxTotalDataSize = 1L; maxTotalDataSize < 50L; maxTotalDataSize++) {
+                for (int blobCount = 1; blobCount < 25; blobCount++) {
+                    getBlobSizesOrException(maxBlobSize, maxTotalDataSize, blobCount);
+                }
+            }
+        }
+    }
+
+    public void testGetBlobSizesWithoutRepeats() {
+        final int logBlobSize = 10;
+        final long blobSize = 1 << logBlobSize;
+        final long totalSize = 2047;
+        assertThat(
+            getBlobSizes(blobSize, totalSize, logBlobSize + 1).stream().mapToLong(l -> l).sorted().toArray(),
+            equalTo(new long[] { 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 }) // one of each size
+        );
+    }
+
+    public void testGetBlobSizesWithRepeats() {
+        final int logBlobSize = 10;
+        final long blobSize = 1 << logBlobSize;
+        final long totalSize = 6014; // large enough for repeats
+        assertThat(
+            getBlobSizes(blobSize, totalSize, 26).stream().mapToLong(l -> l).sorted().toArray(),
+            equalTo(
+                new long[] { 1, 1, 2, 2, 4, 4, 8, 8, 16, 16, 32, 32, 64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512, 1024, 1024, 1024 }
+            )
+        );
+    }
+
+    public void testGetBlobSizesTotalTooSmall() {
+        final int logBlobSize = 10;
+        final long blobSize = 1 << logBlobSize;
+        final long totalSize = 2046; // not large enough for one of each size
+        assertThat(
+            getBlobSizes(blobSize, totalSize, logBlobSize + 1).stream().mapToLong(l -> l).sorted().toArray(),
+            equalTo(new long[] { 2, 4, 8, 16, 32, 64, 128, 128, 256, 256, 1024 })
+        );
+    }
+
+    public void testGetBlobSizesDefaultExample() {
+        // The defaults are 10MB max blob size, 1GB max total size, 100 blobs, which gives 4 blobs of each size (either powers of 2 or 10MB)
+        assertThat(
+            RepositoryAnalyzeAction.getBlobSizes(new RepositoryAnalyzeAction.Request("repo")).stream().mapToLong(l -> l).sorted().toArray(),
+            equalTo(
+                LongStream.range(0, 25)
+                    .flatMap(power -> IntStream.range(0, 4).mapToLong(i -> power == 24 ? ByteSizeValue.ofMb(10).getBytes() : 1L << power))
+                    .toArray()
+            )
+        );
+    }
+
+    public void testGetBlobSizesRealisticExample() {
+        // The docs suggest a realistic example needs 2GB max blob size, 1TB max total size, 2000 blobs, which gives 62 or 63 blobs of each
+        // size (powers of 2 up to 2GB)
+        assertThat(
+            getBlobSizes(ByteSizeValue.ofGb(2).getBytes(), ByteSizeValue.ofTb(1).getBytes(), 2000).stream()
+                .mapToLong(l -> l)
+                .sorted()
+                .toArray(),
+            equalTo(
+                LongStream.range(0, 32).flatMap(power -> IntStream.range(0, power < 16 ? 62 : 63).mapToLong(i -> 1L << power)).toArray()
+            )
+        );
+    }
+
+    private static void getBlobSizesOrException(long maxBlobSize, long maxTotalDataSize, int blobCount) {
+        try {
+            getBlobSizes(maxBlobSize, maxTotalDataSize, blobCount);
+        } catch (IllegalArgumentException e) {
+            // ok
+        }
+    }
+
+    private static List<Long> getBlobSizes(long maxBlobSize, long maxTotalDataSize, int blobCount) {
+        final RepositoryAnalyzeAction.Request request = new RepositoryAnalyzeAction.Request("repo");
+        request.maxBlobSize(new ByteSizeValue(maxBlobSize));
+        request.maxTotalDataSize(new ByteSizeValue(maxTotalDataSize));
+        request.blobCount(blobCount);
+        return RepositoryAnalyzeAction.getBlobSizes(request);
+    }
+
+}