[TEST] Force close failed job before skipping test (#53128)

The assumption added in #52631 skips a problematic test
if it fails to create the required conditions for the
scenario it is supposed to be testing.  (This happens
very rarely.)

However, before skipping the test it needs to remove the
failed job it has created because the standard test
cleanup code treats failed jobs as fatal errors.

Closes #52608

x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/integration/MlDistributedFailureIT.java

@@ -248,8 +248,15 @@ public class MlDistributedFailureIT extends BaseMlIntegTestCase {
 
         // It is possible that the datafeed has already detected the job failure and
         // terminated itself. In this happens there is no persistent task to stop
-        assumeFalse("The datafeed task is null most likely because the datafeed detected the job had failed. " +
+        if (task == null) {
+            // We have to force close the job, because the standard cleanup
+            // will treat a leftover failed job as a fatal error
+            CloseJobAction.Request closeJobRequest = new CloseJobAction.Request(jobId);
+            closeJobRequest.setForce(true);
+            client().execute(CloseJobAction.INSTANCE, closeJobRequest).actionGet();
+            assumeFalse("The datafeed task is null most likely because the datafeed detected the job had failed. " +
                 "This is expected to happen extremely rarely but the test cannot continue in these circumstances.", task == null);
+        }
 
         UpdatePersistentTaskStatusAction.Request updatePersistentTaskStatusRequest =
                 new UpdatePersistentTaskStatusAction.Request(task.getId(), task.getAllocationId(), DatafeedState.STOPPING);