[docs] explainer for java packaging tests (#30825)

qa/vagrant/README.md

@@ -0,0 +1,119 @@
+# packaging tests
+
+This project contains tests that verify the distributions we build work
+correctly on the operating systems we support. They're intended to cover the
+steps a user would take when installing and configuring an Elasticsearch
+distribution. They're not intended to have significant coverage of the behavior
+of Elasticsearch's features.
+
+There are two types of tests in this project. The old tests live in
+`src/test/` and are written in [Bats](https://github.com/sstephenson/bats),
+which is a flavor of bash scripts that run as unit tests. These tests are
+deprecated because Bats is unmaintained and cannot run on Windows.
+
+The new tests live in `src/main/` and are written in Java. Like the old tests,
+this project's tests are run inside the VM, not on your host. All new packaging
+tests should be added to this set of tests if possible.
+
+## Running these tests
+
+See the section in [TESTING.asciidoc](../../TESTING.asciidoc#testing-packaging)
+
+## Adding a new test class
+
+When gradle runs the packaging tests on a VM, it runs the full suite by
+default. To add a test class to the suite, add its `class` to the
+`@SuiteClasses` annotation in [PackagingTests.java](src/main/java/org/elasticsearch/packaging/PackagingTests.java).
+If a test class is added to the project but not to this annotation, it will not
+run in CI jobs. The test classes are run in the order they are listed in the
+annotation.
+
+## Choosing which distributions to test
+
+Distributions are represented by [enum values](src/main/java/org/elasticsearch/packaging/util/Distribution.java)
+which know if they are compatible with the platform the tests are currently
+running on. To skip a test if the distribution it's using isn't compatible with
+the current platform, put this [assumption](https://github.com/junit-team/junit4/wiki/assumptions-with-assume)
+in your test method or in a `@Before` method
+
+```java
+assumeTrue(distribution.packaging.compatible);
+```
+
+Similarly if you write a test that is intended only for particular platforms,
+you can make an assumption using the constants and methods in [Platforms.java](src/main/java/org/elasticsearch/packaging/util/Platforms.java)
+
+```java
+assumeTrue("only run on windows", Platforms.WINDOWS);
+
+assumeTrue("only run if using systemd", Platforms.isSystemd());
+```
+
+## Writing a test that covers multiple distributions
+
+It seems like the way to do this that makes it the most straightforward to run
+and reproduce specific test cases is to create a test case class with an
+abstract method that provides the distribution
+
+```java
+public abstract class MyTestCase {
+  @Test
+  public void myTest() { /* do something with the value of #distribution() */ }
+  abstract Distribution distribution();
+}
+```
+
+and then for each distribution you want to test, create a subclass
+
+```java
+public class MyTestDefaultTar extends MyTestCase {
+  @Override
+  Distribution distribution() { return Distribution.DEFAULT_TAR; }
+}
+```
+
+That way when a test fails the user gets told explicitly that `MyTestDefaultTar`
+failed, and to reproduce it they should run that class. See [ArchiveTestCase](src/main/java/org/elasticsearch/packaging/test/ArchiveTestCase.java)
+and its children for an example of this.
+
+## Running external commands
+
+In general it's probably best to avoid running external commands when a good
+Java alternative exists. For example most filesystem operations can be done with
+the java.nio.file APIs. For those that aren't, use an instance of [Shell](src/main/java/org/elasticsearch/packaging/util/Shell.java)
+
+Despite the name, commands run with this class are not run in a shell, and any
+familiar features of shells like variables or expansion won't work.
+
+If you do need the shell, you must explicitly invoke the shell's command. For
+example to run a command with Bash, use the `bash -c command` syntax. Note that
+the entire script must be in a single string argument
+
+```java
+Shell sh = new Shell();
+sh.run("bash", "-c", "echo $foo; echo $bar");
+```
+
+Similary for powershell - again, the entire powershell script must go in a
+single string argument
+
+```java
+sh.run("powershell.exe", "-Command", "Write-Host $foo; Write-Host $bar");
+```
+
+On Linux, most commands you'll want to use will be executable files and will
+work fine without a shell
+
+```java
+sh.run("tar", "-xzpf", "elasticsearch-6.1.0.tar.gz");
+```
+
+On Windows you'll mostly want to use powershell as it can do a lot more and
+gives much better feedback than Windows' legacy command line. Unfortunately that
+means that you'll need to use the `powershell.exe -Command` syntax as
+powershell's [Cmdlets](https://msdn.microsoft.com/en-us/library/ms714395.aspx)
+don't correspond to executable files and are not runnable by `Runtime` directly.
+
+When writing powershell commands this way, make sure to test them as some types
+of formatting can cause it to return a successful exit code but not run
+anything.

qa/vagrant/src/main/java/org/elasticsearch/packaging/test/ArchiveTestCase.java

@@ -31,8 +31,7 @@ import org.elasticsearch.packaging.util.Installation;
 import static org.elasticsearch.packaging.util.Cleanup.cleanEverything;
 import static org.elasticsearch.packaging.util.Archives.installArchive;
 import static org.elasticsearch.packaging.util.Archives.verifyArchiveInstallation;
-import static org.hamcrest.CoreMatchers.is;
-import static org.junit.Assume.assumeThat;
+import static org.junit.Assume.assumeTrue;
 
 /**
  * Tests that apply to the archive distributions (tar, zip). To add a case for a distribution, subclass and
@@ -54,7 +53,7 @@ public abstract class ArchiveTestCase {
 
     @Before
     public void onlyCompatibleDistributions() {
-        assumeThat(distribution().packaging.compatible, is(true));
+        assumeTrue("only compatible distributions", distribution().packaging.compatible);
     }
 
     @Test