[Packaging] Run tests in vagrant

This creates a module in qa called vagrant that can be run if you have
vagrant and virtualbox installed and will run the packaging tests in trusty
and centos-7.0. You can ask it to run tests in other linuxes. This is the full
list:
* precise aka Ubuntu 12.04
* trusty aka Ubuntu 14.04
* vivid aka Ubuntun 15.04
* wheezy aka Debian 7, the current debian oldstable distribution
* jessie aka Debian 8, the current debina stable distribution
* centos-6
* centos-7
* fedora-22
* oel-7

There is lots of documentation on how to do this in the TESTING.asciidoc.

Closes #12611

.gitignore

@@ -15,9 +15,11 @@ docs/build.log
 /tmp/
 backwards/
 html_docs
+.vagrant/
+
 ## eclipse ignores (use 'mvn eclipse:eclipse' to build eclipse projects)
 ## All files (.project, .classpath, .settings/*) should be generated through Maven which
-## will correctly set the classpath based on the declared dependencies and write settings 
+## will correctly set the classpath based on the declared dependencies and write settings
 ## files to ensure common coding style across Eclipse and IDEA.
 .project
 .classpath

TESTING.asciidoc

@@ -81,7 +81,7 @@ You can also filter tests by certain annotations ie:
 Those annotation names can be combined into a filter expression like:
 
 ------------------------------------------------
-mvn test -Dtests.filter="@nightly and not @backwards" 
+mvn test -Dtests.filter="@nightly and not @backwards"
 ------------------------------------------------
 
 to run all nightly test but not the ones that are backwards tests. `tests.filter` supports
@@ -89,7 +89,7 @@ the boolean operators `and, or, not` and grouping ie:
 
 
 ---------------------------------------------------------------
-mvn test -Dtests.filter="@nightly and not(@badapple or @backwards)" 
+mvn test -Dtests.filter="@nightly and not(@badapple or @backwards)"
 ---------------------------------------------------------------
 
 === Seed and repetitions.
@@ -102,7 +102,7 @@ mvn test -Dtests.seed=DEADBEEF
 
 === Repeats _all_ tests of ClassName N times.
 
-Every test repetition will have a different method seed 
+Every test repetition will have a different method seed
 (derived from a single random master seed).
 
 --------------------------------------------------
@@ -149,7 +149,7 @@ mvn test -Dtests.awaitsfix=[false] - known issue (@AwaitsFix)
 
 === Load balancing and caches.
 
-By default, the tests run sequentially on a single forked JVM. 
+By default, the tests run sequentially on a single forked JVM.
 
 To run with more forked JVMs than the default use:
 
@@ -158,7 +158,7 @@ mvn test -Dtests.jvms=8 test
 ----------------------------
 
 Don't count hypercores for CPU-intense tests and leave some slack
-for JVM-internal threads (like the garbage collector). Make sure there is 
+for JVM-internal threads (like the garbage collector). Make sure there is
 enough RAM to handle child JVMs.
 
 === Test compatibility.
@@ -208,7 +208,7 @@ mvn test -Dtests.output=always
 Configure the heap size.
 
 ------------------------------
-mvn test -Dtests.heap.size=512m 
+mvn test -Dtests.heap.size=512m
 ------------------------------
 
 Pass arbitrary jvm arguments.
@@ -231,7 +231,7 @@ mvn test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.bwc.path=
 Note that backwards tests must be run with security manager disabled.
 If the elasticsearch release is placed under `./backwards/elasticsearch-x.y.z` the path
 can be omitted:
- 
+
 ---------------------------------------------------------------------------
 mvn test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.security.manager=false
 ---------------------------------------------------------------------------
@@ -242,7 +242,7 @@ already in your elasticsearch clone):
 ---------------------------------------------------------------------------
 $ mkdir backwards && cd backwards
 $ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.tar.gz
-$ tar -xzf elasticsearch-1.2.1.tar.gz 
+$ tar -xzf elasticsearch-1.2.1.tar.gz
 ---------------------------------------------------------------------------
 
 == Running integration tests
@@ -314,25 +314,180 @@ mvn test -Pdev
 
 == Testing scripts
 
-Shell scripts can be tested with the Bash Automate Testing System tool available
-at https://github.com/sstephenson/bats. Once the tool is installed, you can
-execute a .bats test file with the following command:
+The simplest way to test scripts and the packaged distributions is to use
+Vagrant. You can get started by following there five easy steps:
 
----------------------------------------------------------------------------
-bats test_file.bats
----------------------------------------------------------------------------
+. Install Virtual Box and Vagrant.
 
-When executing the test files located in the `/packaging/scripts` folder,
-it's possible to add the flag `ES_CLEAN_BEFORE_TEST=true` to clean the test
-environment before the tests are executed:
+. (Optional) Install vagrant-cachier to squeeze a bit more performance out of
+the process:
+--------------------------------------
+vagrant plugin install vagrant-cachier
+--------------------------------------
 
----------------------------------------------------------------------------
-ES_CLEAN_BEFORE_TEST=true bats 30_deb_package.bats
----------------------------------------------------------------------------
+. Validate your installed dependencies:
+-------------------------------------
+mvn -Pvagrant -pl qa/vagrant validate
+-------------------------------------
+
+. Download the VMs. Since Maven or ant or something eats the progress reports
+from Vagrant when you run it inside mvn its probably best if you run this one
+time to setup all the VMs one at a time. Run this to download and setup the VMs
+we use for testing by default:
+--------------------------------------------------------
+vagrant up --provision trusty && vagrant halt trusty
+vagrant up --provision centos-7 && vagrant halt centos-7
+--------------------------------------------------------
+or run this to download and setup all the VMs:
+-------------------------------------------------------------------------------
+vagrant halt
+for box in $(vagrant status | grep 'poweroff\|not created' | cut -f1 -d' '); do
+  vagrant up --provision $box
+  vagrant halt $box
+done
+-------------------------------------------------------------------------------
+
+. Smoke test the maven/ant dance that we use to get vagrant involved in
+integration testing is working:
+---------------------------------------------
+mvn -Pvagrant,smoke-vms -pl qa/vagrant verify
+---------------------------------------------
+or this to validate all the VMs:
+-------------------------------------------------
+mvn -Pvagrant,smoke-vms,all -pl qa/vagrant verify
+-------------------------------------------------
+That will start up the VMs and then immediate quit.
+
+. Finally run the tests. The fastest way to get this started is to run:
+-----------------------------------
+mvn clean install -DskipTests
+mvn -Pvagrant -pl qa/vagrant verify
+-----------------------------------
+You could just run:
+--------------------
+mvn -Pvagrant verify
+--------------------
+but that will run all the tests. Which is probably a good thing, but not always
+what you want.
+
+Whichever snippet you run mvn will build the tar, zip and deb packages. If you
+have rpmbuild installed it'll build the rpm package as well. Then mvn will
+spin up trusty and verify the tar, zip, and deb package. If you have rpmbuild
+installed it'll spin up centos-7 and verify the tar, zip and rpm packages. We
+chose those two distributions as the default because they cover deb and rpm
+packaging and SyvVinit and systemd.
+
+You can control the boxes that are used for testing like so. Run just
+fedora-22 with:
+--------------------------------------------
+mvn -Pvagrant -pl qa/vagrant verify -DboxesToTest=fedora-22
+--------------------------------------------
+or run wheezy and trusty:
+------------------------------------------------------------------
+mvn -Pvagrant -pl qa/vagrant verify -DboxesToTest='wheezy, trusty'
+------------------------------------------------------------------
+or run all the boxes:
+---------------------------------------
+mvn -Pvagrant,all -pl qa/vagrant verify
+---------------------------------------
+
+Its important to know that if you ctrl-c any of these `mvn` runs that you'll
+probably leave a VM up. You can terminate it by running:
+------------
+vagrant halt
+------------
+
+This is just regular vagrant so you can run normal multi box vagrant commands
+to test things manually. Just run:
+---------------------------------------
+vagrant up trusty && vagrant ssh trusty
+---------------------------------------
+to get an Ubuntu or
+-------------------------------------------
+vagrant up centos-7 && vagrant ssh centos-7
+-------------------------------------------
+to get a CentOS. Once you are done with them you should halt them:
+-------------------
+vagrant halt trusty
+-------------------
+
+These are the linux flavors the Vagrantfile currently supports:
+* precise aka Ubuntu 12.04
+* trusty aka Ubuntu 14.04
+* vivid aka Ubuntun 15.04
+* wheezy aka Debian 7, the current debian oldstable distribution
+* jessie aka Debian 8, the current debina stable distribution
+* centos-6
+* centos-7
+* fedora-22
+* oel-7 aka Oracle Enterprise Linux 7
+
+We're missing the following from the support matrix because there aren't high
+quality boxes available in vagrant atlas:
+* sles-11
+* sles-12
+* opensuse-13
+* oel-6
+
+We're missing the follow because our tests are very linux/bash centric:
+* Windows Server 2012
+
+Its important to think of VMs like cattle: if they become lame you just shoot
+them and let vagrant reprovision them. Say you've hosed your precise VM:
+----------------------------------------------------
+vagrant ssh precise -c 'sudo rm -rf /bin'; echo oops
+----------------------------------------------------
+All you've got to do to get another one is
+----------------------------------------------
+vagrant destroy -f trusty && vagrant up trusty
+----------------------------------------------
+The whole process takes a minute and a half on a modern laptop, two and a half
+without vagrant-cachier.
+
+Its possible that some downloads will fail and it'll be impossible to restart
+them. This is a bug in vagrant. See the instructions here for how to work
+around it:
+https://github.com/mitchellh/vagrant/issues/4479
+
+Some vagrant commands will work on all VMs at once:
+------------------
+vagrant halt
+vagrant destroy -f
+------------------
+
+----------
+vagrant up
+----------
+would normally start all the VMs but we've prevented that because that'd
+consume a ton of ram.
+
+== Testing scripts more directly
+
+In general its best to stick to testing in vagrant because the bats scripts are
+destructive. When working with a single package its generally faster to run its
+tests in a tighter loop than maven provides. In one window:
+--------------------------------
+mvn -pl distribution/rpm package
+--------------------------------
+and in another window:
+----------------------------------------------------
+vagrant up centos-7 && vagrant ssh centos-7
+cd $RPM
+sudo ES_CLEAN_BEFORE_TEST=true bats $BATS/*rpm*.bats
+----------------------------------------------------
 
-The current mode of execution is to copy all the packages that should be tested
-into one directory, then copy the bats files into the same directory and run
-those.
+At this point `ES_CLEAN_BEFORE_TEST=true` is required or tests fail spuriously.
+
+If you wanted to retest all the release artifacts on a single VM you could:
+-------------------------------------------------
+# Build all the distributions fresh but skip recompiling elasticsearch:
+mvn -amd -pl distribution install -DskipTests
+# Copy them all the testroot
+mvn -Pvagrant -pl qa/vagrant pre-integration-test
+vagrant up trusty && vagrant ssh trusty
+cd $TESTROOT
+sudo ES_CLEAN_BEFORE_TEST=true bats $BATS/*.bats
+-------------------------------------------------
 
 == Coverage analysis
 

Vagrantfile

@@ -0,0 +1,156 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# This Vagrantfile exists to test packaging. Read more about its use in the
+# vagrant section in TESTING.asciidoc.
+
+# Licensed to Elasticsearch under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+Vagrant.configure(2) do |config|
+  config.vm.define "precise", autostart: false do |config|
+    config.vm.box = "ubuntu/precise64"
+    ubuntu_common config
+  end
+  config.vm.define "trusty", autostart: false do |config|
+    config.vm.box = "ubuntu/trusty64"
+    ubuntu_common config
+  end
+  config.vm.define "vivid", autostart: false do |config|
+    config.vm.box = "ubuntu/vivid64"
+    ubuntu_common config
+  end
+  config.vm.define "wheezy", autostart: false do |config|
+    config.vm.box = "debian/wheezy64"
+    deb_common(config)
+  end
+  config.vm.define "jessie", autostart: false do |config|
+    config.vm.box = "debian/jessie64"
+    deb_common(config)
+  end
+  config.vm.define "centos-6", autostart: false do |config|
+    # TODO switch from chef to boxcutter to provide?
+    config.vm.box = "chef/centos-6.6"
+    rpm_common(config)
+  end
+  config.vm.define "centos-7", autostart: false do |config|
+    # There is a centos/7 box but it doesn't have rsync or virtualbox guest
+    # stuff on there so its slow to use. So chef it is....
+    # TODO switch from chef to boxcutter to provide?
+    config.vm.box = "chef/centos-7.0"
+    rpm_common(config)
+  end
+  # This box hangs _forever_ on ```yum check-update```. I have no idea why.
+  # config.vm.define "oel-6", autostart: false do |config|
+  #   config.vm.box = "boxcutter/oel66"
+  #   rpm_common(config)
+  # end
+  config.vm.define "oel-7", autostart: false do |config|
+    config.vm.box = "boxcutter/oel70"
+    rpm_common(config)
+  end
+  config.vm.define "fedora-22", autostart: false do |config|
+    # Fedora hosts their own 'cloud' images that aren't in Vagrant's Atlas but
+    # and are missing required stuff like rsync. It'd be nice if we could use
+    # them but they much slower to get up and running then the boxcutter image.
+    config.vm.box = "boxcutter/fedora22"
+    dnf_common(config)
+  end
+  # Switch the default share for the project root from /vagrant to
+  # /elasticsearch because /vagrant is confusing when there is a project inside
+  # the elasticsearch project called vagrant....
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "", "/elasticsearch"
+end
+
+def ubuntu_common(config)
+  # Ubuntu is noisy
+  # http://foo-o-rama.com/vagrant--stdin-is-not-a-tty--fix.html
+  config.vm.provision "fix-no-tty", type: "shell" do |s|
+      s.privileged = false
+      s.inline = "sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile"
+  end
+  deb_common(config)
+end
+
+def deb_common(config)
+  provision(config, "apt-get update", "/var/cache/apt/archives/last_update",
+    "apt-get install -y", "openjdk-7-jdk")
+  if Vagrant.has_plugin?("vagrant-cachier")
+    config.cache.scope = :box
+  end
+end
+
+def rpm_common(config)
+  provision(config, "yum check-update", "/var/cache/yum/last_update",
+    "yum install -y", "java-1.7.0-openjdk-devel")
+  if Vagrant.has_plugin?("vagrant-cachier")
+    config.cache.scope = :box
+  end
+end
+
+def dnf_common(config)
+  provision(config, "dnf check-update", "/var/cache/dnf/last_update",
+    "dnf install -y", "java-1.8.0-openjdk-devel")
+  if Vagrant.has_plugin?("vagrant-cachier")
+    config.cache.scope = :box
+    # Autodetect doesn't work....
+    config.cache.auto_detect = false
+    config.cache.enable :generic, { :cache_dir => "/var/cache/dnf" }
+  end
+end
+
+
+def provision(config, update_command, update_tracking_file, install_command, java_package)
+  config.vm.provision "elasticsearch bats dependencies", type: "shell", inline: <<-SHELL
+    set -e
+    installed() {
+      command -v $1 2>&1 >/dev/null
+    }
+    install() {
+      # Only apt-get update if we haven't in the last day
+      if [ ! -f /tmp/update ] || [ "x$(find /tmp/update -mtime +0)" == "x/tmp/update" ]; then
+          sudo #{update_command} || true
+          touch #{update_tracking_file}
+      fi
+      sudo #{install_command} $1
+    }
+    ensure() {
+      installed $1 || install $1
+    }
+    installed java || install #{java_package}
+    ensure curl
+    ensure unzip
+
+    installed bats || {
+      # Bats lives in a git repository....
+      ensure git
+      git clone https://github.com/sstephenson/bats /tmp/bats
+      # Centos doesn't add /usr/local/bin to the path....
+      sudo /tmp/bats/install.sh /usr
+      sudo rm -rf /tmp/bats
+    }
+    cat \<\<VARS > /etc/profile.d/elasticsearch_vars.sh
+export ZIP=/elasticsearch/distribution/zip/target/releases
+export TAR=/elasticsearch/distribution/tar/target/releases
+export RPM=/elasticsearch/distribution/rpm/target/releases
+export DEB=/elasticsearch/distribution/deb/target/releases
+export TESTROOT=/elasticsearch/qa/vagrant/target/testroot
+export BATS=/elasticsearch/qa/vagrant/src/test/resources/packaging/scripts/
+VARS
+  SHELL
+end

dev-tools/src/main/resources/ant/integration-tests.xml

@@ -237,7 +237,7 @@
   <target name="setup-workspace-zip" depends="stop-external-cluster">
     <sequential>
       <delete dir="${integ.scratch}"/>
-      <unzip src="${project.build.directory}/releases/${project.artifactId}-${project.version}.zip" 
+      <unzip src="${project.build.directory}/releases/${project.artifactId}-${project.version}.zip"
              dest="${integ.scratch}"/>
     </sequential>
   </target>
@@ -252,7 +252,7 @@
   <target name="setup-workspace-tar" depends="stop-external-cluster">
     <sequential>
       <delete dir="${integ.scratch}"/>
-      <untar src="${project.build.directory}/releases/${project.artifactId}-${project.version}.tar.gz" 
+      <untar src="${project.build.directory}/releases/${project.artifactId}-${project.version}.tar.gz"
              dest="${integ.scratch}"
              compression="gzip"/>
     </sequential>
@@ -273,13 +273,13 @@
       <!-- print some basic package info -->
       <exec executable="dpkg-deb" failonerror="true" taskname="deb-info">
          <arg value="-I"/>
-         <arg value="${debfile}"/> 
+         <arg value="${debfile}"/>
       </exec>
       <!-- extract contents from .deb package -->
       <exec executable="dpkg-deb" failonerror="true">
-         <arg value="-x"/> 
-         <arg value="${debfile}"/> 
-         <arg value="${integ.scratch}/deb-extracted"/> 
+         <arg value="-x"/>
+         <arg value="${debfile}"/>
+         <arg value="${integ.scratch}/deb-extracted"/>
       </exec>
     </sequential>
   </target>
@@ -306,7 +306,7 @@
          <arg value="-q"/>
          <arg value="-i"/>
          <arg value="-p"/>
-         <arg value="${rpm.file}"/> 
+         <arg value="${rpm.file}"/>
       </exec>
       <!-- extract contents from .rpm package -->
       <exec executable="rpm" failonerror="true" taskname="rpm">
@@ -315,11 +315,11 @@
          <arg value="--badreloc"/>
          <arg value="--relocate"/>
          <arg value="/=${rpm.extracted}"/>
-         <arg value="--nodeps"/> 
-         <arg value="--noscripts"/> 
-         <arg value="--notriggers"/> 
+         <arg value="--nodeps"/>
+         <arg value="--noscripts"/>
+         <arg value="--notriggers"/>
          <arg value="-i"/>
-         <arg value="${rpm.file}"/> 
+         <arg value="${rpm.file}"/>
       </exec>
     </sequential>
   </target>
@@ -359,5 +359,4 @@
        </condition>
     </fail>
   </target>
-
 </project>

qa/pom.xml

@@ -148,4 +148,13 @@
         <module>smoke-test-plugins</module>
         <module>smoke-test-shaded</module>
     </modules>
+
+    <profiles>
+        <profile>
+            <id>vagrant</id>
+            <modules>
+                <module>vagrant</module>
+            </modules>
+        </profile>
+    </profiles>
 </project>

qa/vagrant/pom.xml

@@ -0,0 +1,279 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.elasticsearch.qa</groupId>
+        <artifactId>elasticsearch-qa</artifactId>
+        <version>2.0.0-beta1-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>elasticsearch-distribution-tests</artifactId>
+    <name>QA: Elasticsearch Vagrant Tests</name>
+    <description>Tests the Elasticsearch distribution artifacts on virtual
+      machines using vagrant and bats.</description>
+    <packaging>pom</packaging>
+
+    <!-- The documentation for how to run this is in ../../Vagrantfile -->
+    <properties>
+      <testScripts>*.bats</testScripts>
+      <testCommand>sudo ES_CLEAN_BEFORE_TEST=true bats $BATS/${testScripts}</testCommand>
+
+      <allDebBoxes>precise, trusty, vivid, wheezy, jessie</allDebBoxes>
+      <allRpmBoxes>centos-6, centos-7, fedora-22, oel-7</allRpmBoxes>
+
+      <debBoxes>trusty</debBoxes>
+      <rpmBoxes>centos-7</rpmBoxes>
+
+      <!-- Unless rpmbuild is available on the host we can't test rpm based
+           boxes because we can't build the rpm and they fail without the rpm.
+           So to get good coverage you'll need to run this on a system with
+           rpmbuild installed - either osx via homebrew or fedora/centos/rhel.
+           -->
+      <proposedBoxesToTest>${debBoxes}</proposedBoxesToTest>
+
+      <!-- rpmbuild location : default to /usr/bin/rpmbuild -->
+      <packaging.rpm.rpmbuild>/usr/bin/rpmbuild</packaging.rpm.rpmbuild>
+    </properties>
+
+    <build>
+        <plugins>
+            <!-- Clean the location where we keep the distribution artifacts
+                 to make sure that there aren't any old versions in there. -->
+            <plugin>
+                <artifactId>maven-clean-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>clean-testroot</id>
+                        <phase>pre-integration-test</phase>
+                        <goals>
+                            <goal>clean</goal>
+                        </goals>
+                        <configuration>
+                            <excludeDefaultDirectories>true</excludeDefaultDirectories>
+                            <filesets>
+                                <fileset>
+                                    <directory>${project.build.directory}/testroot</directory>
+                                </fileset>
+                            </filesets>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+            <!-- Put the distribution artifacts some place the test can get at
+                 them -->
+            <plugin>
+                <artifactId>maven-dependency-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>copy-common-to-testroot</id>
+                        <phase>pre-integration-test</phase>
+                        <goals>
+                            <goal>copy</goal>
+                        </goals>
+                        <configuration>
+                            <outputDirectory>${project.build.directory}/testroot</outputDirectory>
+                            <artifactItems>
+                                <artifactItem>
+                                    <groupId>org.elasticsearch.distribution.zip</groupId>
+                                    <artifactId>elasticsearch</artifactId>
+                                    <version>${elasticsearch.version}</version>
+                                    <type>zip</type>
+                                </artifactItem>
+                                <artifactItem>
+                                    <groupId>org.elasticsearch.distribution.tar</groupId>
+                                    <artifactId>elasticsearch</artifactId>
+                                    <version>${elasticsearch.version}</version>
+                                    <type>tar.gz</type>
+                                </artifactItem>
+                                <artifactItem>
+                                    <groupId>org.elasticsearch.distribution.deb</groupId>
+                                    <artifactId>elasticsearch</artifactId>
+                                    <version>${elasticsearch.version}</version>
+                                    <type>deb</type>
+                                </artifactItem>
+                            </artifactItems>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-antrun-plugin</artifactId>
+                <dependencies>
+                    <dependency>
+                        <groupId>ant-contrib</groupId>
+                        <artifactId>ant-contrib</artifactId>
+                        <version>1.0b3</version>
+                        <exclusions>
+                            <exclusion>
+                                <groupId>ant</groupId>
+                                <artifactId>ant</artifactId>
+                            </exclusion>
+                        </exclusions>
+                    </dependency>
+                </dependencies>
+                <executions>
+                    <execution>
+                        <id>check-vagrant-version</id>
+                        <phase>validate</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <target>
+                                <taskdef resource="net/sf/antcontrib/antlib.xml"
+                                    classpathref="maven.dependency.classpath" />
+                                <ant antfile="src/dev/ant/vagrant-integration-tests.xml"
+                                    target="check-vagrant-version"/>
+                            </target>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>test-vms</id>
+                        <phase>integration-test</phase>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                        <configuration>
+                            <target unless="${skipTests}">
+                                <taskdef resource="net/sf/antcontrib/antlib.xml"
+                                    classpathref="maven.dependency.classpath" />
+                                <echo message="Running package tests on ${boxesToTest}"/>
+                                <ant antfile="src/dev/ant/vagrant-integration-tests.xml"
+                                    target="vagrant-test-all-boxes"/>
+                            </target>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+
+    <profiles>
+        <!-- The following profiles change which boxes are run and whether or
+             not this build depends on the rpm artifact. We only depend on the
+             rpm artifact if this machine is capable of building it and we only
+             test on the rpm based distributions if the rpm is available
+             because the tests require it to be. -->
+        <profile>
+            <!-- Test on all boxes -->
+            <id>all</id>
+            <properties>
+              <debBoxes>${allDebBoxes}</debBoxes>
+              <rpmBoxes>${allRpmBoxes}</rpmBoxes>
+            </properties>
+        </profile>
+        <profile>
+            <!-- Enable the rpm artifact and rpm-boxes because we're on an
+                 rpm-based distribution. -->
+            <id>rpm</id>
+            <activation>
+                <file>
+                    <exists>${packaging.rpm.rpmbuild}</exists>
+                </file>
+            </activation>
+            <build>
+                <plugins>
+                    <plugin>
+                        <artifactId>maven-dependency-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>copy-rpm-to-testroot</id>
+                                <phase>pre-integration-test</phase>
+                                <goals>
+                                    <goal>copy</goal>
+                                </goals>
+                                <configuration>
+                                    <outputDirectory>${project.build.directory}/testroot</outputDirectory>
+                                    <artifactItems>
+                                        <artifactItem>
+                                            <groupId>org.elasticsearch.distribution.rpm</groupId>
+                                            <artifactId>elasticsearch</artifactId>
+                                            <version>${elasticsearch.version}</version>
+                                            <type>rpm</type>
+                                        </artifactItem>
+                                    </artifactItems>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+            <dependencies>
+                <dependency>
+                    <groupId>org.elasticsearch.distribution</groupId>
+                    <artifactId>elasticsearch-rpm</artifactId>
+                    <version>${elasticsearch.version}</version>
+                    <type>rpm</type>
+                </dependency>
+            </dependencies>
+            <properties>
+                <proposedBoxesToTest>${debBoxes}, ${rpmBoxes}</proposedBoxesToTest>
+            </properties>
+        </profile>
+        <profile>
+            <!-- Enable the rpm artifact and rpm-boxes because we're on an
+               rpm-based distribution. -->
+            <id>rpm-via-homebrew</id>
+            <activation>
+                <file>
+                    <exists>/usr/local/bin/rpmbuild</exists>
+                </file>
+            </activation>
+            <build>
+                <plugins>
+                    <plugin>
+                        <artifactId>maven-dependency-plugin</artifactId>
+                        <executions>
+                            <execution>
+                                <id>copy-rpm-to-testroot</id>
+                                <phase>pre-integration-test</phase>
+                                <goals>
+                                    <goal>copy</goal>
+                                </goals>
+                                <configuration>
+                                    <outputDirectory>${project.build.directory}/testroot</outputDirectory>
+                                    <artifactItems>
+                                        <artifactItem>
+                                            <groupId>org.elasticsearch.distribution.rpm</groupId>
+                                            <artifactId>elasticsearch</artifactId>
+                                            <version>${elasticsearch.version}</version>
+                                            <type>rpm</type>
+                                        </artifactItem>
+                                    </artifactItems>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+            <properties>
+                <proposedBoxesToTest>${debBoxes}, ${rpmBoxes}</proposedBoxesToTest>
+            </properties>
+        </profile>
+        <profile>
+            <!-- Only set boxesToTest if it hasn't been set on the command
+                 line. -->
+            <id>set-boxes-to-test</id>
+            <activation>
+                <property>
+                    <name>!boxesToTest</name>
+                </property>
+            </activation>
+            <properties>
+                <boxesToTest>${proposedBoxesToTest}</boxesToTest>
+            </properties>
+        </profile>
+
+        <!-- This profile manipulates what is run. -->
+        <profile>
+            <!-- Smoke tests the VMs but doesn't actually run the bats tests -->
+            <id>smoke-vms</id>
+            <properties>
+              <testCommand>echo skipping tests</testCommand>
+            </properties>
+        </profile>
+    </profiles>
+</project>

qa/vagrant/src/dev/ant/vagrant-integration-tests.xml

@@ -0,0 +1,74 @@
+<?xml version="1.0"?>
+<project name="elasticsearch-integration-tests">
+  <target name="vagrant-test-all-boxes">
+    <foreach list="${boxesToTest}" trim="true" param="box"
+      target="vagrant-test" inheritall="true" inheritrefs="true"/>
+  </target>
+
+  <target name="vagrant-test" depends="vagrant-up">
+    <trycatch>
+      <try>
+        <exec executable="vagrant" failonerror="true">
+          <arg value="ssh"/>
+          <arg value="${box}"/>
+          <arg value="--command"/>
+          <arg value="
+            set -o pipefail;
+            cd $TESTROOT;
+            ${testCommand} | sed -ue 's/^/${box}: /'
+          "/>
+        </exec>
+      </try>
+      <finally>
+        <exec executable="vagrant" failonerror="true">
+          <arg value="halt"/>
+          <arg value="${box}"/>
+        </exec>
+      </finally>
+    </trycatch>
+  </target>
+
+  <target name="vagrant-up">
+    <exec executable="vagrant" failonerror="true">
+      <arg value="up"/>
+      <arg value="${box}"/>
+      <!-- Its important that we try to reprovision the box even if it already
+        exists. That way updates to the vagrant configuration take automatically.
+        That isn't to say that the updates will always be compatible. Its ok to
+        just destroy the boxes if they get busted. -->
+      <arg value="--provision"/>
+    </exec>
+  </target>
+
+  <target name="check-vagrant-version">
+    <check-version executable="vagrant" ok="^1\.[789]\..+$"
+        message="Only known to work with Vagrant 1.7+"/>
+  </target>
+
+  <macrodef name="check-version">
+    <attribute name="executable" description="The executable to check."/>
+    <attribute name="rewrite" default="(?:\S*\s)*(.+)"
+        description="Regex extracting the version from the output of the executable. Defaults to everything after the last space."/>
+    <attribute name="ok" description="The regex to check the version against."/>
+    <attribute name="message" description="The message to report on failure."/>
+    <sequential>
+      <exec executable="@{executable}" failonerror="true"
+          outputproperty="versionOutput">
+        <arg value="--version" />
+      </exec>
+      <propertyregex property="version" input="${versionOutput}"
+          regexp="@{rewrite}" select="\1" />
+      <echo message="The @{executable} version is ${version}"/>
+      <fail message="@{message}">
+        <condition>
+          <not>
+            <!-- Very simple version checking.... -->
+            <matches string="${version}" pattern="@{ok}"/>
+          </not>
+        </condition>
+      </fail>
+    </sequential>
+  </macrodef>
+
+
+</project>

distribution/src/test/resources/packaging/scripts/25_tar_plugins.bats → qa/vagrant/src/test/resources/packaging/scripts/25_tar_plugins.bats

@@ -50,6 +50,7 @@ setup() {
 # Install plugins with a tar archive
 ##################################
 @test "[TAR] install shield plugin" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the archive
     install_archive
@@ -90,6 +91,7 @@ setup() {
 }
 
 @test "[TAR] install shield plugin with a custom path.plugins" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the archive
     install_archive
@@ -143,6 +145,7 @@ setup() {
 }
 
 @test "[TAR] install shield plugin with a custom CONFIG_DIR" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the archive
     install_archive
@@ -199,6 +202,7 @@ setup() {
 }
 
 @test "[TAR] install shield plugin with a custom ES_JAVA_OPTS" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the archive
     install_archive
@@ -259,6 +263,8 @@ setup() {
 }
 
 @test "[TAR] install shield plugin to elasticsearch directory with a space" {
+    skip "awaits public release of shield for 2.0"
+
     export ES_DIR="/tmp/elastic search"
 
     # Install the archive
@@ -307,6 +313,7 @@ setup() {
 }
 
 @test "[TAR] install shield plugin from a directory with a space" {
+    skip "awaits public release of shield for 2.0"
 
     export SHIELD_ZIP_WITH_SPACE="/tmp/plugins with space/shield.zip"
 

distribution/src/test/resources/packaging/scripts/50_plugins.bats → qa/vagrant/src/test/resources/packaging/scripts/50_plugins.bats

@@ -63,6 +63,7 @@ install_package() {
 # Install plugins with DEB/RPM package
 ##################################
 @test "[PLUGINS] install shield plugin" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the package
     install_package
@@ -103,6 +104,7 @@ install_package() {
 }
 
 @test "[PLUGINS] install shield plugin with a custom path.plugins" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the package
     install_package
@@ -160,6 +162,7 @@ install_package() {
 }
 
 @test "[PLUGINS] install shield plugin with a custom CONFIG_DIR" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the package
     install_package
@@ -227,6 +230,7 @@ install_package() {
 }
 
 @test "[PLUGINS] install shield plugin with a custom ES_JAVA_OPTS" {
+    skip "awaits public release of shield for 2.0"
 
     # Install the package
     install_package

distribution/src/test/resources/packaging/scripts/70_sysv_initd.bats → qa/vagrant/src/test/resources/packaging/scripts/70_sysv_initd.bats

@@ -97,7 +97,8 @@ setup() {
     skip_not_sysvinit
 
     run service elasticsearch status
-    [ "$status" -eq 3 ]
+    # precise returns 4, trusty 3
+    [ "$status" -eq 3 ] || [ "$status" -eq 4 ]
 }
 
 # Simulates the behavior of a system restart:
@@ -120,4 +121,4 @@ setup() {
 
     run service elasticsearch stop
     [ "$status" -eq 0 ]
-}
+}