Prevent in-place downgrades and invalid upgrades (#41731)

Downgrading an Elasticsearch node to an earlier version is unsupported, because
we do not make any attempt to guarantee that a node can read any of the on-disk
data written by a future version. Yet today we do not actively prevent
downgrades, and sometimes users will attempt to roll back a failed upgrade with
an in-place downgrade and get into an unrecoverable state.

This change adds the current version of the node to the node metadata file, and
checks the version found in this file against the current version at startup.
If the node cannot be sure of its ability to read the on-disk data then it
refuses to start, preserving any on-disk data in its upgraded state.

This change also adds a command-line tool to overwrite the node metadata file
without performing any version checks, to unsafely bypass these checks and
recover the historical and lenient behaviour.

docs/reference/commands/node-tool.asciidoc

@@ -4,14 +4,15 @@
 The `elasticsearch-node` command enables you to perform certain unsafe
 operations on a node that are only possible while it is shut down. This command
 allows you to adjust the <<modules-node,role>> of a node and may be able to
-recover some data after a disaster.
+recover some data after a disaster or start a node even if it is incompatible
+with the data on disk.
 
 [float]
 === Synopsis
 
 [source,shell]
 --------------------------------------------------
-bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster
+bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster|override-version
   [--ordinal <Integer>] [-E <KeyValuePair>]
   [-h, --help] ([-s, --silent] | [-v, --verbose])
 --------------------------------------------------
@@ -19,7 +20,7 @@ bin/elasticsearch-node repurpose|unsafe-bootstrap|detach-cluster
 [float]
 === Description
 
-This tool has three modes:
+This tool has four modes:
 
 * `elasticsearch-node repurpose` can be used to delete unwanted data from a
   node if it used to be a <<data-node,data node>> or a
@@ -36,6 +37,11 @@ This tool has three modes:
   cluster bootstrapping was not possible, it also enables you to move nodes
   into a brand-new cluster.
 
+* `elasticsearch-node override-version` enables you to start up a node
+  even if the data in the data path was written by an incompatible version of
+  {es}. This may sometimes allow you to downgrade to an earlier version of
+  {es}.
+
 [[node-tool-repurpose]]
 [float]
 ==== Changing the role of a node
@@ -109,6 +115,25 @@ way forward that does not risk data loss, but it may be possible to use the
 `elasticsearch-node` tool to construct a new cluster that contains some of the
 data from the failed cluster.
 
+[[node-tool-override-version]]
+[float]
+==== Bypassing version checks
+
+The data that {es} writes to disk is designed to be read by the current version
+and a limited set of future versions. It cannot generally be read by older
+versions, nor by versions that are more than one major version newer. The data
+stored on disk includes the version of the node that wrote it, and {es} checks
+that it is compatible with this version when starting up.
+
+In rare circumstances it may be desirable to bypass this check and start up an
+{es} node using data that was written by an incompatible version. This may not
+work if the format of the stored data has changed, and it is a risky process
+because it is possible for the format to change in ways that {es} may
+misinterpret, silently leading to data loss.
+
+To bypass this check, you can use the `elasticsearch-node override-version`
+tool to overwrite the version number stored in the data path with the current
+version, causing {es} to believe that it is compatible with the on-disk data.
 
 [[node-tool-unsafe-bootstrap]]
 [float]
@@ -262,6 +287,9 @@ one-node cluster.
 `detach-cluster`:: Specifies to unsafely detach this node from its cluster so
 it can join a different cluster.
 
+`override-version`:: Overwrites the version number stored in the data path so
+that a node can start despite being incompatible with the on-disk data.
+
 `--ordinal <Integer>`:: If there is <<max-local-storage-nodes,more than one
 node sharing a data path>> then this specifies which node to target. Defaults
 to `0`, meaning to use the first node in the data path.
@@ -423,3 +451,32 @@ Do you want to proceed?
 Confirm [y/N] y
 Node was successfully detached from the cluster
 ----
+
+[float]
+==== Bypassing version checks
+
+Run the `elasticsearch-node override-version` command to overwrite the version
+stored in the data path so that a node can start despite being incompatible
+with the data stored in the data path:
+
+[source, txt]
+----
+node$ ./bin/elasticsearch-node override-version
+
+    WARNING: Elasticsearch MUST be stopped before running this tool.
+
+This data path was last written by Elasticsearch version [x.x.x] and may no
+longer be compatible with Elasticsearch version [y.y.y]. This tool will bypass
+this compatibility check, allowing a version [y.y.y] node to start on this data
+path, but a version [y.y.y] node may not be able to read this data or may read
+it incorrectly leading to data loss.
+
+You should not use this tool. Instead, continue to use a version [x.x.x] node
+on this data path. If necessary, you can use reindex-from-remote to copy the
+data from here into an older cluster.
+
+Do you want to proceed?
+
+Confirm [y/N] y
+Successfully overwrote this node's metadata to bypass its version compatibility checks.
+----

server/src/main/java/org/elasticsearch/cluster/coordination/ElasticsearchNodeCommand.java

@@ -44,7 +44,7 @@ import java.util.Objects;
 public abstract class ElasticsearchNodeCommand extends EnvironmentAwareCommand {
     private static final Logger logger = LogManager.getLogger(ElasticsearchNodeCommand.class);
     protected final NamedXContentRegistry namedXContentRegistry;
-    static final String DELIMITER = "------------------------------------------------------------------------\n";
+    protected static final String DELIMITER = "------------------------------------------------------------------------\n";
 
     static final String STOP_WARNING_MSG =
             DELIMITER +
@@ -81,9 +81,8 @@ public abstract class ElasticsearchNodeCommand extends EnvironmentAwareCommand {
                 throw new ElasticsearchException(NO_NODE_FOLDER_FOUND_MSG);
             }
             processNodePaths(terminal, dataPaths, env);
-        } catch (LockObtainFailedException ex) {
-            throw new ElasticsearchException(
-                    FAILED_TO_OBTAIN_NODE_LOCK_MSG + " [" + ex.getMessage() + "]");
+        } catch (LockObtainFailedException e) {
+            throw new ElasticsearchException(FAILED_TO_OBTAIN_NODE_LOCK_MSG, e);
         }
     }
 
@@ -177,6 +176,17 @@ public abstract class ElasticsearchNodeCommand extends EnvironmentAwareCommand {
         MetaData.FORMAT.cleanupOldFiles(newGeneration, dataPaths);
     }
 
+    protected NodeEnvironment.NodePath[] toNodePaths(Path[] dataPaths) {
+        return Arrays.stream(dataPaths).map(ElasticsearchNodeCommand::createNodePath).toArray(NodeEnvironment.NodePath[]::new);
+    }
+
+    private static NodeEnvironment.NodePath createNodePath(Path path) {
+        try {
+            return new NodeEnvironment.NodePath(path);
+        } catch (IOException e) {
+            throw new ElasticsearchException("Unable to investigate path [" + path + "]", e);
+        }
+    }
 
     //package-private for testing
     OptionParser getParser() {

server/src/main/java/org/elasticsearch/cluster/coordination/NodeToolCli.java

@@ -22,6 +22,7 @@ import org.elasticsearch.cli.CommandLoggingConfigurator;
 import org.elasticsearch.cli.MultiCommand;
 import org.elasticsearch.cli.Terminal;
 import org.elasticsearch.env.NodeRepurposeCommand;
+import org.elasticsearch.env.OverrideNodeVersionCommand;
 
 // NodeToolCli does not extend LoggingAwareCommand, because LoggingAwareCommand performs logging initialization
 // after LoggingAwareCommand instance is constructed.
@@ -39,6 +40,7 @@ public class NodeToolCli extends MultiCommand {
         subcommands.put("repurpose", new NodeRepurposeCommand());
         subcommands.put("unsafe-bootstrap", new UnsafeBootstrapMasterCommand());
         subcommands.put("detach-cluster", new DetachClusterCommand());
+        subcommands.put("override-version", new OverrideNodeVersionCommand());
     }
 
     public static void main(String[] args) throws Exception {

server/src/main/java/org/elasticsearch/env/NodeEnvironment.java

@@ -31,6 +31,7 @@ import org.apache.lucene.store.LockObtainFailedException;
 import org.apache.lucene.store.NativeFSLockFactory;
 import org.apache.lucene.store.SimpleFSDirectory;
 import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.Version;
 import org.elasticsearch.cluster.metadata.IndexMetaData;
 import org.elasticsearch.cluster.node.DiscoveryNode;
 import org.elasticsearch.common.CheckedFunction;
@@ -248,7 +249,7 @@ public final class NodeEnvironment  implements Closeable {
             sharedDataPath = null;
             locks = null;
             nodeLockId = -1;
-            nodeMetaData = new NodeMetaData(generateNodeId(settings));
+            nodeMetaData = new NodeMetaData(generateNodeId(settings), Version.CURRENT);
             return;
         }
         boolean success = false;
@@ -393,7 +394,6 @@ public final class NodeEnvironment  implements Closeable {
         logger.info("heap size [{}], compressed ordinary object pointers [{}]", maxHeapSize, useCompressedOops);
     }
 
-
     /**
      * scans the node paths and loads existing metaData file. If not found a new meta data will be generated
      * and persisted into the nodePaths
@@ -403,10 +403,15 @@ public final class NodeEnvironment  implements Closeable {
         final Path[] paths = Arrays.stream(nodePaths).map(np -> np.path).toArray(Path[]::new);
         NodeMetaData metaData = NodeMetaData.FORMAT.loadLatestState(logger, NamedXContentRegistry.EMPTY, paths);
         if (metaData == null) {
-            metaData = new NodeMetaData(generateNodeId(settings));
+            metaData = new NodeMetaData(generateNodeId(settings), Version.CURRENT);
+        } else {
+            metaData = metaData.upgradeToCurrentVersion();
         }
+
         // we write again to make sure all paths have the latest state file
+        assert metaData.nodeVersion().equals(Version.CURRENT) : metaData.nodeVersion() + " != " + Version.CURRENT;
         NodeMetaData.FORMAT.writeAndCleanup(metaData, paths);
+
         return metaData;
     }
 

server/src/main/java/org/elasticsearch/env/NodeMetaData.java

@@ -19,6 +19,7 @@
 
 package org.elasticsearch.env;
 
+import org.elasticsearch.Version;
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.xcontent.ObjectParser;
 import org.elasticsearch.common.xcontent.XContentBuilder;
@@ -31,66 +32,104 @@ import java.io.OutputStream;
 import java.util.Objects;
 
 /**
- * Metadata associated with this node. Currently only contains the unique uuid describing this node.
+ * Metadata associated with this node: its persistent node ID and its version.
  * The metadata is persisted in the data folder of this node and is reused across restarts.
  */
 public final class NodeMetaData {
 
     private static final String NODE_ID_KEY = "node_id";
+    private static final String NODE_VERSION_KEY = "node_version";
 
     private final String nodeId;
 
-    public NodeMetaData(final String nodeId) {
+    private final Version nodeVersion;
+
+    public NodeMetaData(final String nodeId, final Version nodeVersion) {
         this.nodeId = Objects.requireNonNull(nodeId);
+        this.nodeVersion = Objects.requireNonNull(nodeVersion);
     }
 
     @Override
     public boolean equals(Object o) {
-        if (this == o) {
-            return true;
-        }
-        if (o == null || getClass() != o.getClass()) {
-            return false;
-        }
-
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
         NodeMetaData that = (NodeMetaData) o;
-
-        return Objects.equals(this.nodeId, that.nodeId);
+        return nodeId.equals(that.nodeId) &&
+            nodeVersion.equals(that.nodeVersion);
     }
 
     @Override
     public int hashCode() {
-        return this.nodeId.hashCode();
+        return Objects.hash(nodeId, nodeVersion);
     }
 
     @Override
     public String toString() {
-        return "node_id [" + nodeId + "]";
+        return "NodeMetaData{" +
+            "nodeId='" + nodeId + '\'' +
+            ", nodeVersion=" + nodeVersion +
+            '}';
     }
 
     private static ObjectParser<Builder, Void> PARSER = new ObjectParser<>("node_meta_data", Builder::new);
 
     static {
         PARSER.declareString(Builder::setNodeId, new ParseField(NODE_ID_KEY));
+        PARSER.declareInt(Builder::setNodeVersionId, new ParseField(NODE_VERSION_KEY));
     }
 
     public String nodeId() {
         return nodeId;
     }
 
+    public Version nodeVersion() {
+        return nodeVersion;
+    }
+
+    public NodeMetaData upgradeToCurrentVersion() {
+        if (nodeVersion.equals(Version.V_EMPTY)) {
+            assert Version.CURRENT.major <= Version.V_7_0_0.major + 1 : "version is required in the node metadata from v9 onwards";
+            return new NodeMetaData(nodeId, Version.CURRENT);
+        }
+
+        if (nodeVersion.before(Version.CURRENT.minimumIndexCompatibilityVersion())) {
+            throw new IllegalStateException(
+                "cannot upgrade a node from version [" + nodeVersion + "] directly to version [" + Version.CURRENT + "]");
+        }
+
+        if (nodeVersion.after(Version.CURRENT)) {
+            throw new IllegalStateException(
+                "cannot downgrade a node from version [" + nodeVersion + "] to version [" + Version.CURRENT + "]");
+        }
+
+        return nodeVersion.equals(Version.CURRENT) ? this : new NodeMetaData(nodeId, Version.CURRENT);
+    }
+
     private static class Builder {
         String nodeId;
+        Version nodeVersion;
 
         public void setNodeId(String nodeId) {
             this.nodeId = nodeId;
         }
 
+        public void setNodeVersionId(int nodeVersionId) {
+            this.nodeVersion = Version.fromId(nodeVersionId);
+        }
+
         public NodeMetaData build() {
-            return new NodeMetaData(nodeId);
+            final Version nodeVersion;
+            if (this.nodeVersion == null) {
+                assert Version.CURRENT.major <= Version.V_7_0_0.major + 1 : "version is required in the node metadata from v9 onwards";
+                nodeVersion = Version.V_EMPTY;
+            } else {
+                nodeVersion = this.nodeVersion;
+            }
+
+            return new NodeMetaData(nodeId, nodeVersion);
         }
     }
 
-
     public static final MetaDataStateFormat<NodeMetaData> FORMAT = new MetaDataStateFormat<NodeMetaData>("node-") {
 
         @Override
@@ -103,10 +142,11 @@ public final class NodeMetaData {
         @Override
         public void toXContent(XContentBuilder builder, NodeMetaData nodeMetaData) throws IOException {
             builder.field(NODE_ID_KEY, nodeMetaData.nodeId);
+            builder.field(NODE_VERSION_KEY, nodeMetaData.nodeVersion.id);
         }
 
         @Override
-        public NodeMetaData fromXContent(XContentParser parser) throws IOException {
+        public NodeMetaData fromXContent(XContentParser parser) {
             return PARSER.apply(parser, null).build();
         }
     };

server/src/main/java/org/elasticsearch/env/NodeRepurposeCommand.java

@@ -172,10 +172,6 @@ public class NodeRepurposeCommand extends ElasticsearchNodeCommand {
         }
     }
 
-    private NodeEnvironment.NodePath[] toNodePaths(Path[] dataPaths) {
-        return Arrays.stream(dataPaths).map(NodeRepurposeCommand::createNodePath).toArray(NodeEnvironment.NodePath[]::new);
-    }
-
     private Set<String> indexUUIDsFor(Set<Path> indexPaths) {
         return indexPaths.stream().map(Path::getFileName).map(Path::toString).collect(Collectors.toSet());
     }
@@ -221,19 +217,11 @@ public class NodeRepurposeCommand extends ElasticsearchNodeCommand {
 
     @SafeVarargs
     @SuppressWarnings("varargs")
-    private final Set<Path> uniqueParentPaths(Collection<Path>... paths) {
+    private Set<Path> uniqueParentPaths(Collection<Path>... paths) {
         // equals on Path is good enough here due to the way these are collected.
         return Arrays.stream(paths).flatMap(Collection::stream).map(Path::getParent).collect(Collectors.toSet());
     }
 
-    private static NodeEnvironment.NodePath createNodePath(Path path) {
-        try {
-            return new NodeEnvironment.NodePath(path);
-        } catch (IOException e) {
-            throw new ElasticsearchException("Unable to investigate path: " + path + ": " + e.getMessage());
-        }
-    }
-
     //package-private for testing
     OptionParser getParser() {
         return parser;

server/src/main/java/org/elasticsearch/env/OverrideNodeVersionCommand.java

@@ -0,0 +1,103 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.env;
+
+import joptsimple.OptionParser;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.Version;
+import org.elasticsearch.cli.Terminal;
+import org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Arrays;
+
+public class OverrideNodeVersionCommand extends ElasticsearchNodeCommand {
+    private static final Logger logger = LogManager.getLogger(OverrideNodeVersionCommand.class);
+
+    private static final String TOO_NEW_MESSAGE =
+        DELIMITER +
+            "\n" +
+            "This data path was last written by Elasticsearch version [V_NEW] and may no\n" +
+            "longer be compatible with Elasticsearch version [V_CUR]. This tool will bypass\n" +
+            "this compatibility check, allowing a version [V_CUR] node to start on this data\n" +
+            "path, but a version [V_CUR] node may not be able to read this data or may read\n" +
+            "it incorrectly leading to data loss.\n" +
+            "\n" +
+            "You should not use this tool. Instead, continue to use a version [V_NEW] node\n" +
+            "on this data path. If necessary, you can use reindex-from-remote to copy the\n" +
+            "data from here into an older cluster.\n" +
+            "\n" +
+            "Do you want to proceed?\n";
+
+    private static final String TOO_OLD_MESSAGE =
+        DELIMITER +
+            "\n" +
+            "This data path was last written by Elasticsearch version [V_OLD] which may be\n" +
+            "too old to be readable by Elasticsearch version [V_CUR].  This tool will bypass\n" +
+            "this compatibility check, allowing a version [V_CUR] node to start on this data\n" +
+            "path, but this version [V_CUR] node may not be able to read this data or may\n" +
+            "read it incorrectly leading to data loss.\n" +
+            "\n" +
+            "You should not use this tool. Instead, upgrade this data path from [V_OLD] to\n" +
+            "[V_CUR] using one or more intermediate versions of Elasticsearch.\n" +
+            "\n" +
+            "Do you want to proceed?\n";
+
+    static final String NO_METADATA_MESSAGE = "no node metadata found, so there is no version to override";
+    static final String SUCCESS_MESSAGE = "Successfully overwrote this node's metadata to bypass its version compatibility checks.";
+
+    public OverrideNodeVersionCommand() {
+        super("Overwrite the version stored in this node's data path with [" + Version.CURRENT +
+            "] to bypass the version compatibility checks");
+    }
+
+    @Override
+    protected void processNodePaths(Terminal terminal, Path[] dataPaths, Environment env) throws IOException {
+        final Path[] nodePaths = Arrays.stream(toNodePaths(dataPaths)).map(p -> p.path).toArray(Path[]::new);
+        final NodeMetaData nodeMetaData = NodeMetaData.FORMAT.loadLatestState(logger, namedXContentRegistry, nodePaths);
+        if (nodeMetaData == null) {
+            throw new ElasticsearchException(NO_METADATA_MESSAGE);
+        }
+
+        try {
+            nodeMetaData.upgradeToCurrentVersion();
+            throw new ElasticsearchException("found [" + nodeMetaData + "] which is compatible with current version [" + Version.CURRENT
+                + "], so there is no need to override the version checks");
+        } catch (IllegalStateException e) {
+            // ok, means the version change is not supported
+        }
+
+        confirm(terminal, (nodeMetaData.nodeVersion().before(Version.CURRENT) ? TOO_OLD_MESSAGE : TOO_NEW_MESSAGE)
+            .replace("V_OLD", nodeMetaData.nodeVersion().toString())
+            .replace("V_NEW", nodeMetaData.nodeVersion().toString())
+            .replace("V_CUR", Version.CURRENT.toString()));
+
+        NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(nodeMetaData.nodeId(), Version.CURRENT), nodePaths);
+
+        terminal.println(SUCCESS_MESSAGE);
+    }
+
+    //package-private for testing
+    OptionParser getParser() {
+        return parser;
+    }
+}

server/src/main/java/org/elasticsearch/gateway/MetaDataStateFormat.java

@@ -382,7 +382,7 @@ public abstract class MetaDataStateFormat<T> {
         return files;
     }
 
-    private String getStateFileName(long generation) {
+    public String getStateFileName(long generation) {
         return prefix + generation + STATE_FILE_EXTENSION;
     }
 
@@ -466,7 +466,7 @@ public abstract class MetaDataStateFormat<T> {
         IOUtils.rm(stateDirectories);
     }
 
-    String getPrefix() {
+    public String getPrefix() {
         return prefix;
     }
 }

server/src/test/java/org/elasticsearch/env/NodeEnvironmentIT.java

@@ -19,12 +19,18 @@
 
 package org.elasticsearch.env;
 
+import org.elasticsearch.Version;
+import org.elasticsearch.common.CheckedConsumer;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.node.Node;
 import org.elasticsearch.test.ESIntegTestCase;
 import org.elasticsearch.test.InternalTestCluster;
 
+import java.nio.file.Path;
+
+import static org.hamcrest.Matchers.allOf;
 import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.endsWith;
 import static org.hamcrest.Matchers.startsWith;
 
 @ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 0)
@@ -86,4 +92,35 @@ public class NodeEnvironmentIT extends ESIntegTestCase {
                 + Node.NODE_DATA_SETTING.getKey()
                 + "=false, but has shard data"));
     }
+
+    private IllegalStateException expectThrowsOnRestart(CheckedConsumer<Path[], Exception> onNodeStopped) {
+        internalCluster().startNode();
+        final Path[] dataPaths = internalCluster().getInstance(NodeEnvironment.class).nodeDataPaths();
+        return expectThrows(IllegalStateException.class,
+            () -> internalCluster().restartRandomDataNode(new InternalTestCluster.RestartCallback() {
+                @Override
+                public Settings onNodeStopped(String nodeName) {
+                    try {
+                        onNodeStopped.accept(dataPaths);
+                    } catch (Exception e) {
+                        throw new AssertionError(e);
+                    }
+                    return Settings.EMPTY;
+                }
+            }));
+    }
+
+    public void testFailsToStartIfDowngraded() {
+        final IllegalStateException illegalStateException = expectThrowsOnRestart(dataPaths ->
+            NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(randomAlphaOfLength(10), NodeMetaDataTests.tooNewVersion()), dataPaths));
+        assertThat(illegalStateException.getMessage(),
+            allOf(startsWith("cannot downgrade a node from version ["), endsWith("] to version [" + Version.CURRENT + "]")));
+    }
+
+    public void testFailsToStartIfUpgradedTooFar() {
+        final IllegalStateException illegalStateException = expectThrowsOnRestart(dataPaths ->
+            NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(randomAlphaOfLength(10), NodeMetaDataTests.tooOldVersion()), dataPaths));
+        assertThat(illegalStateException.getMessage(),
+            allOf(startsWith("cannot upgrade a node from version ["), endsWith("] directly to version [" + Version.CURRENT + "]")));
+    }
 }

server/src/test/java/org/elasticsearch/env/NodeMetaDataTests.java

@@ -0,0 +1,118 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.env;
+
+import org.elasticsearch.Version;
+import org.elasticsearch.common.collect.Tuple;
+import org.elasticsearch.gateway.MetaDataStateFormat;
+import org.elasticsearch.test.ESTestCase;
+import org.elasticsearch.test.EqualsHashCodeTestUtils;
+import org.elasticsearch.test.VersionUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import static org.hamcrest.Matchers.allOf;
+import static org.hamcrest.Matchers.endsWith;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.hamcrest.Matchers.startsWith;
+
+public class NodeMetaDataTests extends ESTestCase {
+    private Version randomVersion() {
+        // VersionUtils.randomVersion() only returns known versions, which are necessarily no later than Version.CURRENT; however we want
+        // also to consider our behaviour with all versions, so occasionally pick up a truly random version.
+        return rarely() ? Version.fromId(randomInt()) : VersionUtils.randomVersion(random());
+    }
+
+    public void testEqualsHashcodeSerialization() {
+        final Path tempDir = createTempDir();
+        EqualsHashCodeTestUtils.checkEqualsAndHashCode(new NodeMetaData(randomAlphaOfLength(10), randomVersion()),
+            nodeMetaData -> {
+                final long generation = NodeMetaData.FORMAT.writeAndCleanup(nodeMetaData, tempDir);
+                final Tuple<NodeMetaData, Long> nodeMetaDataLongTuple
+                    = NodeMetaData.FORMAT.loadLatestStateWithGeneration(logger, xContentRegistry(), tempDir);
+                assertThat(nodeMetaDataLongTuple.v2(), equalTo(generation));
+                return nodeMetaDataLongTuple.v1();
+            }, nodeMetaData -> {
+                if (randomBoolean()) {
+                    return new NodeMetaData(randomAlphaOfLength(21 - nodeMetaData.nodeId().length()), nodeMetaData.nodeVersion());
+                } else {
+                    return new NodeMetaData(nodeMetaData.nodeId(), randomValueOtherThan(nodeMetaData.nodeVersion(), this::randomVersion));
+                }
+            });
+    }
+
+    public void testReadsFormatWithoutVersion() throws IOException {
+        // the behaviour tested here is only appropriate if the current version is compatible with versions 7 and earlier
+        assertTrue(Version.CURRENT.minimumIndexCompatibilityVersion().onOrBefore(Version.V_7_0_0));
+        // when the current version is incompatible with version 7, the behaviour should change to reject files like the given resource
+        // which do not have the version field
+
+        final Path tempDir = createTempDir();
+        final Path stateDir = Files.createDirectory(tempDir.resolve(MetaDataStateFormat.STATE_DIR_NAME));
+        final InputStream resource = this.getClass().getResourceAsStream("testReadsFormatWithoutVersion.binary");
+        assertThat(resource, notNullValue());
+        Files.copy(resource, stateDir.resolve(NodeMetaData.FORMAT.getStateFileName(between(0, Integer.MAX_VALUE))));
+        final NodeMetaData nodeMetaData = NodeMetaData.FORMAT.loadLatestState(logger, xContentRegistry(), tempDir);
+        assertThat(nodeMetaData.nodeId(), equalTo("y6VUVMSaStO4Tz-B5BxcOw"));
+        assertThat(nodeMetaData.nodeVersion(), equalTo(Version.V_EMPTY));
+    }
+
+    public void testUpgradesLegitimateVersions() {
+        final String nodeId = randomAlphaOfLength(10);
+        final NodeMetaData nodeMetaData = new NodeMetaData(nodeId,
+            randomValueOtherThanMany(v -> v.after(Version.CURRENT) || v.before(Version.CURRENT.minimumIndexCompatibilityVersion()),
+                this::randomVersion)).upgradeToCurrentVersion();
+        assertThat(nodeMetaData.nodeVersion(), equalTo(Version.CURRENT));
+        assertThat(nodeMetaData.nodeId(), equalTo(nodeId));
+    }
+
+    public void testUpgradesMissingVersion() {
+        final String nodeId = randomAlphaOfLength(10);
+        final NodeMetaData nodeMetaData = new NodeMetaData(nodeId, Version.V_EMPTY).upgradeToCurrentVersion();
+        assertThat(nodeMetaData.nodeVersion(), equalTo(Version.CURRENT));
+        assertThat(nodeMetaData.nodeId(), equalTo(nodeId));
+    }
+
+    public void testDoesNotUpgradeFutureVersion() {
+        final IllegalStateException illegalStateException = expectThrows(IllegalStateException.class,
+            () -> new NodeMetaData(randomAlphaOfLength(10), tooNewVersion())
+                .upgradeToCurrentVersion());
+        assertThat(illegalStateException.getMessage(),
+            allOf(startsWith("cannot downgrade a node from version ["), endsWith("] to version [" + Version.CURRENT + "]")));
+    }
+
+    public void testDoesNotUpgradeAncientVersion() {
+        final IllegalStateException illegalStateException = expectThrows(IllegalStateException.class,
+            () -> new NodeMetaData(randomAlphaOfLength(10), tooOldVersion()).upgradeToCurrentVersion());
+        assertThat(illegalStateException.getMessage(),
+            allOf(startsWith("cannot upgrade a node from version ["), endsWith("] directly to version [" + Version.CURRENT + "]")));
+    }
+
+    public static Version tooNewVersion() {
+        return Version.fromId(between(Version.CURRENT.id + 1, 99999999));
+    }
+
+    public static Version tooOldVersion() {
+        return Version.fromId(between(1, Version.CURRENT.minimumIndexCompatibilityVersion().id - 1));
+    }
+}

server/src/test/java/org/elasticsearch/env/OverrideNodeVersionCommandTests.java

@@ -0,0 +1,155 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.env;
+
+import org.elasticsearch.ElasticsearchException;
+import org.elasticsearch.Version;
+import org.elasticsearch.cli.MockTerminal;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.gateway.WriteStateException;
+import org.elasticsearch.test.ESTestCase;
+import org.junit.Before;
+
+import java.io.IOException;
+import java.nio.file.Path;
+
+import static org.hamcrest.Matchers.allOf;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
+
+public class OverrideNodeVersionCommandTests extends ESTestCase {
+
+    private Environment environment;
+    private Path[] nodePaths;
+
+    @Before
+    public void createNodePaths() throws IOException {
+        final Settings settings = buildEnvSettings(Settings.EMPTY);
+        environment = TestEnvironment.newEnvironment(settings);
+        try (NodeEnvironment nodeEnvironment = new NodeEnvironment(settings, environment)) {
+            nodePaths = nodeEnvironment.nodeDataPaths();
+        }
+    }
+
+    public void testFailsOnEmptyPath() {
+        final Path emptyPath = createTempDir();
+        final MockTerminal mockTerminal = new MockTerminal();
+        final ElasticsearchException elasticsearchException = expectThrows(ElasticsearchException.class, () ->
+            new OverrideNodeVersionCommand().processNodePaths(mockTerminal, new Path[]{emptyPath}, environment));
+        assertThat(elasticsearchException.getMessage(), equalTo(OverrideNodeVersionCommand.NO_METADATA_MESSAGE));
+        expectThrows(IllegalStateException.class, () -> mockTerminal.readText(""));
+    }
+
+    public void testFailsIfUnnecessary() throws WriteStateException {
+        final Version nodeVersion = Version.fromId(between(Version.CURRENT.minimumIndexCompatibilityVersion().id, Version.CURRENT.id));
+        NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(randomAlphaOfLength(10), nodeVersion), nodePaths);
+        final MockTerminal mockTerminal = new MockTerminal();
+        final ElasticsearchException elasticsearchException = expectThrows(ElasticsearchException.class, () ->
+            new OverrideNodeVersionCommand().processNodePaths(mockTerminal, nodePaths, environment));
+        assertThat(elasticsearchException.getMessage(), allOf(
+            containsString("compatible with current version"),
+            containsString(Version.CURRENT.toString()),
+            containsString(nodeVersion.toString())));
+        expectThrows(IllegalStateException.class, () -> mockTerminal.readText(""));
+    }
+
+    public void testWarnsIfTooOld() throws Exception {
+        final String nodeId = randomAlphaOfLength(10);
+        final Version nodeVersion = NodeMetaDataTests.tooOldVersion();
+        NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(nodeId, nodeVersion), nodePaths);
+        final MockTerminal mockTerminal = new MockTerminal();
+        mockTerminal.addTextInput("n\n");
+        final ElasticsearchException elasticsearchException = expectThrows(ElasticsearchException.class, () ->
+            new OverrideNodeVersionCommand().processNodePaths(mockTerminal, nodePaths, environment));
+        assertThat(elasticsearchException.getMessage(), equalTo("aborted by user"));
+        assertThat(mockTerminal.getOutput(), allOf(
+            containsString("too old"),
+            containsString("data loss"),
+            containsString("You should not use this tool"),
+            containsString(Version.CURRENT.toString()),
+            containsString(nodeVersion.toString())));
+        expectThrows(IllegalStateException.class, () -> mockTerminal.readText(""));
+
+        final NodeMetaData nodeMetaData = NodeMetaData.FORMAT.loadLatestState(logger, xContentRegistry(), nodePaths);
+        assertThat(nodeMetaData.nodeId(), equalTo(nodeId));
+        assertThat(nodeMetaData.nodeVersion(), equalTo(nodeVersion));
+    }
+
+    public void testWarnsIfTooNew() throws Exception {
+        final String nodeId = randomAlphaOfLength(10);
+        final Version nodeVersion = NodeMetaDataTests.tooNewVersion();
+        NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(nodeId, nodeVersion), nodePaths);
+        final MockTerminal mockTerminal = new MockTerminal();
+        mockTerminal.addTextInput(randomFrom("yy", "Yy", "n", "yes", "true", "N", "no"));
+        final ElasticsearchException elasticsearchException = expectThrows(ElasticsearchException.class, () ->
+            new OverrideNodeVersionCommand().processNodePaths(mockTerminal, nodePaths, environment));
+        assertThat(elasticsearchException.getMessage(), equalTo("aborted by user"));
+        assertThat(mockTerminal.getOutput(), allOf(
+            containsString("data loss"),
+            containsString("You should not use this tool"),
+            containsString(Version.CURRENT.toString()),
+            containsString(nodeVersion.toString())));
+        expectThrows(IllegalStateException.class, () -> mockTerminal.readText(""));
+
+        final NodeMetaData nodeMetaData = NodeMetaData.FORMAT.loadLatestState(logger, xContentRegistry(), nodePaths);
+        assertThat(nodeMetaData.nodeId(), equalTo(nodeId));
+        assertThat(nodeMetaData.nodeVersion(), equalTo(nodeVersion));
+    }
+
+    public void testOverwritesIfTooOld() throws Exception {
+        final String nodeId = randomAlphaOfLength(10);
+        final Version nodeVersion = NodeMetaDataTests.tooOldVersion();
+        NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(nodeId, nodeVersion), nodePaths);
+        final MockTerminal mockTerminal = new MockTerminal();
+        mockTerminal.addTextInput(randomFrom("y", "Y"));
+        new OverrideNodeVersionCommand().processNodePaths(mockTerminal, nodePaths, environment);
+        assertThat(mockTerminal.getOutput(), allOf(
+            containsString("too old"),
+            containsString("data loss"),
+            containsString("You should not use this tool"),
+            containsString(Version.CURRENT.toString()),
+            containsString(nodeVersion.toString()),
+            containsString(OverrideNodeVersionCommand.SUCCESS_MESSAGE)));
+        expectThrows(IllegalStateException.class, () -> mockTerminal.readText(""));
+
+        final NodeMetaData nodeMetaData = NodeMetaData.FORMAT.loadLatestState(logger, xContentRegistry(), nodePaths);
+        assertThat(nodeMetaData.nodeId(), equalTo(nodeId));
+        assertThat(nodeMetaData.nodeVersion(), equalTo(Version.CURRENT));
+    }
+
+    public void testOverwritesIfTooNew() throws Exception {
+        final String nodeId = randomAlphaOfLength(10);
+        final Version nodeVersion = NodeMetaDataTests.tooNewVersion();
+        NodeMetaData.FORMAT.writeAndCleanup(new NodeMetaData(nodeId, nodeVersion), nodePaths);
+        final MockTerminal mockTerminal = new MockTerminal();
+        mockTerminal.addTextInput(randomFrom("y", "Y"));
+        new OverrideNodeVersionCommand().processNodePaths(mockTerminal, nodePaths, environment);
+        assertThat(mockTerminal.getOutput(), allOf(
+            containsString("data loss"),
+            containsString("You should not use this tool"),
+            containsString(Version.CURRENT.toString()),
+            containsString(nodeVersion.toString()),
+            containsString(OverrideNodeVersionCommand.SUCCESS_MESSAGE)));
+        expectThrows(IllegalStateException.class, () -> mockTerminal.readText(""));
+
+        final NodeMetaData nodeMetaData = NodeMetaData.FORMAT.loadLatestState(logger, xContentRegistry(), nodePaths);
+        assertThat(nodeMetaData.nodeId(), equalTo(nodeId));
+        assertThat(nodeMetaData.nodeVersion(), equalTo(Version.CURRENT));
+    }
+}