Added upgrade docs explaining how to reindex in place or reindex from remote

Closes #20675

docs/reference/docs/reindex.asciidoc

@@ -367,6 +367,7 @@ POST _reindex
 // TEST[s/^/PUT source\n/]
 
 [float]
+[[reindex-from-remote]]
 === Reindex from Remote
 
 Reindex supports reindexing from a remote Elasticsearch cluster:

docs/reference/setup/cluster_restart.asciidoc

@@ -8,7 +8,9 @@ required.
 
 The process to perform an upgrade with a full cluster restart is as follows:
 
-==== Step 1: Disable shard allocation
+. *Disable shard allocation*
++
+--
 
 When you shut down a node, the allocation process will immediately try to
 replicate the shards that were on that node to other nodes in the cluster,
@@ -26,8 +28,11 @@ PUT _cluster/settings
 --------------------------------------------------
 // CONSOLE
 // TEST[skip:indexes don't assign]
+--
 
-==== Step 2: Perform a synced flush
+. *Perform a synced flush*
++
+--
 
 Shard recovery will be much faster if you stop indexing and issue a
 <<indices-synced-flush, synced-flush>> request:
@@ -41,19 +46,28 @@ POST _flush/synced
 A synced flush request is a ``best effort'' operation. It will fail if there
 are any pending indexing operations, but it is safe to reissue the request
 multiple times if necessary.
+--
 
-==== Step 3: Shutdown and upgrade all nodes
+. *Shutdown and upgrade all nodes*
++
+--
 
 Stop all Elasticsearch services on all nodes in the cluster. Each node can be
 upgraded following the same procedure described in <<upgrade-node>>.
+--
 
-==== Step 4: Upgrade any plugins
+. *Upgrade any plugins*
++
+--
 
 Elasticsearch plugins must be upgraded when upgrading a node.  Use the
 `elasticsearch-plugin` script to install the correct version of any plugins
 that you need.
+--
 
-==== Step 5: Start the cluster
+. *Start the cluster*
++
+--
 
 If you have dedicated master nodes -- nodes with `node.master` set to
 `true`(the default) and `node.data` set to `false` --  then it is a good idea
@@ -75,8 +89,11 @@ GET _cat/nodes
 // CONSOLE
 
 Use these APIs to check that all nodes have successfully joined the cluster.
+--
 
-==== Step 6: Wait for yellow
+. *Wait for yellow*
++
+--
 
 As soon as each node has joined the cluster, it will start to recover any
 primary shards that are stored locally.  Initially, the
@@ -87,8 +104,11 @@ Once each node has recovered its local shards, the `status` will become
 `yellow`, meaning all primary shards have been recovered, but not all replica
 shards are allocated.  This is to be expected because allocation is still
 disabled.
+--
 
-==== Step 7: Reenable allocation
+. *Reenable allocation*
++
+--
 
 Delaying the allocation of replicas until all nodes have joined the cluster
 allows the master to allocate replicas to nodes which already have local shard
@@ -124,3 +144,4 @@ GET _cat/recovery
 
 Once the `status` column in the `_cat/health` output has reached `green`, all
 primary and replica shards have been successfully allocated.
+--

docs/reference/setup/reindex_upgrade.asciidoc

@@ -0,0 +1,92 @@
+[[reindex-upgrade]]
+=== Reindex to upgrade
+
+Elasticsearch is able to use indices created in the previous major version
+only.  For instance, Elasticsearch 6.x can use indices created in
+Elasticsearch 5.x, but not those created in Elasticsearch 2.x or before.
+
+NOTE: Elasticsearch 6.x nodes will fail to start in the presence of too old indices.
+
+If you are running an Elasticsearch 5.x cluster which contains indices that
+were created before 5.x, you will either need to delete those old indices or
+to reindex them before upgrading to 6.x.  See <<reindex-upgrade-inplace>>.
+
+If you are running an Elasticsearch 2.x cluster or older, you have two options:
+
+* First upgrade to Elasticsearch 5.x, reindex the old indices, then upgrade
+  to 6.x. See <<reindex-upgrade-inplace>>.
+
+* Create a new 6.x cluster and use reindex-from-remote to import indices
+  directly from the 2.x cluster. See <<reindex-upgrade-remote>>.
+
+[[reindex-upgrade-inplace]]
+==== Reindex in place
+
+If you are running a 5.x cluster which contains indices created in
+Elasticsearch 2.x, you will need to reindex (or delete) those indices before
+upgrading to Elasticsearch 6.x.
+
+The reindex process works as follows:
+
+* Create a new index, copying the mappings and settings from the old index.
+  Set the `refresh_interval` to `-1` and the `number_of_replicas` to `0` for
+  efficient reindexing.
+
+* Reindex all documents from the old index to the new index using the
+  <<docs-reindex,reindex API>>.
+
+* Reset the `refresh_interval` and `number_of_replicas` to the values
+  used in the old index, and wait for the index to become green.
+
+* In a single <<indices-aliases,update aliases>> request:
+
+  * Delete the old index.
+  * Add an alias with the old index name to the new index.
+  * Add any aliases that existed on the old index to the new index.
+
+At the end of this process, you will have a new 5.x index which can be used
+by an Elasticsearch 6.x cluster.
+
+[[reindex-upgrade-remote]]
+==== Upgrading with reindex-from-remote
+
+If you are running a 1.x or 2.x cluster and would like to migrate directly to 6.x
+without first migrating to 5.x, you can do so using
+<<reindex-from-remote,reindex-from-remote>>.
+
+[WARNING]
+=============================================
+
+Elasticsearch includes backwards compatibility code that allows indices from
+the previous major version to be upgraded to the current major version.  By
+moving directly from Elasticsearch 2.x or before to 6.x, you will have to solve any
+backwards compatibility issues yourself.
+
+=============================================
+
+You will need to set up a 6.x cluster alongside your existing old cluster.
+The 6.x cluster needs to have access to the REST API of the old cluster.
+
+For each old index that you want to transfer to the 6.x cluster, you will need
+to:
+
+* Create a new index in 6.x with the appropriate mappings and settings.  Set
+  the `refresh_interval` to `-1` and set `number_of_replicas` to `0` for
+  faster reindexing.
+
+* Use <<reindex-from-remote,reindex-from-remote>> to pull documents from the
+  old index into the new 6.x index.
+
+* If you run the reindex job in the background (with `wait_for_completion` set
+  to `false`), the reindex request will return a `task_id` which can be used to
+  monitor progress of the reindex job in the <<tasks,task API>>:
+  `GET _tasks/TASK_ID`.
+
+* Once reindex has completed, set the `refresh_interval` and
+  `number_of_replicas` to the desired values (the defaults are `30s` and `1`
+  respectively).
+
+* Once the new index has finished replication, you can delete the old index.
+
+The 6.x cluster can start out small, and you can gradually move nodes from the
+old cluster to the 6.x cluster as you migrate indices across.

docs/reference/setup/rolling_upgrade.asciidoc

@@ -12,7 +12,9 @@ supported for your version of Elasticsearch.
 
 To perform a rolling upgrade:
 
-==== Step 1: Disable shard allocation
+. *Disable shard allocation*
++
+--
 
 When you shut down a node, the allocation process will wait for one minute
 before starting to replicate the shards that were on that node to other nodes
@@ -30,8 +32,11 @@ PUT _cluster/settings
 --------------------------------------------------
 // CONSOLE
 // TEST[skip:indexes don't assign]
+--
 
-==== Step 2: Stop non-essential indexing and perform a synced flush (Optional)
+. *Stop non-essential indexing and perform a synced flush (Optional)*
++
+--
 
 You may happily continue indexing during the upgrade.  However, shard recovery
 will be much faster if you temporarily stop non-essential indexing and issue a
@@ -46,9 +51,11 @@ POST _flush/synced
 A synced flush request is a ``best effort'' operation. It will fail if there
 are any pending indexing operations, but it is safe to reissue the request
 multiple times if necessary.
+--
 
-[[upgrade-node]]
-==== Step 3: Stop and upgrade a single node
+. [[upgrade-node]] *Stop and upgrade a single node*
++
+--
 
 Shut down one of the nodes in the cluster *before* starting the upgrade.
 
@@ -87,14 +94,20 @@ To upgrade using a zip or compressed tarball:
 *   Either copy the files in the `data` directory from your old installation
     to your new installation, or configure the location of the data directory
     in the `config/elasticsearch.yml` file, with the `path.data` setting.
+--
 
-==== Step 4: Upgrade any plugins
+. *Upgrade any plugins*
++
+--
 
 Elasticsearch plugins must be upgraded when upgrading a node.  Use the
 `elasticsearch-plugin` script to install the correct version of any plugins
 that you need.
+--
 
-==== Step 5: Start the upgraded node
+. *Start the upgraded node*
++
+--
 
 Start the now upgraded node and confirm that it joins the cluster by checking
 the log file or by checking the output of this request:
@@ -104,8 +117,11 @@ the log file or by checking the output of this request:
 GET _cat/nodes
 --------------------------------------------------
 // CONSOLE
+--
 
-==== Step 6: Reenable shard allocation
+. *Reenable shard allocation*
++
+--
 
 Once the node has joined the cluster, reenable shard allocation to start using
 the node:
@@ -120,8 +136,11 @@ PUT _cluster/settings
 }
 --------------------------------------------------
 // CONSOLE
+--
 
-==== Step 7: Wait for the node to recover
+. *Wait for the node to recover*
++
+--
 
 You should wait for the cluster to finish shard allocation before upgrading
 the next node.  You can check on progress with the <<cat-health,`_cat/health`>>
@@ -168,8 +187,12 @@ GET _cat/recovery
 
 If you stopped indexing, then it is safe to resume indexing as soon as
 recovery has completed.
+--
 
-==== Step 8: Repeat
+. *Repeat*
++
+--
 
 When the cluster is stable and the node has recovered, repeat the above steps
 for all remaining nodes.
+--

docs/reference/setup/upgrade.asciidoc

@@ -1,5 +1,5 @@
 [[setup-upgrade]]
-== Upgrading
+== Upgrading Elasticsearch
 
 [IMPORTANT]
 ===========================================
@@ -22,14 +22,34 @@ consult this table:
 [cols="1<m,1<m,3",options="header",]
 |=======================================================================
 |Upgrade From   |Upgrade To     |Supported Upgrade Type
-|2.x            |2.y            |<<rolling-upgrades,Rolling upgrade>> (where `y > x`)
+|< 5.x          |6.x            |<<reindex-upgrade,Reindex to upgrade>>
 |5.x            |5.y            |<<rolling-upgrades,Rolling upgrade>> (where `y > x`)
-|2.x            |5.x            |<<restart-upgrade,Full cluster restart>>
-|5.0.0-alpha1   |5.y            |<<restart-upgrade,Full cluster restart>>
-|5.0.0-alpha2   |5.y            |<<restart-upgrade,Full cluster restart>>
-|5.0.0-beta1    |5.y            |<<restart-upgrade,Full cluster restart>>
+|5.x            |6.x            |<<restart-upgrade,Full cluster restart>>
+|6.0.0 pre GA   |6.x            |<<restart-upgrade,Full cluster restart>>
+|6.x            |6.y            |<<rolling-upgrades,Rolling upgrade>> (where `y > x`)
 |=======================================================================
 
+[IMPORTANT]
+.Indices created in Elasticsearch 2.x or before
+===============================================
+
+Elasticsearch is able to read indices created in the *previous major version
+only*.  For instance, Elasticsearch 6.x can use indices created in
+Elasticsearch 5.x, but not those created in Elasticsearch 2.x or before.
+
+This condition also applies to indices backed up with
+<<modules-snapshots,snapshot and restore>>.  If an index was originally
+created in 2.x, it cannot be restored into a 6.x cluster even if the
+snapshot was made by a 5.x cluster.
+
+Elasticsearch 6.x nodes will fail to start in the presence of too old indices.
+
+See <<reindex-upgrade>> for more information about how to upgrade old indices.
+===============================================
+
+
 include::rolling_upgrade.asciidoc[]
 
 include::cluster_restart.asciidoc[]
+
+include::reindex_upgrade.asciidoc[]