[DOCS] First pass at upgrade updates for 7.0. (#39944)

* [DOCS] First pass at upgrade updates for 7.0.

* [DOCS] Updates X-Pack terminology

* [DOCS] Incorporated feedback from lcawl.

docs/Versions.asciidoc

@@ -1,5 +1,6 @@
 :version:               8.0.0-alpha1
 :major-version:         8.x
+:prev-major-version:    7.x
 :lucene_version:        8.0.0
 :lucene_version_path:   8_0_0
 :branch:                master

docs/reference/upgrade.asciidoc

@@ -1,71 +1,46 @@
 [[setup-upgrade]]
-= Upgrade Elasticsearch
+= Upgrade {es}
 
 [partintro]
 --
-Elasticsearch can usually be upgraded using a <<rolling-upgrades,Rolling upgrade>>
-process so upgrading does not interrupt service. However, you might
-need to <<reindex-upgrade,Reindex to upgrade>> indices created in older versions.
-Upgrades across major versions prior to 6.0 require a <<restart-upgrade,Full cluster restart>>.
+{es} can usually be upgraded using a <<rolling-upgrades,Rolling upgrade>>
+process so upgrading does not interrupt service. Rolling upgrades are supported:
+
+* Between minor versions
+* From 5.6 to 6.7
+* From 6.7 to {version}
+
+{es} can read indices created in the previous major version. If you
+have indices created in 5.x or before, you must reindex or delete them
+before upgrading to {version}. {es} nodes will fail to start if
+incompatible indices are present. Snapshots of 5.x or earlier indices cannot be
+restored to a 7.x cluster even if they were created by a 6.x cluster. For
+information about upgrading old indices, see <<reindex-upgrade, Reindex to upgrade>>.
+
+When upgrading to a new version of {es}, you need to upgrade each
+of the products in your Elastic Stack. For more information, see the
+{stack-ref}/upgrading-elastic-stack.html[Elastic Stack Installation and Upgrade Guide].
+
+To upgrade directly to {version} from 6.6 or earlier, you must shut down the
+cluster, install {version}, and restart. For more information, see
+<<restart-upgrade, Full cluster restart upgrade>>.
+
+[float]
+== Preparing to upgrade
+
+Before upgrading {es}:
+
+. Check the <<deprecation-logging, deprecation log>> to see if you are using
+any deprecated features and update your code accordingly. By default,
+deprecation warnings are logged when the log level is set to `WARN`.
+. Review the <<breaking-changes,breaking changes>> and make any necessary changes
+to your code and configuration for {version}.
+. If you use custom plugins, make sure compatible versions are available.
+. Test upgrades in a dev environment before upgrading your production cluster.
+before upgrading.
+. <<modules-snapshots,Back up your data!>> You must have a snapshot of your
+data to roll back to an earlier version.
 
-When upgrading to a new version of Elasticsearch, you need to upgrade each
-of the products in your Elastic Stack. The steps you need to take to upgrade
-differ depending on which products you are using. Want a list that's tailored
-to your stack? Try out our {upgrade_guide}[Interactive Upgrade Guide]. For
-more information about upgrading your stack, see {stack-ref}[Upgrading the
-Elastic Stack].
-
-[IMPORTANT]
-===========================================
-Before upgrading Elasticsearch:
-
-* Review the <<breaking-changes,breaking changes>> for changes that
-affect your application.
-* Check the <<deprecation-logging, deprecation log>> to see if you are using
-any deprecated features.
-* If you use custom plugins, make sure compatible versions are available.
-* Test upgrades in a dev environment before upgrading your production cluster.
-* <<modules-snapshots,Back up your data>> before upgrading.
-You **cannot roll back** to an earlier version unless you have a backup of
-your data.
-
-===========================================
-
-
-The following table shows when you can perform a rolling upgrade, when you
-need to reindex or delete old indices, and when a full cluster restart is
-required.
-
-[[upgrade-paths]]
-[cols="1<m,1<m,3",options="header",]
-|=======================================================================
-|Upgrade From   |Upgrade To     |Supported Upgrade Type
-|5.x            |5.y            |<<rolling-upgrades,Rolling upgrade>> (where `y > x`)
-|5.6            |6.x            |<<rolling-upgrades,Rolling upgrade>> footnoteref:[reindexfn, You must delete or reindex any indices created in 2.x before upgrading.]
-|5.0-5.5        |6.x            |<<restart-upgrade,Full cluster restart>> footnoteref:[reindexfn]
-|<5.x           |6.x            |<<reindex-upgrade,Reindex to upgrade>>
-|6.x            |6.y            |<<rolling-upgrades,Rolling upgrade>> (where `y > x`) 
-|=======================================================================
-
-[IMPORTANT]
-===============================================
-
-Elasticsearch can read indices created in the *previous major version*.
-Older indices must be reindexed or deleted. Elasticsearch 6.x
-can use indices created in Elasticsearch 5.x, but not those created in
-Elasticsearch 2.x or before. Elasticsearch 5.x can use indices created in
-Elasticsearch 2.x, but not those created in 1.x or before.
-
-This also applies to indices backed up with <<modules-snapshots,snapshot
-and restore>>. If an index was originally created in 2.x, it cannot be
-restored to a 6.x cluster even if the snapshot was created by a 5.x cluster.
-
-Elasticsearch nodes will fail to start if incompatible indices are present.
-
-For information about how to upgrade old indices, see <<reindex-upgrade,
-Reindex to upgrade>>.
-
-===============================================
 --
 
 include::upgrade/rolling_upgrade.asciidoc[]

docs/reference/upgrade/close-ml.asciidoc

@@ -13,9 +13,9 @@ POST _ml/set_upgrade_mode?enabled=false
 
 If your {ml} indices were created earlier than the previous major version, they
 must be reindexed. In those circumstances, there must be no machine learning
-jobs running during the upgrade. 
+jobs running during the upgrade.
 
-In all other circumstances, there is no requirement to close your {ml} jobs. 
+In all other circumstances, there is no requirement to close your {ml} jobs.
 There are, however, advantages to doing so. If you choose to leave your jobs
 running during the upgrade, they are affected when you stop the {ml} nodes. The
 jobs move to another {ml} node and restore the model states. This scenario has
@@ -26,7 +26,7 @@ To close all {ml} jobs before you upgrade, see
 {stack-ov}/stopping-ml.html[Stopping {ml}]. This method persists the model
 state at the moment of closure, which means that when you open your jobs after
 the upgrade, they use the exact same model. This scenario takes the most time,
-however, especially if you have many jobs or jobs with large model states.  
+however, especially if you have many jobs or jobs with large model states.
 
 To temporarily halt the tasks associated with your {ml} jobs and {dfeeds} and
 prevent new jobs from opening, use the <<ml-set-upgrade-mode,set upgrade mode API>>:
@@ -40,4 +40,4 @@ POST _ml/set_upgrade_mode?enabled=true
 This method does not persist the absolute latest model state, rather it uses the
 last model state that was automatically saved. By halting the tasks, you avoid
 incurring the cost of managing active jobs during the upgrade and it's quicker
-than stopping {dfeeds} and closing jobs. 
+than stopping {dfeeds} and closing jobs.

docs/reference/upgrade/cluster_restart.asciidoc

@@ -1,15 +1,15 @@
 [[restart-upgrade]]
 == Full cluster restart upgrade
 
-A full cluster restart upgrade requires that you shut all nodes in the cluster
-down, upgrade them, and restart the cluster. A full cluster restart was required
-when upgrading to major versions prior to 6.x. Elasticsearch 6.x supports
-<<rolling-upgrades, rolling upgrades>> from *Elasticsearch 5.6*. Upgrading to
-6.x from earlier versions requires a full cluster restart. See the
-<<upgrade-paths,Upgrade paths table>> to verify the type of upgrade you need
-to perform.
+To upgrade directly to {es} {version} from versions 6.0-6.6, you must shut down
+all nodes in the cluster, upgrade each node to {version}, and restart the cluster.
 
-To perform a full cluster restart upgrade:
+NOTE: If you are running a version prior to 6.0,
+https://www.elastic.co/guide/en/elastic-stack/6.7/upgrading-elastic-stack.html[upgrade to 6.7]
+and reindex your old indices or bring up a new {version} cluster and
+<<reindex-upgrade-remote, reindex from remote>>.
+
+To perform a  full cluster restart upgrade to {version}:
 
 . *Disable shard allocation.*
 +
@@ -26,7 +26,7 @@ recovery.
 include::synced-flush.asciidoc[]
 --
 
-. *Stop any machine learning jobs that are running.* 
+. *Stop any machine learning jobs that are running.*
 +
 --
 include::close-ml.asciidoc[]
@@ -52,7 +52,7 @@ include::set-paths-tip.asciidoc[]
 . *Upgrade any plugins.*
 +
 Use the `elasticsearch-plugin` script to install the upgraded version of each
-installed Elasticsearch plugin. All plugins must be upgraded when you upgrade
+installed {es} plugin. All plugins must be upgraded when you upgrade
 a node.
 
 . If you use {es} {security-features} to define realms, verify that your realm

docs/reference/upgrade/open-ml.asciidoc

@@ -1,8 +1,5 @@
 [testenv="platinum"]
-If you closed all {ml} jobs before the upgrade, you must open them. Use {kib} or
-the <<ml-open-job,open jobs API>>. 
-
-Alternatively, if you temporarily halted the tasks associated with your {ml} jobs,
+If you temporarily halted the tasks associated with your {ml} jobs,
 use the <<ml-set-upgrade-mode,set upgrade mode API>> to return them to active
 states:
 
@@ -11,3 +8,7 @@ states:
 POST _ml/set_upgrade_mode?enabled=false
 --------------------------------------------------
 // CONSOLE
+
+If you closed all {ml} jobs before the upgrade, open the jobs and start the
+datafeeds from {kib} or with the <<ml-open-job,open jobs>> and
+<<ml-start-datafeed,start datafeed>> APIs.

docs/reference/upgrade/reindex_upgrade.asciidoc

@@ -1,55 +1,28 @@
 [[reindex-upgrade]]
 == Reindex before upgrading
 
-Elasticsearch can read indices created in the *previous major version*.
-Older indices must be reindexed or deleted. Elasticsearch 6.x
-can use indices created in Elasticsearch 5.x, but not those created in
-Elasticsearch 2.x or before. Elasticsearch 5.x can use indices created in
-Elasticsearch 2.x, but not those created in 1.x or before.
+{es} can read indices created in the previous major version. If you
+have indices created in 5.x or before, you must reindex or delete them
+before upgrading to {version}. {es} nodes will fail to start if
+incompatible indices are present. Snapshots of 5.x or earlier indices cannot be
+restored to a 7.x cluster even if they were created by a 6.x cluster.
 
-Elasticsearch nodes will fail to start if incompatible indices are present.
+You have two options for reindexing old indices:
 
-To upgrade an Elasticsearch 5.x cluster that contains indices created in 2.x,
-you must reindex or delete them before upgrading to 6.x.
-For more information, see <<reindex-upgrade-inplace, Reindex in place>>.
-
-To upgrade an Elasticsearch cluster running 2.x, you have two options:
-
-* Perform a <<restart-upgrade, full cluster restart upgrade>> to 5.6,
-  <<reindex-upgrade-inplace, reindex>> the 2.x indices, then perform a
-  <<rolling-upgrades, rolling upgrade>> to 6.x. If your Elasticsearch 2.x
-  cluster contains indices that were created before 2.x, you must either
-  delete or reindex them before upgrading to 5.6. For more information about
-  upgrading from 2.x to 5.6, see https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html[
-  Upgrading Elasticsearch] in the Elasticsearch 5.6 Reference.
-
-* Create a new 6.x cluster and <<reindex-upgrade-remote, reindex from
-  remote>> to import indices directly from the 2.x cluster.
-
-To upgrade an Elasticsearch 1.x cluster, you have two options:
-
-* Perform a <<restart-upgrade, full cluster restart upgrade>> to Elasticsearch
-  2.4.x and <<reindex-upgrade-inplace, reindex>> or delete the 1.x indices.
-  Then, perform a full cluster restart upgrade to 5.6 and reindex or delete
-  the 2.x indices. Finally, perform a <<rolling-upgrades, rolling upgrade>>
-  to 6.x. For more information about upgrading from 1.x to 2.4, see https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-upgrade.html[
-  Upgrading Elasticsearch] in the Elasticsearch 2.4 Reference.
-  For more information about upgrading from 2.4 to 5.6, see https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-upgrade.html[
-  Upgrading Elasticsearch] in the Elasticsearch 5.6 Reference.
-
-* Create a new 6.x cluster and <<reindex-upgrade-remote, reindex from
-  remote>> to import indices directly from the 1.x cluster.
+* <<reindex-upgrade-inplace, Reindex in place>> on your 6.x cluster before upgrading.
+* Create a new {version} cluster and <<reindex-upgrade-remote, Reindex from remote>>.
+This enables you to reindex indices that reside on clusters running any version of {es}.
 
 .Upgrading time-based indices
 *******************************************
 
 If you use time-based indices, you likely won't need to carry
-pre-5.x indices forward to 6.x. Data in time-based indices
+pre-6.x indices forward to {version}. Data in time-based indices
 generally becomes less useful as time passes and are
 deleted as they age past your retention period.
 
 Unless you have an unusually long retention period, you can just
-wait to upgrade to 6.x until all of your pre-5.x indices have
+wait to upgrade to 6.x until all of your pre-6.x indices have
 been deleted.
 
 *******************************************
@@ -58,45 +31,36 @@ been deleted.
 [[reindex-upgrade-inplace]]
 === Reindex in place
 
-To manually reindex your old indices with the <<docs-reindex,`reindex` API>>:
+You can use the Upgrade Assistant in {kib} 6.7 to automatically reindex 5.x
+indices you need to carry forward to {version}.
 
-. Create a new index and copy the mappings and settings from the old index.
+To manually reindex your old indices in place:
+
+. Create an index with 7.x compatible mappings.
 . Set the `refresh_interval` to `-1` and the `number_of_replicas` to `0` for
   efficient reindexing.
-. Reindex all documents from the old index into the new index using the
-  <<docs-reindex,reindex API>>.
+. Use the <<docs-reindex,`reindex` API>> to copy documents from the
+5.x index into the new index. You can use a script to perform any necessary
+modifications to the document data and metadata during reindexing.
 . Reset the `refresh_interval` and `number_of_replicas` to the values
   used in the old index.
 . Wait for the index status to change to `green`.
 . In a single <<indices-aliases,update aliases>> request:
-
 .. Delete the old index.
 .. Add an alias with the old index name to the new index.
 .. Add any aliases that existed on the old index to the new index.
 
-
-// Need to update the CSS to override sidebar titles.
-[role="xpack"]
-.Migration assistance and upgrade tools
-*******************************************
-{xpack} 5.6 provides migration assistance and upgrade tools that simplify
-reindexing and upgrading to 6.x. These tools are free with the X-Pack trial
-and Basic licenses and you can use them to upgrade whether or not X-Pack is a
-regular part of your Elastic Stack. For more information, see
-{stack-ref}/upgrading-elastic-stack.html.
-*******************************************
-
 [[reindex-upgrade-remote]]
 === Reindex from a remote cluster
 
 You can use <<reindex-from-remote,reindex from remote>> to migrate indices from
-your old cluster to a new 6.x cluster. This enables you move to 6.x from a
-pre-5.6 cluster without interrupting service.
+your old cluster to a new {version} cluster. This enables you move to {version}
+from a pre-6.7 cluster without interrupting service.
 
 [WARNING]
 =============================================
 
-Elasticsearch provides backwards compatibility support that enables
+{es} provides backwards compatibility support that enables
 indices from the previous major version to be upgraded to the
 current major version. Skipping a major version means that you must
 resolve any backward compatibility issues yourself.
@@ -105,8 +69,8 @@ resolve any backward compatibility issues yourself.
 
 To migrate your indices:
 
-. Set up a new 6.x cluster alongside your old cluster. Enable it to access
-your old cluster by adding your old cluster to the `reindex.remote.whitelist` in `elasticsearch.yml`:
+. Set up a new {version} cluster and add the existing cluster to the
+`reindex.remote.whitelist` in `elasticsearch.yml`.
 +
 --
 [source,yaml]
@@ -123,14 +87,14 @@ cluster and remove nodes from the old one.
 =============================================
 --
 
-. For each index that you need to migrate to the 6.x cluster:
+. For each index that you need to migrate to the new cluster:
 
-.. Create a new index in 6.x with the appropriate mappings and settings. Set the
+.. Create an index the appropriate mappings and settings. Set the
   `refresh_interval` to `-1` and set `number_of_replicas` to `0` for
   faster reindexing.
 
-.. <<reindex-from-remote,Reindex from remote>> to pull documents from the
-  old index into the new 6.x index:
+.. Use the <<docs-reindex,`reindex` API>> to pull documents from the
+  remote index into the new {version} index:
 +
 --
 [source,js]
@@ -172,5 +136,5 @@ monitor progress of the reindex job with the <<tasks,task API>>:
   `number_of_replicas` to the desired values (the default settings are
   `30s` and `1`).
 
-.. Once replication is complete and the status of the new index is `green`,
+.. Once reindexing is complete and the status of the new index is `green`,
   you can delete the old index.

docs/reference/upgrade/remove-xpack.asciidoc

@@ -1,8 +1,6 @@
-IMPORTANT: If you are upgrading from a version prior to 6.3 and use {xpack}
-then you must remove the {xpack} plugin before upgrading with
-`bin/elasticsearch-plugin remove x-pack`. As of 6.3, {xpack} is included in
-the default distribution so make sure to upgrade to that one. If you upgrade
-without removing the {xpack} plugin first the node will fail to start. If you
-did not remove the {xpack} plugin and the node fails to start then you must
-downgrade to your previous version, remove {xpack}, and then upgrade again.
-In general downgrading is not supported but in this particular case it is.
+IMPORTANT: If you are upgrading from 6.2 or earlier and use {xpack},
+run `bin/elasticsearch-plugin remove x-pack` to remove the {xpack} plugin before
+you upgrade. The {xpack} functionality is now included in the default distribution
+and is no longer installed separately. The node won't start after upgrade if
+the {xpack} plugin is present. You will need to downgrade, remove the plugin,
+and reapply the upgrade.

docs/reference/upgrade/rolling_upgrade.asciidoc

@@ -1,30 +1,22 @@
 [[rolling-upgrades]]
 == Rolling upgrades
 
-A rolling upgrade allows an Elasticsearch cluster to be upgraded one node at
+A rolling upgrade allows an {es} cluster to be upgraded one node at
 a time so upgrading does not interrupt service. Running multiple versions of
-Elasticsearch in the same cluster beyond the duration of an upgrade is
+{es} in the same cluster beyond the duration of an upgrade is
 not supported, as shards cannot be replicated from upgraded nodes to nodes
 running the older version.
 
-Rolling upgrades can be performed between minor versions. Elasticsearch
-6.x supports rolling upgrades from *Elasticsearch 5.6*.
-Upgrading from earlier 5.x versions requires a <<restart-upgrade,
-full cluster restart>>. You must <<reindex-upgrade,reindex to upgrade>> from
-versions prior to 5.x.
+Rolling upgrades are supported:
 
-WARNING: If the {es} {security-features} are enabled on your 5.x cluster, before
-you can do a rolling upgrade you must encrypt the internode-communication with
-SSL/TLS, which requires a full cluster restart. For more information about this
-requirement and the associated bootstrap check, see <<bootstrap-checks-tls>>.
+* Between minor versions
+* https://www.elastic.co/guide/en/elastic-stack/6.7/upgrading-elastic-stack.html[From 5.6 to 6.7]
+* From 6.7 to {version}
 
-WARNING: The format used for the internal indices used by Kibana and {xpack}
-has changed in 6.x. When upgrading from 5.6 to 6.x, these internal indices have
-to be {stack-ref}/upgrading-elastic-stack.html#upgrade-internal-indices[upgraded]
-before the rolling upgrade procedure can start. Otherwise the upgraded node will
-refuse to join the cluster.
+Upgrading directly to {version} from 6.6 or earlier requires a
+<<restart-upgrade, full cluster restart>>.
 
-To perform a rolling upgrade:
+To perform a rolling upgrade from 6.7 to {version}:
 
 . *Disable shard allocation*.
 +
@@ -58,10 +50,6 @@ include::shut-down-node.asciidoc[]
 . *Upgrade the node you shut down.*
 +
 --
-include::remove-xpack.asciidoc[]
---
-+
---
 include::upgrade-node.asciidoc[]
 include::set-paths-tip.asciidoc[]
 --
@@ -69,7 +57,7 @@ include::set-paths-tip.asciidoc[]
 . *Upgrade any plugins.*
 +
 Use the `elasticsearch-plugin` script to install the upgraded version of each
-installed Elasticsearch plugin. All plugins must be upgraded when you upgrade
+installed {es} plugin. All plugins must be upgraded when you upgrade
 a node.
 
 . If you use {es} {security-features} to define realms, verify that your realm

docs/reference/upgrade/set-paths-tip.asciidoc

@@ -2,14 +2,14 @@
 ================================================
 
 When you extract the zip or tarball packages, the `elasticsearch-n.n.n`
-directory contains the Elasticsearch `config`, `data`, `logs` and
+directory contains the {es} `config`, `data`, `logs` and
 `plugins` directories.
 
-We recommend moving these directories out of the Elasticsearch directory
-so that there is no chance of deleting them when you upgrade Elasticsearch.
+We recommend moving these directories out of the {es} directory
+so that there is no chance of deleting them when you upgrade {es}.
 To specify the new locations, use the `ES_PATH_CONF` environment
 variable and the `path.data` and `path.logs` settings. For more information,
-see <<important-settings,Important Elasticsearch configuration>>.
+see <<important-settings,Important {es} configuration>>.
 
 The <<deb,Debian>> and <<rpm,RPM>> packages place these directories in the
 appropriate place for each operating system. In production, we recommend

docs/reference/upgrade/shut-down-node.asciidoc

@@ -1,20 +1,20 @@
-* If you are running Elasticsearch with `systemd`:
+* If you are running {es} with `systemd`:
 +
 [source,sh]
 --------------------------------------------------
 sudo systemctl stop elasticsearch.service
 --------------------------------------------------
 
-* If you are running Elasticsearch with SysV `init`:
+* If you are running {es} with SysV `init`:
 +
 [source,sh]
 --------------------------------------------------
 sudo -i service elasticsearch stop
 --------------------------------------------------
 
-* If you are running Elasticsearch as a daemon:
+* If you are running {es} as a daemon:
 +
 [source,sh]
 --------------------------------------------------
 kill $(cat pid)
---------------------------------------------------
+--------------------------------------------------

docs/reference/upgrade/upgrade-node.asciidoc

@@ -2,7 +2,7 @@ To upgrade using a <<deb,Debian>> or <<rpm,RPM>> package:
 
 *   Use `rpm` or `dpkg` to install the new package.  All files are
     installed in the appropriate location for the operating system
-    and Elasticsearch config files are not overwritten.
+    and {es} config files are not overwritten.
 
 To upgrade using a zip or compressed tarball:
 
@@ -19,7 +19,7 @@ To upgrade using a zip or compressed tarball:
    your old data directory over to the new installation. +
 +
 --
-IMPORTANT: If you use {monitoring}, re-use the data directory when you upgrade
+IMPORTANT: If you use {monitor-features}, re-use the data directory when you upgrade
 {es}. Monitoring identifies unique {es} nodes by using the persistent UUID, which
 is stored in the data directory.