Browse Source

[DOCS] Update snapshot defaults for system indices (#81226)

Updates the snapshot and restore docs for https://github.com/elastic/elasticsearch/pull/79670.

Closes https://github.com/elastic/elasticsearch/issues/81183
James Rodewig 3 years ago
parent
commit
1519bb6ecb

+ 38 - 0
docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc

@@ -800,6 +800,44 @@ Assign users with the `kibana_user` role to the `kibana_admin` role.
 Discontinue use of the `kibana_user` role.
 ====
 
+[[snapshot-resolve-system-indices]]
+.For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to system indices or system data streams.
+[%collapsible]
+====
+*Details* +
+For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to
+system indices or system data streams.
+{ref}/snapshot-restore.html#feature-state[Feature states] are now the only way
+to back up and restore system indices or system data streams from a snapshot.
+
+You can no longer use the `indices` parameter for the
+{ref}/slm-api-put-policy.html[create {slm-init} policy API] or the
+{ref}/create-snapshot-api.html[create snapshot API] to include a system index in
+a snapshot. To back up a system index, use the `include_global_state` and
+`feature_states` parameters to include the corresponding feature state instead.
+By default, the `include_global_state` and `feature_states` parameters include
+all system indices.
+
+Similarly, you can no longer use the {ref}/restore-snapshot-api.html[restore snapshot
+API]'s `indices` parameter to restore a system index from a snapshot. To restore
+a system index, use the `include_global_state` and `feature_states` parameters
+to restore the corresponding feature state instead. By default, the
+`include_global_state` and `feature_states` parameters don't restore any system
+indices.
+
+*Impact* +
+If you previously used the `indices` parameter to back up or restore system
+indices, update your {slm-init} policies and application to use the
+`include_global_state` and `feature_states` parameters instead.
+
+An {slm-init} policy that explicitly specifies a system index in the `indices`
+parameter will fail to create snapshots. Similarly, a create snapshot API or
+restore snapshot API request that explicitly specifies a system index in the
+`indices` parameter will fail and return an error. If the `indices` value
+includes a wildcard (`*`) pattern, the pattern will no longer match system
+indices.
+====
+
 .Snapshots compress metadata files by default.
 [%collapsible]
 ====

+ 5 - 2
docs/reference/snapshot-restore/apis/create-snapshot-api.asciidoc

@@ -114,9 +114,12 @@ include::restore-snapshot-api.asciidoc[tag=cluster-state-contents]
 (Optional, string or array of strings)
 Comma-separated list of data streams and indices to include in the snapshot.
 Supports <<api-multi-index,multi-index syntax>>. Defaults to an empty array
-(`[]`), which includes all data streams and indices, including system indices.
+(`[]`), which includes all regular data streams and regular indices. To exclude
+all data streams and indices, use `-*` or `none`.
 +
-To exclude all data streams and indices, use `-*` or `none`.
+You can't use this parameter to include or exclude <<system-indices,system
+indices or system data streams>> from a snapshot. Use
+<<{page-id}-feature-states,`feature_states`>> instead.
 
 [id="{page-id}-feature-states"]
 `feature_states`::

+ 7 - 7
docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc

@@ -53,14 +53,8 @@ POST /index_4/_close
 [source,console]
 ----
 POST /_snapshot/my_repository/my_snapshot/_restore
-{
-  "indices": "*,-.*",
-  "feature_states": [ "geoip" ]
-}
 ----
 // TEST[s/_restore/_restore?wait_for_completion=true/]
-// TEST[s/",/"/]
-// TEST[s/"feature_states": \[ "geoip" \]//]
 
 [[restore-snapshot-api-request]]
 ==== {api-request-title}
@@ -198,7 +192,13 @@ For data streams, this option only applies to restored backing indices. New
 backing indices are configured using the data stream's matching index template.
 
 `indices`::
-(Optional, string or array of strings) Comma-separated list of indices and data streams to restore. Supports <<api-multi-index,multi-target syntax>>. Defaults to all indices and data streams in the snapshot, including system indices.
+(Optional, string or array of strings) Comma-separated list of indices and data
+streams to restore. Supports <<api-multi-index,multi-target syntax>>. Defaults
+to all regular indices and regular data streams in the snapshot.
++
+You can't use this parameter to restore <<system-indices,system indices or
+system data streams>>. Use
+<<restore-snapshot-api-feature-states,`feature_states`>> instead.
 
 [[restore-snapshot-api-partial]]
 `partial`::

+ 5 - 2
docs/reference/snapshot-restore/index.asciidoc

@@ -31,8 +31,8 @@ recover or transfer its data.
 [[snapshot-contents]]
 == Snapshot contents
 
-By default, a snapshot of a cluster contains the cluster state, all data
-streams, and all indices, including system indices. The cluster state includes:
+By default, a snapshot of a cluster contains the cluster state, all regular data
+streams, and all regular indices. The cluster state includes:
 
 include::apis/restore-snapshot-api.asciidoc[tag=cluster-state-contents]
 
@@ -61,6 +61,9 @@ used by the feature. For example, a feature state may include a regular index
 that contains the feature's execution history. Storing this history in a regular
 index lets you more easily search it.
 
+In {es} 8.0 and later versions, feature states are the only way to back up and
+restore system indices and system data streams.
+
 [discrete]
 [[how-snapshots-work]]
 == How snapshots work

+ 4 - 38
docs/reference/snapshot-restore/restore-snapshot.asciidoc

@@ -82,10 +82,10 @@ GET _snapshot/my_repository/*?verbose=false
 You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the
 <<restore-snapshot-api,restore snapshot API>>.
 
-By default, a restore request attempts to restore all indices and data streams
-in a snapshot, including <<system-indices,system indices and system data
-streams>>. In most cases, you only need to restore a specific index or data
-stream from a snapshot. However, you can't restore an existing open index.
+By default, a restore request attempts to restore all regular indices and
+regular data streams in a snapshot. In most cases, you only need to restore a
+specific index or data stream from a snapshot. However, you can't restore an
+existing open index.
 
 If you're restoring data to a pre-existing cluster, use one of the
 following methods to avoid conflicts with existing indices and data streams:
@@ -93,10 +93,6 @@ following methods to avoid conflicts with existing indices and data streams:
 * <<delete-restore>>
 * <<rename-on-restore>>
 
-Some {es} features automatically create system indices on cluster startup. To
-avoid conflicts with these system indices when restoring data to a new cluster,
-see <<restore-exclude-system-indices>>.
-
 [discrete]
 [[delete-restore]]
 ==== Delete and restore
@@ -210,37 +206,12 @@ POST _reindex
 ----
 // TEST[continued]
 
-[discrete]
-[[restore-exclude-system-indices]]
-==== Exclude system indices
-
-Some {es} features, such as the <<geoip-processor,GeoIP processor>>,
-automatically create system indices at startup. To avoid naming conflicts with
-these indices, use the `-.*` wildcard pattern to exclude system indices and
-other dot (`.`) indices from your restore request.
-
-For example, the following request uses the `*,-.*` wildcard pattern to restore
-all indices and data streams except dot indices.
-
-[source,console]
-----
-POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
-{
-  "indices": "*,-.*"
-}
-----
-// TEST[setup:setup-snapshots]
-// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
-// TEST[s/_restore/_restore?wait_for_completion=true/]
-
 [discrete]
 [[restore-feature-state]]
 === Restore a feature state
 
 You can restore a <<feature-state,feature state>> to recover system indices,
 system data streams, and other configuration data for a feature from a snapshot.
-Restoring a feature state is the preferred way to restore system indices and
-system data streams.
 
 If you restore a snapshot's cluster state, the operation restores all feature
 states in the snapshot by default. Similarly, if you don't restore a snapshot's
@@ -271,21 +242,16 @@ Console before restoring the `security` feature state. If you run {es} on your
 own hardware, <<restore-create-file-realm-user,create a superuser in the file
 realm>> to ensure you'll still be able to access your cluster.
 
-To avoid accidentally restoring system indices, system data streams, and other
-dot indices, append the `-.*` wildcard pattern to the `indices` value.
-
 [source,console]
 ----
 POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 {
-  "indices": "*,-.*",
   "feature_states": [ "geoip" ]
 }
 ----
 // TEST[setup:setup-snapshots]
 // TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
 // TEST[s/_restore/_restore?wait_for_completion=true/]
-// TEST[s/",/"/]
 // TEST[s/"feature_states": \[ "geoip" \]//]
 
 [discrete]

+ 4 - 7
docs/reference/snapshot-restore/take-snapshot.asciidoc

@@ -207,9 +207,7 @@ PUT _slm/policy/nightly-snapshots
     naming conflicts, the policy also appends a UUID to each snapshot name.
 <3> <<snapshots-register-repository,Registered snapshot repository>> used to
     store the policy's snapshots.
-<4> Data streams and indices to include in the policy's snapshots. This
-    configuration includes all data streams and indices, including system
-    indices.
+<4> Data streams and indices to include in the policy's snapshots.
 <5> If `true`, the policy's snapshots include the cluster state. This also
     includes all feature states by default. To only include specific feature
     states, see <<back-up-specific-feature-state>>.
@@ -498,7 +496,7 @@ PUT _slm/policy/nightly-cluster-state-snapshots
 <2> Excludes regular data streams and indices.
 
 If you take dedicated snapshots of the cluster state, you'll need to exclude the
-cluster state and system indices from your other snapshots. For example:
+cluster state from your other snapshots. For example:
 
 [source,console]
 ----
@@ -509,7 +507,7 @@ PUT _slm/policy/nightly-snapshots
   "repository": "my_repository",
   "config": {
     "include_global_state": false,    <1>
-    "indices": "*,-.*"                <2>
+    "indices": "*"                    <2>
   },
   "retention": {
     "expire_after": "30d",
@@ -521,8 +519,7 @@ PUT _slm/policy/nightly-snapshots
 
 <1> Excludes the cluster state. This also excludes all feature states by
     default.
-<2> Includes all data streams and indices except system indices and other
-    indices that begin with a dot (`.`).
+<2> Includes all regular data streams and indices.
 
 [discrete]
 [[create-snapshots-different-time-intervals]]