Browse Source

[DOCS] Update snapshot defaults for system indices (#81226)

Updates the snapshot and restore docs for https://github.com/elastic/elasticsearch/pull/79670.

Closes https://github.com/elastic/elasticsearch/issues/81183
James Rodewig 3 years ago
parent
commit
1519bb6ecb

+ 38 - 0
docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc

@@ -800,6 +800,44 @@ Assign users with the `kibana_user` role to the `kibana_admin` role.
 Discontinue use of the `kibana_user` role.
 Discontinue use of the `kibana_user` role.
 ====
 ====
 
 
+[[snapshot-resolve-system-indices]]
+.For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to system indices or system data streams.
+[%collapsible]
+====
+*Details* +
+For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to
+system indices or system data streams.
+{ref}/snapshot-restore.html#feature-state[Feature states] are now the only way
+to back up and restore system indices or system data streams from a snapshot.
+
+You can no longer use the `indices` parameter for the
+{ref}/slm-api-put-policy.html[create {slm-init} policy API] or the
+{ref}/create-snapshot-api.html[create snapshot API] to include a system index in
+a snapshot. To back up a system index, use the `include_global_state` and
+`feature_states` parameters to include the corresponding feature state instead.
+By default, the `include_global_state` and `feature_states` parameters include
+all system indices.
+
+Similarly, you can no longer use the {ref}/restore-snapshot-api.html[restore snapshot
+API]'s `indices` parameter to restore a system index from a snapshot. To restore
+a system index, use the `include_global_state` and `feature_states` parameters
+to restore the corresponding feature state instead. By default, the
+`include_global_state` and `feature_states` parameters don't restore any system
+indices.
+
+*Impact* +
+If you previously used the `indices` parameter to back up or restore system
+indices, update your {slm-init} policies and application to use the
+`include_global_state` and `feature_states` parameters instead.
+
+An {slm-init} policy that explicitly specifies a system index in the `indices`
+parameter will fail to create snapshots. Similarly, a create snapshot API or
+restore snapshot API request that explicitly specifies a system index in the
+`indices` parameter will fail and return an error. If the `indices` value
+includes a wildcard (`*`) pattern, the pattern will no longer match system
+indices.
+====
+
 .Snapshots compress metadata files by default.
 .Snapshots compress metadata files by default.
 [%collapsible]
 [%collapsible]
 ====
 ====

+ 5 - 2
docs/reference/snapshot-restore/apis/create-snapshot-api.asciidoc

@@ -114,9 +114,12 @@ include::restore-snapshot-api.asciidoc[tag=cluster-state-contents]
 (Optional, string or array of strings)
 (Optional, string or array of strings)
 Comma-separated list of data streams and indices to include in the snapshot.
 Comma-separated list of data streams and indices to include in the snapshot.
 Supports <<api-multi-index,multi-index syntax>>. Defaults to an empty array
 Supports <<api-multi-index,multi-index syntax>>. Defaults to an empty array
-(`[]`), which includes all data streams and indices, including system indices.
+(`[]`), which includes all regular data streams and regular indices. To exclude
+all data streams and indices, use `-*` or `none`.
 +
 +
-To exclude all data streams and indices, use `-*` or `none`.
+You can't use this parameter to include or exclude <<system-indices,system
+indices or system data streams>> from a snapshot. Use
+<<{page-id}-feature-states,`feature_states`>> instead.
 
 
 [id="{page-id}-feature-states"]
 [id="{page-id}-feature-states"]
 `feature_states`::
 `feature_states`::

+ 7 - 7
docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc

@@ -53,14 +53,8 @@ POST /index_4/_close
 [source,console]
 [source,console]
 ----
 ----
 POST /_snapshot/my_repository/my_snapshot/_restore
 POST /_snapshot/my_repository/my_snapshot/_restore
-{
-  "indices": "*,-.*",
-  "feature_states": [ "geoip" ]
-}
 ----
 ----
 // TEST[s/_restore/_restore?wait_for_completion=true/]
 // TEST[s/_restore/_restore?wait_for_completion=true/]
-// TEST[s/",/"/]
-// TEST[s/"feature_states": \[ "geoip" \]//]
 
 
 [[restore-snapshot-api-request]]
 [[restore-snapshot-api-request]]
 ==== {api-request-title}
 ==== {api-request-title}
@@ -198,7 +192,13 @@ For data streams, this option only applies to restored backing indices. New
 backing indices are configured using the data stream's matching index template.
 backing indices are configured using the data stream's matching index template.
 
 
 `indices`::
 `indices`::
-(Optional, string or array of strings) Comma-separated list of indices and data streams to restore. Supports <<api-multi-index,multi-target syntax>>. Defaults to all indices and data streams in the snapshot, including system indices.
+(Optional, string or array of strings) Comma-separated list of indices and data
+streams to restore. Supports <<api-multi-index,multi-target syntax>>. Defaults
+to all regular indices and regular data streams in the snapshot.
++
+You can't use this parameter to restore <<system-indices,system indices or
+system data streams>>. Use
+<<restore-snapshot-api-feature-states,`feature_states`>> instead.
 
 
 [[restore-snapshot-api-partial]]
 [[restore-snapshot-api-partial]]
 `partial`::
 `partial`::

+ 5 - 2
docs/reference/snapshot-restore/index.asciidoc

@@ -31,8 +31,8 @@ recover or transfer its data.
 [[snapshot-contents]]
 [[snapshot-contents]]
 == Snapshot contents
 == Snapshot contents
 
 
-By default, a snapshot of a cluster contains the cluster state, all data
-streams, and all indices, including system indices. The cluster state includes:
+By default, a snapshot of a cluster contains the cluster state, all regular data
+streams, and all regular indices. The cluster state includes:
 
 
 include::apis/restore-snapshot-api.asciidoc[tag=cluster-state-contents]
 include::apis/restore-snapshot-api.asciidoc[tag=cluster-state-contents]
 
 
@@ -61,6 +61,9 @@ used by the feature. For example, a feature state may include a regular index
 that contains the feature's execution history. Storing this history in a regular
 that contains the feature's execution history. Storing this history in a regular
 index lets you more easily search it.
 index lets you more easily search it.
 
 
+In {es} 8.0 and later versions, feature states are the only way to back up and
+restore system indices and system data streams.
+
 [discrete]
 [discrete]
 [[how-snapshots-work]]
 [[how-snapshots-work]]
 == How snapshots work
 == How snapshots work

+ 4 - 38
docs/reference/snapshot-restore/restore-snapshot.asciidoc

@@ -82,10 +82,10 @@ GET _snapshot/my_repository/*?verbose=false
 You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the
 You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the
 <<restore-snapshot-api,restore snapshot API>>.
 <<restore-snapshot-api,restore snapshot API>>.
 
 
-By default, a restore request attempts to restore all indices and data streams
-in a snapshot, including <<system-indices,system indices and system data
-streams>>. In most cases, you only need to restore a specific index or data
-stream from a snapshot. However, you can't restore an existing open index.
+By default, a restore request attempts to restore all regular indices and
+regular data streams in a snapshot. In most cases, you only need to restore a
+specific index or data stream from a snapshot. However, you can't restore an
+existing open index.
 
 
 If you're restoring data to a pre-existing cluster, use one of the
 If you're restoring data to a pre-existing cluster, use one of the
 following methods to avoid conflicts with existing indices and data streams:
 following methods to avoid conflicts with existing indices and data streams:
@@ -93,10 +93,6 @@ following methods to avoid conflicts with existing indices and data streams:
 * <<delete-restore>>
 * <<delete-restore>>
 * <<rename-on-restore>>
 * <<rename-on-restore>>
 
 
-Some {es} features automatically create system indices on cluster startup. To
-avoid conflicts with these system indices when restoring data to a new cluster,
-see <<restore-exclude-system-indices>>.
-
 [discrete]
 [discrete]
 [[delete-restore]]
 [[delete-restore]]
 ==== Delete and restore
 ==== Delete and restore
@@ -210,37 +206,12 @@ POST _reindex
 ----
 ----
 // TEST[continued]
 // TEST[continued]
 
 
-[discrete]
-[[restore-exclude-system-indices]]
-==== Exclude system indices
-
-Some {es} features, such as the <<geoip-processor,GeoIP processor>>,
-automatically create system indices at startup. To avoid naming conflicts with
-these indices, use the `-.*` wildcard pattern to exclude system indices and
-other dot (`.`) indices from your restore request.
-
-For example, the following request uses the `*,-.*` wildcard pattern to restore
-all indices and data streams except dot indices.
-
-[source,console]
-----
-POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
-{
-  "indices": "*,-.*"
-}
-----
-// TEST[setup:setup-snapshots]
-// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
-// TEST[s/_restore/_restore?wait_for_completion=true/]
-
 [discrete]
 [discrete]
 [[restore-feature-state]]
 [[restore-feature-state]]
 === Restore a feature state
 === Restore a feature state
 
 
 You can restore a <<feature-state,feature state>> to recover system indices,
 You can restore a <<feature-state,feature state>> to recover system indices,
 system data streams, and other configuration data for a feature from a snapshot.
 system data streams, and other configuration data for a feature from a snapshot.
-Restoring a feature state is the preferred way to restore system indices and
-system data streams.
 
 
 If you restore a snapshot's cluster state, the operation restores all feature
 If you restore a snapshot's cluster state, the operation restores all feature
 states in the snapshot by default. Similarly, if you don't restore a snapshot's
 states in the snapshot by default. Similarly, if you don't restore a snapshot's
@@ -271,21 +242,16 @@ Console before restoring the `security` feature state. If you run {es} on your
 own hardware, <<restore-create-file-realm-user,create a superuser in the file
 own hardware, <<restore-create-file-realm-user,create a superuser in the file
 realm>> to ensure you'll still be able to access your cluster.
 realm>> to ensure you'll still be able to access your cluster.
 
 
-To avoid accidentally restoring system indices, system data streams, and other
-dot indices, append the `-.*` wildcard pattern to the `indices` value.
-
 [source,console]
 [source,console]
 ----
 ----
 POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
 {
 {
-  "indices": "*,-.*",
   "feature_states": [ "geoip" ]
   "feature_states": [ "geoip" ]
 }
 }
 ----
 ----
 // TEST[setup:setup-snapshots]
 // TEST[setup:setup-snapshots]
 // TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
 // TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
 // TEST[s/_restore/_restore?wait_for_completion=true/]
 // TEST[s/_restore/_restore?wait_for_completion=true/]
-// TEST[s/",/"/]
 // TEST[s/"feature_states": \[ "geoip" \]//]
 // TEST[s/"feature_states": \[ "geoip" \]//]
 
 
 [discrete]
 [discrete]

+ 4 - 7
docs/reference/snapshot-restore/take-snapshot.asciidoc

@@ -207,9 +207,7 @@ PUT _slm/policy/nightly-snapshots
     naming conflicts, the policy also appends a UUID to each snapshot name.
     naming conflicts, the policy also appends a UUID to each snapshot name.
 <3> <<snapshots-register-repository,Registered snapshot repository>> used to
 <3> <<snapshots-register-repository,Registered snapshot repository>> used to
     store the policy's snapshots.
     store the policy's snapshots.
-<4> Data streams and indices to include in the policy's snapshots. This
-    configuration includes all data streams and indices, including system
-    indices.
+<4> Data streams and indices to include in the policy's snapshots.
 <5> If `true`, the policy's snapshots include the cluster state. This also
 <5> If `true`, the policy's snapshots include the cluster state. This also
     includes all feature states by default. To only include specific feature
     includes all feature states by default. To only include specific feature
     states, see <<back-up-specific-feature-state>>.
     states, see <<back-up-specific-feature-state>>.
@@ -498,7 +496,7 @@ PUT _slm/policy/nightly-cluster-state-snapshots
 <2> Excludes regular data streams and indices.
 <2> Excludes regular data streams and indices.
 
 
 If you take dedicated snapshots of the cluster state, you'll need to exclude the
 If you take dedicated snapshots of the cluster state, you'll need to exclude the
-cluster state and system indices from your other snapshots. For example:
+cluster state from your other snapshots. For example:
 
 
 [source,console]
 [source,console]
 ----
 ----
@@ -509,7 +507,7 @@ PUT _slm/policy/nightly-snapshots
   "repository": "my_repository",
   "repository": "my_repository",
   "config": {
   "config": {
     "include_global_state": false,    <1>
     "include_global_state": false,    <1>
-    "indices": "*,-.*"                <2>
+    "indices": "*"                    <2>
   },
   },
   "retention": {
   "retention": {
     "expire_after": "30d",
     "expire_after": "30d",
@@ -521,8 +519,7 @@ PUT _slm/policy/nightly-snapshots
 
 
 <1> Excludes the cluster state. This also excludes all feature states by
 <1> Excludes the cluster state. This also excludes all feature states by
     default.
     default.
-<2> Includes all data streams and indices except system indices and other
-    indices that begin with a dot (`.`).
+<2> Includes all regular data streams and indices.
 
 
 [discrete]
 [discrete]
 [[create-snapshots-different-time-intervals]]
 [[create-snapshots-different-time-intervals]]