|
|
@@ -0,0 +1,281 @@
|
|
|
+[[indices-synced-flush-api]]
|
|
|
+=== Synced flush API
|
|
|
+++++
|
|
|
+<titleabbrev>Synced flush</titleabbrev>
|
|
|
+++++
|
|
|
+
|
|
|
+Performs a synced flush on one or more indices.
|
|
|
+
|
|
|
+[source,console]
|
|
|
+--------------------------------------------------
|
|
|
+POST /twitter/_flush/synced
|
|
|
+--------------------------------------------------
|
|
|
+// TEST[setup:twitter]
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-request]]
|
|
|
+==== {api-request-title}
|
|
|
+
|
|
|
+`POST /<index>/flush/synced`
|
|
|
+
|
|
|
+`GET /<index>/flush/synced`
|
|
|
+
|
|
|
+`POST /flush/synced`
|
|
|
+
|
|
|
+`GET /flush/synced`
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-desc]]
|
|
|
+==== {api-description-title}
|
|
|
+
|
|
|
+[[synced-flush-using-api]]
|
|
|
+===== Use the synced flush API
|
|
|
+
|
|
|
+Use the synced flush API to manually initiate a synced flush.
|
|
|
+This can be useful for a planned cluster restart where
|
|
|
+you can stop indexing but don't want to wait for 5 minutes until all indices
|
|
|
+are marked as inactive and automatically sync-flushed.
|
|
|
+
|
|
|
+You can request a synced flush even if there is ongoing indexing activity, and
|
|
|
+{es} will perform the synced flush on a "best-effort" basis: shards that do not
|
|
|
+have any ongoing indexing activity will be successfully sync-flushed, and other
|
|
|
+shards will fail to sync-flush. The successfully sync-flushed shards will have
|
|
|
+faster recovery times as long as the `sync_id` marker is not removed by a
|
|
|
+subsequent flush.
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-overview]]
|
|
|
+===== Synced flush overview
|
|
|
+
|
|
|
+{es} keeps track of which shards have received indexing activity recently, and
|
|
|
+considers shards that have not received any indexing operations for 5 minutes to
|
|
|
+be inactive.
|
|
|
+
|
|
|
+When a shard becomes inactive {es} performs a special kind of flush
|
|
|
+known as a *synced flush*. A synced flush performs a normal
|
|
|
+<<indices-flush,flush>> on each replica of the shard, and then adds a marker known
|
|
|
+as the `sync_id` to each replica to indicate that these copies have identical
|
|
|
+Lucene indices. Comparing the `sync_id` markers of the two copies is a very
|
|
|
+efficient way to check whether they have identical contents.
|
|
|
+
|
|
|
+When allocating shard replicas, {es} must ensure that each replica contains the
|
|
|
+same data as the primary. If the shard copies have been synced-flushed and the
|
|
|
+replica shares a `sync_id` with the primary then {es} knows that the two copies
|
|
|
+have identical contents. This means there is no need to copy any segment files
|
|
|
+from the primary to the replica, which saves a good deal of time during
|
|
|
+recoveries and restarts.
|
|
|
+
|
|
|
+This is particularly useful for clusters having lots of indices which are very
|
|
|
+rarely updated, such as with time-based indices. Without the synced flush
|
|
|
+marker, recovery of this kind of cluster would be much slower.
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-sync-id-markers]]
|
|
|
+===== Check for `sync_id` markers
|
|
|
+
|
|
|
+To check whether a shard has a `sync_id` marker or not, look for the `commit`
|
|
|
+section of the shard stats returned by the <<indices-stats,indices stats>> API:
|
|
|
+
|
|
|
+[source,console]
|
|
|
+--------------------------------------------------
|
|
|
+GET /twitter/_stats?filter_path=**.commit&level=shards <1>
|
|
|
+--------------------------------------------------
|
|
|
+// TEST[s/^/PUT twitter\nPOST twitter\/_flush\/synced\n/]
|
|
|
+
|
|
|
+<1> `filter_path` is used to reduce the verbosity of the response, but is entirely optional
|
|
|
+
|
|
|
+The API returns the following response:
|
|
|
+
|
|
|
+[source,console-result]
|
|
|
+--------------------------------------------------
|
|
|
+{
|
|
|
+ "indices": {
|
|
|
+ "twitter": {
|
|
|
+ "shards": {
|
|
|
+ "0": [
|
|
|
+ {
|
|
|
+ "commit" : {
|
|
|
+ "id" : "3M3zkw2GHMo2Y4h4/KFKCg==",
|
|
|
+ "generation" : 3,
|
|
|
+ "user_data" : {
|
|
|
+ "translog_uuid" : "hnOG3xFcTDeoI_kvvvOdNA",
|
|
|
+ "history_uuid" : "XP7KDJGiS1a2fHYiFL5TXQ",
|
|
|
+ "local_checkpoint" : "-1",
|
|
|
+ "translog_generation" : "2",
|
|
|
+ "max_seq_no" : "-1",
|
|
|
+ "sync_id" : "AVvFY-071siAOuFGEO9P", <1>
|
|
|
+ "max_unsafe_auto_id_timestamp" : "-1",
|
|
|
+ "min_retained_seq_no" : "0"
|
|
|
+ },
|
|
|
+ "num_docs" : 0
|
|
|
+ }
|
|
|
+ }
|
|
|
+ ]
|
|
|
+ }
|
|
|
+ }
|
|
|
+ }
|
|
|
+}
|
|
|
+--------------------------------------------------
|
|
|
+// TESTRESPONSE[s/"id" : "3M3zkw2GHMo2Y4h4\/KFKCg=="/"id": $body.indices.twitter.shards.0.0.commit.id/]
|
|
|
+// TESTRESPONSE[s/"translog_uuid" : "hnOG3xFcTDeoI_kvvvOdNA"/"translog_uuid": $body.indices.twitter.shards.0.0.commit.user_data.translog_uuid/]
|
|
|
+// TESTRESPONSE[s/"history_uuid" : "XP7KDJGiS1a2fHYiFL5TXQ"/"history_uuid": $body.indices.twitter.shards.0.0.commit.user_data.history_uuid/]
|
|
|
+// TESTRESPONSE[s/"sync_id" : "AVvFY-071siAOuFGEO9P"/"sync_id": $body.indices.twitter.shards.0.0.commit.user_data.sync_id/]
|
|
|
+<1> the `sync id` marker
|
|
|
+
|
|
|
+NOTE: The `sync_id` marker is removed as soon as the shard is flushed again, and
|
|
|
+{es} may trigger an automatic flush of a shard at any time if there are
|
|
|
+unflushed operations in the shard's translog. In practice this means that one
|
|
|
+should consider any indexing operation on an index as having removed its
|
|
|
+`sync_id` markers.
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-path-params]]
|
|
|
+==== {api-path-parms-title}
|
|
|
+
|
|
|
+include::{docdir}/rest-api/common-parms.asciidoc[tag=index]
|
|
|
++
|
|
|
+To sync-flush all indices,
|
|
|
+omit this parameter
|
|
|
+or use a value of `_all` or `*`.
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-query-params]]
|
|
|
+==== {api-query-parms-title}
|
|
|
+
|
|
|
+include::{docdir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
|
|
|
+
|
|
|
+include::{docdir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
|
|
|
++
|
|
|
+Defaults to `open`.
|
|
|
+
|
|
|
+include::{docdir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-response-codes]]
|
|
|
+==== {api-response-codes-title}
|
|
|
+
|
|
|
+`200`::
|
|
|
+All shards successfully sync-flushed.
|
|
|
+
|
|
|
+`409`::
|
|
|
+A replica shard failed to sync-flush.
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-example]]
|
|
|
+==== {api-examples-title}
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-specific-ex]]
|
|
|
+===== Sync-flush a specific index
|
|
|
+
|
|
|
+[source,console]
|
|
|
+----
|
|
|
+POST /kimchy/_flush
|
|
|
+----
|
|
|
+// TEST[s/^/PUT kimchy\n/]
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-multi-ex]]
|
|
|
+===== Synch-flush several indices
|
|
|
+
|
|
|
+[source,console]
|
|
|
+--------------------------------------------------
|
|
|
+POST /kimchy,elasticsearch/_flush/synced
|
|
|
+--------------------------------------------------
|
|
|
+// TEST[s/^/PUT elasticsearch\n/]
|
|
|
+// TEST[continued]
|
|
|
+
|
|
|
+
|
|
|
+[[synced-flush-api-all-ex]]
|
|
|
+===== Sync-flush all indices
|
|
|
+
|
|
|
+[source,console]
|
|
|
+--------------------------------------------------
|
|
|
+POST /_flush/synced
|
|
|
+--------------------------------------------------
|
|
|
+// TEST[setup:twitter]
|
|
|
+
|
|
|
+The response contains details about how many shards were successfully
|
|
|
+sync-flushed and information about any failure.
|
|
|
+
|
|
|
+The following response indicates two shards
|
|
|
+and one replica shard
|
|
|
+successfully sync-flushed:
|
|
|
+
|
|
|
+[source,console-result]
|
|
|
+--------------------------------------------------
|
|
|
+{
|
|
|
+ "_shards": {
|
|
|
+ "total": 2,
|
|
|
+ "successful": 2,
|
|
|
+ "failed": 0
|
|
|
+ },
|
|
|
+ "twitter": {
|
|
|
+ "total": 2,
|
|
|
+ "successful": 2,
|
|
|
+ "failed": 0
|
|
|
+ }
|
|
|
+}
|
|
|
+--------------------------------------------------
|
|
|
+// TESTRESPONSE[s/"successful": 2/"successful": 1/]
|
|
|
+
|
|
|
+The following response indicates one shard group failed
|
|
|
+due to pending operations:
|
|
|
+
|
|
|
+[source,js]
|
|
|
+--------------------------------------------------
|
|
|
+{
|
|
|
+ "_shards": {
|
|
|
+ "total": 4,
|
|
|
+ "successful": 2,
|
|
|
+ "failed": 2
|
|
|
+ },
|
|
|
+ "twitter": {
|
|
|
+ "total": 4,
|
|
|
+ "successful": 2,
|
|
|
+ "failed": 2,
|
|
|
+ "failures": [
|
|
|
+ {
|
|
|
+ "shard": 1,
|
|
|
+ "reason": "[2] ongoing operations on primary"
|
|
|
+ }
|
|
|
+ ]
|
|
|
+ }
|
|
|
+}
|
|
|
+--------------------------------------------------
|
|
|
+// NOTCONSOLE
|
|
|
+
|
|
|
+Sometimes the failures are specific to a shard replica. The copies that failed
|
|
|
+will not be eligible for fast recovery but those that succeeded still will be.
|
|
|
+This case is reported as follows:
|
|
|
+
|
|
|
+[source,js]
|
|
|
+--------------------------------------------------
|
|
|
+{
|
|
|
+ "_shards": {
|
|
|
+ "total": 4,
|
|
|
+ "successful": 1,
|
|
|
+ "failed": 1
|
|
|
+ },
|
|
|
+ "twitter": {
|
|
|
+ "total": 4,
|
|
|
+ "successful": 3,
|
|
|
+ "failed": 1,
|
|
|
+ "failures": [
|
|
|
+ {
|
|
|
+ "shard": 1,
|
|
|
+ "reason": "unexpected error",
|
|
|
+ "routing": {
|
|
|
+ "state": "STARTED",
|
|
|
+ "primary": false,
|
|
|
+ "node": "SZNr2J_ORxKTLUCydGX4zA",
|
|
|
+ "relocating_node": null,
|
|
|
+ "shard": 1,
|
|
|
+ "index": "twitter"
|
|
|
+ }
|
|
|
+ }
|
|
|
+ ]
|
|
|
+ }
|
|
|
+}
|
|
|
+--------------------------------------------------
|
|
|
+// NOTCONSOLE
|
James Rodewig