elasticsearch/docs/reference/ccr/getting-started.asciidoc

[role="xpack"]
[testenv="platinum"]
[[ccr-getting-started]]
=== Set up {ccr}
You can manually create follower indices to replicate specific indices on a
remote cluster, or configure auto-follow patterns to automatically create
follower indices for new time series.

After the follower index is created, the
<<ccr-remote-recovery, remote recovery>> process copies all of the Lucene
segment files from the remote cluster to the local cluster.

To set up {ccr}:

. <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote cluster>>
. <<ccr-getting-started-leader-index,Identify the index (or time series indices) you want to replicate on the remote cluster>>
. <<ccr-enable-soft-deletes,Enable soft deletes on the leader index>>
. Manually create a follower index or create an auto-follow pattern:
  * To replicate the leader index, <<ccr-getting-started-follower-index,manually create a follower index>>
  * To automatically follow time series indices, <<ccr-getting-started-auto-follow,create an auto-follow pattern>>

[[ccr-getting-started-prerequisites]]
==== Prerequisites
If the Elastic {security-features} are enabled in your local and remote
clusters, you need a user with appropriate authority to complete the steps
in this tutorial.

By default, you can complete the following steps as the built-in
`elastic` user. However, you must <<get-started-built-in-users,set a password>>
for this user before proceeding.

WARNING: If you are performing these steps in a production environment, do
not use the `elastic` user.

Alternatively, you can assign the appropriate privileges to a user ID of your
choice. On the remote cluster that contains the leader index, a user must have
the `read_ccr` cluster privilege and `monitor` and `read` privileges on the
leader index.

[source,yml]
--------------------------------------------------
ccr_user:
  cluster:
    - read_ccr
  indices:
    - names: [ 'leader-index' ]
      privileges:
        - monitor
        - read
--------------------------------------------------

On the local cluster that contains the follower index, the same user will need
the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
`manage_follow_index` privileges on the follower index.

[source,yml]
--------------------------------------------------
ccr_user:
  cluster:
    - manage_ccr
  indices:
    - names: [ 'follower-index' ]
      privileges:
        - monitor
        - read
        - write
        - manage_follow_index
--------------------------------------------------

If you are managing
<<ccr-getting-started-remote-cluster,connecting to the remote cluster>> using
the cluster update settings API, you will also need a user with the `all`
cluster privilege.

[[ccr-getting-started-remote-cluster]]
==== Connect to a remote cluster
Connect your local cluster to a
<<modules-remote-clusters,remote cluster>> to begin using cross-cluster
replication.

To configure a {kibana-ref}/working-remote-clusters.html[remote cluster],
access {kib} and go to
*Management > Stack Management*. In the side navigation, select
*Remote Clusters*.

Add a remote cluster by specifying the IP address or host name, followed by the
transport port of the remote cluster.

[role="screenshot"]
image::images/ccr-add-remote-cluster.png["The Add remote clusters page in {kib}"]

[%collapsible]
.API example
====
Use the <<cluster-update-settings,cluster update settings API>> to add a remote cluster:

[source,console]
--------------------------------------------------
PUT /_cluster/settings
{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "leader" : {
          "seeds" : [
            "127.0.0.1:9300" <1>
          ]
        }
      }
    }
  }
}
--------------------------------------------------
// TEST[setup:host]
// TEST[s/127.0.0.1:9300/\${transport_host}/]
<1> Specifies the hostname and transport port of a seed node in the remote
    cluster.

You can verify that the local cluster is successfully connected to the remote
cluster.

[source,console]
--------------------------------------------------
GET /_remote/info
--------------------------------------------------
// TEST[continued]

The API will respond by showing that the local cluster is connected to the
remote cluster.

[source,console-result]
--------------------------------------------------
{
  "leader" : {
    "seeds" : [
      "127.0.0.1:9300"
    ],
    "connected" : true, <1>
    "num_nodes_connected" : 1, <2>
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false,
    "mode" : "sniff"
  }
}
--------------------------------------------------
// TESTRESPONSE[s/127.0.0.1:9300/$body.leader.seeds.0/]
// TEST[s/"connected" : true/"connected" : $body.leader.connected/]
// TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/]
<1> This shows the local cluster is connected to the remote cluster with cluster
    alias `leader`
<2> This shows the number of nodes in the remote cluster the local cluster is
    connected to.
====

[[ccr-getting-started-leader-index]]
==== Create a leader index
To create a leader index, access {kib} on your _remote_ cluster and go to
*Management > Dev Tools*.

Copy the following example into the Console to create a leader index named
`server-metrics` in your remote cluster:

[%collapsible]
.Leader index example
====
[source,console]
--------------------------------------------------
PUT /server-metrics
{
  "settings" : {
    "index" : {
      "number_of_shards" : 1,
      "number_of_replicas" : 0
    }
  },
  "mappings" : {
    "properties" : {
      "@timestamp" : {
        "type" : "date"
      },
      "accept" : {
        "type" : "long"
      },
      "deny" : {
        "type" : "long"
      },
      "host" : {
        "type" : "keyword"
      },
      "response" : {
        "type" : "float"
      },
      "service" : {
        "type" : "keyword"
      },
      "total" : {
        "type" : "long"
      }
    }
  }
}
--------------------------------------------------
// TEST[continued]
====

[[ccr-enable-soft-deletes]]
==== Enable soft deletes on leader indices
<<ccr-leader-requirements,Soft deletes>> must be enabled for indices that you want to
use as leader indices. Soft deletes are enabled by default on new indices
created on or after {es} 7.0.0, so
*no further action is required if your cluster is running {es} 7.0.0 or later*.

include::{es-ref-dir}/ccr/index.asciidoc[tag=ccr-existing-indices-tag]

To enable soft deletes on indices created on versions of
{es} between 6.5.0 and 7.0.0, set <<ccr-index-soft-deletes,`index.soft_deletes.enabled`>> to `true`.

[[ccr-getting-started-follower-index]]
==== Create a follower index
When you create a {kibana-ref}/managing-cross-cluster-replication.html#_create_specific_follower_indices[follower index], you
must reference the
<<ccr-getting-started-remote-cluster,remote cluster>> and the
<<ccr-getting-started-leader-index,leader index>> that you created in the remote
cluster.

To create a follower index, access {kib} and go to
*Management > Stack Management*. In the side navigation, select
*Cross-Cluster Replication* and choose the *Follower Indices* tab.

. Choose the remote cluster containing the index you want to replicate, which
is `leader` if you are following the tutorial.
. Enter the name of the leader index, which is `server-metrics` if you are
following the tutorial.

image::images/ccr-add-follower-index.png["Adding a follower index named server-metrics in {kib}"]

The follower index is initialized using the
<<ccr-remote-recovery, remote recovery>>
process, which transfers the existing Lucene segment files from the leader
index to the follower index. The index status changes to *Paused*. When the
remote recovery process is complete, the index following begins and the status
changes to *Active*.

When you index documents into your leader index, the documents are replicated
in the follower index.

[role="screenshot"]
image::images/ccr-follower-index.png["The Cross-Cluster Replication page in {kib}"]

[%collapsible]
.API example
====
Use the <<ccr-put-follow,create follower API>> to create follower indices.
When you create a follower index, you must reference the
<<ccr-getting-started-remote-cluster,remote cluster>> and the
<<ccr-getting-started-leader-index,leader index>> that you created in the
remote cluster.

When initiating the follower request, the response returns before the
<<ccr-remote-recovery, remote recovery>> process completes. To wait for the process
to complete, add the `wait_for_active_shards` parameter to your request.

[source,console]
--------------------------------------------------
PUT /server-metrics-follower/_ccr/follow?wait_for_active_shards=1
{
  "remote_cluster" : "leader",
  "leader_index" : "server-metrics"
}
--------------------------------------------------
// TEST[continued]

//////////////////////////

[source,console-result]
--------------------------------------------------
{
  "follow_index_created" : true,
  "follow_index_shards_acked" : true,
  "index_following_started" : true
}
--------------------------------------------------

//////////////////////////

Use the
<<ccr-get-follow-stats,get follower stats API>> to inspect the status of
replication

//////////////////////////

[source,console]
--------------------------------------------------
POST /server-metrics-follower/_ccr/pause_follow

POST /server-metrics-follower/_close

POST /server-metrics-follower/_ccr/unfollow
--------------------------------------------------
// TEST[continued]

//////////////////////////
====

[[ccr-getting-started-auto-follow]]
==== Automatically create follower indices
Create <<ccr-auto-follow,auto-follow patterns>> to automatically follow time
series indices that are periodically created in a remote cluster (such as daily
{beats} indices).

With an auto-follow pattern, you reference the
<<ccr-getting-started-remote-cluster,remote cluster>> connected to your
local cluster. You must also specify a collection of patterns that match the
indices you want to automatically follow.

// tag::ccr-create-auto-follow-pattern-tag[]
To create follower indices from an {kibana-ref}/managing-cross-cluster-replication.html#_create_follower_indices_from_an_auto_follow_pattern[auto-follow pattern],
access {kib} on your remote cluster and go to
*Management > Stack Management*. In the side navigation, select
*Cross Cluster Replication* and choose the *Auto-follow patterns* tab.

[role="screenshot"]
image::images/auto-follow-patterns.png["The Auto-follow patterns page in {kib}"]

* Enter a name for the auto-follow pattern. For this tutorial, enter `beats`
as the name.
* Choose the remote cluster containing the index you want to replicate, which
is `leader` if you are following the tutorial.
* Enter one or more index patterns that identify the indices you want to
replicate from the remote cluster. For this tutorial, enter
`metricbeat-*,packetbeat-*` as the index pattern.
* Enter *copy-* as the prefix to apply to the names of the follower indices so
you can more easily identify replicated indices.

As new indices matching these patterns are
created, they are replicated to the follower indices.

// end::ccr-create-auto-follow-pattern-tag[]

[%collapsible]
.API example
====
Use the <<ccr-put-auto-follow-pattern,create auto-follow pattern API>> to
configure auto-follow patterns.

[source,console]
--------------------------------------------------
PUT /_ccr/auto_follow/beats
{
  "remote_cluster" : "leader",
  "leader_index_patterns" :
  [
    "metricbeat-*", <1>
    "packetbeat-*" <2>
  ],
  "follow_index_pattern" : "{{leader_index}}-copy" <3>
}
--------------------------------------------------
// TEST[continued]
<1> Automatically follow new {metricbeat} indices.
<2> Automatically follow new {packetbeat} indices.
<3> The name of the follower index is derived from the name of the leader index
    by adding the suffix `-copy` to the name of the leader index.

//////////////////////////

[source,console-result]
--------------------------------------------------
{
  "acknowledged" : true
}
--------------------------------------------------

//////////////////////////

//////////////////////////

[source,console]
--------------------------------------------------
DELETE /_ccr/auto_follow/beats
--------------------------------------------------
// TEST[continued]

//////////////////////////
====