getting-started.asciidoc 9.2 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318
  1. [role="xpack"]
  2. [testenv="platinum"]
  3. [[ccr-getting-started]]
  4. == Getting started with {ccr}
  5. This getting-started guide for {ccr} shows you how to:
  6. * <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote
  7. cluster>>
  8. * <<ccr-getting-started-leader-index,Create a leader index>> in a remote cluster
  9. * <<ccr-getting-started-follower-index,Create a follower index>> that replicates
  10. a leader index
  11. * <<ccr-getting-started-auto-follow,Automatically create follower indices>>
  12. [float]
  13. [[ccr-getting-started-before-you-begin]]
  14. === Before you begin
  15. . {stack-gs}/get-started-elastic-stack.html#install-elasticsearch[Install {es}]
  16. on your local and remote clusters.
  17. . Obtain a license that includes the {ccr} features. See
  18. https://www.elastic.co/subscriptions[subscriptions] and
  19. <<license-management>>.
  20. . If the Elastic {security-features} are enabled in your local and remote
  21. clusters, you need a user that has appropriate authority to perform the steps
  22. in this tutorial.
  23. +
  24. --
  25. [[ccr-getting-started-security]]
  26. The {ccr} features use cluster privileges and built-in roles to make it easier
  27. to control which users have authority to manage {ccr}.
  28. By default, you can perform all of the steps in this tutorial by
  29. using the built-in `elastic` user. However, a password must be set for this user
  30. before the user can do anything. For information about how to set that password,
  31. see <<security-getting-started>>.
  32. If you are performing these steps in a production environment, take extra care
  33. because the `elastic` user has the `superuser` role and you could inadvertently
  34. make significant changes.
  35. Alternatively, you can assign the appropriate privileges to a user ID of your
  36. choice. On the remote cluster that contains the leader index, a user will need
  37. the `read_ccr` cluster privilege and `monitor` and `read` privileges on the
  38. leader index.
  39. [source,yml]
  40. --------------------------------------------------
  41. ccr_user:
  42. cluster:
  43. - read_ccr
  44. indices:
  45. - names: [ 'leader-index' ]
  46. privileges:
  47. - monitor
  48. - read
  49. --------------------------------------------------
  50. On the local cluster that contains the follower index, the same user will need
  51. the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
  52. `manage_follow_index` privileges on the follower index.
  53. [source,yml]
  54. --------------------------------------------------
  55. ccr_user:
  56. cluster:
  57. - manage_ccr
  58. indices:
  59. - names: [ 'follower-index' ]
  60. privileges:
  61. - monitor
  62. - read
  63. - write
  64. - manage_follow_index
  65. --------------------------------------------------
  66. If you are managing
  67. <<ccr-getting-started-remote-cluster,connecting to the remote cluster>> via the
  68. cluster update settings API, you will also need a user with the `all` cluster
  69. privilege.
  70. --
  71. [float]
  72. [[ccr-getting-started-remote-cluster]]
  73. === Connecting to a remote cluster
  74. The {ccr} features require that you
  75. {ref}/modules-remote-clusters.html[connect your local cluster to a remote
  76. cluster]. In this tutorial, we will connect our local cluster to a remote
  77. cluster with the cluster alias `leader`.
  78. [source,console]
  79. --------------------------------------------------
  80. PUT /_cluster/settings
  81. {
  82. "persistent" : {
  83. "cluster" : {
  84. "remote" : {
  85. "leader" : {
  86. "seeds" : [
  87. "127.0.0.1:9300" <1>
  88. ]
  89. }
  90. }
  91. }
  92. }
  93. }
  94. --------------------------------------------------
  95. // TEST[setup:host]
  96. // TEST[s/127.0.0.1:9300/\${transport_host}/]
  97. <1> Specifies the hostname and transport port of a seed node in the remote
  98. cluster.
  99. You can verify that the local cluster is successfully connected to the remote
  100. cluster.
  101. [source,console]
  102. --------------------------------------------------
  103. GET /_remote/info
  104. --------------------------------------------------
  105. // TEST[continued]
  106. The API will respond by showing that the local cluster is connected to the
  107. remote cluster.
  108. [source,console-result]
  109. --------------------------------------------------
  110. {
  111. "leader" : {
  112. "seeds" : [
  113. "127.0.0.1:9300"
  114. ],
  115. "connected" : true, <1>
  116. "num_nodes_connected" : 1, <2>
  117. "max_connections_per_cluster" : 3,
  118. "initial_connect_timeout" : "30s",
  119. "skip_unavailable" : false
  120. }
  121. }
  122. --------------------------------------------------
  123. // TESTRESPONSE[s/127.0.0.1:9300/$body.leader.seeds.0/]
  124. // TEST[s/"connected" : true/"connected" : $body.leader.connected/]
  125. // TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/]
  126. <1> This shows the local cluster is connected to the remote cluster with cluster
  127. alias `leader`
  128. <2> This shows the number of nodes in the remote cluster the local cluster is
  129. connected to.
  130. Alternatively, you can manage remote clusters on the
  131. *Management / Elasticsearch / Remote Clusters* page in {kib}:
  132. [role="screenshot"]
  133. image::images/remote-clusters.jpg["The Remote Clusters page in {kib}"]
  134. [float]
  135. [[ccr-getting-started-leader-index]]
  136. === Creating a leader index
  137. In the following example, we will create a leader index in the remote cluster:
  138. [source,console]
  139. --------------------------------------------------
  140. PUT /server-metrics
  141. {
  142. "settings" : {
  143. "index" : {
  144. "number_of_shards" : 1,
  145. "number_of_replicas" : 0
  146. }
  147. },
  148. "mappings" : {
  149. "properties" : {
  150. "@timestamp" : {
  151. "type" : "date"
  152. },
  153. "accept" : {
  154. "type" : "long"
  155. },
  156. "deny" : {
  157. "type" : "long"
  158. },
  159. "host" : {
  160. "type" : "keyword"
  161. },
  162. "response" : {
  163. "type" : "float"
  164. },
  165. "service" : {
  166. "type" : "keyword"
  167. },
  168. "total" : {
  169. "type" : "long"
  170. }
  171. }
  172. }
  173. }
  174. --------------------------------------------------
  175. // TEST[continued]
  176. [float]
  177. [[ccr-getting-started-follower-index]]
  178. === Creating a follower index
  179. Follower indices are created with the {ref}/ccr-put-follow.html[create follower
  180. API]. When you create a follower index, you must reference the
  181. <<ccr-getting-started-remote-cluster,remote cluster>> and the
  182. <<ccr-getting-started-leader-index,leader index>> that you created in the remote
  183. cluster.
  184. [source,console]
  185. --------------------------------------------------
  186. PUT /server-metrics-copy/_ccr/follow?wait_for_active_shards=1
  187. {
  188. "remote_cluster" : "leader",
  189. "leader_index" : "server-metrics"
  190. }
  191. --------------------------------------------------
  192. // TEST[continued]
  193. //////////////////////////
  194. [source,console-result]
  195. --------------------------------------------------
  196. {
  197. "follow_index_created" : true,
  198. "follow_index_shards_acked" : true,
  199. "index_following_started" : true
  200. }
  201. --------------------------------------------------
  202. //////////////////////////
  203. The follower index is initialized using the <<remote-recovery, remote recovery>>
  204. process. The remote recovery process transfers the existing Lucene segment files
  205. from the leader to the follower. When the remote recovery process is complete,
  206. the index following begins.
  207. Now when you index documents into your leader index, you will see these
  208. documents replicated in the follower index. You can
  209. inspect the status of replication using the
  210. {ref}/ccr-get-follow-stats.html[get follower stats API].
  211. //////////////////////////
  212. [source,console]
  213. --------------------------------------------------
  214. POST /server-metrics-copy/_ccr/pause_follow
  215. POST /server-metrics-copy/_close
  216. POST /server-metrics-copy/_ccr/unfollow
  217. --------------------------------------------------
  218. // TEST[continued]
  219. //////////////////////////
  220. [float]
  221. [[ccr-getting-started-auto-follow]]
  222. === Automatically create follower indices
  223. The <<ccr-auto-follow,auto-follow>> feature in {ccr} helps for time series use
  224. cases where you want to follow new indices that are periodically created in the
  225. remote cluster (such as daily Beats indices). Auto-following is configured using
  226. the {ref}/ccr-put-auto-follow-pattern.html[create auto-follow pattern API]. With
  227. an auto-follow pattern, you reference the
  228. <<ccr-getting-started-remote-cluster,remote cluster>> that you connected your
  229. local cluster to. You must also specify a collection of patterns that match the
  230. indices you want to automatically follow.
  231. For example:
  232. [source,console]
  233. --------------------------------------------------
  234. PUT /_ccr/auto_follow/beats
  235. {
  236. "remote_cluster" : "leader",
  237. "leader_index_patterns" :
  238. [
  239. "metricbeat-*", <1>
  240. "packetbeat-*" <2>
  241. ],
  242. "follow_index_pattern" : "{{leader_index}}-copy" <3>
  243. }
  244. --------------------------------------------------
  245. // TEST[continued]
  246. <1> Automatically follow new {metricbeat} indices.
  247. <2> Automatically follow new {packetbeat} indices.
  248. <3> The name of the follower index is derived from the name of the leader index
  249. by adding the suffix `-copy` to the name of the leader index.
  250. //////////////////////////
  251. [source,console-result]
  252. --------------------------------------------------
  253. {
  254. "acknowledged" : true
  255. }
  256. --------------------------------------------------
  257. //////////////////////////
  258. //////////////////////////
  259. [source,console]
  260. --------------------------------------------------
  261. DELETE /_ccr/auto_follow/beats
  262. --------------------------------------------------
  263. // TEST[continued]
  264. //////////////////////////
  265. Alternatively, you can manage auto-follow patterns on the
  266. *Management / Elasticsearch / Cross Cluster Replication* page in {kib}:
  267. [role="screenshot"]
  268. image::images/auto-follow-patterns.jpg["The Auto-follow patterns page in {kib}"]