Browse Source

[DOCS] Update size your shards for `max_primary_shard_size` (#71367)

James Rodewig 4 years ago
parent
commit
1f361ad2d1
1 changed files with 39 additions and 19 deletions
  1. 39 19
      docs/reference/how-to/size-your-shards.asciidoc

+ 39 - 19
docs/reference/how-to/size-your-shards.asciidoc

@@ -105,10 +105,9 @@ image:images/ilm/index-lifecycle-policies.png[]
 
 
 One advantage of this setup is
 One advantage of this setup is
 <<getting-started-index-lifecycle-management,automatic rollover>>, which creates
 <<getting-started-index-lifecycle-management,automatic rollover>>, which creates
-a new write index when the current one meets a defined `max_age`, `max_docs`, or
-`max_size` threshold. You can use these thresholds to create indices based on
-your retention intervals. When an index is no longer needed, you can use
-{ilm-init} to automatically delete it and free up resources.
+a new write index when the current one meets a defined `max_primary_shard_size`,
+`max_age`, `max_docs`, or `max_size` threshold. When an index is no longer
+needed, you can use {ilm-init} to automatically delete it and free up resources.
 
 
 {ilm-init} also makes it easy to change your sharding strategy over time:
 {ilm-init} also makes it easy to change your sharding strategy over time:
 
 
@@ -129,12 +128,36 @@ Every new backing index is an opportunity to further tune your strategy.
 
 
 [discrete]
 [discrete]
 [[shard-size-recommendation]]
 [[shard-size-recommendation]]
-==== Aim for shard sizes between 10GB and 50GB
+==== Aim for shard sizes between 10GB and 65GB
 
 
-Shards larger than 50GB may make a cluster less likely to recover from failure.
+Shards larger than 65GB may make a cluster less likely to recover from failure.
 When a node fails, {es} rebalances the node's shards across the data tier's
 When a node fails, {es} rebalances the node's shards across the data tier's
-remaining nodes. Shards larger than 50GB can be harder to move across a network
-and may tax node resources.
+remaining nodes. Larger shards can be harder to move across a network and may
+tax node resources.
+
+If you use {ilm-init}, set the <<ilm-rollover,rollover action>>'s
+`max_primary_shard_size` threshold to `65gb` to avoid larger shards.
+
+To see the current size of your shards, use the <<cat-shards,_cat shards API>>.
+
+[source,console]
+----
+GET _cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb
+----
+// TEST[setup:my_index]
+
+The `pri.store.size` value shows the combined size of all primary shards for
+the index.
+
+[source,txt]
+----
+index                                 prirep shard store
+.ds-my-data-stream-2099.05.06-000001  p      0      65gb
+...
+----
+// TESTRESPONSE[non_json]
+// TESTRESPONSE[s/\.ds-my-data-stream-2099\.05\.06-000001/my-index-000001/]
+// TESTRESPONSE[s/65gb/.*/]
 
 
 [discrete]
 [discrete]
 [[shard-count-recommendation]]
 [[shard-count-recommendation]]
@@ -198,19 +221,16 @@ If your cluster is experiencing stability issues due to oversharded indices,
 you can use one or more of the following methods to fix them.
 you can use one or more of the following methods to fix them.
 
 
 [discrete]
 [discrete]
-[[reindex-indices-from-shorter-periods-into-longer-periods]]
-==== Create time-based indices that cover longer periods
-
-For time series data, you can create indices that cover longer time intervals.
-For example, instead of daily indices, you can create indices on a monthly or
-yearly basis.
+[[create-indices-that-cover-longer-time-periods]]
+==== Create indices that cover longer time periods
 
 
-If you're using {ilm-init}, you can do this by increasing the `max_age`
-threshold for the <<ilm-rollover,rollover action>>.
+If you use {ilm-init} and your retention policy allows it, avoid using a
+`max_age` threshold for the rollover action. Instead, use
+`max_primary_shard_size` to avoid creating empty indices or many small shards.
 
 
-If your retention policy allows it, you can also create larger indices by
-omitting a `max_age` threshold and using `max_docs` and/or `max_size`
-thresholds instead.
+If your retention policy requires a `max_age` threshold, increase it to create
+indices that cover longer time intervals. For example, instead of creating daily
+indices, you can create indices on a weekly or monthly basis.
 
 
 [discrete]
 [discrete]
 [[delete-empty-indices]]
 [[delete-empty-indices]]