|
@@ -105,10 +105,9 @@ image:images/ilm/index-lifecycle-policies.png[]
|
|
|
|
|
|
One advantage of this setup is
|
|
|
<<getting-started-index-lifecycle-management,automatic rollover>>, which creates
|
|
|
-a new write index when the current one meets a defined `max_age`, `max_docs`, or
|
|
|
-`max_size` threshold. You can use these thresholds to create indices based on
|
|
|
-your retention intervals. When an index is no longer needed, you can use
|
|
|
-{ilm-init} to automatically delete it and free up resources.
|
|
|
+a new write index when the current one meets a defined `max_primary_shard_size`,
|
|
|
+`max_age`, `max_docs`, or `max_size` threshold. When an index is no longer
|
|
|
+needed, you can use {ilm-init} to automatically delete it and free up resources.
|
|
|
|
|
|
{ilm-init} also makes it easy to change your sharding strategy over time:
|
|
|
|
|
@@ -129,12 +128,36 @@ Every new backing index is an opportunity to further tune your strategy.
|
|
|
|
|
|
[discrete]
|
|
|
[[shard-size-recommendation]]
|
|
|
-==== Aim for shard sizes between 10GB and 50GB
|
|
|
+==== Aim for shard sizes between 10GB and 65GB
|
|
|
|
|
|
-Shards larger than 50GB may make a cluster less likely to recover from failure.
|
|
|
+Shards larger than 65GB may make a cluster less likely to recover from failure.
|
|
|
When a node fails, {es} rebalances the node's shards across the data tier's
|
|
|
-remaining nodes. Shards larger than 50GB can be harder to move across a network
|
|
|
-and may tax node resources.
|
|
|
+remaining nodes. Larger shards can be harder to move across a network and may
|
|
|
+tax node resources.
|
|
|
+
|
|
|
+If you use {ilm-init}, set the <<ilm-rollover,rollover action>>'s
|
|
|
+`max_primary_shard_size` threshold to `65gb` to avoid larger shards.
|
|
|
+
|
|
|
+To see the current size of your shards, use the <<cat-shards,_cat shards API>>.
|
|
|
+
|
|
|
+[source,console]
|
|
|
+----
|
|
|
+GET _cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb
|
|
|
+----
|
|
|
+// TEST[setup:my_index]
|
|
|
+
|
|
|
+The `pri.store.size` value shows the combined size of all primary shards for
|
|
|
+the index.
|
|
|
+
|
|
|
+[source,txt]
|
|
|
+----
|
|
|
+index prirep shard store
|
|
|
+.ds-my-data-stream-2099.05.06-000001 p 0 65gb
|
|
|
+...
|
|
|
+----
|
|
|
+// TESTRESPONSE[non_json]
|
|
|
+// TESTRESPONSE[s/\.ds-my-data-stream-2099\.05\.06-000001/my-index-000001/]
|
|
|
+// TESTRESPONSE[s/65gb/.*/]
|
|
|
|
|
|
[discrete]
|
|
|
[[shard-count-recommendation]]
|
|
@@ -198,19 +221,16 @@ If your cluster is experiencing stability issues due to oversharded indices,
|
|
|
you can use one or more of the following methods to fix them.
|
|
|
|
|
|
[discrete]
|
|
|
-[[reindex-indices-from-shorter-periods-into-longer-periods]]
|
|
|
-==== Create time-based indices that cover longer periods
|
|
|
-
|
|
|
-For time series data, you can create indices that cover longer time intervals.
|
|
|
-For example, instead of daily indices, you can create indices on a monthly or
|
|
|
-yearly basis.
|
|
|
+[[create-indices-that-cover-longer-time-periods]]
|
|
|
+==== Create indices that cover longer time periods
|
|
|
|
|
|
-If you're using {ilm-init}, you can do this by increasing the `max_age`
|
|
|
-threshold for the <<ilm-rollover,rollover action>>.
|
|
|
+If you use {ilm-init} and your retention policy allows it, avoid using a
|
|
|
+`max_age` threshold for the rollover action. Instead, use
|
|
|
+`max_primary_shard_size` to avoid creating empty indices or many small shards.
|
|
|
|
|
|
-If your retention policy allows it, you can also create larger indices by
|
|
|
-omitting a `max_age` threshold and using `max_docs` and/or `max_size`
|
|
|
-thresholds instead.
|
|
|
+If your retention policy requires a `max_age` threshold, increase it to create
|
|
|
+indices that cover longer time intervals. For example, instead of creating daily
|
|
|
+indices, you can create indices on a weekly or monthly basis.
|
|
|
|
|
|
[discrete]
|
|
|
[[delete-empty-indices]]
|