elasticsearch/docs/reference/index-modules/history-retention.asciidoc

[[index-modules-history-retention]]
== History retention

{es} sometimes needs to replay some of the operations that were performed on a
shard. For instance, if a replica is briefly offline then it may be much more
efficient to replay the few operations it missed while it was offline than to
rebuild it from scratch. Similarly, {ccr} works by performing operations on the
leader cluster and then replaying those operations on the follower cluster.

At the Lucene level there are really only two write operations that {es}
performs on an index: a new document may be indexed, or an existing document may
be deleted. Updates are implemented by atomically deleting the old document and
then indexing the new document. A document indexed into Lucene already contains
all the information needed to replay that indexing operation, but this is not
true of document deletions. To solve this, {es} uses a feature called _soft
deletes_ to preserve recent deletions in the Lucene index so that they can be
replayed.

{es} only preserves certain recently-deleted documents in the index because a
soft-deleted document still takes up some space. Eventually {es} will fully
discard these soft-deleted documents to free up that space so that the index
does not grow larger and larger over time. Fortunately {es} does not need to be
able to replay every operation that has ever been performed on a shard, because
it is always possible to make a full copy of a shard on a remote node. However,
copying the whole shard may take much longer than replaying a few missing
operations, so {es} tries to retain all of the operations it expects to need to
replay in future.

{es} keeps track of the operations it expects to need to replay in future using
a mechanism called _shard history retention leases_. Each shard copy that might
need operations to be replayed must first create a shard history retention lease
for itself. For example, this shard copy might be a replica of a shard or it
might be a shard of a follower index when using {ccr}. Each retention lease
keeps track of the sequence number of the first operation that the corresponding
shard copy has not received. As the shard copy receives new operations, it
increases the sequence number contained in its retention lease to indicate that
it will not need to replay those operations in future. {es} discards
soft-deleted operations once they are not being held by any retention lease.

If a shard copy fails then it stops updating its shard history retention lease,
which means that {es} will preserve all new operations so they can be replayed
when the failed shard copy recovers. However, retention leases only last for a
limited amount of time. If the shard copy does not recover quickly enough then
its retention lease may expire. This protects {es} from retaining history
forever if a shard copy fails permanently, because once a retention lease has
expired {es} can start to discard history again. If a shard copy recovers after
its retention lease has expired then {es} will fall back to copying the whole
index since it can no longer simply replay the missing history. The expiry time
of a retention lease defaults to `12h` which should be long enough for most
reasonable recovery scenarios.

[discrete]
=== History retention settings

include::{es-ref-dir}/index-modules.asciidoc[tag=ccr-index-soft-deletes-tag]

include::{es-ref-dir}/index-modules.asciidoc[tag=ccr-index-soft-deletes-retention-tag]