elasticsearch/docs/reference/ilm/ilm-with-existing-indices.asciidoc

[role="xpack"]
[testenv="basic"]
[[ilm-with-existing-indices]]
== Using {ilm-init} with existing indices

While it is recommended to use {ilm-init} to manage the index lifecycle from
start to finish, it may be useful to use {ilm-init} with existing indices,
particularly when transitioning from an alternative method of managing the index
lifecycle such as Curator, or when migrating from daily indices to
rollover-based indices. Such use cases are fully supported, but there are some
configuration differences from when {ilm-init} can manage the complete index
lifecycle.

This section describes strategies to leverage {ilm-init} for existing periodic
indices when migrating to fully {ilm-init}-manged indices, which can be done in
a few different ways, each providing different tradeoffs. As an example, we'll
walk through a use case of a very simple logging index with just a field for the
log message and a timestamp.

First, we need to create a template for these indices:

[source,js]
-----------------------
PUT _template/mylogs_template
{
  "index_patterns": [
    "mylogs-*"
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}
-----------------------
// CONSOLE
// TEST

And we'll ingest a few documents to create a few daily indices:

[source,js]
-----------------------
POST mylogs-pre-ilm-2019.06.24/_doc
{
  "@timestamp": "2019-06-24T10:34:00",
  "message": "this is one log message"
}
-----------------------
// CONSOLE
// TEST[continued]

[source,js]
-----------------------
POST mylogs-pre-ilm-2019.06.25/_doc
{
  "@timestamp": "2019-06-25T17:42:00",
  "message": "this is another log message"
}
-----------------------
// CONSOLE
// TEST[continued]

Now that we have these indices, we'll look at a few different ways of migrating
these indices to ILM.

[[ilm-with-existing-periodic-indices]]
=== Managing existing periodic indices with {ilm-init}

NOTE: The examples in this section assume daily indices as set up in
<<ilm-with-existing-indices,the previous section>>.

The simplest way to manage existing indices while transitioning to fully
{ilm-init}-managed indices is to allow all new indices to be fully managed by
{ilm-init} before attaching {ilm-init} policies to existing indices. To do this,
all new documents should be directed to {ilm-init}-managed indices - if you are
using Beats or Logstash data shippers, upgrading all of those shippers to
version 7.0.0 or higher will take care of that part for you. If you are not
using Beats or Logstash, you may need to set up ILM for new indices yourself as
demonstrated in the <<getting-started-index-lifecycle-management,getting started
guide>>.

NOTE: If you are using Beats through Logstash, you may need to change your
Logstash output configuration and invoke the Beats setup to use ILM for new
data.

Once all new documents are being written to fully {ilm-init}-managed indices, it
is easy to add an {ilm-init} policy to existing indices. However, there are two
things to keep in mind when doing this, and a trick that makes those two things
much easier to handle.

The two biggest things to keep in mind are:

1. Existing periodic indices shouldn't use policies with rollover, because
rollover is used to manage where new data goes. Since existing indices should no
longer be receiving new documents, there is no point to using rollover for them.

2. {ilm-init} policies attached to existing indices will compare the `min_age`
for each phase to the original creation date of the index, and so might proceed
through multiple phases immediately.

The first one is the most important, because it makes it difficult to use the
same policy for new and existing periodic indices. But that's easy to solve
with one simple trick: Create a second policy for existing indices, in addition
to the one for new indices. {ilm-init} policies are cheap to create, so don't be
afraid to have more than one. Modifying a policy designed for new indices to be
used on existing indices is generally very simple: just remove the `rollover`
action.

For example, if you created a policy for your new indices with each phase
like so:
[source,js]
-----------------------
PUT _ilm/policy/mylogs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "25GB"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
-----------------------
// CONSOLE
// TEST[continued]

You can create a policy for pre-existing indices by removing the `rollover`
action, and in this case, the `hot` phase is now empty so we can remove that
too:

[source,js]
-----------------------
PUT _ilm/policy/mylogs_policy_existing
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
-----------------------
// CONSOLE
// TEST[continued]

Creating a separate policy for existing indices will also allow using different
`min_age` values. You may want to use higher values to prevent many indices from
running through the policy at once, which may be important if your policy
includes potentially resource-intensive operations like force merge.

You can configure the lifecycle for many indices at once by using wildcards in
the index name when calling the <<indices-update-settings,Update Settings API>>
to set the policy name, but be careful that you don't include any indices that
you don't want to change the policy for:

[source,js]
-----------------------
PUT mylogs-pre-ilm*/_settings <1>
{
  "index": {
    "lifecycle": {
      "name": "mylogs_policy_existing"
    }
  }
}
-----------------------
// CONSOLE
// TEST[continued]

<1> This pattern will match all indices with names that start with
`mylogs-pre-ilm`

Once all pre-{ilm-init} indices have aged out and been deleted, the policy for
older periodic indices can be deleted.

[[ilm-reindexing-into-rollover]]
=== Reindexing via {ilm-init}

NOTE: The examples in this section assume daily indices as set up in
<<ilm-with-existing-indices,the previous section>>.

In some cases, it may be useful to reindex data into {ilm-init}-managed indices.
This is more complex than simply attaching policies to existing indices as
described in <<ilm-with-existing-periodic-indices,the previous section>>, and
requires pausing indexing during the reindexing process. However, this technique
may be useful in cases where periodic indices were created with very small
amounts of data leading to excessive shard counts, or for indices which grow
steadily over time, but have not been broken up into time-series indices leading
to shards which are much too large, situations that cause significant
performance problems.

Before getting started with reindexing data, the new index structure should be
set up. For this section, we'll be using the same setup described in
<<ilm-with-existing-indices,{ilm-imit} with existing indices>>.

First, we'll set up a policy with rollover, and can include any additional
phases required. For simplicity, we'll just use rollover:

[source,js]
-----------------------
PUT _ilm/policy/sample_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "50G"
          }
        }
      }
    }
  }
}
-----------------------
// CONSOLE
// TEST[continued]

And now we'll update the index template for our indices to include the relevant
{ilm-init} settings:

[source,js]
-----------------------
PUT _template/mylogs_template
{
  "index_patterns": [
    "ilm-mylogs-*" <1>
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index": {
      "lifecycle": {
        "name": "mylogs_condensed_policy", <2>
        "rollover_alias": "mylogs" <3>
      }
    }
  },
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}
-----------------------
// CONSOLE
// TEST[continued]
<1> The new index pattern has a prefix compared to the old one, this will
    make it easier to reindex later
<2> The name of the policy we defined above
<3> The name of the alias we'll use to write to and query

And create the first index with the alias specified in the `rollover_alias`
setting in the index template:

[source,js]
-----------------------
PUT ilm-mylogs-000001
{
  "aliases": {
    "mylogs": {
      "is_write_index": true
    }
  }
}
-----------------------
// CONSOLE
// TEST[continued]

All new documents should be indexed via the `mylogs` alias at this point. Adding
new data to the old indices during the reindexing process can cause data to be
added to the old indices, but not be reindexed into the new indices.

NOTE: If you do not want to mix new data and old data in the new ILM-managed
indices, indexing of new data should be paused entirely while the reindex
completes. Mixing old and new data within one index is safe, but keep in mind
that the indices with mixed data should be retained in their entirety until you
are ready to delete both the old and new data.

By default, {ilm-init} only checks rollover conditions every 10 minutes. Under
normal indexing load, this usually works well, but during reindexing, indices
can grow very, very quickly. We'll need to set the poll interval to something
shorter to ensure that the new indices don't grow too large while waiting for
the rollover check:

[source,js]
-----------------------
PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1m" <1>
  }
}
-----------------------
// CONSOLE
// TEST[skip:don't want to overwrite this setting for other tests]
<1> This tells ILM to check for rollover conditions every minute

We're now ready to reindex our data using the <<docs-reindex,reindex API>>. If
you have a timestamp or date field in your documents, as in this example, it may
be useful to specify that the documents should be sorted by that field - this
will mean that all documents in `ilm-mylogs-000001` come before all documents in
`ilm-mylogs-000002`, and so on. However, if this is not a requirement, omitting
the sort will allow the data to be reindexed more quickly.

IMPORTANT: If your data uses document IDs generated by means other than
Elasticsearch's automatic ID generation, you may need to do additional
processing to ensure that the document IDs don't conflict during the reindex, as
documents will retain their original IDs. One way to do this is to use a
<<reindex-scripts,script>> in the reindex call to append the original index name
to the document ID.

[source,js]
-----------------------
POST _reindex
{
  "source": {
    "index": "mylogs-*", <1>
    "sort": { "@timestamp": "desc" }
  },
  "dest": {
    "index": "mylogs", <2>
    "op_type": "create" <3>
  }
}
-----------------------
// CONSOLE
// TEST[continued]
<1> This index pattern matches our existing indices. Using the prefix for
    the new indices makes using this index pattern much easier.
<2> The alias set up above
<3> This option will cause the reindex to abort if it encounters multiple
    documents with the same ID. This is optional, but recommended to prevent
    accidentally overwriting documents if two documents from different indices
    have the same ID.

Once this completes, indexing new data can be resumed, as long as all new
documents are indexed into the alias used above. All data, existing and new, can
be queried using that alias as well. We should also be sure to set the
{ilm-init} poll interval back to its default value, because keeping it set too
low can cause unnecessary load on the current master node:

[source,js]
-----------------------
PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": null
  }
}

-----------------------
// CONSOLE
// TEST[skip:don't want to overwrite this setting for other tests]

All of the reindexed data should now be accessible via the alias set up above,
in this case `mylogs`. Once you have verified that all the data has been
reindexed and is available in the new indices, the existing indices can be
safely removed.