|
@@ -4,48 +4,74 @@
|
|
|
|
|
|
To set up a data stream, follow these steps:
|
|
|
|
|
|
-. <<configure-a-data-stream-ilm-policy>>.
|
|
|
-. <<create-a-data-stream-template>>.
|
|
|
-. <<create-a-data-stream>>.
|
|
|
-. <<secure-a-data-stream>>.
|
|
|
+* <<create-index-lifecycle-policy>>
|
|
|
+* <<create-component-templates>>
|
|
|
+* <<create-index-template>>
|
|
|
+* <<create-data-stream>>
|
|
|
+* <<secure-data-stream>>
|
|
|
|
|
|
-You can also <<convert-an-index-alias-to-a-data-stream,convert an index alias to
|
|
|
+You can also <<convert-index-alias-to-data-stream,convert an index alias to
|
|
|
a data stream>>.
|
|
|
|
|
|
-[discrete]
|
|
|
-[[configure-a-data-stream-ilm-policy]]
|
|
|
-=== Optional: Configure an {ilm-init} lifecycle policy
|
|
|
+IMPORTANT: If you use {fleet} or {agent}, skip this tutorial. {fleet} and
|
|
|
+{agent} set up data streams for you. See {fleet-guide}/data-streams.html[Data
|
|
|
+streams] in the {fleet} Guide.
|
|
|
|
|
|
-While optional, we recommend you configure an <<set-up-lifecycle-policy,{ilm}
|
|
|
-({ilm-init}) policy>> to automate the management of your data stream's backing
|
|
|
-indices.
|
|
|
+[discrete]
|
|
|
+[[create-index-lifecycle-policy]]
|
|
|
+=== Step 1. Create an index lifecycle policy
|
|
|
|
|
|
-In {kib}, open the menu and go to *Stack Management > Index Lifecycle Policies*.
|
|
|
-Click *Create policy*.
|
|
|
+While optional, we recommend using {ilm-init} to automate the management of your
|
|
|
+data stream's backing indices. {ilm-init} requires an index lifecycle policy.
|
|
|
|
|
|
-[role="screenshot"]
|
|
|
-image::images/ilm/create-policy.png[Create Policy page]
|
|
|
+To create an index lifecycle policy in {kib}, open the main menu and go to
|
|
|
+*Stack Management > Index Lifecycle Policies*. Click *Create policy*.
|
|
|
|
|
|
-[%collapsible]
|
|
|
-.API example
|
|
|
-====
|
|
|
-Use the <<ilm-put-lifecycle,create lifecycle policy API>> to configure a policy:
|
|
|
+You can also use the <<ilm-put-lifecycle,create lifecycle policy API>>.
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-PUT /_ilm/policy/my-data-stream-policy
|
|
|
+PUT _ilm/policy/my-lifecycle-policy
|
|
|
{
|
|
|
"policy": {
|
|
|
"phases": {
|
|
|
"hot": {
|
|
|
"actions": {
|
|
|
"rollover": {
|
|
|
- "max_primary_shard_size": "25GB"
|
|
|
+ "max_age": "30d",
|
|
|
+ "max_primary_shard_size": "50gb"
|
|
|
}
|
|
|
}
|
|
|
},
|
|
|
- "delete": {
|
|
|
+ "warm": {
|
|
|
"min_age": "30d",
|
|
|
+ "actions": {
|
|
|
+ "shrink": {
|
|
|
+ "number_of_shards": 1
|
|
|
+ },
|
|
|
+ "forcemerge": {
|
|
|
+ "max_num_segments": 1
|
|
|
+ }
|
|
|
+ }
|
|
|
+ },
|
|
|
+ "cold": {
|
|
|
+ "min_age": "60d",
|
|
|
+ "actions": {
|
|
|
+ "searchable_snapshot": {
|
|
|
+ "snapshot_repository": "my-snapshot-repo"
|
|
|
+ }
|
|
|
+ }
|
|
|
+ },
|
|
|
+ "frozen": {
|
|
|
+ "min_age": "90d",
|
|
|
+ "actions": {
|
|
|
+ "searchable_snapshot": {
|
|
|
+ "snapshot_repository": "my-snapshot-repo"
|
|
|
+ }
|
|
|
+ }
|
|
|
+ },
|
|
|
+ "delete": {
|
|
|
+ "min_age": "735d",
|
|
|
"actions": {
|
|
|
"delete": {}
|
|
|
}
|
|
@@ -54,139 +80,158 @@ PUT /_ilm/policy/my-data-stream-policy
|
|
|
}
|
|
|
}
|
|
|
----
|
|
|
-====
|
|
|
|
|
|
[discrete]
|
|
|
-[[create-a-data-stream-template]]
|
|
|
-=== Create an index template
|
|
|
-
|
|
|
-. In {kib}, open the menu and go to *Stack Management > Index Management*.
|
|
|
-. In the *Index Templates* tab, click *Create template*.
|
|
|
-. In the Create template wizard, use the *Data stream* toggle to indicate the
|
|
|
-template is used for data streams.
|
|
|
-. Use the wizard to finish defining your template. Specify:
|
|
|
+[[create-component-templates]]
|
|
|
+=== Step 2. Create component templates
|
|
|
|
|
|
-* One or more index patterns that match the data stream's name. +
|
|
|
-include::{es-repo-dir}/indices/create-data-stream.asciidoc[tag=data-stream-name]
|
|
|
+A data stream requires a matching index template. In most cases, you compose
|
|
|
+this index template using one or more component templates. You typically use
|
|
|
+separate component templates for mappings and index settings. This lets you
|
|
|
+reuse the component templates in multiple index templates.
|
|
|
|
|
|
-* Mappings and settings for the stream's backing indices.
|
|
|
+When creating your component templates, include:
|
|
|
|
|
|
-* A priority for the index template
|
|
|
-+
|
|
|
-include::{es-repo-dir}/indices/index-templates.asciidoc[tag=built-in-index-templates]
|
|
|
+* A <<date,`date`>> or <<date_nanos,`date_nanos`>> mapping for the `@timestamp`
|
|
|
+field. If you don't specify a mapping, {es} maps `@timestamp` as a `date` field
|
|
|
+with default options.
|
|
|
|
|
|
-[[elastic-data-stream-naming-scheme]]
|
|
|
-.The Elastic data stream naming scheme
|
|
|
-****
|
|
|
-The {agent} uses the Elastic data stream naming scheme to name its data streams.
|
|
|
-To help you organize your data consistently and avoid naming collisions, we
|
|
|
-recommend you also use the Elastic naming scheme for your other data streams.
|
|
|
+* Your lifecycle policy in the `index.lifecycle.name` index setting.
|
|
|
|
|
|
-The naming scheme splits data into different data streams based on the following
|
|
|
-components. Each component corresponds to a
|
|
|
-<<constant-keyword-field-type,constant keyword>> field defined in the
|
|
|
-{ecs-ref}[Elastic Common Schema (ECS)].
|
|
|
+To create a component template in {kib}, open the main menu and go to *Stack
|
|
|
+Management > Index Management*. In the *Index Templates* view, click *Create a
|
|
|
+component template*.
|
|
|
|
|
|
-`type`::
|
|
|
-Generic type describing the data, such as `logs`, `metrics`, or `synthetics`.
|
|
|
-Corresponds to the `data_stream.type` field.
|
|
|
+You can also use the <<indices-component-template,create component template
|
|
|
+API>>.
|
|
|
|
|
|
-`dataset`::
|
|
|
-Describes the ingested data and its structure. Corresponds to the
|
|
|
-`data_stream.dataset` field. Defaults to `generic`.
|
|
|
-
|
|
|
-`namespace`::
|
|
|
-User-configurable arbitrary grouping. Corresponds to the `data_stream.dataset`
|
|
|
-field. Defaults to `default`.
|
|
|
+[source,console]
|
|
|
+----
|
|
|
+# Creates a component template for mappings
|
|
|
+PUT _component_template/my-mappings
|
|
|
+{
|
|
|
+ "template": {
|
|
|
+ "mappings": {
|
|
|
+ "properties": {
|
|
|
+ "@timestamp": {
|
|
|
+ "type": "date",
|
|
|
+ "format": "date_optional_time||epoch_millis"
|
|
|
+ },
|
|
|
+ "message": {
|
|
|
+ "type": "wildcard"
|
|
|
+ }
|
|
|
+ }
|
|
|
+ }
|
|
|
+ },
|
|
|
+ "_meta": {
|
|
|
+ "description": "Mappings for @timestamp and message fields",
|
|
|
+ "my-custom-meta-field": "More arbitrary metadata"
|
|
|
+ }
|
|
|
+}
|
|
|
|
|
|
-The naming scheme separates these components with a `-` character:
|
|
|
+# Creates a component template for index settings
|
|
|
+PUT _component_template/my-settings
|
|
|
+{
|
|
|
+ "template": {
|
|
|
+ "settings": {
|
|
|
+ "index.lifecycle.name": "my-lifecycle-policy"
|
|
|
+ }
|
|
|
+ },
|
|
|
+ "_meta": {
|
|
|
+ "description": "Settings for ILM",
|
|
|
+ "my-custom-meta-field": "More arbitrary metadata"
|
|
|
+ }
|
|
|
+}
|
|
|
+----
|
|
|
+// TEST[continued]
|
|
|
|
|
|
-```
|
|
|
-<type>-<dataset>-<namespace>
|
|
|
-```
|
|
|
+[discrete]
|
|
|
+[[create-index-template]]
|
|
|
+=== Step 3. Create an index template
|
|
|
|
|
|
-For example, the {agent} uses the `logs-nginx.access-production` data
|
|
|
-stream to store data with a type of `logs`, a dataset of `nginx.access`, and a
|
|
|
-namespace of `production`. If you use the {agent} to ingest a log file, it
|
|
|
-stores the data in the `logs-generic-default` data stream.
|
|
|
+Use your component templates to create an index template. Specify:
|
|
|
|
|
|
-For more information about the naming scheme and its benefits, see our
|
|
|
-https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme[An
|
|
|
-introduction to the Elastic data stream naming scheme] blog post.
|
|
|
-****
|
|
|
+* One or more index patterns that match the data stream's name. We recommend
|
|
|
+using our {fleet-guide}/data-streams.html#data-streams-naming-scheme[data stream
|
|
|
+naming scheme].
|
|
|
|
|
|
-include::{es-repo-dir}/data-streams/data-streams.asciidoc[tag=timestamp-reqs]
|
|
|
+* That the template is data stream enabled.
|
|
|
|
|
|
-If using {ilm-init}, specify your lifecycle policy in the `index.lifecycle.name`
|
|
|
-setting.
|
|
|
+* Any component templates that contain your mappings and index settings.
|
|
|
|
|
|
-TIP: Carefully consider your template's mappings and settings. Later changes may
|
|
|
-require reindexing. See <<data-streams-change-mappings-and-settings>>.
|
|
|
+* A priority higher than `200` to avoid collisions with built-in templates.
|
|
|
+See <<avoid-index-pattern-collisions>>.
|
|
|
|
|
|
-[role="screenshot"]
|
|
|
-image::images/data-streams/create-index-template.png[Create template page]
|
|
|
+To create an index template in {kib}, open the main menu and go to *Stack
|
|
|
+Management > Index Management*. In the *Index Templates* view, click *Create
|
|
|
+template*.
|
|
|
|
|
|
-[%collapsible]
|
|
|
-.API example
|
|
|
-====
|
|
|
-Use the <<indices-put-template,create or update index template API>> to create
|
|
|
-an index template. The template must include a `data_stream` object, indicating
|
|
|
-it's used for data streams.
|
|
|
+You can also use the <<indices-put-template,create index template API>>.
|
|
|
+Include the `data_stream` object to enable data streams.
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-PUT /_index_template/my-data-stream-template
|
|
|
+PUT _index_template/my-index-template
|
|
|
{
|
|
|
- "index_patterns": [ "my-data-stream*" ],
|
|
|
+ "index_patterns": ["my-data-stream*"],
|
|
|
"data_stream": { },
|
|
|
+ "composed_of": [ "my-mappings", "my-settings" ],
|
|
|
"priority": 500,
|
|
|
- "template": {
|
|
|
- "settings": {
|
|
|
- "index.lifecycle.name": "my-data-stream-policy"
|
|
|
- }
|
|
|
+ "_meta": {
|
|
|
+ "description": "Template for my time series data",
|
|
|
+ "my-custom-meta-field": "More arbitrary metadata"
|
|
|
}
|
|
|
}
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
-====
|
|
|
|
|
|
[discrete]
|
|
|
-[[create-a-data-stream]]
|
|
|
-=== Create the data stream
|
|
|
+[[create-data-stream]]
|
|
|
+=== Step 4. Create the data stream
|
|
|
|
|
|
To automatically create the data stream, submit an
|
|
|
-<<add-documents-to-a-data-stream,indexing request>> to the stream. The stream's
|
|
|
-name must match one of your template's index patterns.
|
|
|
+<<add-documents-to-a-data-stream,indexing request>> that targets the stream's
|
|
|
+name. This name must match one of your index template's index patterns. The
|
|
|
+request must use an `op_type` of `create`. Documents must include a `@timestamp`
|
|
|
+field.
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-POST /my-data-stream/_doc/
|
|
|
+PUT my-data-stream/_bulk
|
|
|
+{ "create":{ } }
|
|
|
+{ "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" }
|
|
|
+{ "create":{ } }
|
|
|
+{ "@timestamp": "2099-05-06T16:25:42.000Z", "message": "192.0.2.255 - - [06/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" }
|
|
|
+
|
|
|
+POST my-data-stream/_doc
|
|
|
{
|
|
|
- "@timestamp": "2099-03-07T11:04:05.000Z",
|
|
|
- "user": {
|
|
|
- "id": "vlb44hny"
|
|
|
- },
|
|
|
- "message": "Login attempt failed"
|
|
|
+ "@timestamp": "2099-05-06T16:21:15.000Z",
|
|
|
+ "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
|
|
|
}
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
|
|
|
-You can also use the <<indices-create-data-stream,create data stream API>> to
|
|
|
-manually create the data stream. The stream's name must match one of your
|
|
|
-template's index patterns.
|
|
|
+You can also manually create the stream using the
|
|
|
+<<indices-create-data-stream,create data stream API>>. The stream's name must
|
|
|
+still match one of your template's index patterns.
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-PUT /_data_stream/my-data-stream
|
|
|
+PUT _data_stream/my-data-stream
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
// TEST[s/my-data-stream/my-data-stream-alt/]
|
|
|
|
|
|
-When you create a data stream, {es} automatically creates a backing index for
|
|
|
-the stream. This index also acts as the stream's first write index.
|
|
|
+[discrete]
|
|
|
+[[secure-data-stream]]
|
|
|
+=== Step 5. Secure the data stream
|
|
|
+
|
|
|
+include::{xes-repo-dir}/security/authorization/alias-privileges.asciidoc[tag=data-stream-security]
|
|
|
+
|
|
|
+For an example, see <<data-stream-privileges>>.
|
|
|
|
|
|
[discrete]
|
|
|
-[[convert-an-index-alias-to-a-data-stream]]
|
|
|
+[[convert-index-alias-to-data-stream]]
|
|
|
=== Convert an index alias to a data stream
|
|
|
|
|
|
// tag::time-series-alias-tip[]
|
|
@@ -196,12 +241,11 @@ functionality, require less maintenance, and automatically integrate with
|
|
|
<<data-tiers,data tiers>>.
|
|
|
// end::time-series-alias-tip[]
|
|
|
|
|
|
-To convert an index alias with a write index to a new data stream with the same
|
|
|
+To convert an index alias with a write index to a data stream with the same
|
|
|
name, use the <<indices-migrate-to-data-stream,migrate to data stream API>>.
|
|
|
During conversion, the alias’s indices become hidden backing indices for the
|
|
|
-stream. The alias’s write index becomes the stream’s write index. Note the data
|
|
|
-stream still requires a matching <<create-a-data-stream-template,index
|
|
|
-template>>.
|
|
|
+stream. The alias’s write index becomes the stream’s write index. The stream
|
|
|
+still requires a matching index template with data stream enabled.
|
|
|
|
|
|
////
|
|
|
[source,console]
|
|
@@ -218,7 +262,7 @@ POST idx2/_doc/
|
|
|
"@timestamp" : "2099-01-01"
|
|
|
}
|
|
|
|
|
|
-POST /_aliases
|
|
|
+POST _aliases
|
|
|
{
|
|
|
"actions": [
|
|
|
{
|
|
@@ -237,7 +281,7 @@ POST /_aliases
|
|
|
]
|
|
|
}
|
|
|
|
|
|
-PUT /_index_template/template
|
|
|
+PUT _index_template/template
|
|
|
{
|
|
|
"index_patterns": ["my-time-series-data"],
|
|
|
"data_stream": { }
|
|
@@ -248,79 +292,58 @@ PUT /_index_template/template
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-POST /_data_stream/_migrate/my-time-series-data
|
|
|
+POST _data_stream/_migrate/my-time-series-data
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
|
|
|
[discrete]
|
|
|
-[[secure-a-data-stream]]
|
|
|
-=== Secure the data stream
|
|
|
-
|
|
|
-To control access to the data stream and its
|
|
|
-data, use <<data-stream-privileges,{es}'s {security-features}>>.
|
|
|
-
|
|
|
-[discrete]
|
|
|
-[[get-info-about-a-data-stream]]
|
|
|
+[[get-info-about-data-stream]]
|
|
|
=== Get information about a data stream
|
|
|
|
|
|
-In {kib}, open the menu and go to *Stack Management > Index Management*. In the
|
|
|
-*Data Streams* tab, click the data stream's name.
|
|
|
-
|
|
|
-[role="screenshot"]
|
|
|
-image::images/data-streams/data-streams-list.png[Data Streams tab]
|
|
|
+To get information about a data stream in {kib}, open the main menu and go to
|
|
|
+*Stack Management > Index Management*. In the *Data Streams* view, click the
|
|
|
+data stream's name.
|
|
|
|
|
|
-[%collapsible]
|
|
|
-.API example
|
|
|
-====
|
|
|
-Use the <<indices-get-data-stream,get data stream API>> to retrieve information
|
|
|
-about one or more data streams:
|
|
|
+You can also use the <<indices-get-data-stream,get data stream API>>.
|
|
|
|
|
|
////
|
|
|
[source,console]
|
|
|
----
|
|
|
-POST /my-data-stream/_rollover/
|
|
|
+POST my-data-stream/_rollover/
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
////
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-GET /_data_stream/my-data-stream
|
|
|
+GET _data_stream/my-data-stream
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
-====
|
|
|
|
|
|
[discrete]
|
|
|
-[[delete-a-data-stream]]
|
|
|
+[[delete-data-stream]]
|
|
|
=== Delete a data stream
|
|
|
|
|
|
-To delete a data stream and its backing indices, open the {kib} menu and go to
|
|
|
-*Stack Management > Index Management*. In the *Data Streams* tab, click the
|
|
|
-trash icon. The trash icon only displays if you have the `delete_index`
|
|
|
+To delete a data stream and its backing indices in {kib}, open the main menu and
|
|
|
+go to *Stack Management > Index Management*. In the *Data Streams* view, click
|
|
|
+the trash icon. The icon only displays if you have the `delete_index`
|
|
|
<<security-privileges, security privilege>> for the data stream.
|
|
|
|
|
|
-[role="screenshot"]
|
|
|
-image::images/data-streams/data-streams-no-delete.png[Data Streams tab]
|
|
|
-
|
|
|
-[%collapsible]
|
|
|
-.API example
|
|
|
-====
|
|
|
-Use the <<indices-delete-data-stream,delete data stream API>> to delete a data
|
|
|
-stream and its backing indices:
|
|
|
+You can also use the <<indices-delete-data-stream,delete data stream API>>.
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
|
-DELETE /_data_stream/my-data-stream
|
|
|
+DELETE _data_stream/my-data-stream
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
-====
|
|
|
|
|
|
////
|
|
|
[source,console]
|
|
|
----
|
|
|
-DELETE /_data_stream/*
|
|
|
-DELETE /_index_template/*
|
|
|
-DELETE /_ilm/policy/my-data-stream-policy
|
|
|
+DELETE _data_stream/*
|
|
|
+DELETE _index_template/*
|
|
|
+DELETE _component_template/my-*
|
|
|
+DELETE _ilm/policy/my-lifecycle-policy
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
////
|