elasticsearch/docs/reference/transform/apis/put-transform.asciidoc

[role="xpack"]
[[put-transform]]
= Create {transform} API

[subs="attributes"]
++++
<titleabbrev>Create {transform}</titleabbrev>
++++

Instantiates a {transform}.

[[put-transform-request]]
== {api-request-title}

`PUT _transform/<transform_id>`

[[put-transform-prereqs]]
== {api-prereq-title}

Requires the following privileges:

* cluster: `manage_transform` (the `transform_admin` built-in role grants this
  privilege)
* source indices: `read`, `view_index_metadata`
* destination index: `read`, `create_index`, `index`. If a `retention_policy` is configured, the `delete` privilege is
  also required.

[[put-transform-desc]]
== {api-description-title}

This API defines a {transform}, which copies data from source indices,
transforms it, and persists it into an entity-centric destination index. If you
choose to use the pivot method for your {transform}, the entities are defined by
the set of `group_by` fields in the `pivot` object. If you choose to use the
latest method, the entities are defined by the `unique_key` field values in the
`latest` object.

You can also think of the destination index as a two-dimensional tabular data
structure (known as a {dataframe}). The ID for each document in the {dataframe}
is generated from a hash of the entity, so there is a unique row per entity. For
more information, see <<transforms>>.

When the {transform} is created, a series of validations occur to ensure its
success. For example, there is a check for the existence of the source indices
and a check that the destination index is not part of the source index pattern.
You can use the `defer_validation` parameter to skip these checks.

Deferred validations are always run when the {transform} is started, with the
exception of privilege checks.

[IMPORTANT]
====

* The {transform} remembers which roles the user that created it had at the time
of creation and uses those same roles. If those roles do not have the required
privileges on the source and destination indices, the {transform} fails when it
attempts unauthorized operations. If you provide
<<http-clients-secondary-authorization,secondary authorization headers>>, those
credentials are used instead.
* You must use {kib} or this API to create a {transform}. Do not add a
{transform} directly into any `.transform-internal*` indices using the {es}
index API. If {es} {security-features} are enabled, do not give users any
privileges on `.transform-internal*` indices. If you used {transforms} prior to
7.5, also do not give users any privileges on `.data-frame-internal*` indices.

====

You must choose either the latest or pivot method for your {transform}; you
cannot use both in a single {transform}.

[[put-transform-path-parms]]
== {api-path-parms-title}

`<transform_id>`::
(Required, string)
Identifier for the {transform}. This identifier can contain lowercase
alphanumeric characters (a-z and 0-9), hyphens, and underscores. It has a 64
character limit and must start and end with alphanumeric characters.

[[put-transform-query-parms]]
== {api-query-parms-title}

`defer_validation`::
(Optional, Boolean) When `true`, deferrable validations are not run. This
behavior may be desired if the source index does not exist until after the
{transform} is created.

`timeout`::
(Optional, time)
Period to wait for a response. If no response is received before the timeout
expires, the request fails and returns an error. Defaults to `30s`.

[role="child_attributes"]
[[put-transform-request-body]]
== {api-request-body-title}

`description`::
  (Optional, string) Free text description of the {transform}.

//Begin dest
`dest`::
(Required, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=dest]
+
.Properties of `dest`
[%collapsible%open]
====

`index`:::
(Required, string)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=dest-index]

`pipeline`:::
(Optional, string)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=dest-pipeline]
====
//End dest

`frequency`::
(Optional, <<time-units, time units>>)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=frequency]

//Begin latest
`latest`::
(Required^*^, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-latest]
+
.Properties of `latest`
[%collapsible%open]
====

`sort`:::
(Required, string)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-sort]

`unique_key`:::
(Required, array of strings)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-unique-key]

====
//End latest

//Begin _meta
`_meta`::
(Optional, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-metadata]
//End _meta

//Begin pivot
`pivot`::
(Required^*^, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pivot]
+
.Properties of `pivot`
[%collapsible%open]
====

`aggregations` or `aggs`:::
(Required, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pivot-aggs]

`group_by`:::
(Required, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pivot-group-by]

====
//End pivot

//Begin retention policy
`retention_policy`::
(Optional, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention]
+
.Properties of `retention_policy`
[%collapsible%open]
====
`time`:::
(Required, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time]
+
.Properties of `time`
[%collapsible%open]
=====
`field`:::
(Required, string)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-field]

`max_age`:::
(Required, <<time-units, time units>>)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-retention-time-max-age]
=====
====
//End retention policy

//Begin settings
`settings`::
(Optional, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings]
+
.Properties of `settings`
[%collapsible%open]
====
`dates_as_epoch_millis`:::
(Optional, boolean)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-dates-as-epoch-milli]
`docs_per_second`:::
(Optional, float)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
`align_checkpoints`:::
(Optional, boolean)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-align-checkpoints]
`deduce_mappings`:::
(Optional, boolean)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-deduce-mappings]
`max_page_search_size`:::
(Optional, integer)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
`num_failure_retries`:::
(Optional, integer)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=transform-settings-num-failure-retries]
====
//End settings

//Begin source
`source`::
(Required, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source-transforms]
+
.Properties of `source`
[%collapsible%open]
====

`index`:::
(Required, string or array)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source-index-transforms]

`query`:::
(Optional, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source-query-transforms]

`runtime_mappings`:::
(Optional, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source-runtime-mappings-transforms]
====
//End source

//Begin sync
`sync`::
(Optional, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=sync]
+
.Properties of `sync`
[%collapsible%open]
====

//Begin time
`time`:::
(Required, object)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=sync-time]
+
.Properties of `time`
[%collapsible%open]
=====

`delay`::::
(Optional, <<time-units, time units>>)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=sync-time-delay]

`field`::::
(Required, string)
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=sync-time-field]
+
--
TIP: It is strongly recommended to use a field that contains the
<<access-ingest-metadata,ingest timestamp>>. If you use a different field,
you might need to set the `delay` such that it accounts for data transmission
delays.

--
=====
//End time
====
//End sync


[[put-transform-example]]
== {api-examples-title}

The following {transform} uses the `pivot` method:

[source,console]
--------------------------------------------------
PUT _transform/ecommerce_transform1
{
  "source": {
    "index": "kibana_sample_data_ecommerce",
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id"
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform1",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  },
  "retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "30d"
    }
  }
}
--------------------------------------------------
// TEST[setup:kibana_sample_data_ecommerce,add_timestamp_pipeline]

When the {transform} is created, you receive the following results:

[source,console-result]
----
{
  "acknowledged" : true
}
----

The following {transform} uses the `latest` method:

[source,console]
--------------------------------------------------
PUT _transform/ecommerce_transform2
{
  "source": {
    "index": "kibana_sample_data_ecommerce"
  },
  "latest": {
    "unique_key": ["customer_id"],
    "sort": "order_date"
  },
  "description": "Latest order for each customer",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform2"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  }
}
--------------------------------------------------
// TEST[setup:kibana_sample_data_ecommerce]