Browse Source

[DOCS] Update transforms tutorial with index mapping details (#73395)

Lisa Cawley 4 years ago
parent
commit
5273011f98

+ 13 - 8
docs/reference/transform/apis/start-transform.asciidoc

@@ -27,16 +27,21 @@ Requires the following privileges:
 [[start-transform-desc]]
 == {api-description-title}
 
-When you start a {transform}, it creates the destination index if it
-does not already exist. The `number_of_shards` is set to `1` and the
+When you start a {transform}, it creates the destination index if it does not 
+already exist. The `number_of_shards` is set to `1` and the
 `auto_expand_replicas` is set to `0-1`.
 
-The {transform} deduces the mapping definitions from the source indices. For
-scripted fields, it uses <<dynamic-mapping,dynamic mappings>>. If a field in the
-destination index is created by `scripted_metric` or `bucket_script`
-aggregations, the {transform} uses dynamic mappings unless a template exists or
-the destination index already exists. Mapping definitions in the destination
-index take precedence over dynamic mappings and templates.
+If it is a pivot {transform}, it deduces the mapping definitions for the
+destination index from the source indices and the transform aggregations. If
+fields in the destination index are derived from scripts (as in the case of
+`scripted_metric` or `bucket_script` aggregations), the {transform} uses
+<<dynamic-mapping,dynamic mappings>> unless an index template exists.
+
+If it is a latest {transform}, it does not deduce mapping definitions; it uses
+dynamic mappings.
+
+TIP: To use explicit mappings, create the destination index before you start the {transform}. Alternatively, you can create an index template, though it does not affect the
+deduced mappings in a pivot {transform}.
 
 When the {transform} starts, a series of validations occur to ensure
 its success. If you deferred validation when you created the

+ 159 - 5
docs/reference/transform/ecommerce-tutorial.asciidoc

@@ -1,5 +1,4 @@
 [role="xpack"]
-[testenv="basic"]
 [[ecommerce-transforms]]
 = Tutorial: Transforming the eCommerce sample data
 
@@ -23,7 +22,7 @@ already familiar with the `kibana_sample_data_ecommerce` index, use the
 might want to derive from this eCommerce data.
 --
 
-. Choose the `pivot` type of {transform} and play with various options for
+. Choose the pivot type of {transform} and play with various options for
 grouping and aggregating the data.
 +
 --
@@ -129,9 +128,9 @@ POST _transform/_preview
 {transform}.
 +
 --
-.. Supply a {transform} ID, the name of the target (or _destination_) index and 
-optionally a description. If the target index does not exist, it will be created 
-automatically.
+.. Supply a {transform} ID, the name of the destination index and optionally a
+description. If the destination index does not exist, it will be created
+automatically when you start the {transform}.
 
 .. Decide whether you want the {transform} to run once or continuously. Since 
 this sample data index is unchanging, let's use the default behavior and just 
@@ -149,6 +148,13 @@ than the configured value are removed from the destination index.
 [role="screenshot"]
 image::images/ecommerce-pivot3.png["Adding transfrom ID and retention policy to a {transform} in {kib}"]
 
+In {kib}, before you finish creating the {transform}, you can copy the preview 
+{transform} API request to your clipboard. This information is useful later when 
+you're deciding whether you want to manually create the destination index.
+
+[role="screenshot"]
+image::images/ecommerce-pivot4.png["Copy the Dev Console statement of the transform preview to the clipboard"]
+
 If you prefer, you can use the
 <<put-transform,create {transforms} API>>.
 
@@ -219,6 +225,147 @@ PUT _transform/ecommerce-customer-transform
 ====
 --
 
+. Optional: Create the destination index.
++
+--
+If the destination index does not exist, it is created the first time you start
+your {transform}. A pivot transform deduces the mappings for the destination
+index from the source indices and the transform aggregations. If there are
+fields in the destination index that are derived from scripts (for example, 
+if you use
+<<search-aggregations-metrics-scripted-metric-aggregation,`scripted_metrics`>>
+or <<search-aggregations-pipeline-bucket-script-aggregation,`bucket_scripts`>>
+aggregations), they're created with <<dynamic-mapping,dynamic mappings>>. You
+can use the preview {transform} API to preview the mappings it will use for the
+destination index. In {kib}, if you copied the API request to your 
+clipboard, paste it into the console, then refer to the `generated_dest_index` 
+object in the API response.
+
+.API example
+[%collapsible]
+====
+
+[source,console-result]
+--------------------------------------------------
+{
+  "preview" : [
+    {
+      "total_quantity" : {
+        "max" : 2,
+        "sum" : 118.0
+      },
+      "taxless_total_price" : {
+        "sum" : 3946.9765625
+      },
+      "customer_id" : "10",
+      "order_id" : {
+        "cardinality" : 59
+      }
+    },
+    ...
+  ],
+  "generated_dest_index" : {
+    "mappings" : {
+      "_meta" : {
+        "_transform" : {
+          "transform" : "transform-preview",
+          "version" : {
+            "created" : "8.0.0"
+          },
+          "creation_date_in_millis" : 1621991264061
+        },
+        "created_by" : "transform"
+      },
+      "properties" : {
+        "total_quantity.sum" : {
+          "type" : "double"
+        },
+        "total_quantity" : {
+          "type" : "object"
+        },
+        "taxless_total_price" : {
+          "type" : "object"
+        },
+        "taxless_total_price.sum" : {
+          "type" : "double"
+        },
+        "order_id.cardinality" : {
+          "type" : "long"
+        },
+        "customer_id" : {
+          "type" : "keyword"
+        },
+        "total_quantity.max" : {
+          "type" : "integer"
+        },
+        "order_id" : {
+          "type" : "object"
+        }
+      }
+    },
+    "settings" : {
+      "index" : {
+        "number_of_shards" : "1",
+        "auto_expand_replicas" : "0-1"
+      }
+    },
+    "aliases" : { }
+  }
+}
+--------------------------------------------------
+// TESTRESPONSE[skip:needs sample data]
+====
+
+In some instances the deduced mappings might be incompatible with the actual
+data. For example, numeric overflows might occur or dynamically mapped fields
+might contain both numbers and strings. To avoid this problem, create your
+destination index before you start the {transform}. For more information, see
+the <<indices-create-index,create index API>>.
+
+.API example
+[%collapsible]
+====
+You can use the information from the {transform} preview to create the
+destination index. For example:
+
+[source,console]
+--------------------------------------------------
+PUT /ecommerce-customers
+{
+  "mappings": {
+    "properties": {
+      "total_quantity.sum" : {
+        "type" : "double"
+      },
+      "total_quantity" : {
+        "type" : "object"
+      },
+      "taxless_total_price" : {
+        "type" : "object"
+      },
+      "taxless_total_price.sum" : {
+        "type" : "double"
+      },
+      "order_id.cardinality" : {
+        "type" : "long"
+      },
+      "customer_id" : {
+        "type" : "keyword"
+      },
+      "total_quantity.max" : {
+        "type" : "integer"
+      },
+      "order_id" : {
+        "type" : "object"
+      }
+    }
+  }
+}
+--------------------------------------------------
+// TEST
+====
+--
+
 . Start the {transform}.
 +
 --
@@ -300,6 +447,13 @@ POST _transform/_preview
 --------------------------------------------------
 // TEST[skip:set up sample data]
 ====
+
+TIP: If the destination index does not exist, it is created the first time you 
+start your {transform}. Unlike pivot {transforms}, however, latest {transforms} 
+do not deduce mapping definitions when they create the index. Instead, they use 
+dynamic mappings. To use explicit mappings, create the destination index 
+before you start the {transform}.
+
 --
 
 . If you do not want to keep a {transform}, you can delete it in

BIN
docs/reference/transform/images/ecommerce-latest1.png


BIN
docs/reference/transform/images/ecommerce-pivot1.png


BIN
docs/reference/transform/images/ecommerce-pivot2.png


BIN
docs/reference/transform/images/ecommerce-pivot3.png


BIN
docs/reference/transform/images/ecommerce-pivot4.png


BIN
docs/reference/transform/images/ecommerce-results.png


BIN
docs/reference/transform/images/manage-transforms.png


+ 4 - 4
docs/reference/transform/limitations.asciidoc

@@ -90,7 +90,7 @@ will not be reassigned until the upgrade is complete. After the upgrade is done,
 [[transform-aggresponse-limitations]]
 === Aggregation responses may be incompatible with destination index mappings
 
-When a {transform} is first started, it will deduce the mappings 
+When a pivot {transform} is first started, it will deduce the mappings 
 required for the destination index. This process is based on the field types of 
 the source index and the aggregations used. If the fields are derived from 
 <<search-aggregations-metrics-scripted-metric-aggregation,`scripted_metrics`>>
@@ -101,13 +101,13 @@ overflows might occur or dynamically mapped fields might contain both numbers
 and strings. Please check {es} logs if you think this may have occurred. 
 
 You can view the deduced mappings by using the 
-https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html[Preview transform API].
+<<preview-transform,preview transform API>>.
 See the `generated_dest_index` object in the API response.
 
 If it's required, you may define custom mappings prior to starting the 
 {transform} by creating a custom destination index using the 
-https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html[Create index API].
-As deduced mappings cannot be overwritten by an index template, use the Create 
+<<indices-create-index,create index API>>.
+As deduced mappings cannot be overwritten by an index template, use the create 
 index API to define custom mappings. The index templates only apply to fields 
 derived from scripts that use dynamic mappings.