Browse Source

[DOCS] Reorganizes Transforms limitations (#70638)

István Zoltán Szabó 4 years ago
parent
commit
178563c224
1 changed files with 92 additions and 82 deletions
  1. 92 82
      docs/reference/transform/limitations.asciidoc

+ 92 - 82
docs/reference/transform/limitations.asciidoc

@@ -7,55 +7,88 @@
 ++++
 
 The following limitations and known problems apply to the {version} release of 
-the Elastic {transform} feature:
+the Elastic {transform} feature. The limitations are grouped into the following 
+categories:
+
+* <<transform-config-limitations>> apply to the configuration process of the 
+  {transforms}.
+* <<transform-operational-limitations>> affect the behavior of the {transforms} 
+  that are running.
+* <<transform-ui-limitations>> only apply to {transforms} managed via the user 
+  interface.
+
 
 [discrete]
-[[transform-space-limitations]]
-== {transforms-cap} are visible in all {kib} spaces
+[[transform-config-limitations]]
+== Configuration limitations
 
-{kibana-ref}/xpack-spaces.html[Spaces] enable you to organize your source and 
-destination indices and other saved objects in {kib} and to see only the objects 
-that belong to your space. However, this limited scope does not apply to 
-{transforms}; they are visible in all spaces.
+[discrete]
+[[transforms-ccs-limitation]]
+=== {transforms-cap} support {ccs} if the remote cluster is configured properly
+
+If you use <<modules-cross-cluster-search,{ccs}>>, the remote cluster must 
+support the search and aggregations you use in your {transforms}. 
+{transforms-cap} validate their configuration; if you use {ccs} and the 
+validation fails, make sure that the remote cluster supports the query and 
+aggregations you use.
 
 [discrete]
-[[transform-ui-limitation]]
-== {transforms-cap} UI will not work during a rolling upgrade from 7.2
+[[transform-painless-limitation]]
+=== Using scripts in {transforms}
 
-If your cluster contains mixed version nodes, for example during a rolling 
-upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2, 
-the {transforms} UI (earler {dataframe} UI) will not work. Please wait until all 
-nodes have been upgraded to the newer version before using the {transforms} UI.
+{transforms-cap} support scripting in every case when aggregations support them. 
+However, there are certain factors you might want to consider when using scripts 
+in {transforms}:
 
+* {transforms-cap} cannot deduce index mappings for output fields when the 
+  fields are created by a script. In this case, you might want to create the 
+  mappings of the destination index yourself prior to creating the transform.
+
+* Scripted fields may increase the runtime of the {transform}.
+  
+* {transforms-cap} cannot optimize queries when you use scripts for all the 
+  groupings defined in `group_by`, you will receive a warning message when you 
+  use scripts this way.
+  
 [discrete]
-[[transform-rolling-upgrade-limitation]]
-== {transforms-cap} reassignment suspended during a rolling upgrade from 7.2 and 7.3
+[[transform-runtime-field-limitation]]
+=== {transforms-cap} perform better on indexed fields
 
-If your cluster contains mixed version nodes, for example during a rolling
-upgrade from 7.2 or 7.3 to a newer version, {transforms} whose nodes are stopped 
-will not be reassigned until the upgrade is complete. After the upgrade is done, 
-{transforms} resume automatically; no action is required.
+{transforms-cap} sort data by a user-defined time field, which is frequently 
+accessed. If the time field is a {ref}/runtime.html[runtime field], the 
+performance impact of calculating field values at query time can significantly 
+slow the {transform}. Use an indexed field as a time field when using 
+{transforms}.
 
 [discrete]
-[[transform-datatype-limitations]]
-== {dataframe-cap} data type limitation
+[[transform-scheduling-limitations]]
+=== {ctransform-cap} scheduling limitations
+
+A {ctransform} periodically checks for changes to source data. The functionality
+of the scheduler is currently limited to a basic periodic timer which can be 
+within the `frequency` range from 1s to 1h. The default is 1m. This is designed 
+to run little and often. When choosing a `frequency` for this timer consider 
+your ingest rate along with the impact that the {transform} 
+search/index operations has other users in your cluster. Also note that retries 
+occur at `frequency` interval.
 
-{dataframes-cap} do not (yet) support fields containing arrays – in the UI or 
-the API. If you try to create one, the UI will fail to show the source index 
-table.
 
 [discrete]
-[[transform-kibana-limitations]]
-== Up to 1,000 {transforms} are supported
+[[transform-operational-limitations]]
+== Operational limitations
+
+[discrete]
+[[transform-rolling-upgrade-limitation]]
+=== {transforms-cap} reassignment suspended during a rolling upgrade from 7.2 and 7.3
 
-A single cluster will support up to 1,000 {transforms}. When using the 
-<<get-transform,GET {transforms} API>> a total `count` of {transforms} 
-is returned. Use the `size` and `from` parameters to enumerate through the full 
-list.
+If your cluster contains mixed version nodes, for example during a rolling
+upgrade from 7.2 or 7.3 to a newer version, {transforms} whose nodes are stopped 
+will not be reassigned until the upgrade is complete. After the upgrade is done, 
+{transforms} resume automatically; no action is required.
 
 [discrete]
 [[transform-aggresponse-limitations]]
-== Aggregation responses may be incompatible with destination index mappings
+=== Aggregation responses may be incompatible with destination index mappings
 
 When a {transform} is first started, it will deduce the mappings 
 required for the destination index. This process is based on the field types of 
@@ -80,7 +113,7 @@ derived from scripts that use dynamic mappings.
 
 [discrete]
 [[transform-batch-limitations]]
-== Batch {transforms} may not account for changed documents
+=== Batch {transforms} may not account for changed documents
 
 A batch {transform} uses a 
 <<search-aggregations-bucket-composite-aggregation,composite aggregation>>
@@ -91,7 +124,7 @@ results may not include these changes.
 
 [discrete]
 [[transform-consistency-limitations]]
-== {ctransform-cap} consistency does not account for deleted or updated documents
+=== {ctransform-cap} consistency does not account for deleted or updated documents
 
 While the process for {transforms} allows the continual recalculation of the 
 {transform} as new data is being ingested, it does also have some limitations.
@@ -115,7 +148,7 @@ viewing the destination index.
 
 [discrete]
 [[transform-deletion-limitations]]
-== Deleting a {transform} does not delete the destination index or {kib} index pattern
+=== Deleting a {transform} does not delete the destination index or {kib} index pattern
 
 When deleting a {transform} using `DELETE _transform/index` 
 neither the destination index nor the {kib} index pattern, should one have been 
@@ -123,7 +156,7 @@ created, are deleted. These objects must be deleted separately.
 
 [discrete]
 [[transform-aggregation-page-limitations]]
-== Handling dynamic adjustment of aggregation page size
+=== Handling dynamic adjustment of aggregation page size
 
 During the development of {transforms}, control was favoured over performance. 
 In the design considerations, it is preferred for the {transform} to take longer 
@@ -152,7 +185,7 @@ its minimum, then the {transform} will be set to a failed state.
 
 [discrete]
 [[transform-dynamic-adjustments-limitations]]
-== Handling dynamic adjustments for many terms
+=== Handling dynamic adjustments for many terms
 
 For each checkpoint, entities are identified that have changed since the last 
 time the check was performed. This list of changed entities is supplied as a 
@@ -172,21 +205,9 @@ is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the
 Using smaller values for `max_page_search_size` may result in a longer duration 
 for the {transform} checkpoint to complete.
 
-[discrete]
-[[transform-scheduling-limitations]]
-== {ctransform-cap} scheduling limitations
-
-A {ctransform} periodically checks for changes to source data. The functionality
-of the scheduler is currently limited to a basic periodic timer which can be 
-within the `frequency` range from 1s to 1h. The default is 1m. This is designed 
-to run little and often. When choosing a `frequency` for this timer consider 
-your ingest rate along with the impact that the {transform} 
-search/index operations has other users in your cluster. Also note that retries 
-occur at `frequency` interval.
-
 [discrete]
 [[transform-failed-limitations]]
-== Handling of failed {transforms}
+=== Handling of failed {transforms}
 
 Failed {transforms} remain as a persistent task and should be handled 
 appropriately, either by deleting it or by resolving the root cause of the 
@@ -197,7 +218,7 @@ When using the API to delete a failed {transform}, first stop it using
 
 [discrete]
 [[transform-availability-limitations]]
-== {ctransforms-cap} may give incorrect results if documents are not yet available to search
+=== {ctransforms-cap} may give incorrect results if documents are not yet available to search
 
 After a document is indexed, there is a very small delay until it is available 
 to search.
@@ -214,7 +235,7 @@ issue will occur.
 
 [discrete]
 [[transform-date-nanos]]
-== Support for date nanoseconds data type
+=== Support for date nanoseconds data type
 
 If your data uses the <<date_nanos,date nanosecond data type>>, aggregations
 are nonetheless on millisecond resolution. This limitation also affects the
@@ -223,7 +244,7 @@ aggregations in your {transforms}.
 
 [discrete]
 [[transform-data-streams-destination]]
-== Data streams as destination indices are not supported
+=== Data streams as destination indices are not supported
 
 {transforms-cap} update data in the destination index which requires writing 
 into the destination. <<data-streams>> are designed to be append-only, which 
@@ -234,7 +255,7 @@ this reason, data streams are not supported as destination indices for
 
 [discrete]
 [[transform-ilm-destination]]
-== ILM as destination index may cause duplicated documents
+=== ILM as destination index may cause duplicated documents
 
 <<index-lifecycle-management,ILM>> is not recommended to use as a {transform} 
 destination index. {transforms-cap} update documents in the current destination, 
@@ -248,40 +269,29 @@ documents if your {transform} contains a `group_by` based on `date_histogram`.
 
 
 [discrete]
-[[transform-painless-limitation]]
-== Using scripts in {transforms}
+[[transform-ui-limitations]]
+== Limitations in {kib}
 
-{transforms-cap} support scripting in every case when aggregations support them. 
-However, there are certain factors you might want to consider when using scripts 
-in {transforms}:
-
-* {transforms-cap} cannot deduce index mappings for output fields when the 
-  fields are created by a script. In this case, you might want to create the 
-  mappings of the destination index yourself prior to creating the transform.
+[discrete]
+[[transform-space-limitations]]
+=== {transforms-cap} are visible in all {kib} spaces
 
-* Scripted fields may increase the runtime of the {transform}.
-  
-* {transforms-cap} cannot optimize queries when you use scripts for all the 
-  groupings defined in `group_by`, you will receive a warning message when you 
-  use scripts this way.
-  
+{kibana-ref}/xpack-spaces.html[Spaces] enable you to organize your source and 
+destination indices and other saved objects in {kib} and to see only the objects 
+that belong to your space. However, this limited scope does not apply to 
+{transforms}; they are visible in all spaces.
 
 [discrete]
-[[transform-runtime-field-limitation]]
-=== {transforms-cap} perform better on indexed fields
-
-{transforms-cap} sort data by a user-defined time field, which is frequently 
-accessed. If the time field is a {ref}/runtime.html[runtime field], the 
-performance impact of calculating field values at query time can significantly 
-slow the {transform}. Use an indexed field as a time field when using 
-{transforms}.
+[[transform-rolling-upgrade-ui-limitation]]
+=== {transforms-cap} UI will not work during a rolling upgrade from 7.2
 
+If your cluster contains mixed version nodes, for example during a rolling 
+upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2, 
+the {transforms} UI (earler {dataframe} UI) will not work. Please wait until all 
+nodes have been upgraded to the newer version before using the {transforms} UI.
 
 [discrete]
-[[transforms-ccs-limitation]]
-=== {transforms-cap} support {ccs} if the remote cluster is configured properly
+[[transform-kibana-limitations]]
+=== Up to 1,000 {transforms} are listed in {kib}
 
-If you use <<modules-cross-cluster-search,{ccs}>>, the remote cluster must 
-support the search and aggregations you use in your {transforms}. 
-{transforms-cap} validate their configuration; if you use {ccs} and the validation fails, 
-make sure that the remote cluster supports the query and aggregations you use.
+The {transforms} management page in {kib} lists up to 1000 {transforms}.