[DOCS] [Enterprise Search] Migrate ingest pipelines/ML docs (#101156)

* WIP, port docs

- Update link syntax
- Update ids
- Fix n^n build failures :/
-

* Fix id for doclink

* Let's try this on for size

* Idem

* Update attributes, Test image rendering

* Update image name

* Fix typo

* Update filename

* Add images, cleanup, standardize naming

* Tweak heading

* Cleanup, rewordings

- Modified introduction in `search-inference-processing.asciidoc`.
- Changed "Search connector" to "Elastic connector".
- Adjusted heading levels in `search-inference-processing.asciidoc`.
- Simplified ingest pipelines intro in `search-ingest-pipelines.asciidoc`.
- Edited ingest pipelines section for the *Content* UI.
- Reordered file inclusions in `search-ingest-pipelines.asciidoc`.
- Formatted inference pipeline creation into steps in `search-nlp-tutorial.asciidoc`.

* Lingering erroneousness

* Delete FAQ

docs/reference/ingest.asciidoc

@@ -990,3 +990,4 @@ GET _nodes/stats/ingest?filter_path=nodes.*.ingest
 include::ingest/common-log-format-example.asciidoc[]
 include::ingest/enrich.asciidoc[]
 include::ingest/processors.asciidoc[]
+include::ingest/search-ingest-pipelines.asciidoc[]

docs/reference/ingest/search-inference-processing.asciidoc

@@ -0,0 +1,187 @@
+[[ingest-pipeline-search-inference]]
+=== Inference processing
+
+When you create an index through the **Content** UI, a set of default ingest pipelines are also created, including a ML inference pipeline.
+The <<ingest-pipeline-search-details-specific-ml-reference, ML inference pipeline>> uses inference processors to analyze fields and enrich documents with the output.
+Inference processors use ML trained models, so you need to use a built-in model or {ml-docs}/ml-nlp-deploy-models.html[deploy a trained model in your cluster^] to use this feature.
+
+This guide focuses on the ML inference pipeline, its use, and how to manage it.
+
+[IMPORTANT]
+====
+This feature is not available at all Elastic subscription levels.
+Refer to the Elastic subscriptions pages for https://www.elastic.co/subscriptions/cloud[Elastic Cloud^] and https://www.elastic.co/subscriptions[self-managed] deployments.
+====
+
+[discrete#ingest-pipeline-search-inference-nlp-use-cases]
+==== NLP use cases
+
+{ml-docs}/ml-nlp-overview.html[Natural Language Processing (NLP)^] allows developers to create rich search experiences that go beyond the standards of lexical search.
+A few examples of ways to improve search experiences through the use of NLP models:
+
+[discrete#ingest-pipeline-search-inference-elser]
+==== ELSER text expansion
+
+Using Elastic's {ml-docs}/ml-nlp-elser.html[ELSER machine learning model^] you can easily incorporate text expansion for your queries.
+This works by using ELSER to provide semantic enrichments to your documents upon ingestion, combined with the power of <<search-application-overview, Elastic Search Application templates>> to provide automated text expansion at query time.
+
+[discrete#ingest-pipeline-search-inference-ner]
+==== Named entity recognition (NER)
+
+Most commonly used to detect entities such as People, Places, and Organization information from text, {ml-docs}/ml-nlp-extract-info.html#ml-nlp-ner[NER^] can be used to extract key information from text and group results based on that information.
+A sports news media site could use NER to automatically extract names of professional athletes, stadiums, and sports teams in their articles and link to season stats or schedules.
+
+[discrete#ingest-pipeline-search-inference-text-classification]
+==== Text classification
+
+{ml-docs}/ml-nlp-classify-text.html#ml-nlp-text-classification[Text classification^] is commonly used for sentiment analysis and can be used for similar tasks, such as labeling content as containing hate speech in public forums, or triaging and labeling support tickets so they reach the correct level of escalation automatically.
+
+[discrete#ingest-pipeline-search-inference-text-embedding]
+==== Text embedding
+
+Analyzing a text field using a {ml-docs}/ml-nlp-search-compare.html#ml-nlp-text-embedding[Text embedding^] model will generate a <<dense-vector, dense vector>> representation of the text.
+This array of numeric values encodes the semantic _meaning_ of the text.
+Using the same model with a user's search query will produce a vector that can then be used to search, ranking results based on vector similarity - semantic similarity - as opposed to traditional word or text similarity.
+
+A common use case is a user searching FAQs, or a support agent searching a knowledge base, where semantically similar content may be indexed with little similarity in phrasing.
+
+[discrete#ingest-pipeline-search-inference-nlp-in-enterprise-search]
+==== NLP in Content UI
+
+[discrete#ingest-pipeline-search-inference-overview]
+===== Overview of ML inference pipeline
+
+The diagram below shows how documents are processed during ingestion.
+
+// Original diagram: https://whimsical.com/ml-in-enterprise-search-ErCetPqrcCPu2QYHvAwrgP@2bsEvpTYSt1Hiuq6UBf68tUWvFiXdzLt6ao
+image::../images/ingest/document-enrichment-diagram.png["ML inference pipeline diagram"]
+
+* Documents are processed by the `my-index-0001` pipeline, which happens automatically when indexing through a an Elastic connector or crawler.
+* The `_run_ml_inference` field is set to `true` to ensure the ML inference pipeline (`my-index-0001@ml-inference`) is executed.
+  This field is removed during the ingestion process.
+* The inference processor analyzes the `message` field on the document using the `my-positivity-model-id` trained model.
+  The inference output is stored in the `ml.inference.positivity_prediction` field.
+* The resulting enriched document is then indexed into the `my-index-0001` index.
+* The `ml.inference.positivity_prediction` field can now be used at query time to search for documents above or below a certain threshold.
+
+[discrete#ingest-pipeline-search-inference-find-deploy-manage-trained-models]
+==== Find, deploy, and manage trained models
+
+This feature is intended to make it easier to use your ML trained models.
+First, you need to figure out which model works best for your data.
+Make sure to use a {ml-docs}/ml-nlp-model-ref.html[compatible third party NLP model^].
+Since these are publicly available, it is not possible to fine-tune models before {ml-docs}/ml-nlp-deploy-models.html[deploying them^].
+
+Trained models must be available in the current {kibana-ref}/xpack-spaces.html[Kibana Space^] and running in order to use them.
+By default, models should be available in all Kibana Spaces that have the *Analytics* > *Machine Learning* feature enabled.
+To manage your trained models, use the Kibana UI and navigate to *Stack Management -> Machine Learning -> Trained Models*.
+Spaces can be controlled in the **spaces** column.
+To stop or start a model, go to the *Machine Learning* tab in the *Analytics* menu of Kibana and click *Trained Models* in the *Model Management* section.
+
+[NOTE]
+=========================
+The `monitor_ml` <<security-privileges, Elasticsearch cluster privilege>> is required to manage ML models and ML inference pipelines which use those models.
+=========================
+
+[discrete#ingest-pipeline-search-inference-add-inference-processors]
+===== Add inference processors to your ML inference pipeline
+
+To create the index-specific ML inference pipeline, go to *Search -> Content -> Indices -> <your index> -> Pipelines* in the Kibana UI.
+
+If you only see the `ent-search-generic-ingestion` pipeline, you will need to click *Copy and customize* to create index-specific pipelines.
+This will create the `{index_name}@ml-inference` pipeline.
+
+Once your index-specific ML inference pipeline is ready, you can add inference processors that use your ML trained models.
+To add an inference processor to the ML inference pipeline, click the *Add Inference Pipeline* button in the *Machine Learning Inference Pipelines* card.
+
+[role="screenshot"]
+image::../images/ingest/document-enrichment-add-inference-pipeline.png["Add Inference Pipeline"]
+
+Here, you'll be able to:
+
+1. Choose a name for your pipeline.
+  - This name will need to be unique across the whole deployment.
+  If you want this pipeline to be index-specific, we recommend including the name of your index in the pipeline name.
+2. Select the ML trained model you want to use.
+3. Select one or more source fields as input for the inference processor.
+  - If there are no source fields available, your index will need a <<mapping, field mapping>>.
+4. (Optional) Choose a name for your target field.
+This is where the output of the inference model will be stored. Changing the default name is only possible if you have a single source field selected.
+5. Add the source-target field mapping to the configuration by clicking the *Add* button.
+6. Repeat steps 3-5 for each field mapping you want to add.
+7. (Optional) Test the pipeline with a sample document.
+8. (Optional) Review the pipeline definition before creating it with the *Create pipeline* button.
+
+[discrete#ingest-pipeline-search-inference-manage-inference-processors]
+===== Manage and delete inference processors from your ML inference pipeline
+
+Inference processors added to your index-specific ML inference pipelines are normal Elasticsearch pipelines.
+Once created, each processor will have options to *View in Stack Management* and *Delete Pipeline*.
+Deleting an inference processor from within the *Content* UI deletes the pipeline and also removes its reference from your index-specific ML inference pipeline.
+
+These pipelines can also be viewed, edited, and deleted in Kibana via *Stack Management -> Ingest Pipelines*, just like all other Elasticsearch ingest pipelines.
+You may also use the <<ingest-apis,Ingest pipeline APIs>>.
+If you delete any of these pipelines outside of the *Content* UI in Kibana, make sure to edit the ML inference pipelines that reference them.
+
+[discrete#ingest-pipeline-search-inference-update-mapping]
+==== Update mappings to use ML inference pipelines
+
+After setting up an ML inference pipeline or attaching an existing one, it may be necessary to manually create the field mappings in order to support the referenced trained ML model's output.
+This needs to happen before the pipeline is first used to index some documents, otherwise the model output fields could be inferred with the wrong type.
+
+[NOTE]
+====
+This doesn't apply when you're creating a pipeline with the ELSER model, for which the index mappings are automatically updated in the process. 
+====
+
+The required field name and type depends on the configuration of the pipeline and the trained model it uses.
+For example, if you configure a `text_embedding` model, select `summary` as a source field, and `ml.inference.summary` as the target field, the inference output will be stored in `ml.inference.<source field name>.predicted_value` as a <<dense-vector, dense_vector>> type.
+In order to support semantic search on this field, it must be added to the mapping:
+
+[source,console]
+----
+PUT my-index-0001/_mapping
+{
+  "properties": {
+    "ml.inference.summary.predicted_value": { <1>
+      "type": "dense_vector", <2>
+      "dims": 768, <3>
+      "index": true,
+      "similarity": "dot_product" 
+    }
+  }
+}
+----
+// NOTCONSOLE
+// TEST[skip:TODO]
+
+<1> The output of the ML model is stored in the configured target field suffixed with `predicted_value`.
+<2> Choose a field type that is compatible with the inference output and supports your search use cases.
+<3> Set additional properties as necessary.
+
+[TIP]
+====
+You can check the shape of the generated output before indexing any documents while creating the ML inference pipeline under the *Test* tab.
+Simply provide a sample document, click *Simulate*, and look for the `ml.inference` object in the results.
+====
+
+[discrete#ingest-pipeline-search-inference-test-inference-pipeline]
+==== Test your ML inference pipeline
+
+To ensure the ML inference pipeline will be run when ingesting documents, you must make sure the documents you are ingesting have a field named `_run_ml_inference` that is set to `true` and you must set the pipeline to `{index_name}`.
+For connector and crawler indices, this will happen automatically if you've configured the settings appropriately for the pipeline name `{index_name}`.
+To manage these settings:
+
+  1. Go to *Search > Content > Indices > <your index> > Pipelines*.
+  2. Click on the *Settings* link in the *Ingest Pipelines* card for the `{index_name}` pipeline.
+  3. Ensure *ML inference pipelines* is selected.
+     If it is not, select it and save the changes.
+
+[discrete#ingest-pipeline-search-inference-learn-more]
+==== Learn More
+
+* See <<ingest-pipeline-search-in-enterprise-search>> for information on the various pipelines that are created.
+* Learn about {ml-docs}/ml-nlp-elser.html[ELSER], Elastic's proprietary retrieval model for semantic search with sparse vectors.
+* https://huggingface.co/models?library=pytorch&pipeline_tag=token-classification&sort=downloads[NER HuggingFace Models^]
+* https://huggingface.co/models?library=pytorch&pipeline_tag=text-classification&sort=downloads[Text Classification HuggingFace Models^]
+* https://huggingface.co/models?library=pytorch&pipeline_tag=sentence-similarity&sort=downloads[Text Embedding HuggingFace Models^]

docs/reference/ingest/search-ingest-pipelines.asciidoc

@@ -0,0 +1,280 @@
+[[ingest-pipeline-search]]
+== Ingest pipelines in Search
+
+You can manage ingest pipelines through Elasticsearch APIs or Kibana UIs.
+
+The *Content* UI under *Search* has a set of tools for creating and managing indices optimized for search use cases (non time series data).
+You can also manage your ingest pipelines in this UI.
+
+[discrete]
+[[ingest-pipeline-search-where]]
+=== Find pipelines in Content UI
+
+To work with ingest pipelines using these UI tools, you'll be using the *Pipelines* tab on your search-optimized Elasticsearch index.
+
+To find this tab in the Kibana UI:
+
+1. Go to *Search > Content > Elasticsearch indices*.
+2. Select the index you want to work with. For example, `search-my-index`.
+3. On the index's overview page, open the *Pipelines* tab.
+4. From here, you can follow the instructions to create custom pipelines, and set up ML inference pipelines.
+
+The tab is highlighted in this screenshot:
+
+[.screenshot]
+image::../images/ingest/ingest-pipeline-ent-search-ui.png[align="center"]
+
+[discrete#ingest-pipeline-search-in-enterprise-search]
+=== Overview
+
+These tools can be particularly helpful by providing a layer of customization and post-processing of documents.
+For example:
+
+* providing consistent extraction of text from binary data types
+* ensuring consistent formatting
+* providing consistent sanitization steps (removing PII like phone numbers or SSN's)
+
+It can be a lot of work to set up and manage production-ready pipelines from scratch.
+Considerations such as error handling, conditional execution, sequencing, versioning, and modularization must all be taken into account.
+
+To this end, when you create indices for search use cases, (including {enterprise-search-ref}/crawler.html[Elastic web crawler], {enterprise-search-ref}/connectors.html[Elastic connector], and API indices), each index already has a pipeline set up with several processors that optimize your content for search.
+
+This pipeline is called `ent-search-generic-ingestion`.
+While it is a "managed" pipeline (meaning it should not be tampered with), you can view its details via the Kibana UI or the Elasticsearch API.
+You can also <<ingest-pipeline-search-details-generic-reference,read more about its contents below>>.
+
+You can control whether you run some of these processors.
+While all features are enabled by default, they are eligible for opt-out.
+For {enterprise-search-ref}/crawler.html[Elastic crawler] and {enterprise-search-ref}/connectors.html[Elastic connectors], you can opt out (or back in) per index, and your choices are saved.
+For API indices, you can opt out (or back in) by including specific fields in your documents.
+<<ingest-pipeline-search-pipeline-settings-using-the-api,See below for details>>.
+
+At the deployment level, you can change the default settings for all new indices.
+This will not effect existing indices.
+
+Each index also provides the capability to easily create index-specific ingest pipelines with customizable processing.
+If you need that extra flexibility, you can create a custom pipeline by going to your pipeline settings and choosing to "copy and customize".
+This will replace the index's use of `ent-search-generic-ingestion` with 3 newly generated pipelines:
+
+1. `<index-name>`
+2. `<index-name>@custom`
+3. `<index-name>@ml-inference`
+
+Like `ent-search-generic-ingestion`, the first of these is "managed", but the other two can and should be modified to fit your needs.
+You can view these pipelines using the platform tools (Kibana UI, Elasticsearch API), and can also 
+<<ingest-pipeline-search-details-specific,read more about their content below>>.
+
+[discrete#ingest-pipeline-search-pipeline-settings]
+=== Pipeline Settings
+
+Aside from the pipeline itself, you have a few configuration options which control individual features of the pipelines.
+
+* **Extract Binary Content** - This controls whether or not binary documents should be processed and any textual content should be extracted.
+* **Reduce Whitespace** - This controls whether or not consecutive, leading, and trailing whitespaces should be removed.
+  This can help to display more content in some search experiences.
+* **Run ML Inference** - Only available on index-specific pipelines.
+  This controls whether or not the optional `<index-name>@ml-inference` pipeline will be run.
+  Enabled by default.
+
+For Elastic web crawler and connectors, you can opt in or out per index.
+These settings are stored in Elasticsearch in the `.elastic-connectors` index, in the document that corresponds to the specific index.
+These settings can be changed there directly, or through the Kibana UI at *Search > Content > Indices > <your index> > Pipelines > Settings*.
+
+You can also change the deployment wide defaults.
+These settings are stored in the Elasticsearch mapping for `.elastic-connectors` in the `_meta` section.
+These settings can be changed there directly, or from the Kibana UI at *Search > Content > Settings* tab.
+Changing the deployment wide defaults will not impact any existing indices, but will only impact any newly created indices defaults.
+Those defaults will still be able to be overriden by the index-specific settings.
+
+[discrete#ingest-pipeline-search-pipeline-settings-using-the-api]
+==== Using the API
+
+These settings are not persisted for indices that "Use the API".
+Instead, changing these settings will, in real time, change the example cURL request displayed.
+Notice that the example document in the cURL request contains three underscore-prefixed fields:
+
+[source,js]
+----
+{
+  ...
+  "_extract_binary_content": true,
+  "_reduce_whitespace": true,
+  "_run_ml_inference": true
+}
+----
+// NOTCONSOLE
+
+Omitting one of these special fields is the same as specifying it with the value `false`.
+
+[NOTE]
+=========================
+You must also specify the pipeline in your indexing request.
+This is also shown in the example cURL request.
+=========================
+
+[WARNING]
+=========================
+If the pipeline is not specified, the underscore-prefixed fields will actually be indexed, and will not impact any processing behaviors.
+=========================
+
+[discrete#ingest-pipeline-search-details]
+=== Details
+
+[discrete#ingest-pipeline-search-details-generic-reference]
+==== `ent-search-generic-ingestion` Reference
+
+You can access this pipeline with the <<get-pipeline-api, Elasticsearch Ingest Pipelines API>> or via Kibana's <<create-manage-ingest-pipelines,Stack Management > Ingest Pipelines>> UI.
+
+[WARNING]
+=========================
+This pipeline is a "managed" pipeline.
+That means that it is not intended to be edited.
+Editing/updating this pipeline manually could result in unintended behaviors, or difficulty in upgrading in the future.
+If you want to make customizations, we recommend you utilize index-specific pipelines (see below), specifically <<ingest-pipeline-search-details-specific-custom-reference,the `<index-name>@custom` pipeline>>.
+=========================
+
+[discrete#ingest-pipeline-search-details-generic-reference-processors]
+===== Processors
+
+1. `attachment` - this uses the <<attachment, Attachment>> processor to convert any binary data stored in a document's `_attachment` field to a nested object of plain text and metadata.
+2. `set_body` - this uses the <<set-processor, Set>> processor to copy any plain text extracted from the previous step and persist it on the document in the `body` field.
+3. `remove_replacement_chars` - this uses the <<gsub-processor, Gsub>> processor to remove characters like "�" from the `body` field.
+4. `remove_extra_whitespace` - this uses the <<gsub-processor, Gsub>> processor to replace consecutive whitespace characters with single spaces in the `body` field.
+  While not perfect for every use case (see below for how to disable), this can ensure that search experiences display more content and highlighting and less empty space for your search results.
+5. `trim` - this uses the <<trim-processor, Trim>> processor to remove any remaining leading or trailing whitespace from the `body` field.
+6. `remove_meta_fields` - this final step of the pipeline uses the <<remove-processor, Remove>> processor to remove special fields that may have been used elsewhere in the pipeline, whether as temporary storage or as control flow parameters.
+
+[discrete#ingest-pipeline-search-details-generic-reference-params]
+===== Control flow parameters
+
+The `ent-search-generic-ingestion` pipeline does not always run all processors.
+It utilizes a feature of ingest pipelines to <<conditionally-run-processor,conditionally run processors>> based on the contents of each individual document.
+
+* `_extract_binary_content` - if this field is present and has a value of `true` on a source document, the pipeline will attempt to run the `attachment`, `set_body`, and `remove_replacement_chars` processors.
+  Note that the document will also need an `_attachment` field populated with base64-encoded binary data in order for the `attachment` processor to have any output.
+  If the `_extract_binary_content` field is missing or `false` on a source document, these processors will be skipped.
+* `_reduce_whitespace` - if this field is present and has a value of `true` on a source document, the pipeline will attempt to run the `remove_extra_whitespace` and `trim` processors.
+  These processors only apply to the `body` field.
+  If the `_reduce_whitespace` field is missing or `false` on a source document, these processors will be skipped.
+
+Crawler, Native Connectors, and Connector Clients will automatically add these control flow parameters based on the settings in the index's Pipeline tab.
+To control what settings any new indices will have upon creation, see the deployment wide content settings.
+See <<ingest-pipeline-search-pipeline-settings>>.
+
+[discrete#ingest-pipeline-search-details-specific]
+==== Index-specific ingest pipelines
+
+In the Kibana UI for your index, by clicking on the Pipelines tab, then *Settings > Copy and customize*, you can quickly generate 3 pipelines which are specific to your index.
+These 3 pipelines replace `ent-search-generic-ingestion` for the index.
+There is nothing lost in this action, as the `<index-name>` pipeline is a superset of functionality over the `ent-search-generic-ingestion` pipeline.
+
+[IMPORTANT]
+====
+The "copy and customize" button is not available at all Elastic subscription levels.
+Refer to the Elastic subscriptions pages for https://www.elastic.co/subscriptions/cloud[Elastic Cloud^] and https://www.elastic.co/subscriptions[self-managed] deployments.
+====
+
+[discrete#ingest-pipeline-search-details-specific-reference]
+===== `<index-name>` Reference
+
+This pipeline looks and behaves a lot like the <<ingest-pipeline-search-details-generic-reference,`ent-search-generic-ingestion` pipeline>>, but with <<ingest-pipeline-search-details-specific-reference-processors,two additional processors>>.
+
+[WARNING]
+=========================
+You should not rename this pipeline.
+=========================
+
+[WARNING]
+=========================
+This pipeline is a "managed" pipeline.
+That means that it is not intended to be edited.
+Editing/updating this pipeline manually could result in unintended behaviors, or difficulty in upgrading in the future.
+If you want to make customizations, we recommend you utilize <<ingest-pipeline-search-details-specific-custom-reference,the `<index-name>@custom` pipeline>>.
+=========================
+
+[discrete#ingest-pipeline-search-details-specific-reference-processors]
+====== Processors
+
+In addition to the processors inherited from the <<ingest-pipeline-search-details-generic-reference,`ent-search-generic-ingestion` pipeline>>, the index-specific pipeline also defines:
+
+* `index_ml_inference_pipeline` - this uses the <<pipeline-processor, Pipeline>> processor to run the `<index-name>@ml-inference` pipeline.
+  This processor will only be run if the source document includes a `_run_ml_inference` field with the value `true`.
+* `index_custom_pipeline` - this uses the <<pipeline-processor, Pipeline>> processor to run the `<index-name>@custom` pipeline.
+
+[discrete#ingest-pipeline-search-details-specific-reference-params]
+====== Control flow parameters
+
+Like the `ent-search-generic-ingestion` pipeline, the `<index-name>` pipeline does not always run all processors.
+In addition to the `_extract_binary_content` and `_reduce_whitespace` control flow parameters, the `<index-name>` pipeline also supports:
+
+* `_run_ml_inference` - if this field is present and has a value of `true` on a source document, the pipeline will attempt to run the `index_ml_inference_pipeline` processor.
+  If the `_run_ml_inference` field is missing or `false` on a source document, this processor will be skipped.
+
+Crawler, Native Connectors, and Connector Clients will automatically add these control flow parameters based on the settings in the index's Pipeline tab.
+To control what settings any new indices will have upon creation, see the deployment wide content settings.
+See <<ingest-pipeline-search-pipeline-settings>>.
+
+[discrete#ingest-pipeline-search-details-specific-ml-reference]
+===== `<index-name>@ml-inference` Reference
+
+This pipeline is empty to start (no processors), but can be added to via the Kibana UI either through the Pipelines tab of your index, or from the *Stack Management > Ingest Pipelines* page.
+Unlike the `ent-search-generic-ingestion` pipeline and the `<index-name>` pipeline, this pipeline is NOT "managed".
+
+It's possible to add one or more ML inference pipelines to an index in the *Content* UI.
+This pipeline will serve as a container for all of the ML inference pipelines configured for the index.
+Each ML inference pipeline added to the index is referenced within `<index-name>@ml-inference` using a `pipeline` processor.
+
+[WARNING]
+=========================
+You should not rename this pipeline.
+=========================
+
+[NOTE]
+=========================
+The `monitor_ml` Elasticsearch cluster permission is required in order to manage ML models and ML inference pipelines which use those models.
+=========================
+
+[discrete#ingest-pipeline-search-details-specific-custom-reference]
+===== `<index-name>@custom` Reference
+
+This pipeline is empty to start (no processors), but can be added to via the Kibana UI either through the Pipelines
+tab of your index, or from the *Stack Management > Ingest Pipelines* page.
+Unlike the `ent-search-generic-ingestion` pipeline and the `<index-name>` pipeline, this pipeline is NOT "managed".
+
+You are encouraged to make additions and edits to this pipeline, provided its name remains the same.
+This provides a convenient hook from which to add custom processing and transformations for your data.
+Be sure to read the <<ingest, docs for ingest pipelines>> to see what options are available.
+
+[WARNING]
+=========================
+You should not rename this pipeline.
+=========================
+
+[discrete#ingest-pipeline-search-upgrading-notes]
+=== Upgrading notes
+
+.Expand to see upgrading notes
+[%collapsible%closed]
+=============
+
+* `app_search_crawler` - Since 8.3, {app-search-crawler} has utilized this pipeline to power its binary content
+extraction.
+  You can read more about this pipeline and its usage in the {app-search-ref}/web-crawler-reference.html#web-crawler-reference-binary-content-extraction[App Search Guide].
+  When upgrading from 8.3 to 8.5+, be sure to note any changes that you made to the `app_search_crawler` pipeline.
+  These changes should be re-applied to each index's `<index-name>@custom` pipeline in order to ensure a consistent data processing experience.
+  In 8.5+, the <<ingest-pipeline-search-pipeline-settings, index setting to enable binary content>> is required *in addition* to the configurations mentioned in the {app-search-ref}/web-crawler-reference.html#web-crawler-reference-binary-content-extraction[App Search Guide].
+
+* `ent_search_crawler` - Since 8.4, the Elastic web crawler has utilized this pipeline to power its binary content extraction.
+  You can read more about this pipeline and its usage in the {enterprise-search-ref}/crawler-managing.html#crawler-managing-binary-content[Elastic web crawler Guide].
+  When upgrading from 8.4 to 8.5+, be sure to note any changes that you made to the `ent_search_crawler` pipeline.
+These changes should be re-applied to each index's `<index-name>@custom` pipeline in order to ensure a consistent data processing experience.
+  In 8.5+, the <<ingest-pipeline-search-pipeline-settings, index setting to enable binary content>> is required *in addition* to the configurations mentioned in the {enterprise-search-ref}/crawler-managing.html#crawler-managing-binary-content[Elastic web crawler Guide].
+
+* `ent-search-generic-ingestion` - Since 8.5, Native Connectors, Connector Clients, and new (>8.4) Elastic web crawler indices will all make use of this pipeline by default.
+  You can <<ingest-pipeline-search-details-generic-reference, read more about this pipeline>> above.
+  As this pipeline is "managed", any modifications that were made to `app_search_crawler` and/or `ent_search_crawler` should NOT be made to `ent-search-generic-ingestion`.
+  Instead, if such customizations are desired, you should utilize <<ingest-pipeline-search-details-specific>>, placing all modifications in the `<index-name>@custom` pipeline(s).
+=============
+
+include::search-inference-processing.asciidoc[]
+include::search-nlp-tutorial.asciidoc[]

docs/reference/ingest/search-nlp-tutorial.asciidoc

@@ -0,0 +1,259 @@
+[[nlp-example]]
+=== Tutorial: Natural language processing (NLP)
+++++
+<titleabbrev>NLP tutorial</titleabbrev>
+++++
+
+This guide focuses on a concrete task: getting a machine learning trained model loaded into Elasticsearch and set up to enrich your documents.
+
+Elasticsearch supports many different ways to use machine learning models.
+In this guide, we will use a trained model to enrich documents at ingest time using ingest pipelines configured within Kibana's *Content* UI.
+
+In this guide, we'll accomplish the above using the following steps:
+
+- *Set up a Cloud deployment*: We will use Elastic Cloud to host our deployment, as it makes it easy to scale machine learning nodes.
+- *Load a model with Eland*: We will use the Eland Elasticsearch client to import our chosen model into Elasticsearch.
+Once we've verified that the model is loaded, we will be able to use it in an ingest pipeline.
+- *Setup an ML inference pipeline*: We will create an Elasticsearch index with a predefined mapping and add an inference pipeline.
+- *Show enriched results*: We will ingest some data into our index and observe that the pipeline enriches our documents.
+
+Follow the instructions to load a text classification model and set it up to enrich some photo comment data.
+Once you're comfortable with the steps involved, use this guide as a blueprint for working with other machine learning trained models.
+
+*Table of contents*:
+
+* <<nlp-example-cloud-deployment>>
+* <<nlp-example-clone-eland>>
+* <<nlp-example-deploy-model>>
+* <<nlp-example-create-index-and-define-ml-inference-pipeline>>
+* <<nlp-example-index-documents>>
+* <<nlp-example-summary>>
+* <<nlp-example-learn-more>>
+
+[discrete#nlp-example-cloud-deployment]
+==== Create an {ecloud} deployment
+
+Your deployment will need a machine learning instance to upload and deploy trained models.
+
+If your team already has an Elastic Cloud deployment, make sure it has at least one machine learning instance.
+If it does not, *Edit* your deployment to add capacity.
+For this tutorial, we'll need at least 2GB of RAM on a single machine learning instance.
+
+If your team does not have an Elastic Cloud deployment, start by signing up for a https://cloud.elastic.co/registration[free Elastic Cloud trial^].
+After creating an account, you'll have an active subscription and you'll be prompted to create your first deployment.
+
+Follow the steps to *Create* a new deployment.
+Make sure to add capacity to the *Machine Learning instances* under the *Advanced settings* before creating the deployment.
+To simplify scaling, turn on the *Autoscale this deployment* feature.
+If you use autoscaling, you should increase the minimum RAM for the machine learning instance.
+For this tutorial, we'll need at least 2GB of RAM.
+For more details, refer to {cloud}/ec-create-deployment.html[Create a deployment^] in the Elastic Cloud documentation.
+
+Enriching documents using machine learning was introduced in Enterprise Search *8.5.0*, so be sure to use version *8.5.0 or later*.
+
+[discrete#nlp-example-clone-eland]
+==== Clone Eland
+
+Elastic's https://github.com/elastic/eland[Eland^] tool makes it easy to upload trained models to your deployment via Docker.
+
+Eland is a specialized Elasticsearch client for exploring and manipulating data, which we can use to upload trained models into Elasticsearch.
+
+To clone and build Eland using Docker, run the following commands:
+
+[source,sh]
+----
+git clone git@github.com:elastic/eland.git
+cd eland
+docker build -t elastic/eland .
+----
+
+[discrete#nlp-example-deploy-model]
+==== Deploy the trained model
+
+Now that you have a deployment and a way to upload models, you will need to choose a trained model that fits your data.
+https://huggingface.co/[Hugging Face^] has a large repository of publicly available trained models.
+The model you choose will depend on your data and what you would like to do with it.
+
+For the purposes of this guide, let's say we have a data set of photo comments.
+In order to promote a positive atmosphere on our platform, we'd like the first few comments on each photo to be positive comments.
+For this task, the https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english?text=I+like+you.+I+love+you[`distilbert-base-uncased-finetuned-sst-2-english`^] model is a good fit.
+
+To upload this model to your deployment, you need a few pieces of data:
+
+- The deployment URL.
+  You can get this via the *Copy endpoint* link next to *Elasticsearch* on the deployment management screen.
+  It will look like `https://ml-test.es.us-west1.gcp.cloud.es.io:443`.
+  Make sure to append the port if it isn't present, as Eland requires the URL to have a scheme, host, and port.
+  443 is the default port for HTTPS.
+- The deployment username and password for your deployment.
+  This is displayed one time when the deployment is created.
+  It will look like `elastic` and `xUjaFNTyycG34tQx5Iq9JIIA`.
+- The trained model id.
+  This comes from Hugging Face.
+  It will look like `distilbert-base-uncased-finetuned-sst-2-english`.
+- The trained model task type.
+  This is the kind of machine learning task the model is designed to achieve.
+  It will be one of: `fill_mask`, `ner`, `text_classification`, `text_embedding`, and `zero_shot_classification`.
+  For our use case, we will use `text_classification`.
+
+We can now upload our chosen model to Elasticsearch by providing these options to Eland.
+
+[source,sh]
+----
+docker run -it --rm --network host \
+    elastic/eland \
+    eland_import_hub_model \
+      --url https://ml-test.es.us-west1.gcp.cloud.es.io:443 \
+      -u elastic -p <PASSWORD> \
+      --hub-model-id distilbert-base-uncased-finetuned-sst-2-english \
+      --task-type text_classification \
+      --start
+----
+
+This script should take roughly 2-3 minutes to run.
+Once your model has been successfully deployed to your Elastic deployment, navigate to Kibana's *Trained Models* page to verify it is ready.
+You can find this page under *Machine Learning > Analytics* menu and then *Trained Models > Model Management*.
+If you do not see your model in the list, you may need to click *Synchronize your jobs and trained models*.
+Your model is now ready to be used.
+
+[discrete#nlp-example-create-index-and-define-ml-inference-pipeline]
+==== Create an index and define an ML inference pipeline
+
+We are now ready to use Kibana's *Content* UI to enrich our documents with inference data.
+Before we ingest photo comments into Elasticsearch, we will first create an ML inference pipeline.
+The pipeline will enrich the incoming photo comments with inference data indicating if the comments are positive.
+
+Let's say our photo comments look like this when they are uploaded as a document into Elasticsearch:
+
+[source,js]
+----
+{
+  "photo_id": "78sdv71-8vdkjaj-knew629-vc8459p",
+  "body": "your dog is so cute!",
+  ...
+}
+----
+// NOTCONSOLE
+
+We want to run our documents through an inference processor that uses the trained model we uploaded to determine if the comments are positive.
+To do this, we first need to set up an Elasticsearch index.
+
+* From the Kibana home page, start by clicking the Search card.
+* Click the button to *Create an Elasticsearch index*.
+* Choose to *Use the API* and give your index a name.
+It will automatically be prefixed with `search-`.
+For this demo, we will name the index `search-photo-comments`.
+* After clicking *Create Index*, you will be redirected to the overview page for your new index.
+
+To configure the ML inference pipeline, we need the index to have an existing field mapping so we can choose which field to analyze.
+This can be done via the <<indices-put-mapping, index mapping API>> in the Kibana Dev Tools or simply through a cURL command:
+
+[source,js]
+----
+PUT search-photo-comments/_mapping
+{
+  "properties": {
+    "photo_id": { "type": "keyword" },
+    "body": { "type": "text" }
+  }
+}
+----
+// NOTCONSOLE
+
+Now it's time to create an inference pipeline.
+
+1. From the overview page for your `search-photo-comments` index in "Search", click the *Pipelines* tab.
+By default, Elasticsearch does not create any index-specific ingest pipelines.
+2. Because we want to customize these pipelines, we need to *Copy and customize* the `ent-search-generic-ingestion` ingest pipeline.
+Find this option above the settings for the `ent-search-generic-ingestion` ingest pipeline.
+This will create two new index-specific ingest pipelines.
+
+Next, we'll add an inference pipeline.
+
+1. Locate the section *Machine Learning Inference Pipelines*, then select *Add inference pipeline*.
+2. Give your inference pipeline a name, select the trained model we uploaded, and select the `body` field to be analyzed.
+3. Optionally, choose a field name to store the output.
+We'll call it `positivity_result`.
+
+You can also run example documents through a simulator and review the pipeline before creating it.
+
+[discrete#nlp-example-index-documents]
+==== Index documents
+
+At this point, everything is ready to enrich documents at index time.
+
+From the Kibana Dev Console, or simply using a cURL command, we can index a document.
+We'll use a `_run_ml_inference` flag to tell the `search-photo-comments` pipeline to run the index-specific ML inference pipeline that we created.
+This field will not be indexed in the document.
+
+[source,js]
+----
+POST search-photo-comments/_doc/my-new-doc?pipeline=search-photo-comments
+{
+  "photo_id": "78sdv71-8vdkjaj-knew629-vc8459p",
+  "body": "your dog is so cute!",
+  "_run_ml_inference": true
+}
+----
+// NOTCONSOLE
+
+Once the document is indexed, use the API to retrieve it and view the enriched data.
+
+[source,js]
+----
+GET search-photo-comments/_doc/my-new-doc
+----
+// NOTCONSOLE
+
+[source,js]
+----
+{
+  "_index": "search-photo-comments",
+  "_id": "_MQggoQBKYghsSwHbDvG",
+  ...
+  "_source": {
+    ...
+    "photo_id": "78sdv71-8vdkjaj-knew629-vc8459p",
+    "body": "your dog is so cute!",
+    "ml": {
+      "inference": {
+        "positivity_result": {
+          "predicted_value": "POSITIVE",
+          "prediction_probability": 0.9998022925461774,
+          "model_id": "distilbert-base-uncased-finetuned-sst-2-english"
+        }
+      }
+    }
+  }
+}
+----
+// NOTCONSOLE
+
+The document has new fields with the enriched data.
+The `ml.inference.positivity_result` field is an object with the analysis from the machine learning model.
+The model we used predicted with 99.98% confidence that the analyzed text is positive.
+
+From here, we can write search queries to boost on `ml.inference.positivity_result.predicted_value`.
+This field will also be stored in a top-level `positivity_result` field if the model was confident enough.
+
+[discrete#nlp-example-summary]
+==== Summary
+
+In this guide, we covered how to:
+
+- Set up a deployment on Elastic Cloud with a machine learning instance.
+- Deploy a machine learning trained model using the Eland Elasticsearch client.
+- Configure an inference pipeline to use the trained model with Elasticsearch.
+- Enrich documents with inference results from the trained model at ingest time.
+- Query your search engine and sort by `positivity_result`.
+
+[discrete#nlp-example-learn-more]
+==== Learn more
+
+* {ml-docs}/ml-nlp-model-ref.html[Compatible third party models^]
+* {ml-docs}/ml-nlp-overview.html[NLP Overview^]
+* https://github.com/elastic/eland#docker[Docker section of Eland readme^]
+* {ml-docs}/ml-nlp-deploy-models.html[Deploying a model ML guide^]
+* {ml-docs}/ml-nlp-import-model.html#ml-nlp-authentication[Eland Authentication methods^]
+* <<ingest-pipeline-search-inference-add-inference-processors,Adding inference pipelines>>
+// * <<elser-text-expansion,Using ELSER for text expansion>>

docs/reference/redirects.asciidoc

@@ -1931,24 +1931,4 @@ Refer to <<remote-clusters>>.
 [role="exclude",id="remote-clusters-privileges"]
 === Configure roles and users for remote clusters
 
-Refer to <<remote-clusters>>.
-
-[role="exclude",id="ingest-pipeline-search"]
-=== Ingest pipelines for Search indices
-
-coming::[8.11.0]
-
-[role="exclude",id="ingest-pipeline-search-inference"]
-=== Inference processing for Search indices
-
-coming::[8.11.0]
-
-[id="ingest-pipeline-search-inference-update-mapping"]
-==== Update mapping
-
-coming::[8.11.0]
-
-[role="exclude",id="nlp-example"]
-=== Tutorial: Natural language processing (NLP)
-
-coming::[8.11.0]
+Refer to <<remote-clusters>>.