[DOCS] Describe how to use Elastic Agent to monitor Elasticsearch (#94348)

* [DOCS] Describe how to use Elastic Agent to monitor Elasticsearch

* Temporarily fix doc build

* Add question about showing Elastic Agent metrics in the monitoring UI

* Apply changes from review

* Activate link to Kibana docs

* Fix broken link

* Update docs/reference/monitoring/indices.asciidoc

docs/reference/monitoring/collecting-monitoring-data.asciidoc

@@ -8,7 +8,7 @@
 include::{es-ref-dir}/settings/monitoring-settings.asciidoc[tag=monitoring-deprecation-notice]
 
 This method for collecting metrics about {es} involves sending the metrics to
-the monitoring cluster by using exporters. For the recommended method, see <<configuring-metricbeat>>.
+the monitoring cluster by using exporters.
 
 Advanced monitoring settings enable you to control how frequently data is
 collected, configure timeouts, and set the retention period for locally-stored

docs/reference/monitoring/collectors.asciidoc

@@ -2,17 +2,7 @@
 [[es-monitoring-collectors]]
 == Collectors
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 Collectors, as their name implies, collect things. Each collector runs once for
 each collection interval to obtain data from the public APIs in {es} and {xpack}

docs/reference/monitoring/configuring-elastic-agent.asciidoc

@@ -0,0 +1,73 @@
+[[configuring-elastic-agent]]
+== Collecting {es} monitoring data with {agent}
+
+[subs="attributes"]
+++++
+<titleabbrev>Collecting monitoring data with {agent}</titleabbrev>
+++++
+
+In 8.5 and later, you can use {agent} to collect data about {es} and ship it to
+the monitoring cluster, rather than <<configuring-metricbeat,using {metricbeat}>>
+or routing it through exporters as described in <<collecting-monitoring-data>>.
+
+[discrete]
+=== Prerequisites
+
+* (Optional) Create a monitoring cluster as described in <<monitoring-production>>.
+
+* Create a user on the production cluster that has the
+`remote_monitoring_collector` {ref}/built-in-roles.html[built-in role].
+
+[discrete]
+=== Add {es} monitoring data
+
+To collect {es} monitoring data, add an {es} integration to an {agent} and
+deploy it to the host where {es} is running.
+
+. Go to the {kib} home page and click **Add integrations**.
+. In the query bar, search for and select the **{es}** integration for
+{agent}.
+. Read the overview to make sure you understand integration requirements and
+other considerations.
+. Click **Add Elasticsearch**.
++
+TIP: If you're installing an integration for the first time, you may be prompted
+to install {agent}. Click **Add integration only (skip agent installation)**.
+
+. Configure the integration name and optionally add a description. Make sure you
+configure all required settings:
+.. Under **Collect Elasticsearch logs**, modify the log paths to match your {es}
+environment.
+.. Under **Collect Elasticsearch metrics**, make sure the hosts setting points to
+your {es} host URLs. By default, the integration collects {es} monitoring
+metrics from `localhost:9200`. If that host and port number are not correct,
+update the `hosts` setting. If you configured {es} to use encrypted
+communications, you must access it via HTTPS. For example, use a `hosts` setting
+like `https://localhost:9200`.
+.. Expand **Advanced options**. If the Elastic {security-features} are enabled,
+enter the username and password of a user that has the
+`remote_monitoring_collector` role.
+.. Specify the scope:
+** Specify `cluster` if each entry in the hosts list indicates a single
+endpoint for a distinct {es} cluster (for example, a load-balancing proxy
+fronting the cluster that directs requests to the master-ineligible nodes in the
+cluster).
+** Otherwise, accept the default scope, `node`. If this scope is set, you
+will need to install {agent} on each {es} node to collect all metrics. {agent}
+will collect most of the metrics from the elected master of the cluster, so you
+must scale up all your master-eligible nodes to account for this extra load. Do
+not use this `node` if you have dedicated master nodes.
+. Choose where to add the integration policy. Click **New hosts** to add it to
+new agent policy or **Existing hosts** to add it to an existing agent policy.
+. Click **Save and continue**. This step takes a minute or two to complete. When
+it's done, you'll have an agent policy that contains an integration for
+collecting monitoring data from {es}.
+. If an {agent} is already assigned to the policy and deployed to the host where
+{es} is running, you're done. Otherwise, you need to deploy an {agent}. To
+deploy an {agent}:
+.. Go to **{fleet} -> Agents**, then click **Add agent**.
+.. Follow the steps in the **Add agent** flyout to download, install,
+and enroll the {agent}. Make sure you choose the agent policy you created
+earlier.
+. Wait a minute or two until incoming data is confirmed.
+. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. 

docs/reference/monitoring/configuring-filebeat.asciidoc

@@ -11,6 +11,9 @@ You can use {filebeat} to monitor the {es} log files, collect log events, and
 ship them to the monitoring cluster. Your recent logs are visible on the
 *Monitoring* page in {kib}.
 
+IMPORTANT: If you're using {agent}, do not deploy {filebeat} for log collection.
+Instead, configure the {es} integration to collect logs.
+
 //NOTE: The tagged regions are re-used in the Stack Overview.
 
 . Verify that {es} is running and that the monitoring cluster is ready to

docs/reference/monitoring/configuring-metricbeat.asciidoc

@@ -9,7 +9,9 @@
 
 In 6.5 and later, you can use {metricbeat} to collect data about {es} 
 and ship it to the monitoring cluster, rather than routing it through exporters 
-as described in <<collecting-monitoring-data>>. 
+as described in <<collecting-monitoring-data>>.
+
+Want to use {agent} instead? Refer to <<configuring-elastic-agent>>.
 
 image::monitoring/images/metricbeat.png[Example monitoring architecture]
 

docs/reference/monitoring/exporters.asciidoc

@@ -2,17 +2,7 @@
 [[es-monitoring-exporters]]
 == Exporters
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 The purpose of exporters is to take data collected from any Elastic Stack
 source and route it to the monitoring cluster. It is possible to configure

docs/reference/monitoring/how-monitoring-works.asciidoc

@@ -5,24 +5,33 @@
 <titleabbrev>How it works</titleabbrev>
 ++++
 
-Each {es} node, {ls} node, {kib} instance, and Beat instance is considered
-unique in the cluster based on its persistent UUID, which is written to the
-<<path-settings,`path.data`>> directory when the node or instance starts.
+Each monitored {stack} component is considered unique in the cluster based on
+its persistent UUID, which is written to the <<path-settings,`path.data`>>
+directory when the node or instance starts.
 
 Monitoring documents are just ordinary JSON documents built by monitoring each
-{stack} component at a specified collection interval. If you want to alter the
-templates for these indices, see <<config-monitoring-indices>>.
+{stack} component at a specified collection interval. If you want to alter how
+these documents are structured or stored, refer to <<config-monitoring-indices>>.
 
-{metricbeat} is used to collect monitoring data and to ship it directly to the
-monitoring cluster.
+You can use {agent} or {metricbeat} to collect monitoring data and to ship it
+directly to the monitoring cluster.
 
-To learn how to collect monitoring data, see:
+To learn how to collect monitoring data, refer to:
 
-* <<collecting-monitoring-data>>
-* <<configuring-metricbeat>>
+* One of the following topics depending on how you want to collect monitoring
+data from {es}:
+** <<configuring-elastic-agent>>: Uses a single agent to
+gather logs and metrics. Can be managed from a central location in {fleet}.
+** <<configuring-metricbeat>>: Uses a lightweight {beats}
+shipper to gather metrics. May be preferred if you have an existing investment
+in {beats} or are not yet ready to use {agent}.
+** <<collecting-monitoring-data>>: Uses internal exporters to
+gather metrics. Not recommended. If you have previously configured legacy
+collection methods, you should migrate to using {agent} or {metricbeat}.
 * {kibana-ref}/xpack-monitoring.html[Monitoring {kib}]
 * {logstash-ref}/configuring-logstash.html[Monitoring {ls}]
-* Monitoring Beats:
+* {enterprise-search-ref}/monitoring.html[Monitoring {ents}]
+* Monitoring {beats}:
 ** {auditbeat-ref}/monitoring.html[{auditbeat}]
 ** {filebeat-ref}/monitoring.html[{filebeat}]
 ** {functionbeat-ref}/monitoring.html[{functionbeat}]
@@ -30,3 +39,7 @@ To learn how to collect monitoring data, see:
 ** {metricbeat-ref}/monitoring.html[{metricbeat}]
 ** {packetbeat-ref}/monitoring.html[{packetbeat}]
 ** {winlogbeat-ref}/monitoring.html[{winlogbeat}]
+* {apm-guide-ref}/monitor-apm.html[Monitoring APM Server]
+* {fleet-guide}/monitor-elastic-agent.html[Monitoring {agent}s]
+{fleet}-managed agents) or
+{fleet-guide}/elastic-agent-monitoring-configuration.html[Configure monitoring for standalone {agent}s]

docs/reference/monitoring/http-export.asciidoc

@@ -2,17 +2,7 @@
 [[http-exporter]]
 === HTTP exporters
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 The `http` exporter is the preferred exporter in the {es} {monitor-features}
 because it enables the use of a separate monitoring cluster. As a secondary

docs/reference/monitoring/index.asciidoc

@@ -10,6 +10,7 @@ performance of your {es} cluster.
 * <<monitoring-overview>>
 * <<how-monitoring-works>>
 * <<monitoring-production>>
+* <<configuring-elastic-agent>>
 * <<configuring-metricbeat>>
 * <<configuring-filebeat>>
 * <<config-monitoring-indices>>
@@ -24,6 +25,8 @@ include::how-monitoring-works.asciidoc[]
 
 include::production.asciidoc[]
 
+include::configuring-elastic-agent.asciidoc[]
+
 include::configuring-metricbeat.asciidoc[]
 
 include::configuring-filebeat.asciidoc[]

docs/reference/monitoring/indices.asciidoc

@@ -2,8 +2,27 @@
 [[config-monitoring-indices]]
 == Configuring indices for monitoring
 
-<<indices-templates-v1,Index templates>> are used to configure the indices
-that store the monitoring data collected from a cluster.
+Sometimes the default index settings might not work for your situation. For
+example, you might want to change index lifecycle management (ILM) settings,
+add custom mappings, or set the number of shards and replicas
+for the monitoring indices.
+
+You can change the default behavior. The steps you follow depend on which data
+collection method is used.
+
+[float]
+=== Configuring data streams created by {agent}
+
+{agent} uses data streams to store time series data across multiple indices
+while giving you a single named resource for requests. You can alter the
+settings of each dataset by configuring an `@custom` component template.
+For more information, refer to {fleet-guide}/data-streams.html[Data streams].
+
+[float]
+=== Configuring indices created by {metricbeat}
+
+<<indices-templates-v1,Index templates>> are used to configure the indices that
+store the monitoring data that {metricbeat} collects from a cluster.
 
 You can retrieve the templates through the `_template` API:
 
@@ -32,7 +51,8 @@ PUT /_index_template/custom_monitoring
   ".monitoring-beats-8-*",
   ".monitoring-es-8-*",
   ".monitoring-kibana-8-*",
-  ".monitoring-logstash-8-*"
+  ".monitoring-logstash-8-*",
+  ".monitoring-ent-search-8-*"
   ],
   "priority": 1,
   "template": {

docs/reference/monitoring/local-export.asciidoc

@@ -2,17 +2,7 @@
 [[local-exporter]]
 === Local exporters
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 The `local` exporter is the default exporter in {monitoring}. It routes data
 back into the same (local) cluster. In other words, it uses the production

docs/reference/monitoring/overview.asciidoc

@@ -6,11 +6,11 @@
 ++++
 
 When you monitor a cluster, you collect data from the {es} nodes, {ls} nodes,
-{kib} instances, and Beats in your cluster. You can also
-<<configuring-filebeat,use {filebeat} to collect {es} logs>>. 
+{kib} instances, {ents}, APM Server, and Beats in your cluster. You can also
+collect logs.
 
 All of the monitoring metrics are stored in {es}, which enables you to easily
-visualize the data from {kib}. By default, the monitoring metrics are stored in
+visualize the data in {kib}. By default, the monitoring metrics are stored in
 local indices.
 
 TIP: In production, we strongly recommend using a separate monitoring cluster.
@@ -20,10 +20,12 @@ monitoring activities from impacting the performance of your production cluster.
 For the same reason, we also recommend using a separate {kib} instance for
 viewing the monitoring data.
 
-You can use {metricbeat} to collect and ship data about {es}, {kib}, {ls}, and
-Beats directly to your monitoring cluster rather than routing it through your
-production cluster. The following diagram illustrates a typical monitoring
-architecture with separate production and monitoring clusters:
+You can use {agent} or {metricbeat} to collect and ship data directly to your
+monitoring cluster rather than routing it through your production cluster.
+
+The following diagram illustrates a typical monitoring architecture with
+separate production and monitoring clusters. This example shows {metricbeat},
+but you can use {agent} instead.
 
 image::images/architecture.png[A typical monitoring environment]
 

docs/reference/monitoring/production.asciidoc

@@ -4,20 +4,20 @@
 
 In production, you should send monitoring data to a separate _monitoring cluster_
 so that historical data is available even when the nodes you are monitoring are
-not. For example, you can use {metricbeat} to ship monitoring data about {kib},
-{es}, {ls}, and Beats to the monitoring cluster.
+not.
 
+// tag::monitoring-rec[]
 [IMPORTANT]
 =========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
+{agent} and {metricbeat} are the recommended methods for collecting and shipping
+monitoring data to a monitoring cluster.
 
 If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
+to using <<configuring-elastic-agent,{agent}>> or
+<<configuring-metricbeat,{metricbeat}>> collection. Do not use legacy collection
+alongside other collection methods.
 =========================
+// end::monitoring-rec[]
 
 If you have at least a Gold Subscription, using a dedicated monitoring cluster
 also enables you to monitor multiple clusters from a central location.
@@ -63,7 +63,7 @@ PUT _cluster/settings
 --
 
 .. If the {es} {security-features} are enabled on the monitoring cluster, create
-users that can send and retrieve monitoring data.
+users that can send and retrieve monitoring data:
 +
 --
 NOTE: If you plan to use {kib} to view monitoring data, username and password
@@ -71,7 +71,11 @@ credentials must be valid on both the {kib} server and the monitoring cluster.
 
 --
 
-*** If you plan to use {metricbeat} to collect data about {es} or {kib},
+*** If you plan to use {agent},
+create a user that has the `remote_monitoring_collector`
+<<built-in-roles-remote-monitoring-agent,built-in role>>.
+
+*** If you plan to use {metricbeat},
 create a user that has the `remote_monitoring_collector` built-in role and a
 user that has the `remote_monitoring_agent`
 <<built-in-roles-remote-monitoring-agent,built-in role>>. Alternatively, use the
@@ -102,16 +106,17 @@ Alternatively, use the `remote_monitoring_user` <<built-in-users,built-in user>>
 
 . Configure your production cluster to collect data and send it to the
 monitoring cluster:
-
+** <<configuring-elastic-agent,{agent} collection methods>>
 ** <<configuring-metricbeat,{metricbeat} collection methods>>
-
 ** <<collecting-monitoring-data,Legacy collection methods>>
 
 . (Optional)
 {logstash-ref}/configuring-logstash.html[Configure {ls} to collect data and send it to the monitoring cluster].
 
-. (Optional) Configure the Beats to collect data and send it to the monitoring
-cluster.
+. (Optional) {enterprise-search-ref}/monitoring.html[Configure {ents} monitoring].
+
+. (Optional) Configure the {beats} to collect data and send it to the monitoring
+cluster. Skip this step for {beats} that are managed by {agent}.
 ** {auditbeat-ref}/monitoring.html[Auditbeat]
 ** {filebeat-ref}/monitoring.html[Filebeat]
 ** {heartbeat-ref}/monitoring.html[Heartbeat]
@@ -119,10 +124,11 @@ cluster.
 ** {packetbeat-ref}/monitoring.html[Packetbeat]
 ** {winlogbeat-ref}/monitoring.html[Winlogbeat]
 
-. (Optional) Configure {kib} to collect data and send it to the monitoring cluster:
+. (Optional) {apm-guide-ref}/monitor-apm.html[Configure APM Server monitoring]
 
+. (Optional) Configure {kib} to collect data and send it to the monitoring cluster:
+** {kibana-ref}/monitoring-elastic-agent.html[{agent} collection methods]
 ** {kibana-ref}/monitoring-metricbeat.html[{metricbeat} collection methods]
-
 ** {kibana-ref}/monitoring-kibana.html[Legacy collection methods]
 
 . (Optional) Create a dedicated {kib} instance for monitoring, rather than using
@@ -132,7 +138,7 @@ cluster.
 --
 NOTE: If you log in to {kib} using SAML, Kerberos, PKI, OpenID Connect, or token
 authentication providers, a dedicated {kib} instance is *required*. The security
-tokens that are used in these contexts are cluster-specific, therefore you
+tokens that are used in these contexts are cluster-specific; therefore you
 cannot use a single {kib} instance to connect to both production and monitoring
 clusters.
 

docs/reference/settings/monitoring-settings.asciidoc

@@ -6,7 +6,7 @@
 ++++
 
 // tag::monitoring-deprecation-notice[]
-deprecated[7.16, "Using the {es} Monitoring plugin to collect and ship monitoring data is deprecated. {metricbeat} is the recommended method for collecting and shipping monitoring data to a monitoring cluster. If you previously configured legacy collection methods, you should migrate to using {metricbeat} collection methods. Refer to <<configuring-metricbeat,Collecting {es} monitoring data with {metricbeat}>>."]
+deprecated[7.16, "Using the {es} Monitoring plugin to collect and ship monitoring data is deprecated. {agent} and {metricbeat} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. If you previously configured legacy collection methods, you should migrate to using <<configuring-elastic-agent,{agent}>> or <<configuring-metricbeat,{metricbeat}>> collection methods."]
 // end::monitoring-deprecation-notice[]
 
 By default, {es} {monitor-features} are enabled but data collection is disabled.