elasticsearch/docs/reference/monitoring/configuring-metricbeat.asciidoc

[role="xpack"]
[testenv="gold"]
[[configuring-metricbeat]]
=== Monitoring {es} with {metricbeat}

beta[]

In 6.5 and later, you can use {metricbeat} to collect data about {es} 
and ship it to the monitoring cluster, rather than routing it through exporters 
as described in <<configuring-monitoring>>. 

image::monitoring/images/metricbeat.png[Example monitoring architecture]

To learn about monitoring in general, see 
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. 

. Enable the collection of monitoring data. Set 
`xpack.monitoring.collection.enabled` to `true` on the production cluster. +
+ 
--
For example, you can use the following APIs to review and change this setting:

[source,js]
----------------------------------
GET _cluster/settings

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}
----------------------------------
// CONSOLE

For more information, see <<monitoring-settings>> and <<cluster-update-settings>>.
--

. Disable the default collection of {es} monitoring metrics. Set 
`xpack.monitoring.elasticsearch.collection.enabled` to `false` on the production 
cluster. +
+ 
--
For example, you can use the following API to change this setting:

[source,js]
----------------------------------
PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.elasticsearch.collection.enabled": false
  }
}
----------------------------------
// CONSOLE

Leave `xpack.monitoring.enabled` set to its default value (`true`). 
--

. On each {es} node in the production cluster:

.. {metricbeat-ref}/metricbeat-installation.html[Install {metricbeat}].

.. Enable the {es} module in {metricbeat}. +
+
--
For example, to enable the default configuration in the `modules.d` directory, 
run the following command:

["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
metricbeat modules enable elasticsearch
----------------------------------------------------------------------

For more information, see 
{metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and 
{metricbeat-ref}/metricbeat-module-elasticsearch.html[{es} module]. 
--

.. Configure the {es} module in {metricbeat}. +
+
--
You must specify the following settings in the `modules.d/elasticsearch.yml` file:

[source,yaml]
----------------------------------
- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"] <1>
  xpack.enabled: true
----------------------------------
<1> This setting identifies the host and port number that are used to access {es}.  
--

.. If {security} is enabled, you must also provide a user ID and password so that 
{metricbeat} can collect metrics successfully. 

... Create or identify a user that you want to use to collect the metrics. 
+
--
TIP: There is a `remote_monitoring_user` built-in user that grants the privileges 
necessary for {metricbeat} to monitor {stack} products. See 
{stack-ov}/built-in-users.html[Built-in users].

Alternatively, you can choose a different user and give them the 
`remote_monitoring_collector` {stack-ov}/built-in-roles.html[built-in role]. 
--

... Add the `username` and `password` settings to the {es} module configuration 
file.
+
--
For example, add the following settings in the `modules.d/kibana.yml` file:

[source,yaml]
----------------------------------
- module: elasticsearch
  ...
  username: remote_monitoring_user
  password: YOUR_PASSWORD
----------------------------------
--

.. If you configured {es} to use <<configuring-tls,encrypted communications>>, 
you must access it via HTTPS. For example, use a `hosts` setting like 
`https://localhost:9200` in the `modules.d/elasticsearch.yml` file.

.. Identify where to send the monitoring data. +
+
--
TIP: In production environments, we strongly recommend using a separate cluster 
(referred to as the _monitoring cluster_) to store the data. Using a separate 
monitoring cluster prevents production cluster outages from impacting your 
ability to access your monitoring data. It also prevents monitoring activities 
from impacting the performance of your production cluster.

For example, specify the {es} output information in the {metricbeat} 
configuration file (`metricbeat.yml`):

[source,yaml]
----------------------------------
output.elasticsearch:
  hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
----------------------------------
<1> In this example, the data is stored on a monitoring cluster with nodes 
`es-mon-1` and `es-mon-2`. 

For more information about these configuration options, see 
{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output].
--

.. If {security} is enabled on the monitoring cluster, you must provide a valid 
user ID and password so that {metricbeat} can send metrics successfully. 

... Create or identify a user that you want to use to send the metrics. 
+
--
TIP: There is a `remote_monitoring_user` built-in user that grants the privileges 
necessary for {metricbeat} to monitor {stack} products. See 
{stack-ov}/built-in-users.html[Built-in users].

Alternatively, you can choose a different user and give them the 
`remote_monitoring_agent` {stack-ov}/built-in-roles.html[built-in role]. 
--

... Add the `username` and `password` settings to the {es} output information in 
the {metricbeat} configuration file (`metricbeat.yml`):
+
--
[source,yaml]
----------------------------------
output.elasticsearch:
  ...
  username: remote_monitoring_user
  password: YOUR_PASSWORD
----------------------------------
--

.. If you configured the monitoring cluster to use 
<<configuring-tls,encrypted communications>>, you must access it via 
HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200` in the 
`metricbeat.yml` file. 

. <<starting-elasticsearch,Start {es}>>.

. {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}]. 

. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].