Document `transport.compress` trade-offs more clearly (#108458)

Spells out explicitly that setting `transport.compress: true` may cost
extra CPU.

docs/reference/modules/cluster/remote-clusters-settings.asciidoc

@@ -59,35 +59,40 @@ you configure the remotes.
 
 `cluster.remote.<cluster_alias>.transport.compress`::
 
-  Per cluster setting that enables you to configure compression for requests
-  to a specific remote cluster. This setting impacts only requests
-  sent to the remote cluster. If the inbound request is compressed,
-  Elasticsearch compresses the response. The setting options are `true`,
-  `indexing_data`, and `false`. If unset, the global `transport.compress` is
-  used as the fallback setting.
+  Per-cluster setting that enables you to configure compression for requests to
+  a specific remote cluster. The handling cluster will automatically compress
+  responses to compressed requests. The setting options are `true`,
+  `indexing_data`, and `false`. If unset, defaults to the behaviour specified
+  by the node-wide `transport.compress` setting. See the
+  <<transport-settings-compress,documentation for the `transport.compress`
+  setting>> for further information.
 
 `cluster.remote.<cluster_alias>.transport.compression_scheme`::
 
-  Per cluster setting that enables you to configure compression scheme for
-  requests to a specific remote cluster. This setting impacts only requests
-  sent to the remote cluster. If an inbound request is compressed, {es}
-  compresses the response using the same compression scheme. The setting options
-  are `deflate` and `lz4`. If unset, the global `transport.compression_scheme`
-  is used as the fallback setting.
+  Per-cluster setting that enables you to configure the compression scheme for
+  requests to a specific cluster if those requests are selected to be
+  compressed by to the `cluster.remote.<cluster_alias>.transport.compress`
+  setting. The handling cluster will automatically use the same compression
+  scheme for responses as for the corresponding requests. The setting options
+  are `deflate` and `lz4`. If unset, defaults to the behaviour specified by the
+  node-wide `transport.compression_scheme` setting. See the
+  <<transport-settings-compression-scheme,documentation for the
+  `transport.compression_scheme` setting>> for further information.
 
-
-`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>, <<reloadable-secure-settings,Reloadable>>)::
 [[remote-cluster-credentials-setting]]
-
-  Per cluster setting for configuring <<remote-clusters-api-key,remote clusters with the API Key based model>>.
-  This setting takes the encoded value of a
-  <<security-api-create-cross-cluster-api-key,cross-cluster API key>> and must be set
-  in the <<secure-settings,{es} keystore>> on each node in the cluster.
-  The presence (or not) of this setting determines which model a remote cluster uses.
-  If present, the remote cluster uses the API key based model.
-  Otherwise, it uses the certificate based model.
-  If the setting is added, removed, or updated in the <<secure-settings,{es} keystore>> and reloaded via the
-  <<cluster-nodes-reload-secure-settings>> API, the cluster will automatically rebuild its connection to the remote.
+`cluster.remote.<cluster_alias>.credentials`::
+
+  (<<secure-settings,Secure>>, <<reloadable-secure-settings,Reloadable>>)
+  Per-cluster setting for configuring <<remote-clusters-api-key,remote clusters
+  with the API Key based model>>. This setting takes the encoded value of a
+  <<security-api-create-cross-cluster-api-key,cross-cluster API key>> and must
+  be set in the <<secure-settings,{es} keystore>> on each node in the cluster.
+  The presence (or not) of this setting determines which model a remote cluster
+  uses. If present, the remote cluster uses the API key based model. Otherwise,
+  it uses the certificate based model. If the setting is added, removed, or
+  updated in the <<secure-settings,{es} keystore>> and reloaded via the
+  <<cluster-nodes-reload-secure-settings>> API, the cluster will automatically
+  rebuild its connection to the remote.
 
 [[remote-cluster-sniff-settings]]
 ==== Sniff mode remote cluster settings

docs/reference/modules/transport.asciidoc

@@ -47,20 +47,44 @@ different from `transport.port`. Defaults to the port assigned via
 The connect timeout for initiating a new connection (in
 time setting format). Defaults to `30s`.
 
+[[transport-settings-compress]]
 `transport.compress`::
 (<<static-cluster-setting,Static>>, string)
-Set to `true`, `indexing_data`, or `false` to configure transport compression
-between nodes. The option `true` will compress all data. The option
-`indexing_data` will compress only the raw index data sent between nodes during
-ingest, ccr following (excluding bootstrap), and operations based shard recovery
-(excluding transferring lucene files). Defaults to `indexing_data`.
+Determines which transport requests are compressed before sending them to
+another node. {es} will compress transport responses if and only if the
+corresponding request was compressed. See also `transport.compression_scheme`,
+which specifies the compression scheme which is used. Accepts the following
+values:
++
+--
+`false`::
+
+No transport requests are compressed. This option uses the most network
+bandwidth, but avoids the CPU overhead of compression and decompression.
+
+`indexing_data`::
+
+Compresses only the raw indexing data sent between nodes during ingest, CCR
+following (excluding bootstrapping) and operations-based shard recovery
+(excluding file-based recovery which copies the raw Lucene data). This option
+is a good trade-off between network bandwidth savings and the extra CPU
+required for compression and decompression. This option is the default.
+
+`true`::
+
+All transport requests are compressed. This option may perform better than
+`indexing_data` in terms of network bandwidth, but will require the most CPU
+for compression and decompression work.
+--
 
+[[transport-settings-compression-scheme]]
 `transport.compression_scheme`::
 (<<static-cluster-setting,Static>>, string)
-Configures the compression scheme for `transport.compress`. The options are
-`deflate` or `lz4`. If `lz4` is configured and the remote node has not been
-upgraded to a version supporting `lz4`, the traffic will be sent uncompressed.
-Defaults to `lz4`.
+Configures the compression scheme for requests which are selected for
+compression by to the `transport.compress` setting. Accepts either `deflate` or
+`lz4`, which offer different trade-offs between compression ratio and CPU
+usage. {es} will use the same compression scheme for responses as for the
+corresponding requests. Defaults to `lz4`.
 
 `transport.tcp.keep_alive`::
 (<<static-cluster-setting,Static>>, boolean)