Docs for hot-reloadable remote cluster credentials (#105483)

Docs PR to accompany
https://github.com/elastic/elasticsearch/pull/103215.

Resolves: ES-7625

docs/reference/modules/cluster/remote-clusters-api-key.asciidoc

@@ -31,7 +31,7 @@ In this model, cross-cluster operations use <<remote_cluster.port,a dedicated
 server port>> (remote cluster interface) for communication between clusters. A
 remote cluster must enable this port for local clusters to connect. Configure
 Transport Layer Security (TLS) for this port to maximize security (as explained
-in <<remote-clusters-security-api-key>>). 
+in <<remote-clusters-security-api-key>>).
 
 The local cluster must trust the remote cluster on the remote cluster interface.
 This means that the local cluster trusts the remote cluster's certificate
@@ -65,15 +65,15 @@ information, refer to https://www.elastic.co/subscriptions.
 ===== On the remote cluster
 
 // tag::remote-cluster-steps[]
-. Enable the remote cluster server on every node of the remote cluster. In 
+. Enable the remote cluster server on every node of the remote cluster. In
 `elasticsearch.yml`:
-.. Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to 
+.. Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to
 `true`.
 .. Configure the bind and publish address for remote cluster server traffic, for
 example using <<remote-cluster-network-settings,`remote_cluster.host`>>. Without
 configuring the address, remote cluster traffic may be bound to the local
 interface, and remote clusters running on other machines can't connect.
-.. Optionally, configure the remote server port using 
+.. Optionally, configure the remote server port using
 <<remote_cluster.port,`remote_cluster.port`>> (defaults to `9443`).
 . Next, generate a certificate authority (CA) and a server certificate/key pair.
 On one of the nodes of the remote cluster, from the directory where {es} has
@@ -86,8 +86,8 @@ been installed:
 ./bin/elasticsearch-certutil ca --pem --out=cross-cluster-ca.zip --pass CA_PASSWORD
 ----
 +
-Replace `CA_PASSWORD` with the password you want to use for the CA. You can 
-remove the `--pass` option and its argument if you are not deploying to a 
+Replace `CA_PASSWORD` with the password you want to use for the CA. You can
+remove the `--pass` option and its argument if you are not deploying to a
 production environment.
 
 .. Unzip the generated `cross-cluster-ca.zip` file. This compressed file
@@ -100,7 +100,7 @@ contains the following content:
 |_ ca.key
 ----
 
-.. Generate a certificate and private key pair for the nodes in the remote 
+.. Generate a certificate and private key pair for the nodes in the remote
 cluster:
 +
 [source,sh]
@@ -183,13 +183,16 @@ Replace `ALIAS` with the same name that you will use to create the remote cluste
 later. When prompted, enter the encoded cross-cluster API key created on the
 remote cluster earlier.
 
-. Restart the local cluster to load the keystore change.
+. Restart the local cluster to load changes to the keystore and settings.
+
+**Note:** If you are configuring only the cross-cluster API key, you can call the <<cluster-nodes-reload-secure-settings>> API, instead of restarting the cluster.
+Configuring the `remote_cluster_client` settings in `elasticsearch.yml` still requires a restart.
 
 [[remote-clusters-connect-api-key]]
 ==== Connect to a remote cluster
 
 :trust-mechanism: api-key
 include::remote-clusters-connect.asciidoc[]
-:!trust-mechanism: 
+:!trust-mechanism:
 
 include::{es-repo-dir}/security/authentication/remote-clusters-privileges-api-key.asciidoc[leveloffset=+1]

docs/reference/modules/cluster/remote-clusters-migration.asciidoc

@@ -51,7 +51,7 @@ include::remote-clusters-api-key.asciidoc[tag=remote-cluster-steps]
 [[remote-clusters-migration-stop]]
 ==== Stop cross-cluster operations
 
-On the local cluster, stop any persistent tasks that refer to the remote 
+On the local cluster, stop any persistent tasks that refer to the remote
 cluster:
 
 * Use the <<stop-transform>> API to stop any transforms.
@@ -74,13 +74,13 @@ roles used for cross-cluster operations. You should be able to copy these
 privileges from the original roles on the remote cluster, where they are defined
 under the certification based security model.
 ** The roles on the local cluster can't exceed the `access` privilege granted by
-the cross-cluster API key. Any extra local privileges will be suppressed by the 
+the cross-cluster API key. Any extra local privileges will be suppressed by the
 cross-cluster API key's privileges.
 ** No update is needed if the {ccr} or {ccs} tasks have been configured with a
 `superuser` role. The `superuser` role is automatically updated to allow access
 to all remote indices.
 ** Tasks that are run as regular users with named roles are immediately updated
-with the new privileges. A task will load a new definition the next time it 
+with the new privileges. A task will load a new definition the next time it
 runs.
 ** You need to restart tasks that are run using an API key (done in a later
 step).
@@ -123,7 +123,7 @@ created on the remote cluster earlier.
 . If you've dynamically configured the remote cluster (via the cluster settings
 API):
 
-.. Restart the local cluster to load changes to the keystore.
+.. Restart the local cluster to load changes to the keystore and settings.
 
 .. Re-add the remote cluster. Use the same remote cluster alias, and change the
 transport port into the remote cluster port. For example:
@@ -188,7 +188,7 @@ remote cluster:
 ----
 // TEST[skip:TODO]
 <1> The remote cluster is connected.
-<2> If present, indicates the remote cluster has connected using API key 
+<2> If present, indicates the remote cluster has connected using API key
 authentication.
 
 [[remote-clusters-migration-resume]]
@@ -204,7 +204,7 @@ task will update the task with the updated API key.
 * Use the <<start-transform>> API to start any transforms.
 * Use the <<ml-open-job>> API to open any anomaly detection jobs.
 * Use the <<ccr-post-resume-follow>> API to resume any auto-follow {ccr}.
-* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or 
+* Use the <<ccr-resume-auto-follow-pattern>> API to resume any manual {ccr} or
 existing indices that were created from the auto-follow pattern.
 
 [[remote-clusters-migration-disable-cert]]
@@ -232,8 +232,8 @@ or distributed, is no longer trusted.
 Another solution is to apply IP filters to the transport interface, blocking
 traffic from outside the cluster.
 
-. Optionally, delete any roles on the remote cluster that were only used for 
-cross-cluster operations. These roles are no longer used under the API key based 
+. Optionally, delete any roles on the remote cluster that were only used for
+cross-cluster operations. These roles are no longer used under the API key based
 security model.
 
 [[remote-clusters-migration-rollback]]
@@ -252,7 +252,7 @@ the migration.
 . On each node, remove the `remote_cluster_client.ssl.*` settings from
 `elasticsearch.yml`.
 
-. Restart the local cluster to apply changes to the keystore and 
+. Restart the local cluster to apply changes to the keystore and
 `elasticsearch.yml`.
 
 . On the local cluster, apply the original remote cluster settings. If the
@@ -263,4 +263,4 @@ remote cluster connection has been configured statically (using the
 local cluster has connected to the remote cluster. The response should have
 `"connected": true` and not have `"cluster_credentials": "::es_redacted::"`.
 
-. Restart any persistent tasks that you've stopped earlier.
+. Restart any persistent tasks that you've stopped earlier.

docs/reference/modules/cluster/remote-clusters-settings.asciidoc

@@ -65,7 +65,8 @@ mode are described separately.
   is used as the fallback setting.
 
 
-`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>)::
+`cluster.remote.<cluster_alias>.credentials` (<<secure-settings,Secure>>, <<reloadable-secure-settings,Reloadable>>)::
+[[remote-cluster-credentials-setting]]
 
 beta:[]
   Per cluster setting for configuring <<remote-clusters-api-key,remote clusters with the API Key based model>>.
@@ -75,6 +76,8 @@ beta:[]
   The presence (or not) of this setting determines which model a remote cluster uses.
   If present, the remote cluster uses the API key based model.
   Otherwise, it uses the certificate based model.
+  If the setting is added, removed, or updated in the <<secure-settings,{es} keystore>> and reloaded via the
+  <<cluster-nodes-reload-secure-settings>> API, the cluster will automatically rebuild its connection to the remote.
 
 [[remote-cluster-sniff-settings]]
 ==== Sniff mode remote cluster settings

docs/reference/modules/cluster/remote-clusters-troubleshooting.asciidoc

@@ -31,13 +31,13 @@ incoming cross-cluster requests by default, while it is ready to send outgoing
 cross-cluster requests. Ensure you've enabled the remote cluster server on every
 node of the remote cluster. In `elasticsearch.yml`:
 
-* Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to 
+* Set <<remote-cluster-network-settings,`remote_cluster_server.enabled`>> to
 `true`.
 * Configure the bind and publish address for remote cluster server traffic, for
 example using <<remote-cluster-network-settings,`remote_cluster.host`>>. Without
 configuring the address, remote cluster traffic may be bound to the local
 interface, and remote clusters running on other machines can't connect.
-* Optionally, configure the remote server port using 
+* Optionally, configure the remote server port using
 <<remote_cluster.port,`remote_cluster.port`>> (defaults to `9443`).
 
 [[remote-clusters-troubleshooting-common-issues]]
@@ -73,7 +73,7 @@ org.elasticsearch.transport.ConnectTransportException: [][192.168.0.42:9443] *co
 ====== Resolution
 
 * Check the host and port for the remote cluster are correct.
-* Ensure the <<remote-clusters-troubleshooting-enable-server,remote cluster 
+* Ensure the <<remote-clusters-troubleshooting-enable-server,remote cluster
 server is enabled>> on the remote cluster.
 * Ensure no firewall is blocking the communication.
 
@@ -108,11 +108,11 @@ cause of the failure. For example:
 
 * Is the remote cluster certificate not signed by a trusted CA? This is the most
 likely cause.
-* Is hostname verification failing? 
+* Is hostname verification failing?
 * Is the certificate expired?
 
 Once you know the cause, you should be able to fix it by adjusting the remote
-cluster related SSL settings on either the local cluster or the remote cluster. 
+cluster related SSL settings on either the local cluster or the remote cluster.
 
 Often, the issue is on the local cluster. For example, fix it by configuring necessary
 trusted CAs (`xpack.security.remote_cluster_client.ssl.certificate_authorities`).
@@ -275,7 +275,7 @@ This does not show up in the logs of the remote cluster.
 ====== Resolution
 
 Add the cross-cluster API key to {es} keystore on every node of the local
-cluster. Restart the local cluster to reload the keystore.
+cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-wrong-api-key-type]]
 ===== Using the wrong API key type
@@ -302,8 +302,7 @@ This does not show up in the logs of the remote cluster.
 Ask the remote cluster administrator to create and distribute a
 <<security-api-create-cross-cluster-api-key,cross-cluster API key>>. Replace the
 existing API key in the {es} keystore with this cross-cluster API key on every
-node of the local cluster. Restart the local cluster for keystore changes to
-take effect.
+node of the local cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-non-valid-api-key]]
 ===== Invalid API key
@@ -334,15 +333,14 @@ The remote cluster logs `Authentication using apikey failed`:
 Ask the remote cluster administrator to create and distribute a
 <<security-api-create-cross-cluster-api-key,cross-cluster API key>>. Replace the
 existing API key in the {es} keystore with this cross-cluster API key on every
-node of the local cluster. Restart the local cluster for keystore changes to
-take effect.
+node of the local cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-insufficient-privileges]]
 ===== API key or local user has insufficient privileges
 
 The effective permission for a local user running requests on a remote cluster
 is determined by the intersection of the cross-cluster API key's privileges and
-the local user's `remote_indices` privileges. 
+the local user's `remote_indices` privileges.
 
 ====== Symptom
 
@@ -366,8 +364,7 @@ This does not show up in any logs.
 create and distribute a
 <<security-api-create-cross-cluster-api-key,cross-cluster API key>>. Replace the
 existing API key in the {es} keystore with this cross-cluster API key on every
-node of the local cluster. Restart the local cluster for keystore changes to
-take effect.
+node of the local cluster. Use the <<cluster-nodes-reload-secure-settings>> API to reload the keystore.
 
 [[remote-clusters-troubleshooting-no-remote_indices-privileges]]
 ===== Local user has no `remote_indices` privileges

docs/reference/setup/secure-settings.asciidoc

@@ -64,3 +64,4 @@ There are reloadable secure settings for:
 * <<ref-jwt-settings, JWT realm>>
 * <<ref-ad-settings, Active Directory realm>>
 * <<ref-ldap-settings, LDAP realm>>
+* <<remote-cluster-credentials-setting, Remote cluster credentials for the API key based security model>>