elasticsearch/docs/reference/security/securing-communications/update-tls-certificates.asciidoc

[[update-node-certs]]
== Updating node security certificates
You might need to update your TLS certificates if your current node
certificates expire soon, you're adding new nodes to your secured cluster, or
a security breach has broken the trust of your certificate chain. Use the 
<<security-api-ssl,SSL certificate>> API to check when your certificates are
expiring.

In instances where you have access to the original Certificate Authority (CA) key and certificate that you used to sign your existing node certificates (and where you can still trust your CA), you can
<<update-node-certs-same,use that CA to sign the new certificates>>.

If you have to trust a new CA from your organization, or you need to generate
a new CA yourself, you need to use this new CA to sign the new node
certificates and instruct your nodes to trust the new CA. In this case, you'll
<<update-node-certs-different,sign node certificates with your new CA>> and
instruct your nodes to trust this certificate chain.

Depending on which certificates are expiring, you might need to update the
certificates for the transport layer, the HTTP layer, or both. 

Regardless of the scenario, {es} monitors the SSL resources for updates
by default, on a five-second interval. You can just copy the new
certificate and key files (or keystore) into the {es} configuration directory
and your nodes will detect the changes and reload the keys and certificates. 

Because {es} doesn't reload the `elasticsearch.yml` configuration,
you must use *the same file names* if you want to take advantage of automatic certificate and key reloading. 

If you need to update the `elasticsearch.yml` configuration or change
passwords for keys or keystores that are stored in the
<<secure-settings,secure settings>>, then you must complete a
<<use-rolling-restarts,rolling restart>>. {es} will not automatically reload changes for
passwords stored in the secure settings. 

[[use-rolling-restarts]]
.Rolling restarts are preferred
****
While it's possible to do an in-place update for security certificates, using
a <<restart-cluster-rolling,rolling restart>> on your cluster is safer. An in-place update avoids some
complications of a rolling restart, but incurs the following risks:

* If you use PEM files, your certificate and key are in separate files. You
must update both files _simultaneously_ or the node might experience a temporary
period where it cannot establish new connections.
* Updating the certificate and key does not automatically force existing
connections to refresh. This means that even if you make a mistake, a node can
seem like it's functioning but only because it still has existing connections.
It's possible that a node will be unable to connect with other nodes, rendering
it unable to recover from a network outage or node restart.
****

[[update-node-certs-same]]
=== Update certificates with the same CA
++++
<titleabbrev>With the same CA</titleabbrev>
++++

This procedure assumes that the you have access to the CA certificate and key
that was originally generated (or otherwise held by your organization) and used
to sign the node certificates currently in use. It also assumes that the
clients connecting to {es} on the HTTP layer are configured to trust the CA
certificate.

If you have access to the CA used to sign your existing certificates, you only
need to replace the certificates and keys for each node in your cluster. If you
replace your existing certificates and keys on each node and use the same
filenames, {es} reloads the files starts using the new certificates and keys.

You don't have to restart each node, but doing so forces new TLS connections and is <<use-rolling-restarts,a recommended practice>> when updating certificates.
Therefore, the following steps include a node restart after updating each
certificate.

The following steps provide instructions for generating new node certificates
and keys for both the transport layer and the HTTP layer. You might only need
to replace one of these layer's certificates depending on which of your
certificates are expiring.

[[cert-password-updates]] 
IMPORTANT: If your keystore is password protected, the password
is stored in the {es} secure settings, _and_ the password needs to change, then
you must perform a <<restart-cluster-rolling,rolling restart>> on your cluster.
You must also use a different file name for the keystore so that {es} doesn't
reload the file before the node is restarted.

TIP: If your CA has changed, complete the steps in 
<<update-node-certs-different,update security certificates with a different CA>>.

[[node-certs-same-transport]]
==== Generate a new certificate for the transport layer
The following examples use PKCS#12 files, but the same steps apply to JKS
keystores.

. Open the `ES_PATH_CONF/elasticsearch.yml` file and check the names and
locations of the keystores that are currently in use. You'll use the same names
for your new certificate.
+
In this example, the keystore and truststore are pointing to different files.
Your configuration might use the same file that contains the certificate and CA.
In this case, include the path to that file for both the keystore and truststore.
+
[NOTE]
====
These instructions assume that the provided certificate is signed by a trusted
CA and the verification mode is set to `certificate`. This setting ensures that
nodes to not attempt to perform hostname verification.
====
+
[source,yaml]
----
xpack.security.transport.ssl.keystore.path: config/elastic-certificates.p12
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: config/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.verification_mode: certificate
----

. Using your existing CA, generate a keystore for your nodes. You must
use the CA that was used to sign the certificate currently in use.
+
[source,shell]
----
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
----
+
[%collapsible%open]
.Command parameters
====
   `--ca <ca_file>`:: Name of the CA keystore used to sign your certificates.
   If you used the `elasticsearch-certutil` tool to generate your existing CA,
   the keystore name defaults to `elastic-stack-ca.p12`.
====

   a. Enter a name for the output file or accept the default of
   `elastic-certificates.p12`.

   b. When prompted, enter a password for the node keystore.

. If you entered a password when creating the node keystore that is different 
from the current keystore password, run the following command to store the
password in the {es} keystore:
+
--
[source,shell]
----
./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
----
--

. [[start-rolling-restart,step 4]]On the current node in your cluster where you're updating the keystore,
start a <<restart-cluster-rolling,rolling restart>>.
+
Stop at the step indicating *Perform any needed changes*, and then proceed to
the next step in this procedure.

. [[replace-keystores]]Replace your existing keystore with the new keystore,
ensuring that the file names match. For example, `elastic-certificates.p12`.
+
IMPORTANT: If your 
<<cert-password-updates,keystore password is changing>>, then save the
keystore with a new filename so that {es} doesn't attempt to reload the file
before you update the password.

. If you needed to save the new keystore with a new filename, update the
`ES_PATH_CONF/elasticsearch.yml` file to use the filename of the new keystore.
For example:
+
[source,yaml]
----
xpack.security.transport.ssl.keystore.path: config/elastic-certificates.p12
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: config/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.type: PKCS12
----

. Start the node where you updated the keystore.

. [[verify-keystore,step 8]](Optional) Use the
<<security-api-ssl,SSL certificate API>> to verify that {es} loaded the new
keystore.
+
--

[source, console]
----
GET /_ssl/certificates
----
// TEST[skip:creates a lot of noise]
--

. If you're only updating certificates for the transport layer (and not the HTTP layer), then complete <<start-rolling-restart>> through <<verify-keystore>> one node at a time until you've updated all keystores in your cluster. You can then
complete the remaining steps for a <<restart-cluster-rolling,rolling restart>>.
+
Otherwise, do not complete a rolling restart. Instead, proceed to the steps for
generating a new certificate for the HTTP layer.

[discrete]
[[transport-layer-sameca-whatsnext]]
==== What's next?
Well done! You've updated the keystore for the transport layer. You can also
<<node-certs-same-http,update the keystore for the HTTP layer>> if
necessary. If you're not updating the keystore for the HTTP layer, then you're
all set. 

[[node-certs-same-http]]
==== Generate a new certificate for the HTTP layer
Other components such as {kib} or any of the Elastic language clients verify
this certificate when they connect to {es}.

NOTE: If your organization has its own CA, you'll need to 
<<certutil-csr,generate Certificate Signing Requests (CSRs)>>. CSRs contain
information that your CA uses to generate and sign a certificate.

. On any node in your cluster where {es} is installed, run the {es} HTTP
certificate tool.
+
[source,shell]
----
./bin/elasticsearch-certutil http
----
+
This command generates a `.zip` file that contains certificates and keys
to use with {es} and {kib}. Each folder contains a `README.txt`
explaining how to use these files.

   a. When asked if you want to generate a CSR, enter `n`.

   b. When asked if you want to use an existing CA, enter `y`.

   c. Enter the absolute path to your CA, such as the path to the
    `elastic-stack-ca.p12` file.

   d. Enter the password for your CA.

   e. Enter an expiration value for your certificate. You can enter the
   validity period in years, months, or days. For example, enter `1y` for one
   year.

   f. When asked if you want to generate one certificate per node, enter `y`.
+
Each certificate will have its own private key, and will be issued for a
specific hostname or IP address.

   g. When prompted, enter the name of the first node in your cluster. It's
   helpful to use the same node name as the value for the `node.name`
   parameter in the `elasticsearch.yml` file.

   h. Enter all hostnames used to connect to your first node. These hostnames
   will be added as DNS names in the Subject Alternative Name (SAN) field in your certificate.
+
List every hostname and variant used to connect to your cluster over HTTPS.

   i. Enter the IP addresses that clients can use to connect to your node.

   j. Repeat these steps for each additional node in your cluster.

. After generating a certificate for each of your nodes, enter a password for
   your private key when prompted.

. Unzip the generated `elasticsearch-ssl-http.zip` file. This compressed file
   contains two directories; one each for {es} and {kib}. Within the `/elasticsearch`
   directory is a directory for each node that you specified with its own
   `http.p12` file. For example:
+
--
[source,txt]
----
/node1
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml
----

[source,txt]
----
/node2
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml
----

[source,txt]
----
/node3
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml
----
--

. If necessary, rename the `http.p12` file to match the name of your existing
certificate for HTTP client communications. For example, `node1-http.p12`.

. [[start-rolling-restart-http,step 5]]On the current node in your cluster where you're updating the keystore,
start a <<restart-cluster-rolling,rolling restart>>.
+
Stop at the step indicating *Perform any needed changes*, and then proceed to
the next step in this procedure.

. Replace your existing keystore with the new keystore, ensuring that the
file names match. For example, `node1-http.p12`.
+
IMPORTANT: If your 
<<cert-password-updates,keystore password is changing>>, then save the
keystore with a new filename so that {es} doesn't attempt to reload the file
before you update the password.

. If you needed to save the new keystore with a new filename, update the
`ES_PATH_CONF/elasticsearch.yml` file to use the filename of the new keystore.
For example:
+
[source,yaml]
----
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: node1-http.p12
----

. If your keystore password is changing, add the password for your private key
to the secure settings in {es}.
+
[source,shell]
----
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
----

. Start the node where you updated the keystore.
+
--
Use the <<cat-nodes,cat nodes API>> to confirm that the node joined the cluster:

[source,console]
----
GET _cat/nodes
----

--

. [[verify-keystore-http,step 10]](Optional) Use the <<security-api-ssl,SSL certificate API>> to verify that {es} loaded the new keystore.
+
--

[source, console]
----
GET /_ssl/certificates
----
// TEST[skip:creates a lot of noise]
--

. One node at a time, complete <<start-rolling-restart-http>> through
<<verify-keystore-http>> until you've updated all keystores in your cluster.

. Complete the remaining steps for a <<restart-cluster-rolling,rolling restart>>,
beginning with the step to *Reenable shard allocation*.

[[update-node-certs-different]]
=== Update security certificates with a different CA
++++
<titleabbrev>With a different CA</titleabbrev>
++++
If you have to trust a new CA from your organization, or you need to generate a new CA yourself, use this new CA to sign the new node certificates and instruct your nodes to trust the new CA. 

[[node-certs-different-transport]]
==== Generate a new certificate for the transport layer
Create a new CA certificate, or get the CA certificate of your organization,
and add it to your existing CA truststore. After you finish updating your certificates for all nodes, you can remove the old CA
certificate from your truststore (but not before!).

NOTE: The following examples use PKCS#12 files, but the same steps apply to JKS
keystores.

. Open the `ES_PATH_CONF/elasticsearch.yml` file and check the names and
locations of the keystores that are currently in use. You'll use the same names
for your new keystores.
+
In this example, the keystore and truststore are using different files.
Your configuration might use the same file for both the keystore and the 
truststore.
+
[NOTE]
====
These instructions assume that the provided certificate is signed by a trusted
CA and the verification mode is set to `certificate`. This setting ensures that
nodes to not attempt to perform hostname verification.
====
+
[source,yaml]
----
xpack.security.transport.ssl.keystore.path: config/elastic-certificates.p12
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: config/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.verification_mode: certificate
----

. On *any* node in your cluster, generate a new CA certificate. You only need
to complete this step one time. If you're using the CA certificate of your
organization, then skip this step.
+
[source,shell]
----
./bin/elasticsearch-certutil ca --pem
----
+
[%collapsible%open]
.Command parameters
====
   `--pem`:: Generates a directory containing a CA certificate and key in PEM
   format instead of PKCS#12. 
====

    a. Enter a name for the compressed output file that will contain your
    certificate and key, or accept the default name of `elastic-stack-ca.zip`.

    b. Unzip the output file. The resulting directory contains a CA certificate
    (`ca.crt`) and a private key (`ca.key`).
+
--
IMPORTANT: Keep these file in a secure location as they contain the private key
for your CA. 
--

. On *every* node in your cluster, import the new `ca.crt` certificate into your
existing CA truststore. This step ensures that your cluster trusts the new CA
certificate. This example uses the Java `keytool` utility to import the
certificate into the `elastic-stack-ca.p12` CA truststore.
+
[source,shell]
----
keytool -importcert -trustcacerts -noprompt -keystore elastic-stack-ca.p12 \
-storepass <password>  -alias new-ca -file ca.crt
----
+
[%collapsible%open]
.Command parameters
====
   `-keystore`:: Name of the truststore that you are importing the new CA
   certificate into. 

   `-storepass`:: Password for the CA truststore. 

   `-alias`:: Name that you want to assign to the new CA certificate entry in the keystore.

   `-file`:: Name of the new CA certificate to import.
====

. [[check-ca-truststore]] Check that the new CA certificate was added to your
truststore. 
+
[source,shell]
----
keytool -keystore config/elastic-stack-ca.p12 -list
----
When prompted, enter the password for the CA truststore.
+
The output should contain both the existing CA certificate and your new
certificate. If you previously used the `elasticsearch-certutil` tool to
generate your keystore, the alias of the old CA defaults to `ca` and the type of
entry is `PrivateKeyEntry`.

[discrete]
[[node-certs-different-nodes]]
==== Generate a new certificate for each node in your cluster
Now that your CA truststore is updated, use your new CA certificate to sign
a certificate for your nodes.

NOTE: If your organization has its own CA, you'll need to 
<<certutil-csr,generate Certificate Signing Requests (CSRs)>>. CSRs contain
information that your CA uses to generate and sign a security certificate.

. Using the new CA certificate and key, create a new certificate for your nodes.
+
[source,shell]
----
./bin/elasticsearch-certutil cert --ca-cert ca/ca.crt --ca-key ca/ca.key
----
+
[%collapsible%open]
.Command parameters
====
   `--ca-cert`:: Specifies the path to your new CA certificate (`ca.crt`) in PEM
   format. You must also specify the `--ca-key` parameter.

   `--ca-key`:: Specifies the path to the private key (`ca.key`) for your CA
   certificate. You must also specify the `--ca-cert` parameter.
====

   a. Enter a name for the output file or accept the default of
   `elastic-certificates.p12`.

   b. When prompted, enter a password for your node certificate.

. [[start-rolling-restart-newca,step 2]]On the current node in your cluster where
you're updating the keystore, start a
<<restart-cluster-rolling,rolling restart>>.
+
Stop at the step indicating *Perform any needed changes*, and then proceed to
the next step in this procedure.

. Replace your existing keystore with the new keystore, ensuring that the
file names match. For example, `elastic-certificates.p12`.
+
IMPORTANT: If your 
<<cert-password-updates,keystore password is changing>>, then save the
keystore with a new filename so that {es} doesn't attempt to reload the file
before you update the password.

. If you needed to save the new keystore with a new filename, update the
`ES_PATH_CONF/elasticsearch.yml` file to use the filename of the new keystore.
For example:
+
[source,yaml]
----
xpack.security.transport.ssl.keystore.path: config/elastic-certificates.p12
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.truststore.path: config/elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.type: PKCS12
----

. Start the node where you updated the keystore.

. [[verify-keystore-newca,step 6]](Optional) Use the <<security-api-ssl,SSL certificate API>> to verify that {es} loaded the new keystore.
+
--

[source, console]
----
GET /_ssl/certificates
----
// TEST[skip:creates a lot of noise]
--

. If you're only updating certificates for the transport layer (and not the HTTP
layer), then complete <<start-rolling-restart-newca>> through
<<verify-keystore-newca>> one node at a time until you've updated all keystores
in your cluster. You can then complete the remaining steps for a
<<restart-cluster-rolling,rolling restart>>.
+
Otherwise, do not complete a rolling restart. Instead, proceed to the steps for
generating a new certificate for the HTTP layer.

. (Optional) After replacing keystores on each node in your cluster,
<<check-ca-truststore,list the certificates in your truststore>> and then remove
the old CA certificate.
+
If you previously used the `elasticsearch-certutil` tool to generate your
keystore, the alias of the old CA defaults to `ca` and the type of entry is
`PrivateKeyEntry`.
+
[source,shell]
----
keytool -delete -noprompt -alias ca  -keystore config/elastic-stack-ca.p12 \
-storepass <password>
----
+
[%collapsible%open]
.Command parameters
====
   `-alias`:: Name of the keystore alias for the old CA certificate that you want to remove from your
   truststore.
====

[discrete]
[[transport-layer-newca-whatsnext]]
==== What's next?
Well done! You've updated the keystore for the transport layer. You can also
<<node-certs-different-http,update the keystore for the HTTP layer>> if
necessary. If you're not updating the keystore for the HTTP layer, then you're
all set. 

[[node-certs-different-http]]
==== Generate a new certificate for the HTTP layer
You can generate certificates for the HTTP layer using your new CA certificate
and private key. Other components such as {kib} or any of the Elastic language
clients verify this certificate when they connect to {es}.

NOTE: If your organization has its own CA, you'll need to 
<<certutil-csr,generate Certificate Signing Requests (CSRs)>>. CSRs contain
information that your CA uses to generate and sign a security certificate
instead of using self-signed certificates that the `elasticsearch-certutil` tool
generates.

.Update clients to trust the new CA
****
After generating (but before using) new certificates for the HTTP layer, you
need to go to all the clients that connect to {es} (such as {beats}, {ls}, and
any language clients) and configure them to also trust the new CA (`ca.crt`)
that you generated.

This process is different for each client, so refer to your client's
documentation for trusting certificates. You'll
<<node-certs-different-kibana,update HTTP encryption between {kib} and {es}>>
after generating the necessary certificates in this procedure.
****

. On any node in your cluster where {es} is installed, run the {es} HTTP
certificate tool.
+
[source,shell]
----
./bin/elasticsearch-certutil http
----
+
This command generates a `.zip` file that contains certificates and keys
to use with {es} and {kib}. Each folder contains a `README.txt`
explaining how to use these files.

   a. When asked if you want to generate a CSR, enter `n`.

   b. When asked if you want to use an existing CA, enter `y`.

   c. Enter the absolute path to your *new* CA certificate, such as the path to
   the `ca.crt` file.

   d. Enter the absolute path to your new CA certificate private key, such as
   the path to the `ca.key` file.

   e. Enter an expiration value for your certificate. You can enter the
   validity period in years, months, or days. For example, enter `1y` for one
   year.

   f. When asked if you want to generate one certificate per node, enter `y`.
+
Each certificate will have its own private key, and will be issued for a
specific hostname or IP address.

   g. When prompted, enter the name of the first node in your cluster. Use the
   same node name as the value for the `node.name` parameter in the
   `elasticsearch.yml` file.

   h. Enter all hostnames used to connect to your first node. These hostnames
   will be added as DNS names in the Subject Alternative Name (SAN) field in your certificate.
+
List every hostname and variant used to connect to your cluster over HTTPS.

   i. Enter the IP addresses that clients can use to connect to your node.

   j. Repeat these steps for each additional node in your cluster.

. After generating a certificate for each of your nodes, enter a password for
   your keystore when prompted.

. Unzip the generated `elasticsearch-ssl-http.zip` file. This compressed file
   contains one directory for both {es} and {kib}. Within the `/elasticsearch`
   directory is a directory for each node that you specified with its own
   `http.p12` file. For example:
+
--
[source,txt]
----
/node1
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml
----

[source,txt]
----
/node2
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml
----

[source,txt]
----
/node3
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml
----
--

. If necessary, rename each `http.p12` file to match the name of your existing
certificate for HTTP client communications. For example, `node1-http.p12`.

. [[start-rolling-restart-http-newca,step 5]]On the current node in your cluster where you're updating the keystore,
start a <<restart-cluster-rolling,rolling restart>>.
+
Stop at the step indicating *Perform any needed changes*, and then proceed to
the next step in this procedure.

. Replace your existing keystore with the new keystore, ensuring that the
file names match. For example, `node1-http.p12`.
+
IMPORTANT: If your 
<<cert-password-updates,keystore password is changing>>, then save the
keystore with a new filename so that {es} doesn't attempt to reload the file
before you update the password.

. If you needed to save the new keystore with a new filename, update the
`ES_PATH_CONF/elasticsearch.yml` file to use the filename of the new keystore.
For example:
+
[source,yaml]
----
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: node1-http.p12
----

. If your keystore password is changing, add the password for your private key
to the secure settings in {es}.
+
[source,shell]
----
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
----

. Start the node where you updated the keystore.
+
--
Use the <<cat-nodes,cat nodes API>> to confirm that the node joined the cluster:

[source,console]
----
GET _cat/nodes
----

--

. [[verify-keystore-http-newca,step 10]](Optional) Use the <<security-api-ssl,SSL certificate API>> to verify that
{es} loaded the new keystore.
+
--

[source, console]
----
GET /_ssl/certificates
----
// TEST[skip:creates a lot of noise]
--

. One node at a time, complete <<start-rolling-restart-http-newca>> through
<<verify-keystore-http-newca>> until you've updated all keystores in your cluster.

. Complete the remaining steps for a <<restart-cluster-rolling,rolling restart>>,
beginning with the step to *Reenable shard allocation*.

[discrete]
[[http-kibana-newca-whatsnext]]
==== What's next?
Well done! You've updated the keystore for the HTTP layer. You can now
<<node-certs-different-kibana,update encryption between {kib} and {es}>>.

[[node-certs-different-kibana]]
==== Update encryption between {kib} and {es}

When you ran the `elasticsearch-certutil` tool with the `http` option, it
created a `/kibana` directory containing an `elasticsearch-ca.pem` file. You
use this file to configure {kib} to trust the {es} CA for the HTTP
layer.

. Copy the `elasticsearch-ca.pem` file to the {kib} configuration directory,
as defined by the `KBN_PATH_CONF` path.
+
NOTE: `KBN_PATH_CONF` contains the path for the {kib} configuration files. If
you installed {kib} using archive distributions (`zip` or `tar.gz`), the
path defaults to `KBN_HOME/config`. If you used package distributions
(Debian or RPM), the path defaults to `/etc/kibana`.

. If you modified the filename for the `elasticsearch-ca.pem` file, edit
`kibana.yml` and update the configuration to specify the location of the
security certificate for the HTTP layer.
+
[source,yaml]
----
elasticsearch.ssl.certificateAuthorities: KBN_PATH_CONF/elasticsearch-ca.pem
----

. Restart {kib}.