|
|
@@ -1,490 +0,0 @@
|
|
|
-[role="xpack"]
|
|
|
-[[security-troubleshooting]]
|
|
|
-== {security} Troubleshooting
|
|
|
-++++
|
|
|
-<titleabbrev>{security}</titleabbrev>
|
|
|
-++++
|
|
|
-
|
|
|
-Use the information in this section to troubleshoot common problems and find
|
|
|
-answers for frequently asked questions.
|
|
|
-
|
|
|
-* <<security-trb-settings>>
|
|
|
-* <<security-trb-roles>>
|
|
|
-* <<security-trb-extraargs>>
|
|
|
-* <<trouble-shoot-active-directory>>
|
|
|
-* <<trb-security-maccurl>>
|
|
|
-* <<trb-security-sslhandshake>>
|
|
|
-* <<trb-security-ssl>>
|
|
|
-* <<trb-security-kerberos>>
|
|
|
-* <<trb-security-internalserver>>
|
|
|
-* <<trb-security-setup>>
|
|
|
-
|
|
|
-
|
|
|
-To get help, see <<help>>.
|
|
|
-
|
|
|
-[[security-trb-settings]]
|
|
|
-=== Some settings are not returned via the nodes settings API
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* When you use the {ref}/cluster-nodes-info.html[nodes info API] to retrieve
|
|
|
-settings for a node, some information is missing.
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-This is intentional. Some of the settings are considered to be highly
|
|
|
-sensitive: all `ssl` settings, ldap `bind_dn`, and `bind_password`.
|
|
|
-For this reason, we filter these settings and do not expose them via
|
|
|
-the nodes info API rest endpoint. You can also define additional
|
|
|
-sensitive settings that should be hidden using the
|
|
|
-`xpack.security.hide_settings` setting. For example, this snippet
|
|
|
-hides the `url` settings of the `ldap1` realm and all settings of the
|
|
|
-`ad1` realm.
|
|
|
-
|
|
|
-[source, yaml]
|
|
|
-------------------------------------------
|
|
|
-xpack.security.hide_settings: xpack.security.authc.realms.ldap1.url,
|
|
|
-xpack.security.authc.realms.ad1.*
|
|
|
-------------------------------------------
|
|
|
-
|
|
|
-[[security-trb-roles]]
|
|
|
-=== Authorization exceptions
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* I configured the appropriate roles and the users, but I still get an
|
|
|
-authorization exception.
|
|
|
-* I can authenticate to LDAP, but I still get an authorization exception.
|
|
|
-
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-. Verify that the role names associated with the users match the roles defined
|
|
|
-in the `roles.yml` file. You can use the `elasticsearch-users` tool to list all
|
|
|
-the users. Any unknown roles are marked with `*`.
|
|
|
-+
|
|
|
---
|
|
|
-[source, shell]
|
|
|
-------------------------------------------
|
|
|
-bin/elasticsearch-users list
|
|
|
-rdeniro : admin
|
|
|
-alpacino : power_user
|
|
|
-jacknich : monitoring,unknown_role* <1>
|
|
|
-------------------------------------------
|
|
|
-<1> `unknown_role` was not found in `roles.yml`
|
|
|
-
|
|
|
-For more information about this command, see the
|
|
|
-{ref}/users-command.html[`elasticsearch-users` command].
|
|
|
---
|
|
|
-
|
|
|
-. If you are authenticating to LDAP, a number of configuration options can cause
|
|
|
-this error.
|
|
|
-+
|
|
|
---
|
|
|
-|======================
|
|
|
-|_group identification_ |
|
|
|
-
|
|
|
-Groups are located by either an LDAP search or by the "memberOf" attribute on
|
|
|
-the user. Also, If subtree search is turned off, it will search only one
|
|
|
-level deep. See the <<ldap-settings, LDAP Settings>> for all the options.
|
|
|
-There are many options here and sticking to the defaults will not work for all
|
|
|
-scenarios.
|
|
|
-
|
|
|
-| _group to role mapping_|
|
|
|
-
|
|
|
-Either the `role_mapping.yml` file or the location for this file could be
|
|
|
-misconfigured. See <<security-files, Security Files>> for more.
|
|
|
-
|
|
|
-|_role definition_|
|
|
|
-
|
|
|
-The role definition might be missing or invalid.
|
|
|
-
|
|
|
-|======================
|
|
|
-
|
|
|
-To help track down these possibilities, add the following lines to the end of
|
|
|
-the `log4j2.properties` configuration file in the `ES_PATH_CONF`:
|
|
|
-
|
|
|
-[source,properties]
|
|
|
-----------------
|
|
|
-logger.authc.name = org.elasticsearch.xpack.security.authc
|
|
|
-logger.authc.level = DEBUG
|
|
|
-----------------
|
|
|
-
|
|
|
-A successful authentication should produce debug statements that list groups and
|
|
|
-role mappings.
|
|
|
---
|
|
|
-
|
|
|
-[[security-trb-extraargs]]
|
|
|
-=== Users command fails due to extra arguments
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* The `elasticsearch-users` command fails with the following message:
|
|
|
-`ERROR: extra arguments [...] were provided`.
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-This error occurs when the `elasticsearch-users` tool is parsing the input and
|
|
|
-finds unexpected arguments. This can happen when there are special characters
|
|
|
-used in some of the arguments. For example, on Windows systems the `,` character
|
|
|
-is considered a parameter separator; in other words `-r role1,role2` is
|
|
|
-translated to `-r role1 role2` and the `elasticsearch-users` tool only
|
|
|
-recognizes `role1` as an expected parameter. The solution here is to quote the
|
|
|
-parameter: `-r "role1,role2"`.
|
|
|
-
|
|
|
-For more information about this command, see
|
|
|
-{ref}/users-command.html[`elasticsearch-users` command].
|
|
|
-
|
|
|
-[[trouble-shoot-active-directory]]
|
|
|
-=== Users are frequently locked out of Active Directory
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* Certain users are being frequently locked out of Active Directory.
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-Check your realm configuration; realms are checked serially, one after another.
|
|
|
-If your Active Directory realm is being checked before other realms and there
|
|
|
-are usernames that appear in both Active Directory and another realm, a valid
|
|
|
-login for one realm might be causing failed login attempts in another realm.
|
|
|
-
|
|
|
-For example, if `UserA` exists in both Active Directory and a file realm, and
|
|
|
-the Active Directory realm is checked first and file is checked second, an
|
|
|
-attempt to authenticate as `UserA` in the file realm would first attempt to
|
|
|
-authenticate against Active Directory and fail, before successfully
|
|
|
-authenticating against the `file` realm. Because authentication is verified on
|
|
|
-each request, the Active Directory realm would be checked - and fail - on each
|
|
|
-request for `UserA` in the `file` realm. In this case, while the authentication
|
|
|
-request completed successfully, the account on Active Directory would have
|
|
|
-received several failed login attempts, and that account might become
|
|
|
-temporarily locked out. Plan the order of your realms accordingly.
|
|
|
-
|
|
|
-Also note that it is not typically necessary to define multiple Active Directory
|
|
|
-realms to handle domain controller failures. When using Microsoft DNS, the DNS
|
|
|
-entry for the domain should always point to an available domain controller.
|
|
|
-
|
|
|
-
|
|
|
-[[trb-security-maccurl]]
|
|
|
-=== Certificate verification fails for curl on Mac
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* `curl` on the Mac returns a certificate verification error even when the
|
|
|
-`--cacert` option is used.
|
|
|
-
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-Apple's integration of `curl` with their keychain technology disables the
|
|
|
-`--cacert` option.
|
|
|
-See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.
|
|
|
-
|
|
|
-You can use another tool, such as `wget`, to test certificates. Alternately, you
|
|
|
-can add the certificate for the signing certificate authority MacOS system
|
|
|
-keychain, using a procedure similar to the one detailed at the
|
|
|
-http://support.apple.com/kb/PH14003[Apple knowledge base]. Be sure to add the
|
|
|
-signing CA's certificate and not the server's certificate.
|
|
|
-
|
|
|
-
|
|
|
-[[trb-security-sslhandshake]]
|
|
|
-=== SSLHandshakeException causes connections to fail
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* A `SSLHandshakeException` causes a connection to a node to fail and indicates
|
|
|
-that there is a configuration issue. Some of the common exceptions are shown
|
|
|
-below with tips on how to resolve these issues.
|
|
|
-
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-`java.security.cert.CertificateException: No name matching node01.example.com found`::
|
|
|
-+
|
|
|
---
|
|
|
-Indicates that a client connection was made to `node01.example.com` but the
|
|
|
-certificate returned did not contain the name `node01.example.com`. In most
|
|
|
-cases, the issue can be resolved by ensuring the name is specified during
|
|
|
-certificate creation. For more information, see <<ssl-tls>>. Another scenario is
|
|
|
-when the environment does not wish to use DNS names in certificates at all. In
|
|
|
-this scenario, all settings in `elasticsearch.yml` should only use IP addresses
|
|
|
-including the `network.publish_host` setting.
|
|
|
---
|
|
|
-
|
|
|
-`java.security.cert.CertificateException: No subject alternative names present`::
|
|
|
-+
|
|
|
---
|
|
|
-Indicates that a client connection was made to an IP address but the returned
|
|
|
-certificate did not contain any `SubjectAlternativeName` entries. IP addresses
|
|
|
-are only used for hostname verification if they are specified as a
|
|
|
-`SubjectAlternativeName` during certificate creation. If the intent was to use
|
|
|
-IP addresses for hostname verification, then the certificate will need to be
|
|
|
-regenerated with the appropriate IP address. See <<ssl-tls>>.
|
|
|
---
|
|
|
-
|
|
|
-`javax.net.ssl.SSLHandshakeException: null cert chain` and `javax.net.ssl.SSLException: Received fatal alert: bad_certificate`::
|
|
|
-+
|
|
|
---
|
|
|
-The `SSLHandshakeException` indicates that a self-signed certificate was
|
|
|
-returned by the client that is not trusted as it cannot be found in the
|
|
|
-`truststore` or `keystore`. This `SSLException` is seen on the client side of
|
|
|
-the connection.
|
|
|
---
|
|
|
-
|
|
|
-`sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` and `javax.net.ssl.SSLException: Received fatal alert: certificate_unknown`::
|
|
|
-+
|
|
|
---
|
|
|
-This `SunCertPathBuilderException` indicates that a certificate was returned
|
|
|
-during the handshake that is not trusted. This message is seen on the client
|
|
|
-side of the connection. The `SSLException` is seen on the server side of the
|
|
|
-connection. The CA certificate that signed the returned certificate was not
|
|
|
-found in the `keystore` or `truststore` and needs to be added to trust this
|
|
|
-certificate.
|
|
|
---
|
|
|
-
|
|
|
-[[trb-security-ssl]]
|
|
|
-=== Common SSL/TLS exceptions
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* You might see some exceptions related to SSL/TLS in your logs. Some of the
|
|
|
-common exceptions are shown below with tips on how to resolve these issues. +
|
|
|
-
|
|
|
-
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-`WARN: received plaintext http traffic on a https channel, closing connection`::
|
|
|
-+
|
|
|
---
|
|
|
-Indicates that there was an incoming plaintext http request. This typically
|
|
|
-occurs when an external applications attempts to make an unencrypted call to the
|
|
|
-REST interface. Please ensure that all applications are using `https` when
|
|
|
-calling the REST interface with SSL enabled.
|
|
|
---
|
|
|
-
|
|
|
-`org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:`::
|
|
|
-+
|
|
|
---
|
|
|
-Indicates that there was incoming plaintext traffic on an SSL connection. This
|
|
|
-typically occurs when a node is not configured to use encrypted communication
|
|
|
-and tries to connect to nodes that are using encrypted communication. Please
|
|
|
-verify that all nodes are using the same setting for
|
|
|
-`xpack.security.transport.ssl.enabled`.
|
|
|
-
|
|
|
-For more information about this setting, see
|
|
|
-{ref}/security-settings.html[Security Settings in {es}].
|
|
|
---
|
|
|
-
|
|
|
-`java.io.StreamCorruptedException: invalid internal transport message format, got`::
|
|
|
-+
|
|
|
---
|
|
|
-Indicates an issue with data received on the transport interface in an unknown
|
|
|
-format. This can happen when a node with encrypted communication enabled
|
|
|
-connects to a node that has encrypted communication disabled. Please verify that
|
|
|
-all nodes are using the same setting for `xpack.security.transport.ssl.enabled`.
|
|
|
-
|
|
|
-For more information about this setting, see
|
|
|
-{ref}/security-settings.html[Security Settings in {es}].
|
|
|
---
|
|
|
-
|
|
|
-`java.lang.IllegalArgumentException: empty text`::
|
|
|
-+
|
|
|
---
|
|
|
-This exception is typically seen when a `https` request is made to a node that
|
|
|
-is not using `https`. If `https` is desired, please ensure the following setting
|
|
|
-is in `elasticsearch.yml`:
|
|
|
-
|
|
|
-[source,yaml]
|
|
|
-----------------
|
|
|
-xpack.security.http.ssl.enabled: true
|
|
|
-----------------
|
|
|
-
|
|
|
-For more information about this setting, see
|
|
|
-{ref}/security-settings.html[Security Settings in {es}].
|
|
|
---
|
|
|
-
|
|
|
-`ERROR: unsupported ciphers [...] were requested but cannot be used in this JVM`::
|
|
|
-+
|
|
|
---
|
|
|
-This error occurs when a SSL/TLS cipher suite is specified that cannot supported
|
|
|
-by the JVM that {es} is running in. Security tries to use the specified cipher
|
|
|
-suites that are supported by this JVM. This error can occur when using the
|
|
|
-Security defaults as some distributions of OpenJDK do not enable the PKCS11
|
|
|
-provider by default. In this case, we recommend consulting your JVM
|
|
|
-documentation for details on how to enable the PKCS11 provider.
|
|
|
-
|
|
|
-Another common source of this error is requesting cipher suites that use
|
|
|
-encrypting with a key length greater than 128 bits when running on an Oracle JDK.
|
|
|
-In this case, you must install the
|
|
|
-<<ciphers, JCE Unlimited Strength Jurisdiction Policy Files>>.
|
|
|
---
|
|
|
-
|
|
|
-[[trb-security-kerberos]]
|
|
|
-=== Common Kerberos exceptions
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* User authentication fails due to either GSS negotiation failure
|
|
|
-or a service login failure (either on the server or in the {es} http client).
|
|
|
-Some of the common exceptions are listed below with some tips to help resolve
|
|
|
-them.
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-`Failure unspecified at GSS-API level (Mechanism level: Checksum failed)`::
|
|
|
-+
|
|
|
---
|
|
|
-
|
|
|
-When you see this error message on the HTTP client side, then it may be
|
|
|
-related to an incorrect password.
|
|
|
-
|
|
|
-When you see this error message in the {es} server logs, then it may be
|
|
|
-related to the {es} service keytab. The keytab file is present but it failed
|
|
|
-to log in as the user. Please check the keytab expiry. Also check whether the
|
|
|
-keytab contain up-to-date credentials; if not, replace them.
|
|
|
-
|
|
|
-You can use tools like `klist` or `ktab` to list principals inside
|
|
|
-the keytab and validate them. You can use `kinit` to see if you can acquire
|
|
|
-initial tickets using the keytab. Please check the tools and their documentation
|
|
|
-in your Kerberos environment.
|
|
|
-
|
|
|
-Kerberos depends on proper hostname resolution, so please check your DNS infrastructure.
|
|
|
-Incorrect DNS setup, DNS SRV records or configuration for KDC servers in `krb5.conf`
|
|
|
-can cause problems with hostname resolution.
|
|
|
-
|
|
|
---
|
|
|
-
|
|
|
-`Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))`::
|
|
|
-
|
|
|
-`Failure unspecified at GSS-API level (Mechanism level: Clock skew too great (37))`::
|
|
|
-+
|
|
|
---
|
|
|
-
|
|
|
-To prevent replay attacks, Kerberos V5 sets a maximum tolerance for computer
|
|
|
-clock synchronization and it is typically 5 minutes. Please check whether
|
|
|
-the time on the machines within the domain is in sync.
|
|
|
-
|
|
|
---
|
|
|
-
|
|
|
-As Kerberos logs are often cryptic in nature and many things can go wrong
|
|
|
-as it depends on external services like DNS and NTP. You might
|
|
|
-have to enable additional debug logs to determine the root cause of the issue.
|
|
|
-
|
|
|
-{es} uses a JAAS (Java Authentication and Authorization Service) Kerberos login
|
|
|
-module to provide Kerberos support. To enable debug logs on {es} for the login
|
|
|
-module use following Kerberos realm setting:
|
|
|
-[source,yaml]
|
|
|
-----------------
|
|
|
-xpack.security.authc.realms.<realm-name>.krb.debug: true
|
|
|
-----------------
|
|
|
-
|
|
|
-For detailed information, see {ref}/security-settings.html#ref-kerberos-settings[Kerberos realm settings].
|
|
|
-
|
|
|
-Sometimes you may need to go deeper to understand the problem during SPNEGO
|
|
|
-GSS context negotiation or look at the Kerberos message exchange. To enable
|
|
|
-Kerberos/SPNEGO debug logging on JVM, add following JVM system properties:
|
|
|
-
|
|
|
-`-Dsun.security.krb5.debug=true`
|
|
|
-
|
|
|
-`-Dsun.security.spnego.debug=true`
|
|
|
-
|
|
|
-For more information about JVM system properties, see {ref}/jvm-options.html[configuring JVM options].
|
|
|
-
|
|
|
-[[trb-security-internalserver]]
|
|
|
-=== Internal Server Error in Kibana
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-* In 5.1.1, an `UnhandledPromiseRejectionWarning` occurs and {kib} displays an
|
|
|
-Internal Server Error.
|
|
|
-//TBD: Is the same true for later releases?
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-If the Security plugin is enabled in {es} but disabled in {kib}, you must
|
|
|
-still set `elasticsearch.username` and `elasticsearch.password` in `kibana.yml`.
|
|
|
-Otherwise, {kib} cannot connect to {es}.
|
|
|
-
|
|
|
-
|
|
|
-[[trb-security-setup]]
|
|
|
-=== Setup-passwords command fails due to connection failure
|
|
|
-
|
|
|
-The {ref}/setup-passwords.html[elasticsearch-setup-passwords command] sets
|
|
|
-passwords for the built-in users by sending user management API requests. If
|
|
|
-your cluster uses SSL/TLS for the HTTP (REST) interface, the command attempts to
|
|
|
-establish a connection with the HTTPS protocol. If the connection attempt fails,
|
|
|
-the command fails.
|
|
|
-
|
|
|
-*Symptoms:*
|
|
|
-
|
|
|
-. {es} is running HTTPS, but the command fails to detect it and returns the
|
|
|
-following errors:
|
|
|
-+
|
|
|
---
|
|
|
-[source, shell]
|
|
|
-------------------------------------------
|
|
|
-Cannot connect to elasticsearch node.
|
|
|
-java.net.SocketException: Unexpected end of file from server
|
|
|
-...
|
|
|
-ERROR: Failed to connect to elasticsearch at
|
|
|
-http://127.0.0.1:9200/_xpack/security/_authenticate?pretty.
|
|
|
-Is the URL correct and elasticsearch running?
|
|
|
-------------------------------------------
|
|
|
---
|
|
|
-
|
|
|
-. SSL/TLS is configured, but trust cannot be established. The command returns
|
|
|
-the following errors:
|
|
|
-+
|
|
|
---
|
|
|
-[source, shell]
|
|
|
-------------------------------------------
|
|
|
-SSL connection to
|
|
|
-https://127.0.0.1:9200/_xpack/security/_authenticate?pretty
|
|
|
-failed: sun.security.validator.ValidatorException:
|
|
|
-PKIX path building failed:
|
|
|
-sun.security.provider.certpath.SunCertPathBuilderException:
|
|
|
-unable to find valid certification path to requested target
|
|
|
-Please check the elasticsearch SSL settings under
|
|
|
-xpack.security.http.ssl.
|
|
|
-...
|
|
|
-ERROR: Failed to establish SSL connection to elasticsearch at
|
|
|
-https://127.0.0.1:9200/_xpack/security/_authenticate?pretty.
|
|
|
-------------------------------------------
|
|
|
---
|
|
|
-
|
|
|
-. The command fails because hostname verification fails, which results in the
|
|
|
-following errors:
|
|
|
-+
|
|
|
---
|
|
|
-[source, shell]
|
|
|
-------------------------------------------
|
|
|
-SSL connection to
|
|
|
-https://idp.localhost.test:9200/_xpack/security/_authenticate?pretty
|
|
|
-failed: java.security.cert.CertificateException:
|
|
|
-No subject alternative DNS name matching
|
|
|
-elasticsearch.example.com found.
|
|
|
-Please check the elasticsearch SSL settings under
|
|
|
-xpack.security.http.ssl.
|
|
|
-...
|
|
|
-ERROR: Failed to establish SSL connection to elasticsearch at
|
|
|
-https://elasticsearch.example.com:9200/_xpack/security/_authenticate?pretty.
|
|
|
-------------------------------------------
|
|
|
---
|
|
|
-
|
|
|
-*Resolution:*
|
|
|
-
|
|
|
-. If your cluster uses TLS/SSL for the HTTP interface but the
|
|
|
-`elasticsearch-setup-passwords` command attempts to establish a non-secure
|
|
|
-connection, use the `--url` command option to explicitly specify an HTTPS URL.
|
|
|
-Alternatively, set the `xpack.security.http.ssl.enabled` setting to `true`.
|
|
|
-
|
|
|
-. If the command does not trust the {es} server, verify that you configured the
|
|
|
-`xpack.security.http.ssl.certificate_authorities` setting or the
|
|
|
-`xpack.security.http.ssl.truststore.path` setting.
|
|
|
-
|
|
|
-. If hostname verification fails, you can disable this verification by setting
|
|
|
-`xpack.security.http.ssl.verification_mode` to `certificate`.
|
|
|
-
|
|
|
-For more information about these settings, see
|
|
|
-{ref}/security-settings.html[Security Settings in {es}].
|
lcawl
