Remove index audit output type (#37707)

This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e6770. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881

docs/reference/migration/migrate_7_0/settings.asciidoc

@@ -154,3 +154,18 @@ node's keystore, i.e., they are not to be specified via the cluster settings API
 `xpack.notification.pagerduty.account.<id>.secure_service_api_key`
 - `xpack.notification.slack.account.<id>.url`, instead use
 `xpack.notification.slack.account.<id>.secure_url`
+
+[float]
+[[remove-audit-index-output]]
+==== Audit index output type removed
+
+All the settings under the `xpack.security.audit.index` namespace have been
+removed. In addition, the `xpack.security.audit.outputs` setting has been
+removed as well.
+
+These settings enabled and configured the audit index output type. This output
+type has been removed because it was unreliable in certain scenarios and this
+could have lead to dropping audit events while the operations on the system
+were allowed to continue as usual. The recommended replacement is the
+use of the `logfile` audit output type and using other components from the
+Elastic Stack to handle the indexing part.

docs/reference/settings/audit-settings.asciidoc

@@ -14,38 +14,19 @@ file. For more information, see
 
 `xpack.security.audit.enabled`::
 Set to `true` to enable auditing on the node. The default value is `false`.
-
-`xpack.security.audit.outputs`::
-Specifies where audit logs are output. For example: `[ index, logfile ]`. The
-default value is `logfile`, which puts the auditing events in a dedicated
-file named `<clustername>_audit.log` on each node.
-You can also specify `index`, which puts the auditing events in an {es} index
-that is prefixed with `.security_audit_log`. The index can reside on the same
-cluster or a separate cluster.
-+
-For backwards compatibility reasons, if you use the logfile output type, a
-`<clustername>_access.log` file is also created. It contains the same
-information, but it uses the older (pre-6.5.0) formatting style.
-If the backwards compatible format is not required, it should be disabled.
-To do that, change its logger level to `off` in the `log4j2.properties` file.
-For more information, see <<configuring-logging-levels>>.
-+
---
-TIP: If the index is unavailable, it is possible for auditing events to
-be lost. The `index` output type should therefore be used in conjunction with
-the `logfile` output type and the latter should be the official record of events.
-
---
+This puts the auditing events in a dedicated file named `<clustername>_audit.log`
+on each node. For more information, see <<configuring-logging-levels>>.
 
 [[event-audit-settings]]
 ==== Audited Event Settings
 
-The events and some other information about what gets logged can be
-controlled by using the following settings:
+The events and some other information about what gets logged can be controlled
+by using the following settings:
 
 `xpack.security.audit.logfile.events.include`::
 Specifies which events to include in the auditing output. The default value is:
-`access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted`.
+`access_denied, access_granted, anonymous_access_denied, authentication_failed,
+connection_denied, tampered_request, run_as_denied, run_as_granted`.
 
 `xpack.security.audit.logfile.events.exclude`::
 Excludes the specified events from the output. By default, no events are
@@ -113,98 +94,3 @@ A list of index names or wildcards. The specified policy will
 not print audit events when all the indices in the event match
 these values. If the event concerns several indices, some of which are
 *not* covered by the policy, the policy will *not* cover this event.
-
-[[index-audit-settings]]
-==== Audit Log Indexing Configuration Settings
-
-`xpack.security.audit.index.bulk_size`::
-Controls how many audit events are batched into a single write. The default
-value is `1000`.
-
-`xpack.security.audit.index.flush_interval`::
-Controls how often buffered events are flushed to the index. The default value
-is `1s`.
-
-`xpack.security.audit.index.rollover`::
-Controls how often to roll over to a new index: `hourly`, `daily`, `weekly`, or
-`monthly`. The default value is `daily`.
-
-`xpack.security.audit.index.events.include`::
-Specifies the audit events to be indexed. The default value is
-`anonymous_access_denied, authentication_failed, realm_authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied, run_as_granted, run_as_denied`.
-See {xpack-ref}/audit-event-types.html[Audit Entry Types] for the
-complete list.
-
-`xpack.security.audit.index.events.exclude`::
-Excludes the specified auditing events from indexing. By default, no events are
-excluded.
-
-`xpack.security.audit.index.events.emit_request_body`::
-Specifies whether to include the request body from REST requests on certain
-event types such as `authentication_failed`. The default value is `false`.
-
-`xpack.security.audit.index.settings`::
-Specifies settings for the indices that the events are stored in. For example,
-the following configuration sets the number of shards and replicas to 1 for the
-audit indices:
-+
---
-[source,yaml]
-----------------------------
-xpack.security.audit.index.settings:
-  index:
-    number_of_shards: 1
-    number_of_replicas: 1
-----------------------------
---
-+
---
-NOTE: These settings apply to the local audit indices, as well as to the
-<<remote-audit-settings, remote audit indices>>, but only if the remote cluster
-does *not* have {security-features} enabled or the {es} versions are different.
-If the remote cluster has {security-features} enabled and the versions coincide,
-the settings for the audit indices there will take precedence,
-even if they are unspecified (i.e. left to defaults).
---
-
-[[remote-audit-settings]]
-==== Remote Audit Log Indexing Configuration Settings
-
-To index audit events to a remote {es} cluster, you configure the following
-`xpack.security.audit.index.client` settings:
-
-`xpack.security.audit.index.client.hosts`::
-Specifies a comma-separated list of `host:port` pairs. These hosts should be
-nodes in the remote cluster. If you are using default values for the 
-<<common-network-settings,`transport.port`>> setting, you can omit the
-`port` value. Otherwise, it must match the `transport.port` setting.
-
-`xpack.security.audit.index.client.cluster.name`::
-Specifies the name of the remote cluster.
-
-`xpack.security.audit.index.client.xpack.security.user`::
-Specifies the `username:password` pair that is used to authenticate with the
-remote cluster. This user must have authority to create the `.security-audit` 
-index on the remote cluster. 
-
-If the remote {es} cluster has Transport Layer Security (TLS/SSL) enabled, you 
-must set the following setting to `true`:
-
-`xpack.security.audit.index.client.xpack.security.transport.ssl.enabled`::
-Used to enable or disable TLS/SSL for the transport client that forwards audit 
-logs to the remote cluster. The default is `false`. 
-
-You must also specify the information necessary to access certificates. See 
-<<auditing-tls-ssl-settings>>. 
-
-You can pass additional settings to the remote client by specifying them in the
-`xpack.security.audit.index.client` namespace. For example, you can add 
-<<modules-transport,transport settings>> and 
-<<tcp-settings,advanced TCP settings>> in that namespace. To allow the remote
-client to discover all of the nodes in the remote cluster you can specify the
-`client.transport.sniff` setting:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.index.client.transport.sniff: true
-----------------------------

docs/reference/settings/security-settings.asciidoc

@@ -1303,16 +1303,6 @@ transport profile, use the prefix `transport.profiles.$PROFILE.xpack.security.`
 append the portion of the setting after `xpack.security.transport.`. For the key
 setting, this would be `transport.profiles.$PROFILE.xpack.security.ssl.key`.
 
-[[auditing-tls-ssl-settings]]
-:ssl-prefix:             xpack.security.audit.index.client.xpack.security.transport
-:component:              Auditing
-:client-auth-default!:
-:server!:
-
-include::ssl-settings.asciidoc[]
-
-See also <<remote-audit-settings>>. 
-
 [float]
 [[ip-filtering-settings]]
 ==== IP filtering settings

x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc

@@ -13,26 +13,13 @@ Search queries are contained inside HTTP request bodies, however, and some
 audit events that are generated by the REST layer can be toggled to output
 the request body to the audit log.
 
-To make certain audit events include the request body, edit the following 
-settings in the `elasticsearch.yml` file:
+To make certain audit events include the request body, edit the following
+setting in the `elasticsearch.yml` file:
 
-* For the `logfile` audit output:
-+  
---
 [source,yaml]
 ----------------------------
 xpack.security.audit.logfile.events.emit_request_body: true
 ----------------------------
---
-
-* For the `index` output:
-+
---
-[source,yaml]
-----------------------------
-xpack.security.audit.index.events.emit_request_body: true
-----------------------------
---
 
 IMPORTANT: No filtering is performed when auditing, so sensitive data might be
 audited in plain text when audit events include the request body. Also, the
@@ -44,10 +31,8 @@ generated in the REST layer and can access the request body. Most of them are no
 included by default.
 
 A good practical piece of advice is to add `authentication_success` to the event
-types that are audited. Add it to the list in the 
-`xpack.security.audit.logfile.events.include` or 
-`xpack.security.audit.index.events.include` settings. This type is not audited
-by default.
+types that are audited (add it to the list in the `xpack.security.audit.logfile.events.include`),
+as this event type is not audited by default.
 
 NOTE: Typically, the include list contains other event types as well, such as
 `access_granted` or `access_denied`.

x-pack/docs/en/security/auditing/event-types.asciidoc

@@ -210,8 +210,7 @@ that have been previously described:
 === Audit event attributes for the deprecated formats
 
 The following table shows the common attributes that can be associated with
-every event, when it is output to the `<clustername>_access.log` file or to the
-<<audit-index, index>>.
+every event, when it is output to the `<clustername>_access.log` file.
 
 .Common attributes
 [cols="2,7",options="header"]
@@ -229,10 +228,8 @@ every event, when it is output to the `<clustername>_access.log` file or to the
                         `run_as_denied`, `run_as_granted`.
 |======
 
-For an event in the <<audit-log-output,audit log file output>>, these are
-positional attributes, which are printed at the beginning of each log line and
-are not adjoined by the attribute name. As a matter of course, the names are
-present for each attribute when the event is forwarded to the <<audit-index, index audit output>>.
+These are positional attributes, which are printed at the beginning of each log line and
+are not adjoined by the attribute name.
 
 The attribute `origin_address` is also common to every audit event. It is always
 named, that is, it is not positional. It denotes the source IP address of the

x-pack/docs/en/security/auditing/forwarding-logs.asciidoc

@@ -1,99 +0,0 @@
-[role="xpack"]
-[[forwarding-audit-logfiles]]
-=== Forwarding audit logs to a remote cluster
-
-When you are auditing security events, you can optionally store the logs in an 
-{es} index on a remote cluster.  The logs are sent to the remote cluster by 
-using the {javaclient}/transport-client.html[transport client]. 
-
-. Configure auditing such that the logs are stored in {es} rolling indices. 
-See <<audit-index>>. 
-
-. Establish a connection to the remote cluster by configuring the following 
-`xpack.security.audit.index.client` settings: 
-+
---
-[source, yaml]
---------------------------------------------------
-xpack.security.audit.index.client.hosts: 192.168.0.1, 192.168.0.2 <1> 
-xpack.security.audit.index.client.cluster.name: logging-prod <2>
-xpack.security.audit.index.client.xpack.security.user: myuser:mypassword <3>
---------------------------------------------------
-<1> A list of hosts in the remote cluster. If you are not using the default 
-value for the `transport.port` setting on the remote cluster, you must
-specify the appropriate port number (prefixed by a colon) after each host. 
-<2> The remote cluster name.
-<3> A valid user and password, which must have authority to create the 
-`.security-audit` index on the remote cluster. 
-
-For more information about these settings, see
-{ref}/auditing-settings.html#remote-audit-settings[Remote audit log indexing configuration settings].
-
---
-
-. If the remote cluster has Transport Layer Security (TLS/SSL) enabled, you 
-must specify extra security settings: 
-
-.. {ref}/configuring-tls.html#node-certificates[Generate a node certificate on 
-the remote cluster], then copy that certificate to the client. 
-
-.. Enable TLS and specify the information required to access the node certificate.
-
-*** If the signed certificate is in PKCS#12 format, add the following information 
-to the `elasticsearch.yml` file:
-+
---
-[source,yaml]
------------------------------------------------------------
-xpack.security.audit.index.client.xpack.security.transport.ssl.enabled: true
-xpack.security.audit.index.client.xpack.security.transport.ssl.keystore.path: certs/remote-elastic-certificates.p12
-xpack.security.audit.index.client.xpack.security.transport.ssl.truststore.path: certs/remote-elastic-certificates.p12
------------------------------------------------------------
-
-For more information about these settings, see 
-{ref}/security-settings.html#auditing-tls-ssl-settings[Auditing TLS settings].
---
-
-*** If the certificate is in PEM format, add the following information to the
-`elasticsearch.yml` file:
-+
---
-[source, yaml]
---------------------------------------------------
-xpack.security.audit.index.client.xpack.security.transport.ssl.enabled: true
-xpack.security.audit.index.client.xpack.security.transport.ssl.key: /home/es/config/audit-client.key
-xpack.security.audit.index.client.xpack.security.transport.ssl.certificate: /home/es/config/audit-client.crt
-xpack.security.audit.index.client.xpack.security.transport.ssl.certificate_authorities: [ "/home/es/config/remote-ca.crt" ]
---------------------------------------------------
-    
-For more information about these settings, see 
-{ref}/security-settings.html#auditing-tls-ssl-settings[Auditing TLS settings].    
---
-
-.. If you secured the certificate with a password, add the password to
-your {es} keystore:
-
-*** If the signed certificate is in PKCS#12 format, use the following commands:
-+
---
-[source,shell]
------------------------------------------------------------
-bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.security.transport.ssl.keystore.secure_password
-
-bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.security.transport.ssl.truststore.secure_password
------------------------------------------------------------
---
-
-*** If the certificate is in PEM format, use the following commands:
-+
---
-[source,shell]
------------------------------------------------------------
-bin/elasticsearch-keystore add xpack.security.audit.index.client.xpack.security.transport.ssl.secure_key_passphrase
------------------------------------------------------------
---
-
-. Restart {es}.
-
-When these steps are complete, your audit logs are stored in {es} rolling 
-indices on the remote cluster. 

x-pack/docs/en/security/auditing/index.asciidoc

@@ -8,11 +8,5 @@ include::event-types.asciidoc[]
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/output-logfile.asciidoc
 include::output-logfile.asciidoc[]
 
-:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/output-index.asciidoc
-include::output-index.asciidoc[]
-
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc
 include::auditing-search-queries.asciidoc[]
-
-:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/forwarding-logs.asciidoc
-include::forwarding-logs.asciidoc[]

x-pack/docs/en/security/auditing/output-index.asciidoc

@@ -1,49 +0,0 @@
-[role="xpack"]
-[[audit-index]]
-=== Index audit output
-
-In addition to logging to a file, you can store audit logs in Elasticsearch
-rolling indices. These indices can be either on the same cluster, or on a
-remote cluster. You configure the following settings in
-`elasticsearch.yml` to control how audit entries are indexed. To enable
-this output, you need to configure the setting `xpack.security.audit.outputs`
-in the `elasticsearch.yml` file:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.outputs: [ index, logfile ]
-----------------------------
-
-For more configuration options, see
-{ref}/auditing-settings.html#index-audit-settings[Audit log indexing configuration settings].
-
-IMPORTANT: No filtering is performed when auditing, so sensitive data may be
-audited in plain text when including the request body in audit events.
-
-[float]
-==== Audit index settings
-
-You can also configure settings for the indices that the events are stored in.
-These settings are configured in the `xpack.security.audit.index.settings` namespace
-in `elasticsearch.yml`. For example, the following configuration sets the
-number of shards and replicas to 1 for the audit indices:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.index.settings:
-  index:
-    number_of_shards: 1
-    number_of_replicas: 1
-----------------------------
-
-These settings apply to the local audit indices, as well as to the
-<<forwarding-audit-logfiles, remote audit indices>>, but only if the remote cluster
-does *not* have {security-features} enabled or the {es} versions are different.
-If the remote cluster has {security-features} enabled and the versions coincide,
-the settings for the audit indices there will take precedence,
-even if they are unspecified (i.e. left to defaults).
-
-NOTE: Audit events are batched for indexing so there is a lag before
-events appear in the index. You can control how frequently batches of
-events are pushed to the index by setting
-`xpack.security.audit.index.flush_interval` in `elasticsearch.yml`.

x-pack/docs/en/security/auditing/overview.asciidoc

@@ -13,26 +13,5 @@ Audit logs are **disabled** by default. To enable this functionality, you
 must set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
 ============================================================================
 
-The {es} {security-features} provide two ways to persist audit logs:
-
-* The <<audit-log-output, `logfile`>> output, which persists events to
-  a dedicated `<clustername>_audit.log` file on the host's file system.
-  For backwards compatibility reasons, a file named `<clustername>_access.log`
-  is also generated.
-* The <<audit-index, `index`>> output, which persists events to an Elasticsearch
-  index. The audit index can reside on the same cluster, or a separate cluster.
-
-By default, only the `logfile` output is used when enabling auditing,
-implicitly outputting to both `<clustername>_audit.log` and `<clustername>_access.log`.
-To facilitate browsing and analyzing the events, you can also enable
-indexing by setting `xpack.security.audit.outputs` in `elasticsearch.yml`:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.outputs: [ index, logfile ]
-----------------------------
-
-TIP: If you choose to enable the `index` output type, we strongly recommend that 
-you still use the `logfile` output as the official record of events. If the 
-target index is unavailable (for example, during a rolling upgrade), the `index` 
-output can lose messages.
+The audit log persists events to a dedicated `<clustername>_audit.log` file on
+the host's file system (on each node).

x-pack/docs/en/security/configuring-es.asciidoc

@@ -131,9 +131,8 @@ and <<auditing-settings>>.
 
 .. Restart {es}.
 
-By default, events are logged to a dedicated `elasticsearch-access.log` file in
-`ES_HOME/logs`. You can also store the events in an {es} index for
-easier analysis and control what events are logged. 
+Events are logged to a dedicated `<clustername>_audit.log` file in
+`ES_HOME/logs`, on each cluster node.
 --
 
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/docs/reference/security/securing-communications/securing-elasticsearch.asciidoc

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java

@@ -23,15 +23,12 @@ import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
 import org.elasticsearch.cluster.node.DiscoveryNode;
 import org.elasticsearch.cluster.node.DiscoveryNodes;
 import org.elasticsearch.cluster.service.ClusterService;
-import org.elasticsearch.common.Booleans;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.inject.Module;
 import org.elasticsearch.common.inject.util.Providers;
 import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
-import org.elasticsearch.common.logging.LoggerMessageFormat;
 import org.elasticsearch.common.network.NetworkModule;
 import org.elasticsearch.common.network.NetworkService;
-import org.elasticsearch.common.regex.Regex;
 import org.elasticsearch.common.settings.ClusterSettings;
 import org.elasticsearch.common.settings.IndexScopedSettings;
 import org.elasticsearch.common.settings.Setting;
@@ -42,12 +39,7 @@ import org.elasticsearch.common.util.BigArrays;
 import org.elasticsearch.common.util.PageCacheRecycler;
 import org.elasticsearch.common.util.concurrent.ThreadContext;
 import org.elasticsearch.common.util.set.Sets;
-import org.elasticsearch.common.xcontent.LoggingDeprecationHandler;
 import org.elasticsearch.common.xcontent.NamedXContentRegistry;
-import org.elasticsearch.common.xcontent.XContent;
-import org.elasticsearch.common.xcontent.XContentFactory;
-import org.elasticsearch.common.xcontent.XContentParser;
-import org.elasticsearch.common.xcontent.XContentType;
 import org.elasticsearch.discovery.DiscoveryModule;
 import org.elasticsearch.env.Environment;
 import org.elasticsearch.env.NodeEnvironment;
@@ -126,7 +118,6 @@ import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions;
 import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissionsCache;
 import org.elasticsearch.xpack.core.security.authz.store.ReservedRolesStore;
 import org.elasticsearch.xpack.core.security.authz.store.RoleRetrievalResult;
-import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
 import org.elasticsearch.xpack.core.security.support.Automatons;
 import org.elasticsearch.xpack.core.security.user.AnonymousUser;
 import org.elasticsearch.xpack.core.ssl.SSLConfiguration;
@@ -136,7 +127,6 @@ import org.elasticsearch.xpack.core.ssl.TLSLicenseBootstrapCheck;
 import org.elasticsearch.xpack.core.ssl.action.GetCertificateInfoAction;
 import org.elasticsearch.xpack.core.ssl.action.TransportGetCertificateInfoAction;
 import org.elasticsearch.xpack.core.ssl.rest.RestGetCertificateInfoAction;
-import org.elasticsearch.xpack.core.template.TemplateUtils;
 import org.elasticsearch.xpack.security.action.filter.SecurityActionFilter;
 import org.elasticsearch.xpack.security.action.interceptor.BulkShardRequestInterceptor;
 import org.elasticsearch.xpack.security.action.interceptor.IndicesAliasesRequestInterceptor;
@@ -172,8 +162,6 @@ import org.elasticsearch.xpack.security.action.user.TransportPutUserAction;
 import org.elasticsearch.xpack.security.action.user.TransportSetEnabledAction;
 import org.elasticsearch.xpack.security.audit.AuditTrail;
 import org.elasticsearch.xpack.security.audit.AuditTrailService;
-import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail;
-import org.elasticsearch.xpack.security.audit.index.IndexNameResolver;
 import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
 import org.elasticsearch.xpack.security.authc.AuthenticationService;
 import org.elasticsearch.xpack.security.authc.InternalRealms;
@@ -224,11 +212,7 @@ import org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServe
 import org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport;
 import org.elasticsearch.xpack.security.transport.nio.SecurityNioHttpServerTransport;
 import org.elasticsearch.xpack.security.transport.nio.SecurityNioTransport;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
 
-import java.io.IOException;
-import java.nio.charset.StandardCharsets;
 import java.nio.file.Path;
 import java.time.Clock;
 import java.util.ArrayList;
@@ -237,7 +221,6 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
-import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
@@ -262,14 +245,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
 
     private static final Logger logger = LogManager.getLogger(Security.class);
 
-    static final Setting<List<String>> AUDIT_OUTPUTS_SETTING =
-        Setting.listSetting(SecurityField.setting("audit.outputs"),
-                Function.identity(),
-                s -> s.keySet().contains(SecurityField.setting("audit.outputs"))
-                        ? Collections.emptyList()
-                        : Collections.singletonList(LoggingAuditTrail.NAME),
-                Property.NodeScope);
-
     private final Settings settings;
     private final Environment env;
     private final boolean enabled;
@@ -286,7 +261,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
     private final SetOnce<TokenService> tokenService = new SetOnce<>();
     private final SetOnce<SecurityActionFilter> securityActionFilter = new SetOnce<>();
     private final SetOnce<SecurityIndexManager> securityIndex = new SetOnce<>();
-    private final SetOnce<IndexAuditTrail> indexAuditTrail = new SetOnce<>();
     private final SetOnce<NioGroupFactory> groupFactory = new SetOnce<>();
     private final List<BootstrapCheck> bootstrapChecks;
     private final List<SecurityExtension> securityExtensions = new ArrayList<>();
@@ -324,7 +298,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
     }
 
     private static void runStartupChecks(Settings settings) {
-        validateAutoCreateIndex(settings);
         validateRealmSettings(settings);
     }
 
@@ -402,31 +375,11 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
         securityContext.set(new SecurityContext(settings, threadPool.getThreadContext()));
         components.add(securityContext.get());
 
-        // audit trails construction
-        Set<AuditTrail> auditTrails = new LinkedHashSet<>();
-        if (XPackSettings.AUDIT_ENABLED.get(settings)) {
-            List<String> outputs = AUDIT_OUTPUTS_SETTING.get(settings);
-            if (outputs.isEmpty()) {
-                throw new IllegalArgumentException("Audit logging is enabled but there are zero output types in "
-                    + XPackSettings.AUDIT_ENABLED.getKey());
-            }
-
-            for (String output : outputs) {
-                switch (output) {
-                    case LoggingAuditTrail.NAME:
-                        auditTrails.add(new LoggingAuditTrail(settings, clusterService, threadPool));
-                        break;
-                    case IndexAuditTrail.NAME:
-                        indexAuditTrail.set(new IndexAuditTrail(settings, client, threadPool, clusterService));
-                        auditTrails.add(indexAuditTrail.get());
-                        break;
-                    default:
-                        throw new IllegalArgumentException("Unknown audit trail output [" + output + "]");
-                }
-            }
-        }
-        final AuditTrailService auditTrailService =
-                new AuditTrailService(new ArrayList<>(auditTrails), getLicenseState());
+        // audit trail service construction
+        final List<AuditTrail> auditTrails = XPackSettings.AUDIT_ENABLED.get(settings)
+                ? Collections.singletonList(new LoggingAuditTrail(settings, clusterService, threadPool))
+                : Collections.emptyList();
+        final AuditTrailService auditTrailService = new AuditTrailService(auditTrails, getLicenseState());
         components.add(auditTrailService);
         this.auditTrailService.set(auditTrailService);
 
@@ -613,9 +566,7 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
         IPFilter.addSettings(settingsList);
 
         // audit settings
-        settingsList.add(AUDIT_OUTPUTS_SETTING);
         LoggingAuditTrail.registerSettings(settingsList);
-        IndexAuditTrail.registerSettings(settingsList);
 
         // authentication and authorization settings
         AnonymousUser.addSettings(settingsList);
@@ -828,92 +779,6 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
         }
     }
 
-    static boolean indexAuditLoggingEnabled(Settings settings) {
-        if (XPackSettings.AUDIT_ENABLED.get(settings)) {
-            List<String> outputs = AUDIT_OUTPUTS_SETTING.get(settings);
-            for (String output : outputs) {
-                if (output.equals(IndexAuditTrail.NAME)) {
-                    return true;
-                }
-            }
-        }
-        return false;
-    }
-
-    static void validateAutoCreateIndex(Settings settings) {
-        String value = settings.get("action.auto_create_index");
-        if (value == null) {
-            return;
-        }
-
-        final boolean indexAuditingEnabled = Security.indexAuditLoggingEnabled(settings);
-        if (indexAuditingEnabled) {
-            String auditIndex = IndexAuditTrailField.INDEX_NAME_PREFIX + "*";
-            String errorMessage = LoggerMessageFormat.format(
-                    "the [action.auto_create_index] setting value [{}] is too" +
-                            " restrictive. disable [action.auto_create_index] or set it to include " +
-                            "[{}]", (Object) value, auditIndex);
-            if (Booleans.isFalse(value)) {
-                throw new IllegalArgumentException(errorMessage);
-            }
-
-            if (Booleans.isTrue(value)) {
-                return;
-            }
-
-            String[] matches = Strings.commaDelimitedListToStringArray(value);
-            List<String> indices = new ArrayList<>();
-            DateTime now = new DateTime(DateTimeZone.UTC);
-            // just use daily rollover
-
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now, IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusDays(1),
-                    IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(1),
-                    IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(2),
-                    IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(3),
-                    IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(4),
-                    IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(5),
-                    IndexNameResolver.Rollover.DAILY));
-            indices.add(IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now.plusMonths(6),
-                    IndexNameResolver.Rollover.DAILY));
-
-            for (String index : indices) {
-                boolean matched = false;
-                for (String match : matches) {
-                    char c = match.charAt(0);
-                    if (c == '-') {
-                        if (Regex.simpleMatch(match.substring(1), index)) {
-                            throw new IllegalArgumentException(errorMessage);
-                        }
-                    } else if (c == '+') {
-                        if (Regex.simpleMatch(match.substring(1), index)) {
-                            matched = true;
-                            break;
-                        }
-                    } else {
-                        if (Regex.simpleMatch(match, index)) {
-                            matched = true;
-                            break;
-                        }
-                    }
-                }
-                if (!matched) {
-                    throw new IllegalArgumentException(errorMessage);
-                }
-            }
-
-            logger.warn("the [action.auto_create_index] setting is configured to be restrictive [{}]. " +
-                    " for the next 6 months audit indices are allowed to be created, but please make sure" +
-                    " that any future history indices after 6 months with the pattern " +
-                    "[.security_audit_log*] are allowed to be created", value);
-        }
-    }
-
     @Override
     public List<TransportInterceptor> getTransportInterceptors(NamedWriteableRegistry namedWriteableRegistry, ThreadContext threadContext) {
         if (transportClientMode || enabled == false) { // don't register anything if we are not enabled
@@ -998,23 +863,9 @@ public class Security extends Plugin implements ActionPlugin, IngestPlugin, Netw
     @Override
     public UnaryOperator<Map<String, IndexTemplateMetaData>> getIndexTemplateMetaDataUpgrader() {
         return templates -> {
+            // .security index is not managed by using templates anymore
             templates.remove(SECURITY_TEMPLATE_NAME);
-            final XContent xContent = XContentFactory.xContent(XContentType.JSON);
-            final byte[] auditTemplate = TemplateUtils.loadTemplate("/" + IndexAuditTrail.INDEX_TEMPLATE_NAME + ".json",
-                    Version.CURRENT.toString(), SecurityIndexManager.TEMPLATE_VERSION_PATTERN).getBytes(StandardCharsets.UTF_8);
-
-            try (XContentParser parser = xContent
-                    .createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, auditTemplate)) {
-                IndexTemplateMetaData auditMetadata = new IndexTemplateMetaData.Builder(
-                        IndexTemplateMetaData.Builder.fromXContent(parser, IndexAuditTrail.INDEX_TEMPLATE_NAME))
-                        .settings(IndexAuditTrail.customAuditIndexSettings(settings, logger))
-                        .build();
-                templates.put(IndexAuditTrail.INDEX_TEMPLATE_NAME, auditMetadata);
-            } catch (IOException e) {
-                // TODO: should we handle this with a thrown exception?
-                logger.error("Error loading template [{}] as part of metadata upgrading", IndexAuditTrail.INDEX_TEMPLATE_NAME);
-            }
-
+            templates.remove("security_audit_log");
             return templates;
         };
     }

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/SecurityFeatureSet.java

@@ -16,11 +16,13 @@ import org.elasticsearch.xpack.core.XPackField;
 import org.elasticsearch.xpack.core.XPackSettings;
 import org.elasticsearch.xpack.core.security.SecurityFeatureSetUsage;
 import org.elasticsearch.xpack.core.security.user.AnonymousUser;
+import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
 import org.elasticsearch.xpack.security.authc.Realms;
 import org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore;
 import org.elasticsearch.xpack.security.authz.store.CompositeRolesStore;
 import org.elasticsearch.xpack.security.transport.filter.IPFilter;
 
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
@@ -153,7 +155,10 @@ public class SecurityFeatureSet implements XPackFeatureSet {
     static Map<String, Object> auditUsage(Settings settings) {
         Map<String, Object> map = new HashMap<>(2);
         map.put("enabled", XPackSettings.AUDIT_ENABLED.get(settings));
-        map.put("outputs", Security.AUDIT_OUTPUTS_SETTING.get(settings));
+        if (XPackSettings.AUDIT_ENABLED.get(settings)) {
+            // the only available output type is "logfile", but the optputs=<list> is to keep compatibility with previous reporting format
+            map.put("outputs", Arrays.asList(LoggingAuditTrail.NAME));
+        }
         return map;
     }
 

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrail.java

@@ -1,1240 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security.audit.index;
-
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-import org.apache.logging.log4j.message.ParameterizedMessage;
-import org.elasticsearch.ElasticsearchException;
-import org.elasticsearch.Version;
-import org.elasticsearch.action.ActionListener;
-import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
-import org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest;
-import org.elasticsearch.action.bulk.BulkProcessor;
-import org.elasticsearch.action.bulk.BulkRequest;
-import org.elasticsearch.action.bulk.BulkResponse;
-import org.elasticsearch.action.index.IndexRequest;
-import org.elasticsearch.client.Client;
-import org.elasticsearch.client.transport.TransportClient;
-import org.elasticsearch.cluster.ClusterChangedEvent;
-import org.elasticsearch.cluster.ClusterState;
-import org.elasticsearch.cluster.ClusterStateListener;
-import org.elasticsearch.cluster.metadata.AliasOrIndex;
-import org.elasticsearch.cluster.metadata.IndexMetaData;
-import org.elasticsearch.cluster.metadata.MappingMetaData;
-import org.elasticsearch.cluster.node.DiscoveryNode;
-import org.elasticsearch.cluster.service.ClusterService;
-import org.elasticsearch.common.Nullable;
-import org.elasticsearch.common.Strings;
-import org.elasticsearch.common.collect.Tuple;
-import org.elasticsearch.common.component.LifecycleListener;
-import org.elasticsearch.common.network.NetworkAddress;
-import org.elasticsearch.common.settings.Setting;
-import org.elasticsearch.common.settings.Setting.Property;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.transport.TransportAddress;
-import org.elasticsearch.common.unit.TimeValue;
-import org.elasticsearch.common.util.concurrent.AbstractRunnable;
-import org.elasticsearch.common.util.concurrent.EsExecutors;
-import org.elasticsearch.common.util.concurrent.ThreadContext;
-import org.elasticsearch.common.xcontent.XContentBuilder;
-import org.elasticsearch.common.xcontent.XContentFactory;
-import org.elasticsearch.common.xcontent.XContentType;
-import org.elasticsearch.gateway.GatewayService;
-import org.elasticsearch.node.Node;
-import org.elasticsearch.plugins.Plugin;
-import org.elasticsearch.rest.RestRequest;
-import org.elasticsearch.tasks.Task;
-import org.elasticsearch.threadpool.ThreadPool;
-import org.elasticsearch.transport.TransportMessage;
-import org.elasticsearch.xpack.core.XPackClientPlugin;
-import org.elasticsearch.xpack.core.security.authc.Authentication;
-import org.elasticsearch.xpack.core.security.authc.AuthenticationToken;
-import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
-import org.elasticsearch.xpack.core.security.user.SystemUser;
-import org.elasticsearch.xpack.core.security.user.User;
-import org.elasticsearch.xpack.core.security.user.XPackUser;
-import org.elasticsearch.xpack.core.template.TemplateUtils;
-import org.elasticsearch.xpack.security.audit.AuditLevel;
-import org.elasticsearch.xpack.security.audit.AuditTrail;
-import org.elasticsearch.xpack.security.rest.RemoteHostHeader;
-import org.elasticsearch.xpack.security.support.SecurityIndexManager;
-import org.elasticsearch.xpack.security.transport.filter.SecurityIpFilterRule;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
-
-import java.io.Closeable;
-import java.io.IOException;
-import java.net.InetAddress;
-import java.net.InetSocketAddress;
-import java.net.UnknownHostException;
-import java.nio.charset.StandardCharsets;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.EnumSet;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.function.Consumer;
-import java.util.function.Function;
-import java.util.stream.Collectors;
-
-import static org.elasticsearch.xpack.core.ClientHelper.SECURITY_ORIGIN;
-import static org.elasticsearch.xpack.core.ClientHelper.clientWithOrigin;
-import static org.elasticsearch.xpack.core.security.SecurityField.setting;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.ACCESS_DENIED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.ACCESS_GRANTED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.ANONYMOUS_ACCESS_DENIED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.AUTHENTICATION_FAILED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.AUTHENTICATION_SUCCESS;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.CONNECTION_DENIED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.CONNECTION_GRANTED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.REALM_AUTHENTICATION_FAILED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.RUN_AS_DENIED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.RUN_AS_GRANTED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.SYSTEM_ACCESS_GRANTED;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.TAMPERED_REQUEST;
-import static org.elasticsearch.xpack.security.audit.AuditLevel.parse;
-import static org.elasticsearch.xpack.security.audit.AuditUtil.indices;
-import static org.elasticsearch.xpack.security.audit.AuditUtil.restRequestContent;
-import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.resolve;
-import static org.elasticsearch.xpack.security.support.SecurityIndexManager.SECURITY_VERSION_STRING;
-
-/**
- * Audit trail implementation that writes events into an index.
- */
-public class IndexAuditTrail implements AuditTrail, ClusterStateListener {
-
-    public static final String NAME = "index";
-    public static final String DOC_TYPE = "doc";
-    public static final String INDEX_TEMPLATE_NAME = "security_audit_log";
-
-    private static final int DEFAULT_BULK_SIZE = 1000;
-    private static final int MAX_BULK_SIZE = 10000;
-    private static final int DEFAULT_MAX_QUEUE_SIZE = 10000;
-    private static final TimeValue DEFAULT_FLUSH_INTERVAL = TimeValue.timeValueSeconds(1);
-    private static final IndexNameResolver.Rollover DEFAULT_ROLLOVER = IndexNameResolver.Rollover.DAILY;
-    private static final Setting<IndexNameResolver.Rollover> ROLLOVER_SETTING =
-            new Setting<>(setting("audit.index.rollover"), (s) -> DEFAULT_ROLLOVER.name(),
-                    s -> IndexNameResolver.Rollover.valueOf(s.toUpperCase(Locale.ENGLISH)), Property.NodeScope);
-    private static final Setting<Integer> QUEUE_SIZE_SETTING =
-            Setting.intSetting(setting("audit.index.queue_max_size"), DEFAULT_MAX_QUEUE_SIZE, 1, Property.NodeScope);
-    private static final String DEFAULT_CLIENT_NAME = "security-audit-client";
-
-    private static final List<String> DEFAULT_EVENT_INCLUDES = Arrays.asList(
-            ACCESS_DENIED.toString(),
-            ACCESS_GRANTED.toString(),
-            ANONYMOUS_ACCESS_DENIED.toString(),
-            AUTHENTICATION_FAILED.toString(),
-            REALM_AUTHENTICATION_FAILED.toString(),
-            CONNECTION_DENIED.toString(),
-            CONNECTION_GRANTED.toString(),
-            TAMPERED_REQUEST.toString(),
-            RUN_AS_DENIED.toString(),
-            RUN_AS_GRANTED.toString(),
-            AUTHENTICATION_SUCCESS.toString()
-    );
-    private static final String FORBIDDEN_INDEX_SETTING = "index.mapper.dynamic";
-
-    private static final Setting<Settings> INDEX_SETTINGS =
-            Setting.groupSetting(setting("audit.index.settings.index."), Property.NodeScope);
-    private static final Setting<List<String>> INCLUDE_EVENT_SETTINGS =
-            Setting.listSetting(setting("audit.index.events.include"), DEFAULT_EVENT_INCLUDES, Function.identity(),
-                    Property.NodeScope);
-    private static final Setting<List<String>> EXCLUDE_EVENT_SETTINGS =
-            Setting.listSetting(setting("audit.index.events.exclude"), Collections.emptyList(),
-                    Function.identity(), Property.NodeScope);
-    private static final Setting<Boolean> INCLUDE_REQUEST_BODY =
-            Setting.boolSetting(setting("audit.index.events.emit_request_body"), false, Property.NodeScope);
-    private static final Setting<Settings> REMOTE_CLIENT_SETTINGS =
-            Setting.groupSetting(setting("audit.index.client."), Property.NodeScope);
-    private static final Setting<Integer> BULK_SIZE_SETTING =
-            Setting.intSetting(setting("audit.index.bulk_size"), DEFAULT_BULK_SIZE, 1, MAX_BULK_SIZE, Property.NodeScope);
-    private static final Setting<TimeValue> FLUSH_TIMEOUT_SETTING =
-            Setting.timeSetting(setting("audit.index.flush_interval"), DEFAULT_FLUSH_INTERVAL,
-                    TimeValue.timeValueMillis(1L), Property.NodeScope);
-    private static final Logger logger = LogManager.getLogger(IndexAuditTrail.class);
-
-    private final AtomicReference<State> state = new AtomicReference<>(State.INITIALIZED);
-    private final Settings settings;
-    private final String nodeName;
-    private final Client client;
-    private final QueueConsumer queueConsumer;
-    private final ThreadPool threadPool;
-    private final ClusterService clusterService;
-    private final boolean indexToRemoteCluster;
-    private final EnumSet<AuditLevel> events;
-    private final IndexNameResolver.Rollover rollover;
-    private final boolean includeRequestBody;
-
-    private BulkProcessor bulkProcessor;
-    private String nodeHostName;
-    private String nodeHostAddress;
-
-    @Override
-    public String name() {
-        return NAME;
-    }
-
-    public IndexAuditTrail(Settings settings, Client client, ThreadPool threadPool, ClusterService clusterService) {
-        this.settings = settings;
-        this.threadPool = threadPool;
-        this.clusterService = clusterService;
-        this.nodeName = Node.NODE_NAME_SETTING.get(settings);
-        final int maxQueueSize = QUEUE_SIZE_SETTING.get(settings);
-        this.queueConsumer = new QueueConsumer(EsExecutors.threadName(settings, "audit-queue-consumer"), createQueue(maxQueueSize));
-        this.rollover = ROLLOVER_SETTING.get(settings);
-        this.events = parse(INCLUDE_EVENT_SETTINGS.get(settings), EXCLUDE_EVENT_SETTINGS.get(settings));
-        this.indexToRemoteCluster = REMOTE_CLIENT_SETTINGS.get(settings).names().size() > 0;
-        this.includeRequestBody = INCLUDE_REQUEST_BODY.get(settings);
-
-        if (indexToRemoteCluster == false) {
-            // in the absence of client settings for remote indexing, fall back to the client that was passed in.
-            this.client = clientWithOrigin(client, SECURITY_ORIGIN);
-        } else {
-            this.client = initializeRemoteClient(settings, logger);
-        }
-        clusterService.addListener(this);
-        clusterService.addLifecycleListener(new LifecycleListener() {
-            @Override
-            public void beforeStop() {
-                stop();
-            }
-        });
-
-    }
-
-    public State state() {
-        return state.get();
-    }
-
-    @Override
-    public void clusterChanged(ClusterChangedEvent event) {
-        try {
-            if (state() == IndexAuditTrail.State.INITIALIZED && canStart(event)) {
-                threadPool.generic().execute(new AbstractRunnable() {
-
-                    @Override
-                    public void onFailure(Exception throwable) {
-                        logger.error("failed to start index audit trail services", throwable);
-                        assert false : "security lifecycle services startup failed";
-                    }
-
-                    @Override
-                    public void doRun() {
-                        start();
-                    }
-                });
-            }
-        } catch (Exception e) {
-            logger.error("failed to start index audit trail", e);
-        }
-    }
-
-    /**
-     * This method determines if this service can be started based on the state in the {@link ClusterChangedEvent} and
-     * if the node is the master or not. When using remote indexing, a call to the remote cluster will be made to retrieve
-     * the state and the same rules will be applied. In order for the service to start, the following must be true:
-     * <ol>
-     * <li>The cluster must not have a {@link GatewayService#STATE_NOT_RECOVERED_BLOCK}; in other words the gateway
-     * must have recovered from disk already.</li>
-     * <li>The current node must be the master OR the <code>security_audit_log</code> index template must exist</li>
-     * <li>The current audit index must not exist or have all primary shards active. The current audit index name
-     * is determined by the rollover settings and current time</li>
-     * </ol>
-     *
-     * @param event  the {@link ClusterChangedEvent} containing the up to date cluster state
-     * @return true if all requirements are met and the service can be started
-     */
-    public boolean canStart(ClusterChangedEvent event) {
-        if (indexToRemoteCluster) {
-            // just return true as we do not determine whether we can start or not based on the local cluster state, but must base it off
-            // of the remote cluster state and this method is called on the cluster state update thread, so we do not really want to
-            // execute remote calls on this thread
-            return true;
-        }
-        synchronized (this) {
-            return canStart(event.state());
-        }
-    }
-
-    private boolean canStart(ClusterState clusterState) {
-        if (clusterState.blocks().hasGlobalBlock(GatewayService.STATE_NOT_RECOVERED_BLOCK)) {
-            // wait until the gateway has recovered from disk, otherwise we think may not have audit indices
-            // but they may not have been restored from the cluster state on disk
-            logger.debug("index audit trail waiting until gateway has recovered from disk");
-            return false;
-        }
-
-        if (TemplateUtils.checkTemplateExistsAndVersionMatches(INDEX_TEMPLATE_NAME, SECURITY_VERSION_STRING,
-                clusterState, logger, Version.CURRENT::onOrBefore) == false) {
-            logger.debug("security audit index template [{}] is not up to date", INDEX_TEMPLATE_NAME);
-            return false;
-        }
-
-        String index = getIndexName();
-        IndexMetaData metaData = clusterState.metaData().index(index);
-        if (metaData == null) {
-            logger.debug("security audit index [{}] does not exist, so service can start", index);
-            return true;
-        }
-
-        if (clusterState.routingTable().index(index).allPrimaryShardsActive()) {
-            logger.debug("security audit index [{}] all primary shards started, so service can start", index);
-            return true;
-        }
-        logger.debug("security audit index [{}] does not have all primary shards started, so service cannot start", index);
-        return false;
-    }
-
-    private String getIndexName() {
-        final Message first = peek();
-        final String index;
-        if (first == null) {
-            index = resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, DateTime.now(DateTimeZone.UTC), rollover);
-        } else {
-            index = resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, first.timestamp, rollover);
-        }
-        return index;
-    }
-
-    private boolean hasStaleMessage() {
-        final Message first = peek();
-        if (first == null) {
-            return false;
-        }
-        return false == IndexNameResolver.resolve(first.timestamp, rollover)
-                .equals(IndexNameResolver.resolve(DateTime.now(DateTimeZone.UTC), rollover));
-    }
-
-    /**
-     * Starts the service. The state is moved to {@link org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.State#STARTING}
-     * at the beginning of the method. The service's components are initialized and if the current node is the master, the index
-     * template will be stored. The state is moved {@link org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.State#STARTED}
-     * and before returning the queue of messages that came before the service started is drained.
-     */
-    public void start() {
-        if (state.compareAndSet(State.INITIALIZED, State.STARTING)) {
-            this.nodeHostName = clusterService.localNode().getHostName();
-            this.nodeHostAddress = clusterService.localNode().getHostAddress();
-            if (indexToRemoteCluster) {
-                client.admin().cluster().prepareState().execute(new ActionListener<ClusterStateResponse>() {
-                    @Override
-                    public void onResponse(ClusterStateResponse clusterStateResponse) {
-                        logger.trace("remote cluster state is [{}] [{}]",
-                                clusterStateResponse.getClusterName(), clusterStateResponse.getState());
-                        if (canStart(clusterStateResponse.getState())) {
-                            updateCurrentIndexMappingsIfNecessary(clusterStateResponse.getState());
-                        } else if (TemplateUtils.checkTemplateExistsAndVersionMatches(INDEX_TEMPLATE_NAME,
-                                SECURITY_VERSION_STRING, clusterStateResponse.getState(), logger,
-                                Version.CURRENT::onOrAfter) == false) {
-                            putTemplate(customAuditIndexSettings(settings, logger),
-                                    e -> {
-                                        logger.error("failed to put audit trail template", e);
-                                        transitionStartingToInitialized();
-                                    });
-                        } else {
-                            // for some reason we can't start up since the remote cluster is not fully setup. in this case
-                            // we try to wait for yellow status (all primaries started up) this will also wait for
-                            // state recovery etc.
-                            String indexName = getIndexName();
-                            // if this index doesn't exists the call will fail with a not_found exception...
-                            client.admin().cluster().prepareHealth().setIndices(indexName).setWaitForYellowStatus().execute(
-                                    ActionListener.wrap(
-                                            (x) -> {
-                                                logger.debug("have yellow status on remote index [{}] ", indexName);
-                                                transitionStartingToInitialized();
-                                                start();
-                                            },
-                                            (e) -> {
-                                                logger.error("failed to get wait for yellow status on remote index [" + indexName + "]", e);
-                                                transitionStartingToInitialized();
-                                            }));
-                        }
-                    }
-
-                    @Override
-                    public void onFailure(Exception e) {
-                        transitionStartingToInitialized();
-                        logger.error("failed to get remote cluster state", e);
-                    }
-                });
-            } else {
-                updateCurrentIndexMappingsIfNecessary(clusterService.state());
-            }
-        }
-    }
-
-    // pkg private for tests
-    void updateCurrentIndexMappingsIfNecessary(ClusterState state) {
-        final String index = getIndexName();
-
-        AliasOrIndex aliasOrIndex = state.getMetaData().getAliasAndIndexLookup().get(index);
-        if (aliasOrIndex != null) {
-            // check mappings
-            final List<IndexMetaData> indices = aliasOrIndex.getIndices();
-            if (aliasOrIndex.isAlias() && indices.size() > 1) {
-                throw new IllegalStateException("Alias [" + index + "] points to more than one index: " +
-                        indices.stream().map(imd -> imd.getIndex().getName()).collect(Collectors.toList()));
-            }
-            IndexMetaData indexMetaData = indices.get(0);
-            MappingMetaData docMapping = indexMetaData.getMappings().get("doc");
-            if (docMapping == null) {
-                if (indexToRemoteCluster || state.nodes().isLocalNodeElectedMaster() || hasStaleMessage()) {
-                    putAuditIndexMappingsAndStart(index);
-                } else {
-                    logger.trace("audit index [{}] is missing mapping for type [{}]", index, DOC_TYPE);
-                    transitionStartingToInitialized();
-                }
-            } else {
-                @SuppressWarnings("unchecked")
-                Map<String, Object> meta = (Map<String, Object>) docMapping.sourceAsMap().get("_meta");
-                if (meta == null) {
-                    logger.info("Missing _meta field in mapping [{}] of index [{}]", docMapping.type(), index);
-                    throw new IllegalStateException("Cannot read security-version string in index " + index);
-                }
-
-                final String versionString = (String) meta.get(SECURITY_VERSION_STRING);
-                if (versionString != null && Version.fromString(versionString).onOrAfter(Version.CURRENT)) {
-                    innerStart();
-                } else {
-                    if (indexToRemoteCluster || state.nodes().isLocalNodeElectedMaster() || hasStaleMessage()) {
-                        putAuditIndexMappingsAndStart(index);
-                    } else if (versionString == null) {
-                        logger.trace("audit index [{}] mapping is missing meta field [{}]", index, SECURITY_VERSION_STRING);
-                        transitionStartingToInitialized();
-                    } else {
-                        logger.trace("audit index [{}] has the incorrect version [{}]", index, versionString);
-                        transitionStartingToInitialized();
-                    }
-                }
-            }
-        } else {
-            innerStart();
-        }
-    }
-
-    private void putAuditIndexMappingsAndStart(String index) {
-        putAuditIndexMappings(index, getPutIndexTemplateRequest(Settings.EMPTY).mappings().get(DOC_TYPE),
-                ActionListener.wrap(ignore -> {
-                    logger.trace("updated mappings on audit index [{}]", index);
-                    innerStart();
-                }, e -> {
-                    logger.error(new ParameterizedMessage("failed to update mappings on audit index [{}]", index), e);
-                    transitionStartingToInitialized(); // reset to initialized so we can retry
-                }));
-    }
-
-    private void transitionStartingToInitialized() {
-        if (state.compareAndSet(State.STARTING, State.INITIALIZED) == false) {
-            final String message = "state transition from starting to initialized failed, current value: " + state.get();
-            assert false : message;
-            logger.error(message);
-        }
-    }
-
-    void innerStart() {
-        initializeBulkProcessor();
-        queueConsumer.start();
-        if (state.compareAndSet(State.STARTING, State.STARTED) == false) {
-            final String message = "state transition from starting to started failed, current value: " + state.get();
-            assert false : message;
-            logger.error(message);
-        } else {
-            logger.trace("successful state transition from starting to started, current value: [{}]", state.get());
-        }
-    }
-
-    public synchronized void stop() {
-        if (state.compareAndSet(State.STARTED, State.STOPPING)) {
-            queueConsumer.close();
-        }
-
-        if (state() != State.STOPPED) {
-            try {
-                if (bulkProcessor != null) {
-                    if (bulkProcessor.awaitClose(10, TimeUnit.SECONDS) == false) {
-                        logger.warn("index audit trail failed to store all pending events after waiting for 10s");
-                    }
-                }
-            } catch (InterruptedException exc) {
-                Thread.currentThread().interrupt();
-            } finally {
-                if (indexToRemoteCluster) {
-                    client.close();
-                }
-                state.set(State.STOPPED);
-            }
-        }
-    }
-
-    @Override
-    public void authenticationSuccess(String requestId, String realm, User user, RestRequest request) {
-        if (events.contains(AUTHENTICATION_SUCCESS)) {
-            try {
-                enqueue(message("authentication_success", new Tuple<>(realm, realm), user, null, request), "authentication_success");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [authentication_success]", e);
-            }
-        }
-    }
-
-    @Override
-    public void authenticationSuccess(String requestId, String realm, User user, String action, TransportMessage message) {
-        if (events.contains(AUTHENTICATION_SUCCESS)) {
-            try {
-                enqueue(message("authentication_success", action, user, null, new Tuple<>(realm, realm), null, message),
-                        "authentication_success");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [authentication_success]", e);
-            }
-        }
-    }
-
-    @Override
-    public void anonymousAccessDenied(String requestId, String action, TransportMessage message) {
-        if (events.contains(ANONYMOUS_ACCESS_DENIED)) {
-            try {
-                enqueue(message("anonymous_access_denied", action, (User) null, null, null, indices(message), message),
-                        "anonymous_access_denied");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [anonymous_access_denied]", e);
-            }
-        }
-    }
-
-    @Override
-    public void anonymousAccessDenied(String requestId, RestRequest request) {
-        if (events.contains(ANONYMOUS_ACCESS_DENIED)) {
-            try {
-                enqueue(message("anonymous_access_denied", null, null, null, null, request), "anonymous_access_denied");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [anonymous_access_denied]", e);
-            }
-        }
-    }
-
-    @Override
-    public void authenticationFailed(String requestId, String action, TransportMessage message) {
-        if (events.contains(AUTHENTICATION_FAILED)) {
-            try {
-                enqueue(message("authentication_failed", action, (User) null, null, null, indices(message), message),
-                        "authentication_failed");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [authentication_failed]", e);
-            }
-        }
-    }
-
-    @Override
-    public void authenticationFailed(String requestId, RestRequest request) {
-        if (events.contains(AUTHENTICATION_FAILED)) {
-            try {
-                enqueue(message("authentication_failed", null, null, null, null, request), "authentication_failed");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [authentication_failed]", e);
-            }
-        }
-    }
-
-    @Override
-    public void authenticationFailed(String requestId, AuthenticationToken token, String action, TransportMessage message) {
-        if (events.contains(AUTHENTICATION_FAILED)) {
-            if (XPackUser.is(token.principal()) == false) {
-                try {
-                    enqueue(message("authentication_failed", action, token, null, indices(message), message), "authentication_failed");
-                } catch (Exception e) {
-                    logger.warn("failed to index audit event: [authentication_failed]", e);
-                }
-            }
-        }
-    }
-
-    @Override
-    public void authenticationFailed(String requestId, AuthenticationToken token, RestRequest request) {
-        if (events.contains(AUTHENTICATION_FAILED)) {
-            if (XPackUser.is(token.principal()) == false) {
-                try {
-                    enqueue(message("authentication_failed", null, token, null, null, request), "authentication_failed");
-                } catch (Exception e) {
-                    logger.warn("failed to index audit event: [authentication_failed]", e);
-                }
-            }
-        }
-    }
-
-    @Override
-    public void authenticationFailed(String requestId, String realm, AuthenticationToken token, String action, TransportMessage message) {
-        if (events.contains(REALM_AUTHENTICATION_FAILED)) {
-            if (XPackUser.is(token.principal()) == false) {
-                try {
-                    enqueue(message("realm_authentication_failed", action, token, realm, indices(message), message),
-                            "realm_authentication_failed");
-                } catch (Exception e) {
-                    logger.warn("failed to index audit event: [authentication_failed]", e);
-                }
-            }
-        }
-    }
-
-    @Override
-    public void authenticationFailed(String requestId, String realm, AuthenticationToken token, RestRequest request) {
-        if (events.contains(REALM_AUTHENTICATION_FAILED)) {
-            if (XPackUser.is(token.principal()) == false) {
-                try {
-                    enqueue(message("realm_authentication_failed", null, token, realm, null, request), "realm_authentication_failed");
-                } catch (Exception e) {
-                    logger.warn("failed to index audit event: [authentication_failed]", e);
-                }
-            }
-        }
-    }
-
-    @Override
-    public void accessGranted(String requestId, Authentication authentication, String action, TransportMessage msg, String[] roleNames) {
-        final User user = authentication.getUser();
-        final boolean isSystem = SystemUser.is(user) || XPackUser.is(user);
-        final boolean logSystemAccessGranted = isSystem && events.contains(SYSTEM_ACCESS_GRANTED);
-        final boolean shouldLog = logSystemAccessGranted || (isSystem == false && events.contains(ACCESS_GRANTED));
-        if (shouldLog) {
-            try {
-                assert authentication.getAuthenticatedBy() != null;
-                final String authRealmName = authentication.getAuthenticatedBy().getName();
-                final String lookRealmName = authentication.getLookedUpBy() == null ? null : authentication.getLookedUpBy().getName();
-                enqueue(message("access_granted", action, user, roleNames, new Tuple(authRealmName, lookRealmName), indices(msg),
-                        msg), "access_granted");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [access_granted]", e);
-            }
-        }
-    }
-
-    @Override
-    public void accessDenied(String requestId, Authentication authentication, String action, TransportMessage message, String[] roleNames) {
-        if (events.contains(ACCESS_DENIED) && (XPackUser.is(authentication.getUser()) == false)) {
-            try {
-                assert authentication.getAuthenticatedBy() != null;
-                final String authRealmName = authentication.getAuthenticatedBy().getName();
-                final String lookRealmName = authentication.getLookedUpBy() == null ? null : authentication.getLookedUpBy().getName();
-                enqueue(message("access_denied", action, authentication.getUser(), roleNames, new Tuple(authRealmName, lookRealmName),
-                        indices(message), message), "access_denied");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [access_denied]", e);
-            }
-        }
-    }
-
-    @Override
-    public void tamperedRequest(String requestId, RestRequest request) {
-        if (events.contains(TAMPERED_REQUEST)) {
-            try {
-                enqueue(message("tampered_request", null, null, null, null, request), "tampered_request");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [tampered_request]", e);
-            }
-        }
-    }
-
-    @Override
-    public void tamperedRequest(String requestId, String action, TransportMessage message) {
-        if (events.contains(TAMPERED_REQUEST)) {
-            try {
-                enqueue(message("tampered_request", action, (User) null, null, null, indices(message), message), "tampered_request");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [tampered_request]", e);
-            }
-        }
-    }
-
-    @Override
-    public void tamperedRequest(String requestId, User user, String action, TransportMessage request) {
-        if (events.contains(TAMPERED_REQUEST) && XPackUser.is(user) == false) {
-            try {
-                enqueue(message("tampered_request", action, user, null, null, indices(request), request), "tampered_request");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [tampered_request]", e);
-            }
-        }
-    }
-
-    @Override
-    public void connectionGranted(InetAddress inetAddress, String profile, SecurityIpFilterRule rule) {
-        if (events.contains(CONNECTION_GRANTED)) {
-            try {
-                enqueue(message("ip_filter", "connection_granted", inetAddress, profile, rule), "connection_granted");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [connection_granted]", e);
-            }
-        }
-    }
-
-    @Override
-    public void connectionDenied(InetAddress inetAddress, String profile, SecurityIpFilterRule rule) {
-        if (events.contains(CONNECTION_DENIED)) {
-            try {
-                enqueue(message("ip_filter", "connection_denied", inetAddress, profile, rule), "connection_denied");
-            } catch (Exception e) {
-                logger.warn("failed to index audit event: [connection_denied]", e);
-            }
-        }
-    }
-
-    @Override
-    public void runAsGranted(String requestId, Authentication authentication, String action, TransportMessage message, String[] roleNames) {
-        if (events.contains(RUN_AS_GRANTED)) {
-            try {
-                assert authentication.getAuthenticatedBy() != null;
-                final String authRealmName = authentication.getAuthenticatedBy().getName();
-                final String lookRealmName = authentication.getLookedUpBy() == null ? null : authentication.getLookedUpBy().getName();
-                enqueue(message("run_as_granted", action, authentication.getUser(), roleNames, new Tuple<>(authRealmName, lookRealmName),
-                        null, message), "run_as_granted");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [run_as_granted]", e);
-            }
-        }
-    }
-
-    @Override
-    public void runAsDenied(String requestId, Authentication authentication, String action, TransportMessage message, String[] roleNames) {
-        if (events.contains(RUN_AS_DENIED)) {
-            try {
-                assert authentication.getAuthenticatedBy() != null;
-                final String authRealmName = authentication.getAuthenticatedBy().getName();
-                final String lookRealmName = authentication.getLookedUpBy() == null ? null : authentication.getLookedUpBy().getName();
-                enqueue(message("run_as_denied", action, authentication.getUser(), roleNames, new Tuple<>(authRealmName, lookRealmName),
-                        null, message), "run_as_denied");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [run_as_denied]", e);
-            }
-        }
-    }
-
-    @Override
-    public void runAsDenied(String requestId, Authentication authentication, RestRequest request, String[] roleNames) {
-        if (events.contains(RUN_AS_DENIED)) {
-            try {
-                assert authentication.getAuthenticatedBy() != null;
-                final String authRealmName = authentication.getAuthenticatedBy().getName();
-                final String lookRealmName = authentication.getLookedUpBy() == null ? null : authentication.getLookedUpBy().getName();
-                enqueue(message("run_as_denied", new Tuple<>(authRealmName, lookRealmName), authentication.getUser(), roleNames, request),
-                        "run_as_denied");
-            } catch (final Exception e) {
-                logger.warn("failed to index audit event: [run_as_denied]", e);
-            }
-        }
-    }
-
-    private Message message(String type, @Nullable String action, @Nullable User user, @Nullable String[] roleNames,
-                            @Nullable Tuple<String, String> realms, @Nullable Set<String> indices, TransportMessage message)
-            throws Exception {
-
-        Message msg = new Message().start();
-        common("transport", type, msg.builder);
-        originAttributes(message, msg.builder, clusterService.localNode(), threadPool.getThreadContext());
-
-        if (action != null) {
-            msg.builder.field(Field.ACTION, action);
-        }
-        addUserAndRealmFields(msg.builder, type, user, realms);
-        if (roleNames != null) {
-            msg.builder.array(Field.ROLE_NAMES, roleNames);
-        }
-        if (indices != null) {
-            msg.builder.array(Field.INDICES, indices.toArray(Strings.EMPTY_ARRAY));
-        }
-        msg.builder.field(Field.REQUEST, message.getClass().getSimpleName());
-
-        return msg.end();
-    }
-
-    private void addUserAndRealmFields(XContentBuilder builder, String type, @Nullable User user, @Nullable Tuple<String, String> realms)
-            throws IOException {
-        if (user != null) {
-            if (user.isRunAs()) {
-                if ("run_as_granted".equals(type) || "run_as_denied".equals(type)) {
-                    builder.field(Field.PRINCIPAL, user.authenticatedUser().principal());
-                    builder.field(Field.RUN_AS_PRINCIPAL, user.principal());
-                    if (realms != null) {
-                        // realms.v1() is the authenticating realm
-                        builder.field(Field.REALM, realms.v1());
-                        // realms.v2() is the lookup realm
-                        builder.field(Field.RUN_AS_REALM, realms.v2());
-                    }
-                } else {
-                    // TODO: this doesn't make sense...
-                    builder.field(Field.PRINCIPAL, user.principal());
-                    builder.field(Field.RUN_BY_PRINCIPAL, user.authenticatedUser().principal());
-                    if (realms != null) {
-                        // realms.v2() is the lookup realm
-                        builder.field(Field.REALM, realms.v2());
-                        // realms.v1() is the authenticating realm
-                        builder.field(Field.RUN_BY_REALM, realms.v1());
-                    }
-                }
-            } else {
-                builder.field(Field.PRINCIPAL, user.principal());
-                if (realms != null) {
-                    // realms.v1() is the authenticating realm
-                    builder.field(Field.REALM, realms.v1());
-                }
-            }
-        }
-    }
-
-    // FIXME - clean up the message generation
-    private Message message(String type, @Nullable String action, @Nullable AuthenticationToken token,
-                            @Nullable String realm, @Nullable Set<String> indices, TransportMessage message) throws Exception {
-
-        Message msg = new Message().start();
-        common("transport", type, msg.builder);
-        originAttributes(message, msg.builder, clusterService.localNode(), threadPool.getThreadContext());
-
-        if (action != null) {
-            msg.builder.field(Field.ACTION, action);
-        }
-        if (token != null) {
-            msg.builder.field(Field.PRINCIPAL, token.principal());
-        }
-        if (realm != null) {
-            msg.builder.field(Field.REALM, realm);
-        }
-        if (indices != null) {
-            msg.builder.array(Field.INDICES, indices.toArray(Strings.EMPTY_ARRAY));
-        }
-        msg.builder.field(Field.REQUEST, message.getClass().getSimpleName());
-
-        return msg.end();
-    }
-
-    private Message message(String type, @Nullable String action, @Nullable AuthenticationToken token,
-            @Nullable String realm, @Nullable Set<String> indices, RestRequest request) throws Exception {
-
-        Message msg = new Message().start();
-        common("rest", type, msg.builder);
-
-        if (action != null) {
-            msg.builder.field(Field.ACTION, action);
-        }
-
-        if (token != null) {
-            msg.builder.field(Field.PRINCIPAL, token.principal());
-        }
-
-        if (realm != null) {
-            msg.builder.field(Field.REALM, realm);
-        }
-        if (indices != null) {
-            msg.builder.array(Field.INDICES, indices.toArray(Strings.EMPTY_ARRAY));
-        }
-        if (includeRequestBody) {
-            msg.builder.field(Field.REQUEST_BODY, restRequestContent(request));
-        }
-        msg.builder.field(Field.ORIGIN_TYPE, "rest");
-        InetSocketAddress address = request.getHttpChannel().getRemoteAddress();
-        if (address != null) {
-            msg.builder.field(Field.ORIGIN_ADDRESS, NetworkAddress.format(address.getAddress()));
-        } else {
-            msg.builder.field(Field.ORIGIN_ADDRESS, address);
-        }
-        msg.builder.field(Field.URI, request.uri());
-        return msg.end();
-    }
-
-    private Message message(String type, @Nullable Tuple<String,String> realms, @Nullable User user, @Nullable String[] roleNames,
-                            RestRequest request) throws Exception {
-
-        Message msg = new Message().start();
-        common("rest", type, msg.builder);
-
-        addUserAndRealmFields(msg.builder, type, user, realms);
-        if (roleNames != null) {
-            msg.builder.array(Field.ROLE_NAMES, roleNames);
-        }
-        if (includeRequestBody) {
-            msg.builder.field(Field.REQUEST_BODY, restRequestContent(request));
-        }
-        msg.builder.field(Field.ORIGIN_TYPE, "rest");
-        InetSocketAddress address = request.getHttpChannel().getRemoteAddress();
-        if (address != null) {
-            msg.builder.field(Field.ORIGIN_ADDRESS, NetworkAddress.format(address.getAddress()));
-        } else {
-            msg.builder.field(Field.ORIGIN_ADDRESS, address);
-        }
-        msg.builder.field(Field.URI, request.uri());
-
-        return msg.end();
-    }
-
-    private Message message(String layer, String type, InetAddress originAddress, String profile,
-                            SecurityIpFilterRule rule) throws IOException {
-
-        Message msg = new Message().start();
-        common(layer, type, msg.builder);
-
-        msg.builder.field(Field.ORIGIN_ADDRESS, NetworkAddress.format(originAddress));
-        msg.builder.field(Field.TRANSPORT_PROFILE, profile);
-        msg.builder.field(Field.RULE, rule);
-
-        return msg.end();
-    }
-
-    private XContentBuilder common(String layer, String type, XContentBuilder builder) throws IOException {
-        builder.field(Field.NODE_NAME, nodeName);
-        builder.field(Field.NODE_HOST_NAME, nodeHostName);
-        builder.field(Field.NODE_HOST_ADDRESS, nodeHostAddress);
-        builder.field(Field.LAYER, layer);
-        builder.field(Field.TYPE, type);
-
-        String opaqueId = threadPool.getThreadContext().getHeader(Task.X_OPAQUE_ID);
-        if (opaqueId != null) {
-            builder.field("opaque_id", opaqueId);
-        }
-
-        return builder;
-    }
-
-    private static XContentBuilder originAttributes(TransportMessage message, XContentBuilder builder,
-                                                    DiscoveryNode localNode, ThreadContext threadContext) throws IOException {
-
-        // first checking if the message originated in a rest call
-        InetSocketAddress restAddress = RemoteHostHeader.restRemoteAddress(threadContext);
-        if (restAddress != null) {
-            builder.field(Field.ORIGIN_TYPE, "rest");
-            builder.field(Field.ORIGIN_ADDRESS, NetworkAddress.format(restAddress.getAddress()));
-            return builder;
-        }
-
-        // we'll see if was originated in a remote node
-        TransportAddress address = message.remoteAddress();
-        if (address != null) {
-            builder.field(Field.ORIGIN_TYPE, "transport");
-            builder.field(Field.ORIGIN_ADDRESS,
-                    NetworkAddress.format(address.address().getAddress()));
-            return builder;
-        }
-
-        // the call was originated locally on this node
-        builder.field(Field.ORIGIN_TYPE, "local_node");
-        builder.field(Field.ORIGIN_ADDRESS, localNode.getHostAddress());
-        return builder;
-    }
-
-    void enqueue(Message message, String type) {
-        State currentState = state();
-        if (currentState != State.STOPPING && currentState != State.STOPPED) {
-            boolean accepted = queueConsumer.offer(message);
-            if (!accepted) {
-                logger.warn("failed to index audit event: [{}]. internal queue is full, which may be caused by a high indexing rate or " +
-                        "issue with the destination", type);
-            }
-        }
-    }
-
-    // for testing to ensure we get the proper timestamp and index name...
-    Message peek() {
-        return queueConsumer.peek();
-    }
-
-    Client initializeRemoteClient(Settings settings, Logger logger) {
-        Settings clientSettings = REMOTE_CLIENT_SETTINGS.get(settings);
-        List<String> hosts = clientSettings.getAsList("hosts");
-        if (hosts.isEmpty()) {
-            throw new ElasticsearchException("missing required setting " +
-                    "[" + REMOTE_CLIENT_SETTINGS.getKey() + ".hosts] for remote audit log indexing");
-        }
-
-        final int processors = EsExecutors.PROCESSORS_SETTING.get(settings);
-        if (EsExecutors.PROCESSORS_SETTING.exists(clientSettings)) {
-            final int clientProcessors = EsExecutors.PROCESSORS_SETTING.get(clientSettings);
-            if (clientProcessors != processors) {
-                final String message = String.format(
-                        Locale.ROOT,
-                        "explicit processor setting [%d] for audit trail remote client does not match inherited processor setting [%d]",
-                        clientProcessors,
-                        processors);
-                throw new IllegalStateException(message);
-            }
-        }
-
-        if (clientSettings.get("cluster.name", "").isEmpty()) {
-            throw new ElasticsearchException("missing required setting " +
-                    "[" + REMOTE_CLIENT_SETTINGS.getKey() + ".cluster.name] for remote audit log indexing");
-        }
-
-        List<Tuple<String, Integer>> hostPortPairs = new ArrayList<>();
-
-        for (String host : hosts) {
-            List<String> hostPort = Arrays.asList(host.trim().split(":"));
-            if (hostPort.size() != 1 && hostPort.size() != 2) {
-                logger.warn("invalid host:port specified: [{}] for setting [{}.hosts]", REMOTE_CLIENT_SETTINGS.getKey(), host);
-            }
-            hostPortPairs.add(new Tuple<>(hostPort.get(0), hostPort.size() == 2 ? Integer.valueOf(hostPort.get(1)) : 9300));
-        }
-
-        if (hostPortPairs.size() == 0) {
-            throw new ElasticsearchException("no valid host:port pairs specified for setting ["
-                    + REMOTE_CLIENT_SETTINGS.getKey() + ".hosts]");
-        }
-        final Settings theClientSetting =
-                Settings.builder()
-                        .put(clientSettings.filter((s) -> s.startsWith("hosts") == false)) // hosts is not a valid setting
-                        .put(EsExecutors.PROCESSORS_SETTING.getKey(), processors)
-                        .build();
-        final TransportClient transportClient = new TransportClient(Settings.builder()
-                .put("node.name", DEFAULT_CLIENT_NAME + "-" + Node.NODE_NAME_SETTING.get(settings))
-                .put(theClientSetting).build(), Settings.EMPTY, remoteTransportClientPlugins(), null) {};
-        for (Tuple<String, Integer> pair : hostPortPairs) {
-            try {
-                transportClient.addTransportAddress(new TransportAddress(InetAddress.getByName(pair.v1()), pair.v2()));
-            } catch (UnknownHostException e) {
-                throw new ElasticsearchException("could not find host {}", e, pair.v1());
-            }
-        }
-
-        logger.info("forwarding audit events to remote cluster [{}] using hosts [{}]",
-                clientSettings.get("cluster.name", ""), hostPortPairs.toString());
-        return transportClient;
-    }
-
-    public static Settings customAuditIndexSettings(Settings nodeSettings, Logger logger) {
-        final Settings newSettings = Settings.builder()
-                .put(INDEX_SETTINGS.get(nodeSettings), false)
-                .normalizePrefix(IndexMetaData.INDEX_SETTING_PREFIX)
-                .build();
-        if (newSettings.names().isEmpty()) {
-            return Settings.EMPTY;
-        }
-
-        // Filter out forbidden setting
-        return Settings.builder().put(newSettings.filter(name -> {
-            if (FORBIDDEN_INDEX_SETTING.equals(name)) {
-                logger.warn("overriding the default [{}} setting is forbidden. ignoring...", name);
-                return false;
-            }
-            return true;
-        })).build();
-    }
-
-    private void putTemplate(Settings customSettings, Consumer<Exception> consumer) {
-        try {
-            final PutIndexTemplateRequest request = getPutIndexTemplateRequest(customSettings);
-
-            client.admin().indices().putTemplate(request, ActionListener.wrap((response) -> {
-                        if (response.isAcknowledged()) {
-                            // now we may need to update the mappings of the current index
-                            client.admin().cluster().prepareState().execute(ActionListener.wrap(
-                                    stateResponse -> updateCurrentIndexMappingsIfNecessary(stateResponse.getState()),
-                                    consumer));
-                        } else {
-                            consumer.accept(new IllegalStateException("failed to put index template for audit logging"));
-                        }
-                    }, consumer));
-        } catch (Exception e) {
-            logger.debug("unexpected exception while putting index template", e);
-            consumer.accept(e);
-        }
-    }
-
-    private PutIndexTemplateRequest getPutIndexTemplateRequest(Settings customSettings) {
-        final byte[] template = TemplateUtils.loadTemplate("/" + INDEX_TEMPLATE_NAME + ".json",
-                Version.CURRENT.toString(), SecurityIndexManager.TEMPLATE_VERSION_PATTERN).getBytes(StandardCharsets.UTF_8);
-        final PutIndexTemplateRequest request = new PutIndexTemplateRequest(INDEX_TEMPLATE_NAME).source(template, XContentType.JSON);
-        if (customSettings != null && customSettings.names().size() > 0) {
-            Settings updatedSettings = Settings.builder()
-                    .put(request.settings())
-                    .put(customSettings)
-                    .build();
-            request.settings(updatedSettings);
-        }
-        return request;
-    }
-
-    private void putAuditIndexMappings(String index, String mappings, ActionListener<Void> listener) {
-        client.admin().indices().preparePutMapping(index)
-                .setType(DOC_TYPE)
-                .setSource(mappings, XContentType.JSON)
-                .execute(ActionListener.wrap((response) -> {
-                    if (response.isAcknowledged()) {
-                        listener.onResponse(null);
-                    } else {
-                        listener.onFailure(new IllegalStateException("failed to put mappings for audit logging index [" + index + "]"));
-                    }
-                },
-                listener::onFailure));
-    }
-
-    BlockingQueue<Message> createQueue(int maxQueueSize) {
-        return new LinkedBlockingQueue<>(maxQueueSize);
-    }
-
-    private void initializeBulkProcessor() {
-
-        final int bulkSize = BULK_SIZE_SETTING.get(settings);
-        final TimeValue interval = FLUSH_TIMEOUT_SETTING.get(settings);
-
-        bulkProcessor = BulkProcessor.builder(client, new BulkProcessor.Listener() {
-            @Override
-            public void beforeBulk(long executionId, BulkRequest request) {
-            }
-
-            @Override
-            public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
-                if (response.hasFailures()) {
-                    logger.info("failed to bulk index audit events: [{}]", response.buildFailureMessage());
-                }
-            }
-
-            @Override
-            public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
-                logger.error(new ParameterizedMessage("failed to bulk index audit events: [{}]", failure.getMessage()), failure);
-            }
-        }).setBulkActions(bulkSize)
-                .setFlushInterval(interval)
-                .setConcurrentRequests(1)
-                .build();
-    }
-
-    // method for testing to allow different plugins such as mock transport...
-    List<Class<? extends Plugin>> remoteTransportClientPlugins() {
-         return Arrays.asList(XPackClientPlugin.class);
-    }
-
-    public static void registerSettings(List<Setting<?>> settings) {
-        settings.add(INDEX_SETTINGS);
-        settings.add(EXCLUDE_EVENT_SETTINGS);
-        settings.add(INCLUDE_EVENT_SETTINGS);
-        settings.add(ROLLOVER_SETTING);
-        settings.add(BULK_SIZE_SETTING);
-        settings.add(FLUSH_TIMEOUT_SETTING);
-        settings.add(QUEUE_SIZE_SETTING);
-        settings.add(REMOTE_CLIENT_SETTINGS);
-        settings.add(INCLUDE_REQUEST_BODY);
-    }
-
-    private final class QueueConsumer extends Thread implements Closeable {
-        private final AtomicBoolean open = new AtomicBoolean(true);
-        private final BlockingQueue<Message> eventQueue;
-        private final Message shutdownSentinelMessage;
-
-        QueueConsumer(String name, BlockingQueue eventQueue) {
-            super(name);
-            this.eventQueue = eventQueue;
-            try {
-                shutdownSentinelMessage = new Message();
-            } catch (IOException e) {
-                throw new AssertionError(e);
-            }
-        }
-
-        @Override
-        public void close() {
-            if (open.compareAndSet(true, false)) {
-                try {
-                    eventQueue.put(shutdownSentinelMessage);
-                } catch (InterruptedException e) {
-                    Thread.currentThread().interrupt();
-                }
-            }
-        }
-
-        @Override
-        public void run() {
-            while (open.get()) {
-                try {
-                    final Message message = eventQueue.take();
-                    if (message == shutdownSentinelMessage || open.get() == false) {
-                        break;
-                    }
-                    final IndexRequest indexRequest = client.prepareIndex()
-                            .setIndex(resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, message.timestamp, rollover))
-                            .setType(DOC_TYPE).setSource(message.builder).request();
-                    bulkProcessor.add(indexRequest);
-                } catch (InterruptedException e) {
-                    logger.debug("index audit queue consumer interrupted", e);
-                    close();
-                    break;
-                } catch (Exception e) {
-                    // log the exception and keep going
-                    logger.warn("failed to index audit message from queue", e);
-                }
-            }
-            eventQueue.clear();
-        }
-
-        public boolean offer(Message message) {
-            if (open.get()) {
-                return eventQueue.offer(message);
-            }
-            return false;
-        }
-
-        public Message peek() {
-            return eventQueue.peek();
-        }
-    }
-
-    static class Message {
-
-        final DateTime timestamp;
-        final XContentBuilder builder;
-
-        Message() throws IOException {
-            this.timestamp = DateTime.now(DateTimeZone.UTC);
-            this.builder = XContentFactory.jsonBuilder();
-        }
-
-        Message start() throws IOException {
-            builder.startObject();
-            builder.timeField(Field.TIMESTAMP, timestamp);
-            return this;
-        }
-
-        Message end() throws IOException {
-            builder.endObject();
-            return this;
-        }
-    }
-
-    interface Field {
-        String TIMESTAMP = "@timestamp";
-        String NODE_NAME = "node_name";
-        String NODE_HOST_NAME = "node_host_name";
-        String NODE_HOST_ADDRESS = "node_host_address";
-        String LAYER = "layer";
-        String TYPE = "event_type";
-        String ORIGIN_ADDRESS = "origin_address";
-        String ORIGIN_TYPE = "origin_type";
-        String PRINCIPAL = "principal";
-        String ROLE_NAMES = "roles";
-        String RUN_AS_PRINCIPAL = "run_as_principal";
-        String RUN_AS_REALM = "run_as_realm";
-        String RUN_BY_PRINCIPAL = "run_by_principal";
-        String RUN_BY_REALM = "run_by_realm";
-        String ACTION = "action";
-        String INDICES = "indices";
-        String REQUEST = "request";
-        String REQUEST_BODY = "request_body";
-        String URI = "uri";
-        String REALM = "realm";
-        String TRANSPORT_PROFILE = "transport_profile";
-        String RULE = "rule";
-    }
-
-    public enum State {
-        INITIALIZED,
-        STARTING,
-        STARTED,
-        STOPPING,
-        STOPPED
-    }
-}

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityFeatureSetTests.java

@@ -21,6 +21,7 @@ import org.elasticsearch.xpack.core.XPackSettings;
 import org.elasticsearch.xpack.core.security.SecurityFeatureSetUsage;
 import org.elasticsearch.xpack.core.security.user.AnonymousUser;
 import org.elasticsearch.xpack.core.watcher.support.xcontent.XContentSource;
+import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
 import org.elasticsearch.xpack.security.authc.Realms;
 import org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore;
 import org.elasticsearch.xpack.security.authz.store.CompositeRolesStore;
@@ -97,12 +98,6 @@ public class SecurityFeatureSetTests extends ESTestCase {
         settings.put("xpack.security.transport.ssl.enabled", transportSSLEnabled);
         final boolean auditingEnabled = randomBoolean();
         settings.put(XPackSettings.AUDIT_ENABLED.getKey(), auditingEnabled);
-        final String[] auditOutputs = randomFrom(
-                new String[] { "logfile" },
-                new String[] { "index" },
-                new String[] { "logfile", "index" }
-        );
-        settings.putList(Security.AUDIT_OUTPUTS_SETTING.getKey(), auditOutputs);
         final boolean httpIpFilterEnabled = randomBoolean();
         final boolean transportIPFilterEnabled = randomBoolean();
         when(ipFilter.usageStats())
@@ -192,7 +187,11 @@ public class SecurityFeatureSetTests extends ESTestCase {
 
                 // auditing
                 assertThat(source.getValue("audit.enabled"), is(auditingEnabled));
-                assertThat(source.getValue("audit.outputs"), contains(auditOutputs));
+                if (auditingEnabled) {
+                    assertThat(source.getValue("audit.outputs"), contains(LoggingAuditTrail.NAME));
+                } else {
+                    assertThat(source.getValue("audit.outputs"), is(nullValue()));
+                }
 
                 // ip filter
                 assertThat(source.getValue("ipfilter.http.enabled"), is(httpIpFilterEnabled));

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecuritySettingsTests.java

@@ -1,51 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security;
-
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.test.ESTestCase;
-import org.elasticsearch.xpack.core.XPackSettings;
-import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
-
-
-import static org.hamcrest.Matchers.containsString;
-
-public class SecuritySettingsTests extends ESTestCase {
-
-    public void testValidAutoCreateIndex() {
-        Security.validateAutoCreateIndex(Settings.EMPTY);
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", true).build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", false).build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".security,.security-6").build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".security*").build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", "*s*").build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".s*").build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", "foo").build());
-        Security.validateAutoCreateIndex(Settings.builder().put("action.auto_create_index", ".security_audit_log*").build());
-
-        Security.validateAutoCreateIndex(Settings.builder()
-                        .put("action.auto_create_index", ".security,.security-6")
-                        .put(XPackSettings.AUDIT_ENABLED.getKey(), true)
-                        .build());
-
-        try {
-            Security.validateAutoCreateIndex(Settings.builder()
-                    .put("action.auto_create_index", ".security,.security-6")
-                    .put(XPackSettings.AUDIT_ENABLED.getKey(), true)
-                    .put(Security.AUDIT_OUTPUTS_SETTING.getKey(), randomFrom("index", "logfile,index"))
-                    .build());
-            fail("IllegalArgumentException expected");
-        } catch (IllegalArgumentException e) {
-            assertThat(e.getMessage(), containsString(IndexAuditTrailField.INDEX_NAME_PREFIX));
-        }
-
-        Security.validateAutoCreateIndex(Settings.builder()
-                .put("action.auto_create_index", ".security_audit_log*,.security,.security-6")
-                .put(XPackSettings.AUDIT_ENABLED.getKey(), true)
-                .put(Security.AUDIT_OUTPUTS_SETTING.getKey(), randomFrom("index", "logfile,index"))
-                .build());
-    }
-}

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/SecurityTests.java

@@ -43,7 +43,6 @@ import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions;
 import org.elasticsearch.xpack.core.security.authz.permission.FieldPermissionsDefinition;
 import org.elasticsearch.xpack.core.ssl.SSLService;
 import org.elasticsearch.xpack.security.audit.AuditTrailService;
-import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail;
 import org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail;
 import org.elasticsearch.xpack.security.authc.Realms;
 import org.hamcrest.Matchers;
@@ -180,37 +179,6 @@ public class SecurityTests extends ESTestCase {
         assertEquals(0, auditTrailService.getAuditTrails().size());
     }
 
-    public void testIndexAuditTrail() throws Exception {
-        Settings settings = Settings.builder()
-            .put(XPackSettings.AUDIT_ENABLED.getKey(), true)
-            .put(Security.AUDIT_OUTPUTS_SETTING.getKey(), "index").build();
-        Collection<Object> components = createComponents(settings);
-        AuditTrailService service = findComponent(AuditTrailService.class, components);
-        assertNotNull(service);
-        assertEquals(1, service.getAuditTrails().size());
-        assertEquals(IndexAuditTrail.NAME, service.getAuditTrails().get(0).name());
-    }
-
-    public void testIndexAndLoggingAuditTrail() throws Exception {
-        Settings settings = Settings.builder()
-            .put(XPackSettings.AUDIT_ENABLED.getKey(), true)
-            .put(Security.AUDIT_OUTPUTS_SETTING.getKey(), "index,logfile").build();
-        Collection<Object> components = createComponents(settings);
-        AuditTrailService service = findComponent(AuditTrailService.class, components);
-        assertNotNull(service);
-        assertEquals(2, service.getAuditTrails().size());
-        assertEquals(IndexAuditTrail.NAME, service.getAuditTrails().get(0).name());
-        assertEquals(LoggingAuditTrail.NAME, service.getAuditTrails().get(1).name());
-    }
-
-    public void testUnknownOutput() {
-        Settings settings = Settings.builder()
-            .put(XPackSettings.AUDIT_ENABLED.getKey(), true)
-            .put(Security.AUDIT_OUTPUTS_SETTING.getKey(), "foo").build();
-        IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> createComponents(settings));
-        assertEquals("Unknown audit trail output [foo]", e.getMessage());
-    }
-
     public void testHttpSettingDefaults() throws Exception {
         final Settings defaultSettings = Security.additionalSettings(Settings.EMPTY, true, false);
         assertThat(SecurityField.NAME4, equalTo(NetworkModule.TRANSPORT_TYPE_SETTING.get(defaultSettings)));

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/AuditTrailTests.java

@@ -1,195 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security.audit.index;
-
-import org.elasticsearch.action.ActionFuture;
-import org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsResponse;
-import org.elasticsearch.action.search.SearchRequest;
-import org.elasticsearch.action.support.PlainActionFuture;
-import org.elasticsearch.client.Client;
-import org.elasticsearch.client.Request;
-import org.elasticsearch.client.RequestOptions;
-import org.elasticsearch.client.Requests;
-import org.elasticsearch.client.ResponseException;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.unit.TimeValue;
-import org.elasticsearch.index.query.QueryBuilders;
-import org.elasticsearch.search.SearchHit;
-import org.elasticsearch.test.SecurityIntegTestCase;
-import org.elasticsearch.test.SecuritySettingsSource;
-import org.elasticsearch.xpack.core.security.ScrollHelper;
-import org.elasticsearch.xpack.core.security.authc.AuthenticationServiceField;
-import org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken;
-import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
-import org.elasticsearch.xpack.security.audit.AuditTrail;
-import org.elasticsearch.xpack.security.audit.AuditTrailService;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
-
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicReference;
-
-import static org.elasticsearch.test.SecuritySettingsSourceField.TEST_PASSWORD_SECURE_STRING;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.iterableWithSize;
-import static org.hamcrest.Matchers.nullValue;
-
-public class AuditTrailTests extends SecurityIntegTestCase {
-
-    private static final String AUTHENTICATE_USER = "http_user";
-    private static final String EXECUTE_USER = "exec_user";
-    private static final String ROLE_CAN_RUN_AS = "can_run_as";
-    private static final String ROLES = ROLE_CAN_RUN_AS + ":\n" + "  run_as: [ '" + EXECUTE_USER + "' ]\n";
-
-    @Override
-    protected boolean addMockHttpTransport() {
-        return false; // enable http
-    }
-
-    @Override
-    public Settings nodeSettings(int nodeOrdinal) {
-        return Settings.builder()
-                .put(super.nodeSettings(nodeOrdinal))
-                .put("xpack.security.audit.enabled", true)
-                .put("xpack.security.audit.outputs", "index")
-                .putList("xpack.security.audit.index.events.include", "access_denied", "authentication_failed", "run_as_denied")
-                .build();
-    }
-
-    @Override
-    public String configRoles() {
-        return ROLES + super.configRoles();
-    }
-
-    @Override
-    public String configUsers() {
-        return super.configUsers()
-                + AUTHENTICATE_USER + ":" + SecuritySettingsSource.TEST_PASSWORD_HASHED + "\n"
-                + EXECUTE_USER + ":xx_no_password_xx\n";
-    }
-
-    @Override
-    public String configUsersRoles() {
-        return super.configUsersRoles()
-                + ROLE_CAN_RUN_AS + ":" + AUTHENTICATE_USER + "\n"
-                + "monitoring_user:" + EXECUTE_USER;
-    }
-
-    @Override
-    public boolean transportSSLEnabled() {
-        return true;
-    }
-
-    public void testAuditAccessDeniedWithRunAsUser() throws Exception {
-        try {
-            Request request = new Request("GET", "/.security/_search");
-            RequestOptions.Builder options = request.getOptions().toBuilder();
-            options.addHeader("Authorization", UsernamePasswordToken.basicAuthHeaderValue(AUTHENTICATE_USER, TEST_PASSWORD_SECURE_STRING));
-            options.addHeader(AuthenticationServiceField.RUN_AS_USER_HEADER, EXECUTE_USER);
-            request.setOptions(options);
-            getRestClient().performRequest(request);
-            fail("request should have failed");
-        } catch (final ResponseException e) {
-            assertThat(e.getResponse().getStatusLine().getStatusCode(), is(403));
-        }
-
-        final Collection<Map<String, Object>> events = waitForAuditEvents();
-
-        assertThat(events, iterableWithSize(1));
-        final Map<String, Object> event = events.iterator().next();
-        assertThat(event.get(IndexAuditTrail.Field.TYPE), equalTo("access_denied"));
-        assertThat((List<?>) event.get(IndexAuditTrail.Field.INDICES), containsInAnyOrder(".security"));
-        assertThat(event.get(IndexAuditTrail.Field.PRINCIPAL), equalTo(EXECUTE_USER));
-        assertThat(event.get(IndexAuditTrail.Field.RUN_BY_PRINCIPAL), equalTo(AUTHENTICATE_USER));
-    }
-
-
-    public void testAuditRunAsDeniedEmptyUser() throws Exception {
-        try {
-            Request request = new Request("GET", "/.security/_search");
-            RequestOptions.Builder options = request.getOptions().toBuilder();
-            options.addHeader("Authorization", UsernamePasswordToken.basicAuthHeaderValue(AUTHENTICATE_USER, TEST_PASSWORD_SECURE_STRING));
-            options.addHeader(AuthenticationServiceField.RUN_AS_USER_HEADER, "");
-            request.setOptions(options);
-            getRestClient().performRequest(request);
-            fail("request should have failed");
-        } catch (final ResponseException e) {
-            assertThat(e.getResponse().getStatusLine().getStatusCode(), is(401));
-        }
-
-        final Collection<Map<String, Object>> events = waitForAuditEvents();
-
-        assertThat(events, iterableWithSize(1));
-        final Map<String, Object> event = events.iterator().next();
-        assertThat(event.get(IndexAuditTrail.Field.TYPE), equalTo("run_as_denied"));
-        assertThat(event.get(IndexAuditTrail.Field.PRINCIPAL), equalTo(AUTHENTICATE_USER));
-        assertThat(event.get(IndexAuditTrail.Field.RUN_AS_PRINCIPAL), equalTo(""));
-        assertThat(event.get(IndexAuditTrail.Field.REALM), equalTo("file"));
-        assertThat(event.get(IndexAuditTrail.Field.RUN_AS_REALM), nullValue());
-    }
-
-    private Collection<Map<String, Object>> waitForAuditEvents() throws InterruptedException {
-        waitForAuditTrailToBeWritten();
-        final AtomicReference<Collection<Map<String, Object>>> eventsRef = new AtomicReference<>();
-        awaitBusy(() -> {
-            try {
-                final Collection<Map<String, Object>> events = getAuditEvents();
-                eventsRef.set(events);
-                return events.size() > 0;
-            } catch (final Exception e) {
-                throw new RuntimeException(e);
-            }
-        });
-
-        return eventsRef.get();
-    }
-    private Collection<Map<String, Object>> getAuditEvents() throws Exception {
-        final Client client = client();
-        final DateTime now = new DateTime(DateTimeZone.UTC);
-        final String indexName = IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, now, IndexNameResolver.Rollover.DAILY);
-
-        assertTrue(awaitBusy(() -> indexExists(client, indexName), 5, TimeUnit.SECONDS));
-
-        client.admin().indices().refresh(Requests.refreshRequest(indexName)).get();
-
-        final SearchRequest request = client.prepareSearch(indexName)
-                .setScroll(TimeValue.timeValueMinutes(10L))
-                .setTypes(IndexAuditTrail.DOC_TYPE)
-                .setQuery(QueryBuilders.matchAllQuery())
-                .setSize(1000)
-                .setFetchSource(true)
-                .request();
-        request.indicesOptions().ignoreUnavailable();
-
-        final PlainActionFuture<Collection<Map<String, Object>>> listener = new PlainActionFuture<>();
-        ScrollHelper.fetchAllByEntity(client, request, listener, SearchHit::getSourceAsMap);
-
-        return listener.get();
-    }
-
-    private boolean indexExists(Client client, String indexName) {
-        try {
-            final ActionFuture<IndicesExistsResponse> future = client.admin().indices().exists(Requests.indicesExistsRequest(indexName));
-            return future.get().isExists();
-        } catch (InterruptedException | ExecutionException e) {
-            throw new RuntimeException("Failed to check if " + indexName + " exists", e);
-        }
-    }
-
-    private void waitForAuditTrailToBeWritten() throws InterruptedException {
-        final AuditTrailService auditTrailService = (AuditTrailService) internalCluster().getInstance(AuditTrail.class);
-        assertThat(auditTrailService.getAuditTrails(), iterableWithSize(1));
-
-        final IndexAuditTrail indexAuditTrail = (IndexAuditTrail) auditTrailService.getAuditTrails().get(0);
-        assertTrue(awaitBusy(() -> indexAuditTrail.peek() == null, 5, TimeUnit.SECONDS));
-    }
-}

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrailMutedTests.java

@@ -1,324 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security.audit.index;
-
-import org.elasticsearch.action.Action;
-import org.elasticsearch.action.ActionListener;
-import org.elasticsearch.action.ActionRequest;
-import org.elasticsearch.action.ActionResponse;
-import org.elasticsearch.client.Client;
-import org.elasticsearch.client.FilterClient;
-import org.elasticsearch.client.transport.TransportClient;
-import org.elasticsearch.cluster.ClusterState;
-import org.elasticsearch.cluster.node.DiscoveryNode;
-import org.elasticsearch.cluster.service.ClusterService;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.rest.RestRequest;
-import org.elasticsearch.test.ESTestCase;
-import org.elasticsearch.threadpool.TestThreadPool;
-import org.elasticsearch.threadpool.ThreadPool;
-import org.elasticsearch.transport.MockTransportClient;
-import org.elasticsearch.transport.TransportMessage;
-import org.elasticsearch.xpack.core.security.authc.Authentication;
-import org.elasticsearch.xpack.core.security.authc.Authentication.RealmRef;
-import org.elasticsearch.xpack.core.security.authc.AuthenticationToken;
-import org.elasticsearch.xpack.core.security.user.SystemUser;
-import org.elasticsearch.xpack.core.security.user.User;
-import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.State;
-import org.elasticsearch.xpack.security.transport.filter.SecurityIpFilterRule;
-import org.junit.After;
-import org.junit.Before;
-
-import java.net.InetAddress;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-import static org.hamcrest.Matchers.is;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.verifyZeroInteractions;
-import static org.mockito.Mockito.when;
-
-public class IndexAuditTrailMutedTests extends ESTestCase {
-
-    private Client client;
-    private TransportClient transportClient;
-    private ThreadPool threadPool;
-    private ClusterService clusterService;
-    private IndexAuditTrail auditTrail;
-
-    private AtomicBoolean messageEnqueued;
-    private AtomicBoolean clientCalled;
-
-    @Before
-    public void setup() {
-        DiscoveryNode localNode = mock(DiscoveryNode.class);
-        when(localNode.getHostAddress()).thenReturn(buildNewFakeTransportAddress().toString());
-        clusterService = mock(ClusterService.class);
-        when(clusterService.localNode()).thenReturn(localNode);
-
-        threadPool = new TestThreadPool("index audit trail tests");
-        transportClient = new MockTransportClient(Settings.EMPTY);
-        clientCalled = new AtomicBoolean(false);
-        class IClient extends FilterClient {
-            IClient(Client transportClient){
-                super(Settings.EMPTY, threadPool, transportClient);
-            }
-
-            @Override
-            protected <Request extends ActionRequest, Response extends ActionResponse>
-            void doExecute(Action<Response> action, Request request, ActionListener<Response> listener) {
-                clientCalled.set(true);
-            }
-        }
-        client = new IClient(transportClient);
-        messageEnqueued = new AtomicBoolean(false);
-    }
-
-    @After
-    public void stop() {
-        if (auditTrail != null) {
-            auditTrail.stop();
-        }
-        if (transportClient != null) {
-            transportClient.close();
-        }
-        threadPool.shutdown();
-    }
-
-    public void testAnonymousAccessDeniedMutedTransport() {
-        createAuditTrail(new String[] { "anonymous_access_denied" });
-        TransportMessage message = mock(TransportMessage.class);
-        auditTrail.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), "_action", message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message);
-    }
-
-    public void testAnonymousAccessDeniedMutedRest() {
-        createAuditTrail(new String[] { "anonymous_access_denied" });
-        RestRequest restRequest = mock(RestRequest.class);
-        auditTrail.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), restRequest);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(restRequest);
-    }
-
-    public void testAuthenticationFailedMutedTransport() {
-        createAuditTrail(new String[] { "authentication_failed" });
-        TransportMessage message = mock(TransportMessage.class);
-        AuthenticationToken token = mock(AuthenticationToken.class);
-
-        // without realm
-        auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), token, "_action", message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        // without the token
-        auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_action", message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(token, message);
-    }
-
-    public void testAuthenticationFailedMutedRest() {
-        createAuditTrail(new String[] { "authentication_failed" });
-        RestRequest restRequest = mock(RestRequest.class);
-        AuthenticationToken token = mock(AuthenticationToken.class);
-
-        // without the realm
-        auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), token, restRequest);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        // without the token
-        auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), restRequest);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(token, restRequest);
-    }
-
-    public void testAuthenticationFailedRealmMutedTransport() {
-        createAuditTrail(new String[] { "realm_authentication_failed" });
-        TransportMessage message = mock(TransportMessage.class);
-        AuthenticationToken token = mock(AuthenticationToken.class);
-
-        // with realm
-        auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), randomAlphaOfLengthBetween(2, 10), token, "_action", message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(token, message);
-    }
-
-    public void testAuthenticationFailedRealmMutedRest() {
-        createAuditTrail(new String[]{"realm_authentication_failed"});
-        RestRequest restRequest = mock(RestRequest.class);
-        AuthenticationToken token = mock(AuthenticationToken.class);
-
-        // with realm
-        auditTrail.authenticationFailed(randomAlphaOfLengthBetween(6, 12), randomAlphaOfLengthBetween(2, 10), token, restRequest);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-        verifyZeroInteractions(token, restRequest);
-    }
-
-    public void testAccessGrantedMuted() {
-        createAuditTrail(new String[] { "access_granted" });
-        final TransportMessage message = mock(TransportMessage.class);
-        final Authentication authentication = mock(Authentication.class);
-        auditTrail.accessGranted(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
-            new String[] { "role" });
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-        verifyZeroInteractions(message);
-    }
-
-    public void testSystemAccessGrantedMuted() {
-        createAuditTrail(randomFrom(new String[] { "access_granted" }, null));
-        final TransportMessage message = mock(TransportMessage.class);
-        final Authentication authentication = new Authentication(SystemUser.INSTANCE, new RealmRef(null, null, null), null);
-        auditTrail.accessGranted(randomAlphaOfLengthBetween(6, 12), authentication, "internal:foo", message, new String[] { "role" });
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message);
-    }
-
-    public void testAccessDeniedMuted() {
-        createAuditTrail(new String[] { "access_denied" });
-        final TransportMessage message = mock(TransportMessage.class);
-        final Authentication authentication = mock(Authentication.class);
-        auditTrail.accessDenied(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
-            new String[] { "role" });
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message, authentication);
-    }
-
-    public void testTamperedRequestMuted() {
-        createAuditTrail(new String[] { "tampered_request" });
-        TransportMessage message = mock(TransportMessage.class);
-        User user = mock(User.class);
-
-        // with user
-        auditTrail.tamperedRequest(randomAlphaOfLengthBetween(6, 12), user, randomAlphaOfLengthBetween(6, 40), message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        // without user
-        auditTrail.tamperedRequest(randomAlphaOfLengthBetween(6, 12), randomAlphaOfLengthBetween(6, 40), message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message, user);
-    }
-
-    public void testConnectionGrantedMuted() {
-        createAuditTrail(new String[] { "connection_granted" });
-        InetAddress address = mock(InetAddress.class);
-        SecurityIpFilterRule rule = mock(SecurityIpFilterRule.class);
-
-        auditTrail.connectionGranted(address, randomAlphaOfLengthBetween(1, 12), rule);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(address, rule);
-    }
-
-    public void testConnectionDeniedMuted() {
-        createAuditTrail(new String[] { "connection_denied" });
-        InetAddress address = mock(InetAddress.class);
-        SecurityIpFilterRule rule = mock(SecurityIpFilterRule.class);
-
-        auditTrail.connectionDenied(address, randomAlphaOfLengthBetween(1, 12), rule);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(address, rule);
-    }
-
-    public void testRunAsGrantedMuted() {
-        createAuditTrail(new String[] { "run_as_granted" });
-        TransportMessage message = mock(TransportMessage.class);
-        Authentication authentication = mock(Authentication.class);
-
-        auditTrail.runAsGranted(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
-            new String[] { "role" });
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message, authentication);
-    }
-
-    public void testRunAsDeniedMuted() {
-        createAuditTrail(new String[] { "run_as_denied" });
-        TransportMessage message = mock(TransportMessage.class);
-        Authentication authentication = mock(Authentication.class);
-
-        auditTrail.runAsDenied(randomAlphaOfLengthBetween(6, 12), authentication, randomAlphaOfLengthBetween(6, 40), message,
-            new String[] { "role" });
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message, authentication);
-    }
-
-    public void testAuthenticationSuccessRest() {
-        createAuditTrail(new String[] { "authentication_success" });
-        RestRequest restRequest = mock(RestRequest.class);
-        User user = mock(User.class);
-        String realm = "_realm";
-
-        auditTrail.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, restRequest);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(restRequest);
-    }
-
-    public void testAuthenticationSuccessTransport() {
-        createAuditTrail(new String[] { "authentication_success" });
-        TransportMessage message = mock(TransportMessage.class);
-        User user = mock(User.class);
-        String realm = "_realm";
-        auditTrail.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, randomAlphaOfLengthBetween(6, 40), message);
-        assertThat(messageEnqueued.get(), is(false));
-        assertThat(clientCalled.get(), is(false));
-
-        verifyZeroInteractions(message, user);
-    }
-
-    IndexAuditTrail createAuditTrail(String[] excludes) {
-        Settings settings = IndexAuditTrailTests.levelSettings(null, excludes);
-        auditTrail = new IndexAuditTrail(settings, client, threadPool, clusterService) {
-            @Override
-            void updateCurrentIndexMappingsIfNecessary(ClusterState state) {
-                // skip stuff so we don't have to stub out unnecessary client activities and cluster state
-                innerStart();
-            }
-
-            @Override
-            BlockingQueue<Message> createQueue(int maxQueueSize) {
-                return new LinkedBlockingQueue<Message>(maxQueueSize) {
-                    @Override
-                    public boolean offer(Message message) {
-                        messageEnqueued.set(true);
-                        return super.offer(message);
-                    }
-                };
-            }
-        };
-        auditTrail.start();
-        assertThat(auditTrail.state(), is(State.STARTED));
-        return auditTrail;
-    }
-}

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/IndexAuditTrailTests.java

@@ -1,996 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security.audit.index;
-
-import org.apache.lucene.util.SetOnce;
-import org.elasticsearch.action.IndicesRequest;
-import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
-import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
-import org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse;
-import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesRequest;
-import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
-import org.elasticsearch.action.search.SearchResponse;
-import org.elasticsearch.action.support.IndicesOptions;
-import org.elasticsearch.client.Client;
-import org.elasticsearch.client.Requests;
-import org.elasticsearch.client.transport.NoNodeAvailableException;
-import org.elasticsearch.cluster.ClusterState;
-import org.elasticsearch.cluster.health.ClusterHealthStatus;
-import org.elasticsearch.cluster.metadata.IndexMetaData;
-import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
-import org.elasticsearch.cluster.node.DiscoveryNode;
-import org.elasticsearch.cluster.node.DiscoveryNodes;
-import org.elasticsearch.cluster.service.ClusterService;
-import org.elasticsearch.common.Priority;
-import org.elasticsearch.common.network.NetworkAddress;
-import org.elasticsearch.common.network.NetworkModule;
-import org.elasticsearch.common.settings.KeyStoreWrapper;
-import org.elasticsearch.common.settings.MockSecureSettings;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.transport.TransportAddress;
-import org.elasticsearch.common.util.concurrent.EsExecutors;
-import org.elasticsearch.common.util.set.Sets;
-import org.elasticsearch.discovery.DiscoveryModule;
-import org.elasticsearch.discovery.zen.SettingsBasedHostsProvider;
-import org.elasticsearch.http.HttpChannel;
-import org.elasticsearch.plugins.MetaDataUpgrader;
-import org.elasticsearch.plugins.Plugin;
-import org.elasticsearch.rest.RestRequest;
-import org.elasticsearch.search.SearchHit;
-import org.elasticsearch.test.ESIntegTestCase;
-import org.elasticsearch.test.InternalTestCluster;
-import org.elasticsearch.test.SecurityIntegTestCase;
-import org.elasticsearch.test.SecuritySettingsSource;
-import org.elasticsearch.test.SecuritySettingsSourceField;
-import org.elasticsearch.test.discovery.TestZenDiscovery;
-import org.elasticsearch.threadpool.TestThreadPool;
-import org.elasticsearch.threadpool.ThreadPool;
-import org.elasticsearch.transport.TransportInfo;
-import org.elasticsearch.transport.TransportMessage;
-import org.elasticsearch.transport.TransportRequest;
-import org.elasticsearch.xpack.core.XPackSettings;
-import org.elasticsearch.xpack.core.security.SecurityField;
-import org.elasticsearch.xpack.core.security.authc.Authentication;
-import org.elasticsearch.xpack.core.security.authc.Authentication.RealmRef;
-import org.elasticsearch.xpack.core.security.authc.AuthenticationToken;
-import org.elasticsearch.xpack.core.security.index.IndexAuditTrailField;
-import org.elasticsearch.xpack.core.security.user.SystemUser;
-import org.elasticsearch.xpack.core.security.user.User;
-import org.elasticsearch.xpack.security.LocalStateSecurity;
-import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.Message;
-import org.elasticsearch.xpack.security.support.SecurityIndexManager;
-import org.elasticsearch.xpack.security.transport.filter.IPFilter;
-import org.elasticsearch.xpack.security.transport.filter.SecurityIpFilterRule;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
-import org.joda.time.format.ISODateTimeFormat;
-import org.junit.After;
-import org.junit.AfterClass;
-import org.junit.Before;
-import org.junit.BeforeClass;
-
-import java.net.InetAddress;
-import java.net.InetSocketAddress;
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
-import java.util.function.Function;
-
-import static java.util.Collections.emptyMap;
-import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE;
-import static org.elasticsearch.test.InternalTestCluster.clusterName;
-import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.DAILY;
-import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.HOURLY;
-import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.MONTHLY;
-import static org.elasticsearch.xpack.security.audit.index.IndexNameResolver.Rollover.WEEKLY;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.containsString;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.hasSize;
-import static org.hamcrest.Matchers.hasToString;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.notNullValue;
-import static org.hamcrest.Matchers.nullValue;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-
-@ESIntegTestCase.ClusterScope(scope = SUITE, supportsDedicatedMasters = false, numDataNodes = 1)
-public class IndexAuditTrailTests extends SecurityIntegTestCase {
-    public static final String SECOND_CLUSTER_NODE_PREFIX = "remote_" + SUITE_CLUSTER_NODE_PREFIX;
-
-    private static boolean remoteIndexing;
-    private static boolean useSSL;
-    private static InternalTestCluster remoteCluster;
-    private static Settings remoteSettings;
-    private static int numShards = -1;
-    private static int numReplicas = -1;
-
-    private TransportAddress remoteAddress = buildNewFakeTransportAddress();
-    private TransportAddress localAddress = new TransportAddress(InetAddress.getLoopbackAddress(), 0);
-    private IndexNameResolver.Rollover rollover;
-    private IndexAuditTrail auditor;
-    private SetOnce<Message> enqueuedMessage;
-    private ThreadPool threadPool;
-    private boolean includeRequestBody;
-
-    @BeforeClass
-    public static void configureBeforeClass() {
-        useSSL = randomBoolean();
-        remoteIndexing = randomBoolean();
-        if (remoteIndexing == false) {
-            remoteSettings = Settings.EMPTY;
-        }
-    }
-
-    @AfterClass
-    public static void cleanupAfterTest() {
-        if (remoteCluster != null) {
-            remoteCluster.close();
-            remoteCluster = null;
-
-        }
-        remoteSettings = null;
-    }
-
-    @Override
-    protected boolean transportSSLEnabled() {
-        return useSSL;
-    }
-
-    @Override
-    public Settings nodeSettings(int nodeOrdinal) {
-        if (numShards == -1) {
-            numShards = numberOfShards();
-        }
-        if (numReplicas == -1) {
-            numReplicas = numberOfReplicas();
-        }
-
-        return Settings.builder()
-                .put(super.nodeSettings(nodeOrdinal))
-                .put("xpack.security.audit.index.settings.index.number_of_shards", numShards)
-                .put("xpack.security.audit.index.settings.index.number_of_replicas", numReplicas)
-                .build();
-    }
-
-    @Before
-    public void initializeRemoteClusterIfNecessary() throws Exception {
-        if (remoteIndexing == false) {
-            logger.info("--> remote indexing disabled.");
-            return;
-        }
-
-        if (remoteCluster != null) {
-            return;
-        }
-
-        // create another cluster
-        String cluster2Name = clusterName(Scope.SUITE.name(), randomLong());
-
-        // Setup a second test cluster with randomization for number of nodes, security enabled, and SSL
-        final int numNodes = randomIntBetween(1, 2);
-        final boolean useSecurity = randomBoolean();
-        final boolean remoteUseSSL = useSecurity && useSSL;
-        logger.info("--> remote indexing enabled. security enabled: [{}], SSL enabled: [{}], nodes: [{}]", useSecurity, useSSL,
-                numNodes);
-        SecuritySettingsSource cluster2SettingsSource =
-                new SecuritySettingsSource(useSSL, createTempDir(), Scope.SUITE) {
-            @Override
-            public Settings nodeSettings(int nodeOrdinal) {
-                Settings.Builder builder = Settings.builder()
-                        .put(super.nodeSettings(nodeOrdinal))
-                        .put(DiscoveryModule.DISCOVERY_HOSTS_PROVIDER_SETTING.getKey(), "file")
-                        .putList(SettingsBasedHostsProvider.DISCOVERY_ZEN_PING_UNICAST_HOSTS_SETTING.getKey())
-                        .put(TestZenDiscovery.USE_ZEN2.getKey(), getUseZen2())
-                        .put("xpack.security.audit.index.settings.index.number_of_shards", numShards)
-                        .put("xpack.security.audit.index.settings.index.number_of_replicas", numReplicas)
-                        // Disable native ML autodetect_process as the c++ controller won't be available
-//                        .put(MachineLearningField.AUTODETECT_PROCESS.getKey(), false)
-                        .put(XPackSettings.SECURITY_ENABLED.getKey(), useSecurity);
-                String transport = builder.get(NetworkModule.TRANSPORT_TYPE_KEY);
-                if (useSecurity == false && (transport == null || SecurityField.NAME4.equals(transport)
-                    || SecurityField.NIO.equals(transport))) {
-                    builder.put(NetworkModule.TRANSPORT_TYPE_KEY, getTestTransportType());
-                }
-                return builder.build();
-            }
-
-            @Override
-            public Settings transportClientSettings() {
-                if (useSecurity) {
-                    return super.transportClientSettings();
-                } else {
-                    Settings.Builder builder = Settings.builder()
-                            .put(XPackSettings.SECURITY_ENABLED.getKey(), false)
-                            .put(super.transportClientSettings());
-                    if (builder.get(NetworkModule.TRANSPORT_TYPE_KEY) == null) {
-                        builder.put(NetworkModule.TRANSPORT_TYPE_KEY, getTestTransportType());
-                    }
-                    return builder.build();
-                }
-            }
-
-            @Override
-            protected void addDefaultSecurityTransportType(Settings.Builder builder, Settings settings) {
-                if (useSecurity) {
-                    super.addDefaultSecurityTransportType(builder, settings);
-                }
-            }
-        };
-
-
-        Set<Class<? extends Plugin>> mockPlugins = new HashSet<>(getMockPlugins());
-        if (useSecurity == false) {
-            mockPlugins.add(getTestTransportPlugin());
-        }
-        remoteCluster = new InternalTestCluster(randomLong(), createTempDir(), false, true, numNodes, numNodes, cluster2Name,
-                cluster2SettingsSource, 0, SECOND_CLUSTER_NODE_PREFIX, mockPlugins,
-                useSecurity ? getClientWrapper() : Function.identity());
-        remoteCluster.beforeTest(random(), 0.5);
-
-        NodesInfoResponse response = remoteCluster.client().admin().cluster().prepareNodesInfo().execute().actionGet();
-        TransportInfo info = response.getNodes().get(0).getTransport();
-        TransportAddress inet = info.address().publishAddress();
-
-        Settings.Builder builder = Settings.builder()
-                .put("xpack.security.audit.index.client." + XPackSettings.SECURITY_ENABLED.getKey(), useSecurity)
-                .put(remoteSettings(NetworkAddress.format(inet.address().getAddress()), inet.address().getPort(), cluster2Name))
-                .put("xpack.security.audit.index.client.xpack.security.user", SecuritySettingsSource.TEST_USER_NAME + ":" +
-                        SecuritySettingsSourceField.TEST_PASSWORD);
-
-        if (remoteUseSSL) {
-            cluster2SettingsSource.addClientSSLSettings(builder, "xpack.security.audit.index.client.xpack.security.transport.");
-            builder.put("xpack.security.audit.index.client.xpack.security.transport.ssl.enabled", true);
-        }
-        if (useSecurity == false && builder.get(NetworkModule.TRANSPORT_TYPE_KEY) == null) {
-            builder.put("xpack.security.audit.index.client." + NetworkModule.TRANSPORT_TYPE_KEY, getTestTransportType());
-        }
-        remoteSettings = builder.build();
-    }
-
-    @After
-    public void afterTest() {
-        if (threadPool != null) {
-            threadPool.shutdown();
-        }
-        if (auditor != null) {
-            auditor.stop();
-        }
-
-        if (remoteCluster != null) {
-            remoteCluster.wipe(excludeTemplates());
-        }
-    }
-
-    @Override
-    protected Set<String> excludeTemplates() {
-        return Sets.newHashSet(SecurityIndexManager.SECURITY_TEMPLATE_NAME, IndexAuditTrail.INDEX_TEMPLATE_NAME);
-    }
-
-    @Override
-    protected int maximumNumberOfShards() {
-        return 3;
-    }
-
-    private Settings commonSettings(IndexNameResolver.Rollover rollover) {
-        return Settings.builder()
-                .put("xpack.security.audit.enabled", true)
-                .put("xpack.security.audit.outputs", "index, logfile")
-                .put("xpack.security.audit.index.bulk_size", 1)
-                .put("xpack.security.audit.index.flush_interval", "1ms")
-                .put("xpack.security.audit.index.rollover", rollover.name().toLowerCase(Locale.ENGLISH))
-                .put("xpack.security.audit.index.settings.index.number_of_shards", numShards)
-                .put("xpack.security.audit.index.settings.index.number_of_replicas", numReplicas)
-                .build();
-    }
-
-    static Settings remoteSettings(String address, int port, String clusterName) {
-        return Settings.builder()
-                .put("xpack.security.audit.index.client.hosts", address + ":" + port)
-                .put("xpack.security.audit.index.client.cluster.name", clusterName)
-                .build();
-    }
-
-    static Settings levelSettings(String[] includes, String[] excludes) {
-        Settings.Builder builder = Settings.builder();
-        if (includes != null) {
-            builder.putList("xpack.security.audit.index.events.include", includes);
-        }
-        if (excludes != null) {
-            builder.putList("xpack.security.audit.index.events.exclude", excludes);
-        }
-        return builder.build();
-    }
-
-    private Settings settings(IndexNameResolver.Rollover rollover, String[] includes, String[] excludes) {
-        Settings.Builder builder = Settings.builder();
-        builder.put(levelSettings(includes, excludes));
-        builder.put(commonSettings(rollover));
-        builder.put("xpack.security.audit.index.events.emit_request_body", includeRequestBody);
-        return builder.build();
-    }
-
-    private Client getClient() {
-        return remoteIndexing ? remoteCluster.client() : client();
-    }
-
-    private void initialize() throws Exception {
-        initialize(null, null);
-    }
-
-    private void initialize(String[] includes, String[] excludes) throws Exception {
-        initialize(includes, excludes, Settings.EMPTY);
-    }
-
-    private void initialize(final String[] includes, final String[] excludes, final Settings additionalSettings) throws Exception {
-        rollover = randomFrom(HOURLY, DAILY, WEEKLY, MONTHLY);
-        includeRequestBody = randomBoolean();
-        Settings.Builder builder = Settings.builder();
-        if (remoteIndexing) {
-            builder.put(remoteSettings);
-        }
-        builder.put(settings(rollover, includes, excludes)).put(additionalSettings).build();
-        // IndexAuditTrail should ignore secure settings
-        // they are merged on the master node creating the audit index
-        if (randomBoolean()) {
-            MockSecureSettings ignored = new MockSecureSettings();
-            if (randomBoolean()) {
-                ignored.setString(KeyStoreWrapper.SEED_SETTING.getKey(), "non-empty-secure-settings");
-            }
-            builder.setSecureSettings(ignored);
-        }
-        Settings settings = builder.build();
-
-        logger.info("--> settings: [{}]", settings);
-        DiscoveryNode localNode = mock(DiscoveryNode.class);
-        when(localNode.getHostAddress()).thenReturn(remoteAddress.getAddress());
-        when(localNode.getHostName()).thenReturn(remoteAddress.getAddress());
-        ClusterService clusterService = mock(ClusterService.class);
-        ClusterState state = mock(ClusterState.class);
-        DiscoveryNodes nodes = mock(DiscoveryNodes.class);
-        when(clusterService.localNode()).thenReturn(localNode);
-        when(clusterService.state()).thenReturn(client().admin().cluster().prepareState().get().getState());
-        when(state.getNodes()).thenReturn(nodes);
-        when(nodes.isLocalNodeElectedMaster()).thenReturn(true);
-        threadPool = new TestThreadPool("index audit trail tests");
-        enqueuedMessage = new SetOnce<>();
-        auditor = new IndexAuditTrail(settings, client(), threadPool, clusterService) {
-
-            @Override
-            void enqueue(Message message, String type) {
-                enqueuedMessage.set(message);
-                super.enqueue(message, type);
-            }
-
-            @Override
-            List<Class<? extends Plugin>> remoteTransportClientPlugins() {
-                return Arrays.asList(LocalStateSecurity.class, getTestTransportPlugin());
-            }
-        };
-        auditor.start();
-    }
-
-    public void testIndexTemplateUpgrader() throws Exception {
-        final MetaDataUpgrader metaDataUpgrader = internalCluster().getInstance(MetaDataUpgrader.class);
-        final Map<String, IndexTemplateMetaData> updatedTemplates = metaDataUpgrader.indexTemplateMetaDataUpgraders.apply(emptyMap());
-        final IndexTemplateMetaData indexAuditTrailTemplate = updatedTemplates.get(IndexAuditTrail.INDEX_TEMPLATE_NAME);
-        assertThat(indexAuditTrailTemplate, notNullValue());
-        // test custom index settings override template
-        assertThat(IndexMetaData.INDEX_NUMBER_OF_REPLICAS_SETTING.get(indexAuditTrailTemplate.settings()), is(numReplicas));
-        assertThat(IndexMetaData.INDEX_NUMBER_OF_SHARDS_SETTING.get(indexAuditTrailTemplate.settings()), is(numShards));
-        // test upgrade template and installed template are equal
-        final GetIndexTemplatesRequest request = new GetIndexTemplatesRequest(IndexAuditTrail.INDEX_TEMPLATE_NAME);
-        final GetIndexTemplatesResponse response = client().admin().indices().getTemplates(request).get();
-        assertThat(response.getIndexTemplates(), hasSize(1));
-        assertThat(indexAuditTrailTemplate, is(response.getIndexTemplates().get(0)));
-    }
-
-    public void testProcessorsSetting() {
-        final boolean explicitProcessors = randomBoolean();
-        final int processors;
-        if (explicitProcessors) {
-            processors = randomIntBetween(1, 16);
-        } else {
-            processors = EsExecutors.PROCESSORS_SETTING.get(Settings.EMPTY);
-        }
-        final boolean explicitClientProcessors = randomBoolean();
-        final int clientProcessors;
-        if (explicitClientProcessors) {
-            clientProcessors = randomIntBetween(1, 16);
-        } else {
-            clientProcessors = EsExecutors.PROCESSORS_SETTING.get(Settings.EMPTY);
-        }
-
-        final Settings.Builder additionalSettingsBuilder =
-                Settings.builder()
-                        .put("xpack.security.audit.index.client.cluster.name", "remote")
-                        .put("xpack.security.audit.index.client.hosts", "localhost:9300");
-
-        if (explicitProcessors) {
-            additionalSettingsBuilder.put(EsExecutors.PROCESSORS_SETTING.getKey(), processors);
-        }
-        if (explicitClientProcessors) {
-            additionalSettingsBuilder.put("xpack.security.audit.index.client.processors", clientProcessors);
-        }
-
-        final ThrowingRunnable runnable = () -> initialize(null, null, additionalSettingsBuilder.build());
-        if (processors == clientProcessors || explicitClientProcessors == false) {
-            // okay, the client initialized which is all we care about but no nodes are available because we never set up the remote cluster
-            expectThrows(NoNodeAvailableException.class, runnable);
-        } else {
-            final IllegalStateException e = expectThrows(IllegalStateException.class, runnable);
-            assertThat(
-                    e,
-                    hasToString(containsString(
-                            "explicit processor setting [" + clientProcessors + "]" +
-                                    " for audit trail remote client does not match inherited processor setting [" + processors + "]")));
-        }
-    }
-
-    public void testAnonymousAccessDeniedTransport() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        auditor.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), "_action", message);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        assertAuditMessage(hit, "transport", "anonymous_access_denied");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        if (message instanceof RemoteHostMockMessage) {
-            assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
-        } else {
-            assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
-        }
-
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals("transport", sourceMap.get("origin_type"));
-        if (message instanceof IndicesRequest) {
-            List<Object> indices = (List<Object>) sourceMap.get("indices");
-            assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
-        }
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testAnonymousAccessDeniedRest() throws Exception {
-        initialize();
-        RestRequest request = mockRestRequest();
-        auditor.anonymousAccessDenied(randomAlphaOfLengthBetween(6, 12), request);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "rest", "anonymous_access_denied");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat(NetworkAddress.format(InetAddress.getLoopbackAddress()), equalTo(sourceMap.get("origin_address")));
-        assertThat("_uri", equalTo(sourceMap.get("uri")));
-        assertThat(sourceMap.get("origin_type"), is("rest"));
-        assertRequestBody(sourceMap);
-    }
-
-    public void testAuthenticationFailedTransport() throws Exception {
-        initialize();
-        TransportMessage message = randomBoolean() ? new RemoteHostMockMessage() : new LocalHostMockMessage();
-        auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), new MockToken(), "_action", message);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertAuditMessage(hit, "transport", "authentication_failed");
-
-        if (message instanceof RemoteHostMockMessage) {
-            assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
-        } else {
-            assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
-        }
-
-        assertEquals("_principal", sourceMap.get("principal"));
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals("transport", sourceMap.get("origin_type"));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testAuthenticationFailedTransportNoToken() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_action", message);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "transport", "authentication_failed");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        if (message instanceof RemoteHostMockMessage) {
-            assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
-        } else {
-            assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
-        }
-
-        assertThat(sourceMap.get("principal"), nullValue());
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals("transport", sourceMap.get("origin_type"));
-        if (message instanceof IndicesRequest) {
-            List<Object> indices = (List<Object>) sourceMap.get("indices");
-            assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
-        }
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testAuthenticationFailedRest() throws Exception {
-        initialize();
-        RestRequest request = mockRestRequest();
-        auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), new MockToken(), request);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "rest", "authentication_failed");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat(sourceMap.get("principal"), is((Object) "_principal"));
-        assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
-        assertThat("_uri", equalTo(sourceMap.get("uri")));
-        assertThat(sourceMap.get("origin_type"), is("rest"));
-        assertRequestBody(sourceMap);
-    }
-
-    public void testAuthenticationFailedRestNoToken() throws Exception {
-        initialize();
-        RestRequest request = mockRestRequest();
-        auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), request);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "rest", "authentication_failed");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat(sourceMap.get("principal"), nullValue());
-        assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
-        assertThat("_uri", equalTo(sourceMap.get("uri")));
-        assertThat(sourceMap.get("origin_type"), is("rest"));
-        assertRequestBody(sourceMap);
-    }
-
-    public void testAuthenticationFailedTransportRealm() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_realm", new MockToken(), "_action", message);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "transport", "realm_authentication_failed");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-
-        if (message instanceof RemoteHostMockMessage) {
-            assertEquals(remoteAddress.getAddress(), sourceMap.get("origin_address"));
-        } else {
-            assertEquals(localAddress.getAddress(), sourceMap.get("origin_address"));
-        }
-
-        assertEquals("transport", sourceMap.get("origin_type"));
-        assertEquals("_principal", sourceMap.get("principal"));
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals("_realm", sourceMap.get("realm"));
-        if (message instanceof IndicesRequest) {
-            List<Object> indices = (List<Object>) sourceMap.get("indices");
-            assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
-        }
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testAuthenticationFailedRestRealm() throws Exception {
-        initialize();
-        RestRequest request = mockRestRequest();
-        auditor.authenticationFailed(randomAlphaOfLengthBetween(6, 12), "_realm", new MockToken(), request);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "rest", "realm_authentication_failed");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
-        assertThat("_uri", equalTo(sourceMap.get("uri")));
-        assertEquals("_realm", sourceMap.get("realm"));
-        assertThat(sourceMap.get("origin_type"), is("rest"));
-        assertRequestBody(sourceMap);
-    }
-
-    public void testAccessGranted() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        final boolean runAs = randomBoolean();
-        User user;
-        if (runAs) {
-            user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        } else {
-            user = new User("_username", new String[]{"r1"});
-        }
-        String role = randomAlphaOfLengthBetween(1, 6);
-        auditor.accessGranted(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { role });
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        assertAuditMessage(hit, "transport", "access_granted");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("transport", sourceMap.get("origin_type"));
-        if (runAs) {
-            assertThat(sourceMap.get("principal"), is("running as"));
-            assertThat(sourceMap.get("realm"), is("lookRealm"));
-            assertThat(sourceMap.get("run_by_principal"), is("_username"));
-            assertThat(sourceMap.get("run_by_realm"), is("authRealm"));
-        } else {
-            assertThat(sourceMap.get("principal"), is("_username"));
-            assertThat(sourceMap.get("realm"), is("authRealm"));
-        }
-        assertEquals("_action", sourceMap.get("action"));
-        assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
-        if (message instanceof IndicesRequest) {
-            List<Object> indices = (List<Object>) sourceMap.get("indices");
-            assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
-        }
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testSystemAccessGranted() throws Exception {
-        initialize(new String[] { "system_access_granted" }, null);
-        TransportMessage message = randomBoolean() ? new RemoteHostMockMessage() : new LocalHostMockMessage();
-        String role = randomAlphaOfLengthBetween(1, 6);
-        auditor.accessGranted(randomAlphaOfLength(8), createAuthentication(SystemUser.INSTANCE), "internal:_action", message,
-            new String[] { role });
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        assertAuditMessage(hit, "transport", "access_granted");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("transport", sourceMap.get("origin_type"));
-        assertEquals(SystemUser.INSTANCE.principal(), sourceMap.get("principal"));
-        assertThat(sourceMap.get("realm"), is("authRealm"));
-        assertEquals("internal:_action", sourceMap.get("action"));
-        assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testAccessDenied() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        final boolean runAs = randomBoolean();
-        User user;
-        if (runAs) {
-            user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        } else {
-            user = new User("_username", new String[]{"r1"});
-        }
-        String role = randomAlphaOfLengthBetween(1, 6);
-        auditor.accessDenied(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { role });
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertAuditMessage(hit, "transport", "access_denied");
-        assertEquals("transport", sourceMap.get("origin_type"));
-        if (runAs) {
-            assertThat(sourceMap.get("principal"), is("running as"));
-            assertThat(sourceMap.get("realm"), is("lookRealm"));
-            assertThat(sourceMap.get("run_by_principal"), is("_username"));
-            assertThat(sourceMap.get("run_by_realm"), is("authRealm"));
-        } else {
-            assertThat(sourceMap.get("principal"), is("_username"));
-            assertThat(sourceMap.get("realm"), is("authRealm"));
-        }
-        assertEquals("_action", sourceMap.get("action"));
-        if (message instanceof IndicesRequest) {
-            List<Object> indices = (List<Object>) sourceMap.get("indices");
-            assertThat(indices, containsInAnyOrder((Object[]) ((IndicesRequest) message).indices()));
-        }
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-        assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
-    }
-
-    public void testTamperedRequestRest() throws Exception {
-        initialize();
-        RestRequest request = mockRestRequest();
-        auditor.tamperedRequest(randomAlphaOfLengthBetween(6, 12), request);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        assertAuditMessage(hit, "rest", "tampered_request");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat(sourceMap.get("principal"), nullValue());
-        assertThat("127.0.0.1", equalTo(sourceMap.get("origin_address")));
-        assertThat("_uri", equalTo(sourceMap.get("uri")));
-        assertThat(sourceMap.get("origin_type"), is("rest"));
-        assertRequestBody(sourceMap);
-    }
-
-    public void testTamperedRequest() throws Exception {
-        initialize();
-        TransportRequest message = new RemoteHostMockTransportRequest();
-        auditor.tamperedRequest(randomAlphaOfLengthBetween(6, 12), "_action", message);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertAuditMessage(hit, "transport", "tampered_request");
-        assertEquals("transport", sourceMap.get("origin_type"));
-        assertThat(sourceMap.get("principal"), is(nullValue()));
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testTamperedRequestWithUser() throws Exception {
-        initialize();
-        TransportRequest message = new RemoteHostMockTransportRequest();
-        final boolean runAs = randomBoolean();
-        User user;
-        if (runAs) {
-            user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        } else {
-            user = new User("_username", new String[]{"r1"});
-        }
-        auditor.tamperedRequest(randomAlphaOfLengthBetween(6, 12), user, "_action", message);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "transport", "tampered_request");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("transport", sourceMap.get("origin_type"));
-        if (runAs) {
-            assertThat(sourceMap.get("principal"), is("running as"));
-            assertThat(sourceMap.get("run_by_principal"), is("_username"));
-        } else {
-            assertEquals("_username", sourceMap.get("principal"));
-        }
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testConnectionGranted() throws Exception {
-        initialize();
-        InetAddress inetAddress = InetAddress.getLoopbackAddress();
-        SecurityIpFilterRule rule = IPFilter.DEFAULT_PROFILE_ACCEPT_ALL;
-        auditor.connectionGranted(inetAddress, "default", rule);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "ip_filter", "connection_granted");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("allow default:accept_all", sourceMap.get("rule"));
-        assertEquals("default", sourceMap.get("transport_profile"));
-    }
-
-    public void testConnectionDenied() throws Exception {
-        initialize();
-        InetAddress inetAddress = InetAddress.getLoopbackAddress();
-        SecurityIpFilterRule rule = new SecurityIpFilterRule(false, "_all");
-        auditor.connectionDenied(inetAddress, "default", rule);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "ip_filter", "connection_denied");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("deny _all", sourceMap.get("rule"));
-        assertEquals("default", sourceMap.get("transport_profile"));
-    }
-
-    public void testRunAsGranted() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        User user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        String role = randomAlphaOfLengthBetween(1, 6);
-        auditor.runAsGranted(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { role });
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        assertAuditMessage(hit, "transport", "run_as_granted");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("transport", sourceMap.get("origin_type"));
-        assertThat(sourceMap.get("principal"), is("_username"));
-        assertThat(sourceMap.get("realm"), is("authRealm"));
-        assertThat(sourceMap.get("run_as_principal"), is("running as"));
-        assertThat(sourceMap.get("run_as_realm"), is("lookRealm"));
-        assertThat((Iterable<String>) sourceMap.get(IndexAuditTrail.Field.ROLE_NAMES), containsInAnyOrder(role));
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testRunAsDenied() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        User user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        auditor.runAsDenied(randomAlphaOfLengthBetween(6, 12), createAuthentication(user), "_action", message, new String[] { "r1" });
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        assertAuditMessage(hit, "transport", "run_as_denied");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertEquals("transport", sourceMap.get("origin_type"));
-        assertThat(sourceMap.get("principal"), is("_username"));
-        assertThat(sourceMap.get("realm"), is("authRealm"));
-        assertThat(sourceMap.get("run_as_principal"), is("running as"));
-        assertThat(sourceMap.get("run_as_realm"), is("lookRealm"));
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    public void testAuthenticationSuccessRest() throws Exception {
-        initialize();
-        RestRequest request = mockRestRequest();
-        final boolean runAs = randomBoolean();
-        User user;
-        if (runAs) {
-            user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        } else {
-            user = new User("_username", new String[] { "r1" });
-        }
-        String realm = "_realm";
-        auditor.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, request);
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-
-        assertAuditMessage(hit, "rest", "authentication_success");
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat("_uri", equalTo(sourceMap.get("uri")));
-        assertRequestBody(sourceMap);
-        if (runAs) {
-            assertThat(sourceMap.get("principal"), is("running as"));
-            assertThat(sourceMap.get("run_by_principal"), is("_username"));
-        } else {
-            assertEquals("_username", sourceMap.get("principal"));
-        }
-        assertEquals("_realm", sourceMap.get("realm"));
-    }
-
-    public void testAuthenticationSuccessTransport() throws Exception {
-        initialize();
-        TransportMessage message = randomFrom(new RemoteHostMockMessage(), new LocalHostMockMessage(), new MockIndicesTransportMessage());
-        final boolean runAs = randomBoolean();
-        User user;
-        if (runAs) {
-            user = new User("running as", new String[]{"r2"}, new User("_username", new String[] {"r1"}));
-        } else {
-            user = new User("_username", new String[] { "r1" });
-        }
-        String realm = "_realm";
-        auditor.authenticationSuccess(randomAlphaOfLengthBetween(6, 12), realm, user, "_action", message);
-
-        SearchHit hit = getIndexedAuditMessage(enqueuedMessage.get());
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertAuditMessage(hit, "transport", "authentication_success");
-        assertEquals("transport", sourceMap.get("origin_type"));
-        if (runAs) {
-            assertThat(sourceMap.get("principal"), is("running as"));
-            assertThat(sourceMap.get("run_by_principal"), is("_username"));
-        } else {
-            assertEquals("_username", sourceMap.get("principal"));
-        }
-        assertEquals("_action", sourceMap.get("action"));
-        assertEquals("_realm", sourceMap.get("realm"));
-        assertEquals(sourceMap.get("request"), message.getClass().getSimpleName());
-    }
-
-    private void assertAuditMessage(SearchHit hit, String layer, String type) {
-        Map<String, Object> sourceMap = hit.getSourceAsMap();
-        assertThat(sourceMap.get("@timestamp"), notNullValue());
-        DateTime dateTime = ISODateTimeFormat.dateTimeParser().withZoneUTC().parseDateTime((String) sourceMap.get("@timestamp"));
-        final DateTime now = DateTime.now(DateTimeZone.UTC);
-        assertThat(dateTime + " should be on/before " + now, dateTime.isAfter(now), equalTo(false));
-
-        assertThat(remoteAddress.getAddress(), equalTo(sourceMap.get("node_host_name")));
-        assertThat(remoteAddress.getAddress(), equalTo(sourceMap.get("node_host_address")));
-
-        assertEquals(layer, sourceMap.get("layer"));
-        assertEquals(type, sourceMap.get("event_type"));
-    }
-
-    private void assertRequestBody(Map<String, Object> sourceMap) {
-        if (includeRequestBody) {
-            assertThat(sourceMap.get("request_body"), notNullValue());
-        } else {
-            assertThat(sourceMap.get("request_body"), nullValue());
-        }
-    }
-    private class LocalHostMockMessage extends TransportMessage {
-        LocalHostMockMessage() {
-            remoteAddress(localAddress);
-        }
-    }
-
-    private class RemoteHostMockMessage extends TransportMessage {
-        RemoteHostMockMessage() throws Exception {
-            remoteAddress(remoteAddress);
-        }
-    }
-
-    private class RemoteHostMockTransportRequest extends TransportRequest {
-        RemoteHostMockTransportRequest() throws Exception {
-            remoteAddress(remoteAddress);
-        }
-    }
-
-    private class MockIndicesTransportMessage extends RemoteHostMockMessage implements IndicesRequest {
-        MockIndicesTransportMessage() throws Exception {
-            super();
-        }
-
-        @Override
-        public String[] indices() {
-            return new String[] { "foo", "bar", "baz" };
-        }
-
-        @Override
-        public IndicesOptions indicesOptions() {
-            return null;
-        }
-    }
-
-    private static class MockToken implements AuthenticationToken {
-        @Override
-        public String principal() {
-            return "_principal";
-        }
-
-        @Override
-        public Object credentials() {
-            fail("it's not allowed to print the credentials of the auth token");
-            return null;
-        }
-
-        @Override
-        public void clearCredentials() {
-        }
-    }
-
-    private RestRequest mockRestRequest() {
-        RestRequest request = mock(RestRequest.class);
-        HttpChannel httpChannel = mock(HttpChannel.class);
-        when(request.getHttpChannel()).thenReturn(httpChannel);
-        when(httpChannel.getRemoteAddress()).thenReturn(new InetSocketAddress(InetAddress.getLoopbackAddress(), 9200));
-        when(request.uri()).thenReturn("_uri");
-        return request;
-    }
-
-    private SearchHit getIndexedAuditMessage(Message message) throws InterruptedException {
-        assertNotNull("no audit message was enqueued", message);
-        final String indexName = IndexNameResolver.resolve(IndexAuditTrailField.INDEX_NAME_PREFIX, message.timestamp, rollover);
-        ensureYellowAndNoInitializingShards(indexName);
-        GetSettingsResponse settingsResponse = getClient().admin().indices().prepareGetSettings(indexName).get();
-        assertThat(settingsResponse.getSetting(indexName, "index.number_of_shards"), is(Integer.toString(numShards)));
-        assertThat(settingsResponse.getSetting(indexName, "index.number_of_replicas"), is(Integer.toString(numReplicas)));
-
-        final SetOnce<SearchResponse> searchResponseSetOnce = new SetOnce<>();
-        final boolean found = awaitBusy(() -> {
-            try {
-                SearchResponse searchResponse = getClient()
-                        .prepareSearch(indexName)
-                        .setTypes(IndexAuditTrail.DOC_TYPE)
-                        .get();
-                if (searchResponse.getHits().getTotalHits().value > 0L) {
-                    searchResponseSetOnce.set(searchResponse);
-                    return true;
-                }
-            } catch (Exception e) {
-                logger.debug("caught exception while executing search", e);
-            }
-            return false;
-        });
-        assertThat("no audit document exists!", found, is(true));
-        SearchResponse response = searchResponseSetOnce.get();
-        assertNotNull(response);
-
-        assertEquals(1, response.getHits().getTotalHits().value);
-        return response.getHits().getHits()[0];
-    }
-
-    @Override
-    public ClusterHealthStatus ensureYellowAndNoInitializingShards(String... indices) {
-        if (remoteIndexing == false) {
-            return super.ensureYellowAndNoInitializingShards(indices);
-        }
-
-        // pretty ugly but just a rip of ensureYellowAndNoInitializingShards that uses a different client
-        ClusterHealthResponse actionGet = getClient().admin().cluster().health(Requests.clusterHealthRequest(indices)
-                .waitForNoRelocatingShards(true)
-                .waitForYellowStatus()
-                .waitForEvents(Priority.LANGUID)
-                .waitForNoInitializingShards(true))
-                .actionGet();
-        if (actionGet.isTimedOut()) {
-            logger.info("ensureYellow timed out, cluster state:\n{}\n{}",
-                    getClient().admin().cluster().prepareState().get().getState(),
-                    getClient().admin().cluster().preparePendingClusterTasks().get());
-            assertThat("timed out waiting for yellow", actionGet.isTimedOut(), equalTo(false));
-        }
-
-        logger.debug("indices {} are yellow", indices.length == 0 ? "[_all]" : indices);
-        return actionGet.getStatus();
-    }
-
-    private static Authentication createAuthentication(User user) {
-        final RealmRef lookedUpBy = user.authenticatedUser() == user ? null : new RealmRef("lookRealm", "up", "by");
-        return new Authentication(user, new RealmRef("authRealm", "test", "foo"), lookedUpBy);
-    }
-}
-

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/index/RemoteIndexAuditTrailStartingTests.java

@@ -1,169 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security.audit.index;
-
-import org.elasticsearch.core.internal.io.IOUtils;
-import org.elasticsearch.action.admin.cluster.node.info.NodeInfo;
-import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.common.transport.TransportAddress;
-import org.elasticsearch.common.util.set.Sets;
-import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
-import org.elasticsearch.test.ESIntegTestCase.Scope;
-import org.elasticsearch.test.InternalTestCluster;
-import org.elasticsearch.test.SecurityIntegTestCase;
-import org.elasticsearch.test.SecuritySettingsSource;
-import org.elasticsearch.test.SecuritySettingsSourceField;
-import org.elasticsearch.test.junit.annotations.TestLogging;
-import org.elasticsearch.xpack.security.audit.AuditTrail;
-import org.elasticsearch.xpack.security.audit.AuditTrailService;
-import org.elasticsearch.xpack.security.support.SecurityIndexManager;
-import org.junit.After;
-import org.junit.Before;
-
-import java.io.Closeable;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Optional;
-import java.util.Set;
-import java.util.stream.StreamSupport;
-
-import static org.elasticsearch.test.InternalTestCluster.clusterName;
-import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoTimeout;
-
-/**
- * This test checks to ensure that the IndexAuditTrail starts properly when indexing to a remote cluster. The cluster
- * started by the integration tests is indexed into by the remote cluster started before the test.
- *
- * The cluster started by the integrations tests may also index into itself...
- */
-@ClusterScope(scope = Scope.TEST, numDataNodes = 1, numClientNodes = 0, transportClientRatio = 0.0, supportsDedicatedMasters = false)
-@TestLogging("org.elasticsearch.xpack.security.audit.index:TRACE")
-public class RemoteIndexAuditTrailStartingTests extends SecurityIntegTestCase {
-
-    public static final String SECOND_CLUSTER_NODE_PREFIX = "remote_" + TEST_CLUSTER_NODE_PREFIX;
-
-    private InternalTestCluster remoteCluster;
-
-    private final boolean sslEnabled = randomBoolean();
-    private final boolean localAudit = randomBoolean();
-    private final String outputs = randomFrom("index", "logfile", "index,logfile");
-
-    @Override
-    public boolean transportSSLEnabled() {
-        return sslEnabled;
-    }
-
-    @Override
-    public Settings nodeSettings(int nodeOrdinal) {
-        return Settings.builder()
-                .put(super.nodeSettings(nodeOrdinal))
-                .put("xpack.security.audit.enabled", localAudit)
-                .put("xpack.security.audit.outputs", outputs)
-                .build();
-    }
-
-    @Override
-    protected Set<String> excludeTemplates() {
-        return Sets.newHashSet(SecurityIndexManager.SECURITY_TEMPLATE_NAME, IndexAuditTrail.INDEX_TEMPLATE_NAME);
-    }
-
-    @Override
-    protected int numberOfShards() {
-        return 1; // limit ourselves to a single shard in order to avoid timeout issues with large numbers of shards in tests
-    }
-
-    @Before
-    public void startRemoteCluster() throws IOException, InterruptedException {
-        final List<String> addresses = new ArrayList<>();
-        // get addresses for current cluster
-        NodesInfoResponse response = client().admin().cluster().prepareNodesInfo().execute().actionGet();
-        final String clusterName = response.getClusterName().value();
-        for (NodeInfo nodeInfo : response.getNodes()) {
-            TransportAddress address = nodeInfo.getTransport().address().publishAddress();
-            addresses.add(address.address().getHostString() + ":" + address.address().getPort());
-        }
-
-        // create another cluster
-        String cluster2Name = clusterName(Scope.TEST.name(), randomLong());
-
-        // Setup a second test cluster with a single node, security enabled, and SSL
-        final int numNodes = 1;
-        SecuritySettingsSource cluster2SettingsSource =
-                new SecuritySettingsSource(sslEnabled, createTempDir(), Scope.TEST) {
-            @Override
-            public Settings nodeSettings(int nodeOrdinal) {
-                Settings.Builder builder = Settings.builder()
-                        .put(super.nodeSettings(nodeOrdinal))
-                        // Disable native ML autodetect_process as the c++ controller won't be available
-//                        .put(MachineLearningField.AUTODETECT_PROCESS.getKey(), false)
-                        .put("xpack.security.audit.enabled", true)
-                        .put("xpack.security.audit.outputs", randomFrom("index", "index,logfile"))
-                        .putList("xpack.security.audit.index.client.hosts", addresses.toArray(new String[addresses.size()]))
-                        .put("xpack.security.audit.index.client.cluster.name", clusterName)
-                        .put("xpack.security.audit.index.client.xpack.security.user",
-                             TEST_USER_NAME + ":" + SecuritySettingsSourceField.TEST_PASSWORD)
-                        .put("xpack.security.audit.index.settings.index.number_of_shards", 1)
-                        .put("xpack.security.audit.index.settings.index.number_of_replicas", 0);
-
-                addClientSSLSettings(builder, "xpack.security.audit.index.client.xpack.security.transport.");
-                builder.put("xpack.security.audit.index.client.xpack.security.transport.ssl.enabled", sslEnabled);
-                return builder.build();
-            }
-        };
-        remoteCluster = new InternalTestCluster(randomLong(), createTempDir(), false, true, numNodes, numNodes,
-                cluster2Name, cluster2SettingsSource, 0, SECOND_CLUSTER_NODE_PREFIX, getMockPlugins(), getClientWrapper());
-        remoteCluster.beforeTest(random(), 0.0);
-        assertNoTimeout(remoteCluster.client().admin().cluster().prepareHealth().setWaitForGreenStatus().get());
-    }
-
-    @After
-    public void stopRemoteCluster() throws Exception {
-        List<Closeable> toStop = new ArrayList<>();
-        // stop the index audit trail so that the shards aren't locked causing the test to fail
-        toStop.add(() -> StreamSupport.stream(internalCluster().getInstances(AuditTrailService.class).spliterator(), false)
-                .map(s -> s.getAuditTrails()).flatMap(List::stream)
-                .filter(t -> t.name().equals(IndexAuditTrail.NAME))
-                .forEach((auditTrail) -> ((IndexAuditTrail) auditTrail).stop()));
-        // first stop both audit trails otherwise we keep on indexing
-        if (remoteCluster != null) {
-            toStop.add(() -> StreamSupport.stream(remoteCluster.getInstances(AuditTrailService.class).spliterator(), false)
-                    .map(s -> s.getAuditTrails()).flatMap(List::stream)
-                    .filter(t -> t.name().equals(IndexAuditTrail.NAME))
-                    .forEach((auditTrail) -> ((IndexAuditTrail) auditTrail).stop()));
-            toStop.add(() -> remoteCluster.wipe(excludeTemplates()));
-            toStop.add(remoteCluster::afterTest);
-            toStop.add(remoteCluster);
-        }
-
-
-        IOUtils.close(toStop);
-    }
-
-    public void testThatRemoteAuditInstancesAreStarted() throws Exception {
-        logger.info("Test configuration: ssl=[{}] localAudit=[{}][{}]", sslEnabled, localAudit, outputs);
-        // we ensure that all instances present are started otherwise we will have issues
-        // and race with the shutdown logic
-        for (InternalTestCluster cluster : Arrays.asList(remoteCluster, internalCluster())) {
-            for (AuditTrailService auditTrailService : cluster.getInstances(AuditTrailService.class)) {
-                Optional<AuditTrail> auditTrail = auditTrailService.getAuditTrails().stream()
-                        .filter(t -> t.name().equals(IndexAuditTrail.NAME)).findAny();
-                if (cluster == remoteCluster || (localAudit && outputs.contains("index"))) {
-                    // remote cluster must be present and only if we do local audit and output to an index we are good on the local one
-                    // as well.
-                    assertTrue(auditTrail.isPresent());
-                }
-                if (auditTrail.isPresent()) {
-                    IndexAuditTrail indexAuditTrail = (IndexAuditTrail) auditTrail.get();
-                    assertBusy(() -> assertSame("trail not started remoteCluster: " + (remoteCluster == cluster),
-                            indexAuditTrail.state(), IndexAuditTrail.State.STARTED));
-                }
-            }
-        }
-    }
-}

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/audit/logfile/AuditTrailSettingsUpdateTests.java

@@ -62,7 +62,6 @@ public class AuditTrailSettingsUpdateTests extends SecurityIntegTestCase {
 
         // enable auditing
         settingsBuilder.put("xpack.security.audit.enabled", "true");
-        settingsBuilder.put("xpack.security.audit.outputs", "logfile");
         // add only startup filter policies
         settingsBuilder.put(startupFilterSettings);
         return settingsBuilder.build();

x-pack/plugin/sql/qa/security/build.gradle

@@ -29,7 +29,6 @@ subprojects {
   integTestCluster {
     // Setup auditing so we can use it in some tests
     setting 'xpack.security.audit.enabled', 'true'
-    setting 'xpack.security.audit.outputs', 'logfile'
     setting 'xpack.security.enabled', 'true'
     setting 'xpack.license.self_generated.type', 'trial'
     // Setup roles used by tests
@@ -49,7 +48,6 @@ subprojects {
   runqa {
     // Setup auditing so we can use it in some tests
     setting 'xpack.security.audit.enabled', 'true'
-    setting 'xpack.security.audit.outputs', 'logfile'
     setting 'xpack.security.enabled', 'true'
     setting 'xpack.license.self_generated.type', 'trial'
     // Setup roles used by tests

x-pack/qa/audit-tests/build.gradle

@@ -1,40 +0,0 @@
-apply plugin: 'elasticsearch.standalone-rest-test'
-apply plugin: 'elasticsearch.rest-test'
-
-dependencies {
-  testCompile project(path: xpackModule('security'), configuration: 'testArtifacts')
-}
-
-String outputDir = "${buildDir}/generated-resources/${project.name}"
-task copyXPackPluginProps(type: Copy) { // wth is this?
-  from project(xpackModule('core')).file('src/main/plugin-metadata')
-  from project(xpackModule('core')).tasks.pluginProperties
-  from project(xpackModule('security')).file('src/main/plugin-metadata')
-  from project(xpackModule('security')).tasks.pluginProperties
-  into outputDir
-}
-project.sourceSets.test.output.dir(outputDir, builtBy: copyXPackPluginProps)
-
-integTestCluster {
-  distribution 'default'
-  setting 'xpack.ilm.enabled', 'false'
-  setting 'xpack.ml.enabled', 'false'
-  setting 'xpack.monitoring.enabled', 'false'
-  setting 'xpack.security.enabled', 'true'
-  setting 'xpack.security.audit.enabled', 'true'
-  setting 'xpack.security.audit.outputs', 'index'
-  setting 'xpack.license.self_generated.type', 'trial'
-  setting 'logger.level', 'DEBUG'
-  setupCommand 'setupDummyUser',
-               'bin/elasticsearch-users', 'useradd', 'test_user', '-p', 'x-pack-test-password', '-r', 'superuser'
-  waitCondition = { node, ant ->
-    File tmpFile = new File(node.cwd, 'wait.success')
-    ant.get(src: "http://${node.httpUri()}/_cluster/health?wait_for_nodes=>=${numNodes}&wait_for_status=yellow",
-            dest: tmpFile.toString(),
-            username: 'test_user',
-            password: 'x-pack-test-password',
-            ignoreerrors: true,
-            retries: 10)
-    return tmpFile.exists()
-  }
-}

x-pack/qa/audit-tests/src/test/java/org/elasticsearch/xpack/security/audit/IndexAuditIT.java

@@ -1,226 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.xpack.security.audit;
-
-import com.carrotsearch.hppc.cursors.ObjectCursor;
-import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
-import org.elasticsearch.action.search.SearchResponse;
-import org.elasticsearch.action.support.master.AcknowledgedResponse;
-import org.elasticsearch.client.Client;
-import org.elasticsearch.client.Request;
-import org.elasticsearch.client.RequestOptions;
-import org.elasticsearch.client.Response;
-import org.elasticsearch.cluster.ClusterState;
-import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
-import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
-import org.elasticsearch.common.network.NetworkModule;
-import org.elasticsearch.common.settings.SecureString;
-import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.index.query.QueryBuilder;
-import org.elasticsearch.index.query.QueryBuilders;
-import org.elasticsearch.plugins.Plugin;
-import org.elasticsearch.tasks.Task;
-import org.elasticsearch.test.ESIntegTestCase;
-import org.elasticsearch.test.TestCluster;
-import org.elasticsearch.xpack.core.XPackClientPlugin;
-import org.elasticsearch.xpack.core.security.SecurityField;
-import org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken;
-import org.elasticsearch.xpack.security.audit.index.IndexAuditTrail;
-
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-
-import static org.hamcrest.Matchers.greaterThan;
-import static org.hamcrest.Matchers.is;
-
-public class IndexAuditIT extends ESIntegTestCase {
-    private static final String USER = "test_user";
-    private static final String PASS = "x-pack-test-password";
-
-    @Override
-    protected TestCluster buildTestCluster(Scope scope, long seed) throws IOException {
-        TestCluster testCluster = super.buildTestCluster(scope, seed);
-        return new TestCluster(seed) {
-
-            @Override
-            public void afterTest() throws IOException {
-                testCluster.afterTest();
-            }
-
-            @Override
-            public Client client() {
-                return testCluster.client();
-            }
-
-            @Override
-            public int size() {
-                return testCluster.size();
-            }
-
-            @Override
-            public int numDataNodes() {
-                return testCluster.numDataNodes();
-            }
-
-            @Override
-            public int numDataAndMasterNodes() {
-                return testCluster.numDataAndMasterNodes();
-            }
-
-            @Override
-            public InetSocketAddress[] httpAddresses() {
-                return testCluster.httpAddresses();
-            }
-
-            @Override
-            public void close() throws IOException {
-                testCluster.close();
-            }
-
-            @Override
-            public void ensureEstimatedStats() {
-                // stats are not going to be accurate for these tests since the index audit trail
-                // is running and changing the values so we wrap the test cluster to skip these
-                // checks
-            }
-
-            @Override
-            public String getClusterName() {
-                return testCluster.getClusterName();
-            }
-
-            @Override
-            public Iterable<Client> getClients() {
-                return testCluster.getClients();
-            }
-
-            @Override
-            public NamedWriteableRegistry getNamedWriteableRegistry() {
-                return testCluster.getNamedWriteableRegistry();
-            }
-        };
-    }
-
-    public void testIndexAuditTrailWorking() throws Exception {
-        Request request = new Request("GET", "/");
-        RequestOptions.Builder options = request.getOptions().toBuilder();
-        options.addHeader(UsernamePasswordToken.BASIC_AUTH_HEADER,
-                UsernamePasswordToken.basicAuthHeaderValue(USER, new SecureString(PASS.toCharArray())));
-        request.setOptions(options);
-        Response response = getRestClient().performRequest(request);
-        final AtomicReference<ClusterState> lastClusterState = new AtomicReference<>();
-        final boolean found = awaitSecurityAuditIndex(lastClusterState, QueryBuilders.matchQuery("principal", USER));
-
-        assertTrue("Did not find security audit index. Current cluster state:\n" + lastClusterState.get().toString(), found);
-
-        SearchResponse searchResponse = client().prepareSearch(".security_audit_log*").setQuery(
-                QueryBuilders.matchQuery("principal", USER)).get();
-        assertThat(searchResponse.getHits().getHits().length, greaterThan(0));
-        assertThat(searchResponse.getHits().getAt(0).getSourceAsMap().get("principal"), is(USER));
-    }
-
-    public void testAuditTrailTemplateIsRecreatedAfterDelete() throws Exception {
-        // this is already "tested" by the test framework since we wipe the templates before and after,
-        // but lets be explicit about the behavior
-        awaitIndexTemplateCreation();
-
-        // delete the template
-        AcknowledgedResponse deleteResponse = client().admin().indices()
-                .prepareDeleteTemplate(IndexAuditTrail.INDEX_TEMPLATE_NAME).execute().actionGet();
-        assertThat(deleteResponse.isAcknowledged(), is(true));
-        awaitIndexTemplateCreation();
-    }
-
-    public void testOpaqueIdWorking() throws Exception {
-        Request request = new Request("GET", "/");
-        RequestOptions.Builder options = request.getOptions().toBuilder();
-        options.addHeader(Task.X_OPAQUE_ID, "foo");
-        options.addHeader(UsernamePasswordToken.BASIC_AUTH_HEADER,
-            UsernamePasswordToken.basicAuthHeaderValue(USER, new SecureString(PASS.toCharArray())));
-        request.setOptions(options);
-        Response response = getRestClient().performRequest(request);
-        assertThat(response.getStatusLine().getStatusCode(), is(200));
-        final AtomicReference<ClusterState> lastClusterState = new AtomicReference<>();
-        final boolean found = awaitSecurityAuditIndex(lastClusterState, QueryBuilders.matchQuery("opaque_id", "foo"));
-
-        assertTrue("Did not find security audit index. Current cluster state:\n" + lastClusterState.get().toString(), found);
-
-        SearchResponse searchResponse = client().prepareSearch(".security_audit_log*").setQuery(
-            QueryBuilders.matchQuery("opaque_id", "foo")).get();
-        assertThat(searchResponse.getHits().getHits().length, greaterThan(0));
-
-        assertThat(searchResponse.getHits().getAt(0).getSourceAsMap().get("opaque_id"), is("foo"));
-    }
-
-    private boolean awaitSecurityAuditIndex(AtomicReference<ClusterState> lastClusterState,
-                                            QueryBuilder query) throws InterruptedException {
-        final AtomicBoolean indexExists = new AtomicBoolean(false);
-        return awaitBusy(() -> {
-            if (indexExists.get() == false) {
-                ClusterState state = client().admin().cluster().prepareState().get().getState();
-                lastClusterState.set(state);
-                for (ObjectCursor<String> cursor : state.getMetaData().getIndices().keys()) {
-                    if (cursor.value.startsWith(".security_audit_log")) {
-                        logger.info("found audit index [{}]", cursor.value);
-                        indexExists.set(true);
-                        break;
-                    }
-                }
-
-                if (indexExists.get() == false) {
-                    return false;
-                }
-            }
-
-            ensureYellowAndNoInitializingShards(".security_audit_log*");
-            logger.info("security audit log index is yellow");
-            ClusterState state = client().admin().cluster().prepareState().get().getState();
-            lastClusterState.set(state);
-
-            logger.info("refreshing audit indices");
-            client().admin().indices().prepareRefresh(".security_audit_log*").get();
-            logger.info("refreshed audit indices");
-            return client().prepareSearch(".security_audit_log*").setQuery(query)
-                .get().getHits().getTotalHits().value > 0;
-        }, 60L, TimeUnit.SECONDS);
-    }
-
-    private void awaitIndexTemplateCreation() throws InterruptedException {
-        boolean found = awaitBusy(() -> {
-            GetIndexTemplatesResponse response = client().admin().indices()
-                    .prepareGetTemplates(IndexAuditTrail.INDEX_TEMPLATE_NAME).execute().actionGet();
-            if (response.getIndexTemplates().size() > 0) {
-                for (IndexTemplateMetaData indexTemplateMetaData : response.getIndexTemplates()) {
-                    if (IndexAuditTrail.INDEX_TEMPLATE_NAME.equals(indexTemplateMetaData.name())) {
-                        return true;
-                    }
-                }
-            }
-            return false;
-        });
-
-        assertThat("index template [" + IndexAuditTrail.INDEX_TEMPLATE_NAME + "] was not created", found, is(true));
-    }
-
-    @Override
-    protected Settings externalClusterClientSettings() {
-        return Settings.builder()
-                .put(SecurityField.USER_SETTING.getKey(), USER + ":" + PASS)
-                .put(NetworkModule.TRANSPORT_TYPE_KEY, "security4")
-                .build();
-    }
-
-    @Override
-    protected Collection<Class<? extends Plugin>> transportClientPlugins() {
-        return Arrays.asList(XPackClientPlugin.class);
-    }
-
-}

x-pack/qa/rolling-upgrade/build.gradle

@@ -156,7 +156,6 @@ subprojects {
       setting 'xpack.security.transport.ssl.enabled', 'true'
       setting 'xpack.security.authc.token.enabled', 'true'
       setting 'xpack.security.audit.enabled', 'true'
-      setting 'xpack.security.audit.outputs', 'index'
       setting 'xpack.security.transport.ssl.keystore.path', 'testnode.jks'
       setting 'xpack.security.transport.ssl.keystore.password', 'testnode'
       dependsOn copyTestNodeKeystore
@@ -235,7 +234,6 @@ subprojects {
         setting 'node.attr.upgraded', 'true'
         setting 'xpack.security.authc.token.enabled', 'true'
         setting 'xpack.security.audit.enabled', 'true'
-        setting 'xpack.security.audit.outputs', 'index'
         setting 'node.name', "upgraded-node-${stopNode}"
         dependsOn copyTestNodeKeystore
         extraConfigFile 'testnode.jks', new File(outputDir + '/testnode.jks')

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/IndexAuditUpgradeIT.java

@@ -1,97 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-package org.elasticsearch.upgrades;
-
-import org.elasticsearch.Version;
-import org.elasticsearch.client.Request;
-import org.elasticsearch.client.Response;
-import org.elasticsearch.common.Booleans;
-import org.hamcrest.Matchers;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-
-import static org.hamcrest.Matchers.hasSize;
-
-public class IndexAuditUpgradeIT extends AbstractUpgradeTestCase {
-
-    public void testAuditLogs() throws Exception {
-        assertBusy(() -> {
-            assertAuditDocsExist();
-            assertNumUniqueNodeNameBuckets(expectedNumUniqueNodeNameBuckets());
-        });
-    }
-
-    private int expectedNumUniqueNodeNameBuckets() throws IOException {
-        switch (CLUSTER_TYPE) {
-            case OLD:
-                // There are three nodes in the initial test cluster
-                return 3;
-            case MIXED:
-                if (false == masterIsNewVersion()) {
-                    return 3;
-                }
-                if (Booleans.parseBoolean(System.getProperty("tests.first_round"))) {
-                    // One of the old nodes has been removed and we've added a new node
-                    return 4;
-                }
-                // Two of the old nodes have been removed and we've added two new nodes
-                return 5;
-            case UPGRADED:
-                return 6;
-            default:
-                throw new IllegalArgumentException("Unsupported cluster type [" + CLUSTER_TYPE + "]");
-            }
-    }
-
-    private void assertAuditDocsExist() throws Exception {
-        Response response = client().performRequest(new Request("GET", "/.security_audit_log*/_count"));
-        assertEquals(200, response.getStatusLine().getStatusCode());
-        Map<String, Object> responseMap = entityAsMap(response);
-        assertNotNull(responseMap.get("count"));
-        assertThat((Integer) responseMap.get("count"), Matchers.greaterThanOrEqualTo(1));
-    }
-
-    private void assertNumUniqueNodeNameBuckets(int numBuckets) throws Exception {
-        // call API that will hit all nodes
-        Map<?, ?> nodesResponse = entityAsMap(client().performRequest(new Request("GET", "/_nodes/_all/info/version")));
-        logger.info("all nodes {}", nodesResponse);
-
-        Request aggRequest = new Request("GET", "/.security_audit_log*/_search");
-        aggRequest.setJsonEntity(
-                "{\n" +
-                "    \"aggs\" : {\n" +
-                "        \"nodes\" : {\n" +
-                "            \"terms\" : { \"field\" : \"node_name\" }\n" +
-                "        }\n" +
-                "    }\n" +
-                "}");
-        aggRequest.addParameter("pretty", "true");
-        Response aggResponse = client().performRequest(aggRequest);
-        Map<String, Object> aggResponseMap = entityAsMap(aggResponse);
-        logger.debug("aggResponse {}", aggResponseMap);
-        Map<?, ?> aggregations = (Map<?, ?>) aggResponseMap.get("aggregations");
-        assertNotNull(aggregations);
-        Map<?, ?> nodesAgg = (Map<?, ?>) aggregations.get("nodes");
-        assertNotNull(nodesAgg);
-        List<?> buckets = (List<?>) nodesAgg.get("buckets");
-        assertNotNull(buckets);
-        assertThat("Found node buckets " + buckets, buckets, hasSize(numBuckets));
-    }
-
-    /**
-     * Has the master been upgraded to the new version?
-     */
-    private boolean masterIsNewVersion() throws IOException {
-        Map<?, ?> map = entityAsMap(client().performRequest(new Request("GET", "/_nodes/_master")));
-        map = (Map<?, ?>) map.get("nodes");
-        assertThat(map.values(), hasSize(1));
-        map = (Map<?, ?>) map.values().iterator().next();
-        Version masterVersion = Version.fromString(map.get("version").toString());
-        return Version.CURRENT.equals(masterVersion);
-    }
-}