DOC Audit security config change (#66839)

Audit log doc changes about:
* the new security_config_change event type (main scope of this PR)
* remove mentions of the 6.5 audit format changes (the JSON format)
* mention the new archiving and rotation by size (in v8 only)
* mention the request.id event attribute used to correlate audit events
* mention that audit is only available on certain subscription levels
* add an exhaustive audit event example list (because schema became too complex to explain in words 😢 given the new security_config_change events)
* move the ignore policies are explained on a separate page (it was collocated with the logfile output since we had multiple outputs and the policies were specific the the logfile only).

Co-authored-by: Lisa Cawley lcawley@elastic.co

Relates #62916
Closes #29912

docs/reference/settings/audit-settings.asciidoc

@@ -8,7 +8,15 @@
 [[auditing-settings-description]]
 You can use <<enable-audit-logging,audit logging>> to record security-related
 events, such as authentication failures, refused connections, and data-access
-events.
+events. In addition, changes via the APIs to the security configuration, such
+as creating, updating and removing <<native-realm, native>> and
+<<built-in-users, built-in>> users, <<security-api-put-role, roles>>,
+<<security-api-put-role-mapping, role mappings>> and
+<<security-api-create-api-key, API keys>> are also recorded.
+--
+TIP: Audit logs are only available on certain subscription levels.
+For more information, see {subscriptions}.
+--
 
 If configured, auditing settings must be set on every node in the cluster.
 Static settings, such as `xpack.security.audit.enabled`, must be configured in
@@ -40,29 +48,38 @@ by using the following settings:
 // tag::xpack-sa-lf-events-include-tag[]
 `xpack.security.audit.logfile.events.include`::
 (<<dynamic-cluster-setting,Dynamic>>)
-Specifies which events to include in the auditing output. The default value is:
+Specifies the <<audit-event-types, kind of events>> to print in the auditing output.
+In addition, `_all` can be used to exhaustively audit all the events, but this is usually
+discouraged since it will get very verbose.
+The default list value contains:
 `access_denied, access_granted, anonymous_access_denied, authentication_failed,
-connection_denied, tampered_request, run_as_denied, run_as_granted`.
+connection_denied, tampered_request, run_as_denied, run_as_granted,
+security_config_change`.
 // end::xpack-sa-lf-events-include-tag[]
 
 [[xpack-sa-lf-events-exclude]]
 // tag::xpack-sa-lf-events-exclude-tag[]
 `xpack.security.audit.logfile.events.exclude`::
 (<<dynamic-cluster-setting,Dynamic>>)
-Excludes the specified events from the output. By default, no events are
-excluded.
+Excludes the specified <<audit-event-types, kind of events>> from the include list.
+This is useful in the case where the `events.include` setting contains the special value `_all`.
+The default is the empty list.
 // end::xpack-sa-lf-events-exclude-tag[]
 
 [[xpack-sa-lf-events-emit-request]]
 // tag::xpack-sa-lf-events-emit-request-tag[]
 `xpack.security.audit.logfile.events.emit_request_body`::
 (<<dynamic-cluster-setting,Dynamic>>)
-Specifies whether to include the request body from REST requests on certain
-event types such as `authentication_failed`. The default value is `false`.
+Specifies whether to include the full request body from REST requests as an
+attribute of certain kinds of audit events. This setting can be used to
+<<auditing-search-queries, audit search queries>>.
++
+The default value is `false`, so request bodies are not printed.
 +
 --
-IMPORTANT: No filtering is performed when auditing, so sensitive data may be
-audited in plain text when including the request body in audit events.
+IMPORTANT: Be advised that sensitive data may be audited in plain text when including
+the request body in audit events, even though all the security APIs, such as those that
+change the user's password, have the credentials filtered out when audited.
 --
 
 // end::xpack-sa-lf-events-emit-request-tag[]
@@ -99,8 +116,6 @@ The default value is `false`.
 `xpack.security.audit.logfile.emit_node_id`::
 (<<dynamic-cluster-setting,Dynamic>>)
 Specifies whether to include the node id as a field in each audit event.
-This is available for the new format only. That is to say, this information
-does not exist in the `<clustername>_access.log` file.
 Unlike <<node-name,node name>>, whose value might change if the administrator
 changes the setting in the config file, the node id will persist across cluster
 restarts and the administrator cannot change it.
@@ -110,11 +125,13 @@ The default value is `true`.
 [[audit-event-ignore-policies]]
 ==== Audit Logfile Event Ignore Policies
 
-These settings affect the <<audit-log-ignore-policy,ignore policies>>
+The following settings affect the <<audit-log-ignore-policy,ignore policies>>
 that enable fine-grained control over which audit events are printed to the log file.
 All of the settings with the same policy name combine to form a single policy.
-If an event matches all of the conditions for a specific policy, it is ignored
-and not printed.
+If an event matches all the conditions of any policy, it is ignored and not printed.
+Most audit events are subject to the ignore policies. The sole exception are
+events of the `security_config_change` type, which cannot be filtered out, unless
+<<xpack-sa-lf-events-exclude, excluded>> altogether.
 
 [[xpack-sa-lf-events-ignore-users]]
 // tag::xpack-sa-lf-events-ignore-users-tag[]

x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc

@@ -4,14 +4,15 @@
 === Auditing search queries
 
 There is no <<audit-event-types, audit event type>> specifically
-dedicated to search queries. Search queries are analyzed and then processed; the 
+dedicated to search queries. Search queries are analyzed and then processed; the
 processing triggers authorization actions that are audited.
 However, the original raw query, as submitted by the client, is not accessible
 downstream when authorization auditing occurs.
 
 Search queries are contained inside HTTP request bodies, however, and some
-audit events that are generated by the REST layer can be toggled to output
-the request body to the audit log.
+audit events that are generated by the REST layer, on the coordinating node,
+can be toggled to output the request body to the audit log. Therefore, one
+must audit request bodies in order to audit search queries.
 
 To make certain audit events include the request body, edit the following
 setting in the `elasticsearch.yml` file:
@@ -26,9 +27,15 @@ audited in plain text when audit events include the request body. Also, the
 request body can contain malicious content that can break a parser consuming
 the audit logs.
 
-There are only a handful of <<audit-event-types, audit event types>> that are 
-generated in the REST layer and can access the request body. Most of them are not
-included by default.
+The request body is printed as an escaped JSON string value (RFC 4627) to the `request.body`
+event attribute.
+
+Not all events contain the `request.body` attribute, even when the above setting
+is toggled. The ones that do are: `authentication_success`,
+`authentication_failed`, `realm_authentication_failed`, `tampered_request`, `run_as_denied`,
+and `anonymous_access_denied`. The `request.body` attribute is printed on the coordinating node only
+(the node that handles the REST request). Most of these event types are
+<<xpack-sa-lf-events-include, not included by default>>.
 
 A good practical piece of advice is to add `authentication_success` to the event
 types that are audited (add it to the list in the `xpack.security.audit.logfile.events.include`),

x-pack/docs/en/security/auditing/enable-audit-logging.asciidoc

@@ -3,13 +3,19 @@
 == Enabling audit logging
 
 You can log security-related events such as authentication failures and refused connections
-to monitor your cluster for suspicious activity. 
+to monitor your cluster for suspicious activity (including data access authorization and user
+security configuration changes).
+
 Audit logging also provides forensic evidence in the event of an attack.
 
 [IMPORTANT]
 ============================================================================
 Audit logs are **disabled** by default. You must explicitly enable audit logging.
 ============================================================================
+--
+TIP: Audit logs are only available on certain subscription levels.
+For more information, see {subscriptions}.
+--
 
 To enable enable audit logging:
 
@@ -17,7 +23,8 @@ To enable enable audit logging:
 . Restart {es}.
 
 When audit logging is enabled, <<audit-event-types, security events>> are persisted to 
-a dedicated `<clustername>_audit.json` file on the host's file system (on each node).
+a dedicated `<clustername>_audit.json` file on the host's file system, on every cluster node.
+For more information, see <<audit-log-output>>.
 
 You can configure additional options to control what events are logged and 
 what information is included in the audit log. 

x-pack/docs/en/security/auditing/event-types.asciidoc

@@ -2,63 +2,146 @@
 [[audit-event-types]]
 === Audit event types
 
-When you are <<enable-audit-logging,auditing security events>>, each request can generate
-multiple audit events.
-
-The following is a list of the events that can be generated:
-
-|======
-| `anonymous_access_denied`         | | | Logged when a request is denied due to a missing
-                                          authentication token.
-| `authentication_success`          | | | Logged when a user successfully authenticates.
-| `authentication_failed`           | | | Logged when the authentication token cannot be
-                                          matched to a known user.
-| `realm_authentication_failed`     | | | Logged for every realm that fails to present a valid
-                                          authentication token. `<realm>` represents the
-                                          realm type.
-| `access_denied`                   | | | Logged when an authenticated user attempts to execute
-                                          an action they do not have the necessary
-                                          <<security-privileges,privilege>> to perform.
-| `access_granted`                  | | | Logged when an authenticated user attempts to execute
-                                          an action they have the necessary privilege to perform.
-                                          When the `system_access_granted` event is included, all system
-                                          (internal) actions are also logged. The default setting does
-                                          not log system actions to avoid cluttering the logs.
-| `run_as_granted`                  | | | Logged when an authenticated user attempts to <<run-as-privilege, run as>>
-                                          another user that they have the necessary privileges to do.
-| `run_as_denied`                   | | | Logged when an authenticated user attempts to <<run-as-privilege, run as>>
-                                          another user action they do not have the necessary
-                                          <<security-privileges,privilege>> to do so.
-| `tampered_request`                | | | Logged when the {security-features} detect that the request has
-                                          been tampered with. Typically relates to `search/scroll`
-                                          requests when the scroll ID is believed to have been
-                                          tampered with.
-| `connection_granted`              | | | Logged when an incoming TCP connection passes the
-                                          <<ip-filtering, IP Filter>> for a specific
-                                          profile.
-| `connection_denied`               | | | Logged when an incoming TCP connection does not pass the
-                                          <<ip-filtering, IP Filter>> for a specific
-                                          profile.
+When you are <<enable-audit-logging,auditing security events>>, a single client request
+might generate multiple audit events, across multiple cluster nodes. The common
+`request.id` attribute can be used to correlate the associated events.
+
+|======
+| Event action | Description | Example
+
+| `access_denied`
+| Logged when an authenticated user attempts to execute an action they do not
+have the necessary <<security-privileges,privilege>> to perform.
+| `{"type":"audit", "timestamp":"2020-12-30T22:30:06,949+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"transport", "event.action":"access_denied", "authentication.type":"REALM", "user.name":"user1", "user.realm":"default_native", "user.roles":["test_role"], "origin.type":"rest", "origin.address":"[::1]:52434", "request.id":"yKOgWn2CRQCKYgZRz3phJw", "action":"indices:admin/auto_create", "request.name":"CreateIndexRequest", "indices":["<index-{now/d+1d}>"]}`
+
+| `access_granted`
+| Logged when an authenticated user attempts to execute an action they have the
+necessary privilege to perform. When the `system_access_granted` event is
+included, all system (internal) actions are also logged. The default setting
+does not log system actions to avoid cluttering the logs.
+| `{"type":"audit", "timestamp":"2020-12-30T22:30:06,947+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"transport", "event.action":"access_granted", "authentication.type":"REALM", "user.name":"user1", "user.realm":"default_native", "user.roles":["test_role"], "origin.type":"rest", "origin.address":"[::1]:52434", "request.id":"yKOgWn2CRQCKYgZRz3phJw", "action":"indices:data/write/bulk", "request.name":"BulkRequest"}`
+
+| `anonymous_access_denied`
+| Logged when a request is denied due to missing authentication credentials.
+| `{"type":"audit", "timestamp":"2020-12-30T21:56:43,608+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"rest", "event.action":"anonymous_access_denied", "origin.type":"rest", "origin.address":"[::1]:50543", "url.path":"/twitter/_async_search", "url.query":"pretty", "request.method":"POST", "request.id":"TqA9OisyQ8WTl1ivJUV1AA"}`
+
+| `authentication_failed`
+| Logged when the authentication credentials cannot be matched to a known user.
+| `{"type":"audit", "timestamp":"2020-12-30T22:10:15,510+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"rest", "event.action":"authentication_failed", "user.name":"elastic", "origin.type":"rest", "origin.address":"[::1]:51504", "url.path":"/_security/user/user1", "url.query":"pretty", "request.method":"POST", "request.id":"POv8p_qeTl2tb5xoFl0HIg"}`
+
+| `authentication_success`
+| Logged when a user successfully authenticates.
+| `{"type":"audit", "timestamp":"2020-12-30T22:03:35,018+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"rest", "event.action":"authentication_success", "authentication.type":"REALM", "user.name":"elastic", "user.realm":"reserved", "origin.type":"rest", "origin.address":"[::1]:51014", "realm":"reserved", "url.path":"/twitter/_search", "url.query":"pretty", "request.method":"POST", "request.id":"nHV3UMOoSiu-TaSPWCfxGg"}`
+
+| `change_disable_user`
+| Logged when the <<security-api-enable-user,enable user API>> is invoked to
+disable a native or a built-in user.
+| `{"type":"audit", "timestamp":"2020-12-30T23:17:28,308+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"change_disable_user", "request.id":"qvLIgw_eTvyK3cgV-GaLVg", "change":{"disable":{"user":{"name":"user1"}}}}`
+
+| `change_enable_user`
+| Logged when the <<security-api-enable-user,enable user API>> is invoked to
+enable a native or a built-in user.
+| `{"type":"audit", "timestamp":"2020-12-30T23:17:34,843+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"change_enable_user", "request.id":"BO3QU3qeTb-Ei0G0rUOalQ", "change":{"enable":{"user":{"name":"user1"}}}}`
+
+| `change_password`
+| Logged when the <<security-api-change-password,change password API>> is
+invoked to change the password of a native or built-in user.
+| `{"type":"audit", "timestamp":"2019-12-30T22:19:41,345+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"change_password", "request.id":"bz5a1Cc3RrebDMitMGGNCw", "change":{"password":{"user":{"name":"user1"}}}}`
+
+| `connection_denied`
+| Logged when an incoming TCP connection does not pass the
+<<ip-filtering,IP filter>> for a specific profile.
+| `{"type":"audit", "timestamp":"2020-12-30T21:47:31,526+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"ip_filter", "event.action":"connection_denied", "origin.type":"rest", "origin.address":"10.10.0.20", "transport.profile":".http", "rule":"deny 10.10.0.0/16"}`
+
+| `connection_granted`
+| Logged when an incoming TCP connection passes the <<ip-filtering,IP filter>>
+for a specific profile.
+| `{"type":"audit", "timestamp":"2020-12-30T21:47:31,526+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"ip_filter", "event.action":"connection_granted", "origin.type":"rest", "origin.address":"::1", "transport.profile":".http", "rule":"allow ::1,127.0.0.1"}`
+
+| `create_apikey`
+| Logged when the <<security-api-create-api-key,create API key>> or the
+<<security-api-grant-api-key, grant API key>> APIs are invoked to create a new
+API key.
+| `{"type":"audit", "timestamp":"2020-12-31T00:33:52,521+0200", "node.id":"9clhpgjJRR-iKzOw20xBNQ", "event.type":"security_config_change", "event.action":"create_apikey", "request.id":"9FteCmovTzWHVI-9Gpa_vQ", "create":{"apikey":{"name":"test-api-key-1","expiration":"10d","role_descriptors":[{"cluster":["monitor","manage_ilm"],"indices":[{"names":["index-a*"],"privileges":["read","maintenance"]},{"names":["in*","alias*"],"privileges":["read"],"field_security":{"grant":["field1*","@timestamp"],"except":["field11"]}}],"applications":[],"run_as":[]},{"cluster":["all"],"indices":[{"names":["index-b*"],"privileges":["all"]}],"applications":[],"run_as":[]}]}}}`
+
+| `delete_privileges`
+| Logged when the
+<<security-api-delete-privilege,delete application privileges API>> is invoked
+to remove one or more application privileges.
+| `{"type":"audit", "timestamp":"2020-12-31T00:39:30,246+0200", "node.id":"9clhpgjJRR-iKzOw20xBNQ", "event.type":"security_config_change", "event.action":"delete_privileges", "request.id":"7wRWVxxqTzCKEspeSP7J8g", "delete":{"privileges":{"application":"myapp","privileges":["read"]}}}`
+
+| `delete_role`
+| Logged when the <<security-api-delete-role,delete role API>> is invoked to
+delete a role.
+| `{"type":"audit", "timestamp":"2020-12-31T00:08:11,678+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"delete_role", "request.id":"155IKq3zQdWq-12dgKZRnw", "delete":{"role":{"name":"my_admin_role"}}}`
+
+| `delete_role_mapping`
+| Logged when the <<security-api-delete-role-mapping,delete role mapping API>>
+is invoked to delete a role mapping.
+| `{"type":"audit", "timestamp":"2020-12-31T00:12:09,349+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"delete_role_mapping", "request.id":"Stim-DuoSTCWom0S_xhf8g", "delete":{"role_mapping":{"name":"mapping1"}}}`
+
+| `delete_user`
+| Logged when the <<security-api-delete-user,delete user API>> is invoked to
+delete a specific native user.
+| `{"type":"audit", "timestamp":"2020-12-30T22:19:41,345+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"delete_user", "request.id":"au5a1Cc3RrebDMitMGGNCw", "delete":{"user":{"name":"jacknich"}}}`
+
+| `invalidate_apikeys`
+| Logged when the <<security-api-invalidate-api-key,invalidate API key API>> is
+invoked to invalidate one or more API keys.
+| `{"type":"audit", "timestamp":"2020-12-31T00:36:30,247+0200", "node.id":"9clhpgjJRR-iKzOw20xBNQ", "event.type":"security_config_change", "event.action":"invalidate_apikeys", "request.id":"7lyIQU9QTFqSrTxD0CqnTQ", "invalidate":{"apikeys":{"owned_by_authenticated_user":false,"user":{"name":"myuser","realm":"native1"}}}}`
+
+| `put_privileges`
+| Logged when the <<security-api-put-privileges,put privileges API>> is invoked
+to add or update one or more application privileges.
+| `{"type":"audit", "timestamp":"2020-12-31T00:39:07,779+0200", "node.id":"9clhpgjJRR-iKzOw20xBNQ", "event.type":"security_config_change", "event.action":"put_privileges", "request.id":"1X2VVtNgRYO7FmE0nR_BGA", "put":{"privileges":[{"application":"myapp","name":"read","actions":["data:read/*","action:login"],"metadata":{"description":"Read access to myapp"}}]}}`
+
+| `put_role`
+| Logged when the <<security-api-put-role,put role API>> is invoked to create or
+update a role.
+| `{"type":"audit", "timestamp":"2020-12-30T22:27:01,978+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"put_role", "request.id":"tDYQhv5CRMWM4Sc5Zkk2cQ", "put":{"role":{"name":"test_role","role_descriptor":{"cluster":["all"],"indices":[{"names":["apm*"],"privileges":["all"],"field_security":{"grant":["granted"]},"query":"{\"term\": {\"service.name\": \"bar\"}}"},{"names":["apm-all*"],"privileges":["all"],"query":"{\"term\": {\"service.name\": \"bar2\"}}"}],"applications":[],"run_as":[]}}}}`
+
+| `put_role_mapping`
+| Logged when the <<security-api-put-role-mapping,put role mapping API>> is
+invoked to create or update a role mapping.
+| `{"type":"audit", "timestamp":"2020-12-31T00:11:13,932+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"put_role_mapping", "request.id":"kg4h1l_kTDegnLC-0A-XxA", "put":{"role_mapping":{"name":"mapping1","roles":["user"],"rules":{"field":{"username":"*"}},"enabled":true,"metadata":{"version":1}}}}`
+
+| `put_user`
+| Logged when the <<security-api-put-user,put user API>> is invoked to create or
+update a native user. Note that user updates can also change the
+user's password.
+| `{"type":"audit", "timestamp":"2020-12-30T22:10:09,749+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"security_config_change", "event.action":"put_user", "request.id":"VIiSvhp4Riim_tpkQCVSQA", "put":{"user":{"name":"user1","enabled":false,"roles":["admin","other_role1"],"full_name":"Jack Sparrow","email":"jack@blackpearl.com","has_password":true,"metadata":{"cunning":10}}}}`
+
+| `realm_authentication_failed`
+| Logged for every realm that fails to present a valid authentication token.
+| `{"type":"audit", "timestamp":"2020-12-30T22:10:15,510+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"rest", "event.action":"realm_authentication_failed", "user.name":"elastic", "origin.type":"rest", "origin.address":"[::1]:51504", "realm":"myTestRealm1", "url.path":"/_security/user/user1", "url.query":"pretty", "request.method":"POST", "request.id":"POv8p_qeTl2tb5xoFl0HIg"}`
+
+| `run_as_denied`
+| Logged when an authenticated user attempts to <<run-as-privilege,run as>>
+another user that they do not have the necessary
+<<security-privileges,privileges>> to do so.
+| `{"type":"audit", "timestamp":"2020-12-30T22:49:34,859+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"transport", "event.action":"run_as_denied", "user.name":"user1", "user.run_as.name":"user1", "user.realm":"default_native", "user.run_as.realm":"default_native", "user.roles":["test_role"], "origin.type":"rest", "origin.address":"[::1]:52662", "request.id":"RcaSt872RG-R_WJBEGfYXA", "action":"indices:data/read/search", "request.name":"SearchRequest", "indices":["alias1"]}`
+
+| `run_as_granted`
+| Logged when an authenticated user attempts to <<run-as-privilege,run as>>
+another user that they have the necessary privileges to do so.
+| `{"type":"audit", "timestamp":"2020-12-30T22:44:42,068+0200", "node.id":"0RMNyghkQYCc_gVd1G6tZQ", "event.type":"transport", "event.action":"run_as_granted", "user.name":"elastic", "user.run_as.name":"user1", "user.realm":"reserved", "user.run_as.realm":"default_native", "user.roles":["superuser"], "origin.type":"rest", "origin.address":"[::1]:52623", "request.id":"dGqPTdEQSX2TAPS3cvc1qA", "action":"indices:data/read/search", "request.name":"SearchRequest", "indices":["alias1"]}`
+
+| `tampered_request`
+| Logged when the {security-features} detect that the request has been tampered
+with. Typically relates to `search/scroll` requests when the scroll ID is
+believed to have been tampered with.
+| `{"type":"audit", "timestamp":"2019-11-27T22:00:00,947+0200", "node.id": "0RMNyghkQYCc_gVd1G6tZQ", "event.type": "rest", "event.action": "tampered_request", "origin.address":"[::1]:50543", "url.path":"/twitter/_async_search", "url.query":"pretty", "request.method":"POST", "request.id":"TqA9OisyQ8WTl1ivJUV1AA"}`
 |======
 
 [discrete]
 [[audit-event-attributes]]
 === Audit event attributes
 
-In 6.5.0, there is a new <<audit-log-output, `logfile` audit output>> format.
-This format also brings in a few changes for audit event attributes.
+The audit events are formatted as JSON documents, and each event is printed on a separate
+line in the audit log. The entries themselves do not contain an end-of-line delimiter.
+For more details, see <<audit-log-entry-format>>.
 
-The new format is output to the `<clustername>_audit.json` file.
-The audit entries are formatted as flat JSON documents (that is to say, no
-nested objects), one per line. Hence, the attribute names are JSON keys and they
-follow a dotted name syntax. Any attributes that lack a value (`null`) are not
-output.
-
-The following list shows attributes that are common to all audit events.
-Their names and values are analogous to those in the deprecated `logfile` or
-`index` output formats. However, it is expected that the formats will evolve
-independently during the 6.x releases, so it is advisable to follow the attribute
-descriptions for the format that you are using.
+The following list shows attributes that are common to all audit event types:
 
 `@timestamp`      ::    The time, in ISO9601 format, when the event occurred.
 `node.name`       ::    The name of the node. This can be changed
@@ -68,6 +151,30 @@ descriptions for the format that you are using.
 `host.ip`         ::    The bound IP address of the node, with which the node
                         can be communicated with.
 `host.name`       ::    The unresolved node's hostname.
+`event.type`      ::    The internal processing layer that generated the event:
+                        `rest`, `transport`, `ip_filter` or `security_config_change`.
+                        This is different from `origin.type` because a request
+                        originating from the REST API is translated to a number
+                        of transport messages, generating audit events with
+                        `origin.type: rest` and `event.type: transport`.
+`event.action`    ::    The type of event that occurred: `anonymous_access_denied`,
+                        `authentication_failed`, `authentication_success`,
+                        `realm_authentication_failed`, `access_denied`, `access_granted`,
+                        `connection_denied`, `connection_granted`, `tampered_request`,
+                        `run_as_denied`, or `run_as_granted`. In addition, if
+                        `event.type` equals `security_config_change`, the
+                        `event.action` attribute takes one of the following values:
+                        `put_user`, `change_password`, `put_role`, `put_role_mapping`,
+                        `change_enable_user`, `change_disable_user`, `put_privileges`,
+                        `create_apikey`, `delete_user`, `delete_role`,
+                        `delete_role_mapping`, `invalidate_apikeys` or `delete_privileges`.
+`request.id`      ::    A synthetic identifier that can be used to correlate the events
+                        associated with a particular REST request.
+
+In addition, all the events of types `rest`, `transport` and `ip_filter` (but not
+`security_config_change`) have the following extra attributes, which
+show more details about the requesting client:
+
 `origin.address`  ::    The source IP address of the request associated with
                         this event. This could be the address of the remote client,
                         the address of another cluster node, or the local node's
@@ -79,17 +186,6 @@ descriptions for the format that you are using.
                         `rest` (request originated from a REST API request),
                         `transport` (request was received on the transport channel),
                         or `local_node` (the local node issued the request).
-`event.type`      ::    The internal processing layer that generated the event:
-                        `rest`, `transport` or `ip_filter`.
-                        This is different from `origin.type` because a request
-                        originating from the REST API is translated to a number
-                        of transport messages, generating audit events with
-                        `origin.type: rest` and `event.type: transport`.
-`event.action`    ::    The type of event that occurred: `anonymous_access_denied`,
-                        `authentication_failed`, `authentication_success`,
-                        `realm_authentication_failed`, `access_denied`, `access_granted`,
-                        `connection_denied`, `connection_granted`, `tampered_request`,
-                        `run_as_denied`, or `run_as_granted`.
 `opaque_id`       ::    The value of the `X-Opaque-Id` HTTP header (if present) of
                         the request associated with this event. This header can
                         be used freely by the client to mark API calls, as it has
@@ -110,7 +206,7 @@ descriptions for the format that you are using.
 The events with `event.type` equal to `rest` have one of the following `event.action`
 attribute values: `authentication_success`, `anonymous_access_denied`, `authentication_failed`,
 `realm_authentication_failed`, `tampered_request` or `run_as_denied`.
-These event types also have the following extra attributes (in addition to the
+These events also have the following extra attributes (in addition to the
 common ones):
 
 `url.path`        ::    The path part of the URL (between the port and the query
@@ -122,8 +218,8 @@ common ones):
                         event. It is one of GET, POST, PUT, DELETE, OPTIONS,
                         HEAD, PATCH, TRACE and CONNECT.
 `request.body`    ::    The full content of the REST request associated with this
-                        event, if enabled. This contains the query body. The body
-                        is escaped according to the JSON RFC 4627.
+                        event, if enabled. This contains the HTTP request body.
+                        The body is escaped as a string value according to the JSON RFC 4627.
 
 ==== Audit event attributes of the transport event type
 
@@ -131,7 +227,7 @@ The events with `event.type` equal to `transport` have one of the following `eve
 attribute values: `authentication_success`, `anonymous_access_denied`, `authentication_failed`,
 `realm_authentication_failed`, `access_granted`, `access_denied`, `run_as_granted`,
 `run_as_denied`, or `tampered_request`.
-These event types also have the following extra attributes (in addition to the common
+These events also have the following extra attributes (in addition to the common
 ones):
 
 `action`              ::    The name of the transport action that was executed.
@@ -144,13 +240,79 @@ ones):
 
 The events with `event.type` equal to `ip_filter` have one of the following `event.action`
 attribute values: `connection_granted` or `connection_denied`.
-These event types also have the following extra attributes (in addition to the common
+These events also have the following extra attributes (in addition to the common
 ones):
 
 `transport_profile`   ::    The transport profile the request targeted.
 `rule`                ::    The <<ip-filtering, IP filtering>> rule that denied
                             the request.
 
+==== Audit event attributes of the security_config_change event type
+
+The events with the `event.type` attribute equal to `security_config_change` have one of the following
+`event.action` attribute values: `put_user`, `change_password`, `put_role`, `put_role_mapping`,
+`change_enable_user`, `change_disable_user`, `put_privileges`, `create_apikey`, `delete_user`,
+`delete_role`, `delete_role_mapping`, `invalidate_apikeys`, or `delete_privileges`.
+These events also have *one* of the following extra attributes (in addition to the common
+ones). The attribute's value is a nested JSON object:
+
+`put`                 ::    The object representation of the security config that
+                            is being created, or the overwrite of an existing config.
+                            It contains the config for a `user`, `role`, `role_mapping`, or
+                            for application `privileges`.
+`delete`              ::    The object representation of the security config that
+                            is being deleted. It can be the config for a `user`, `role`,
+                            `role_mapping` or for application `privileges`.
+`change`              ::    The object representation of the security config that
+                            is being changed. It can be the `password`, `enable` or `disable`,
+                            config object for native or built-in users.
+`create`              ::    The object representation of the new security config that is being
+                            created. This is currently only used for API keys auditing.
+                            If the API key is created using the
+                            <<security-api-create-api-key, create API key API>> it only contains
+                            an `apikey` config object. If the API key is created using the
+                            <<security-api-grant-api-key, grant API key API>> it also contains
+                            a `grant` config object.
+`invalidate`          ::    The object representation of the security configuration that is being invalidated.
+                            The only config that currently supports invalidation is `apikeys`, through
+                            the <<security-api-invalidate-api-key, invalidate API key API>>.
+
+The schemas of the security config objects mentioned above are as follows. They are very similar to the
+request bodies of the corresponding security APIs.
+
+`user`                ::     An object like `{"name": <string>, "enabled": <boolean>, "roles": <string_list>,
+                             "full_name": <string>, "email": <string>, "has_password": <boolean>,
+                             "metadata": <object>}`. The `full_name`, `email` and `metadata` fields are omitted if empty.
+
+`role`                ::     An object like `{"name": <string>, "role_descriptor": {"cluster": <string_list>, "global":
+                             {"application":{"manage":{<string>:<string_list>}}}, "indices": [
+                             {"names": <string_list>, "privileges": <string_list>, "field_security":
+                             {"grant": <string_list>, "except": <string_list>}, "query": <string>,
+                             "allow_restricted_indices": <boolean>}], "applications":[{"application": <string>,
+                             "privileges": <string_list>, "resources": <string_list>}], "run_as": <string_list>,
+                             "metadata": <object>}}`. The `global`, `field_security`, `except`, `query`,
+                             `allow_restricted_indices` and `metadata` fields are omitted if empty.
+
+`role_mapping`        ::     An object like `{"name": <string>, "roles": <string_list>, "role_templates": [{"template": <string>,
+                             "format": <string>}], "rules": <object>, "enabled": <boolean>, "metadata": <object>}`.
+                             The `roles` and `role_templates` fields are omitted if empty.
+                             The `rules` object has a recursively nested schema, identical to what is passed in
+                             the <<mapping-roles, API request for mapping roles>>.
+
+`privileges`          ::     An array of objects like `{"application": <string>, "name": <string>, "actions": <string_list>,
+                             "metadata": <object>}`.
+
+`password`            ::     A simple object like `{"user":{"name": <string>}}`.
+`enable`              ::     A simple object like `{"user":{"name": <string>}}`.
+`disable`             ::     A simple object like `{"user":{"name": <string>}}`.
+`apikey`              ::     An object like `{"name": <string>, "expiration": <string>, "role_descriptors" [<object>]}`.
+                             The `role_descriptors` objects have the same schema as the `role_descriptor` object that
+                             is part of the above `role` config object.
+`grant`               ::     An object like `{"type": <string>, "user": {"name": <string>, "has_password": <boolean>},
+                             "has_access_token": <boolean>}`
+`apikeys`             ::     An object like `{"ids": <string_list>, "name": <string>, "owned_by_authenticated_user":
+                             <boolean>, "user":{"name": <string>, "realm": <string>}}`.
+
 ==== Extra audit event attributes for specific events
 
 There are a few events that have some more attributes in addition to those
@@ -168,12 +330,12 @@ that have been previously described:
                             this instead denotes the name of the  _impersonated_ user.
                             If authenticated using an API key, this is
                             the name of the API key owner.
-  `user.realm`         ::   Name of the realm to which the _effective_ user 
+  `user.realm`         ::   Name of the realm to which the _effective_ user
                             belongs. If authenticated using an API key, this is
                             the name of the realm to which the API key owner belongs.
   `user.run_by.name`   ::   This attribute is present only if the request is
                             using the <<run-as-privilege, run as authorization functionality>>
-                            and denotes the name of the  _authenticated_ user, 
+                            and denotes the name of the  _authenticated_ user,
                             which is also known as the _impersonator_.
   `user.run_by.realm`  ::   Name of the realm to which the _authenticated_
                             (_impersonator_) user belongs.
@@ -188,7 +350,7 @@ that have been previously described:
 
 * `authentication_failed`:
   `user.name`          ::    The name of the user that failed authentication.
-                             If the request authentication token is invalid or 
+                             If the request authentication token is invalid or
                              unparsable, this information might be missing.
 
 * `realm_authentication_failed`:
@@ -216,7 +378,7 @@ that have been previously described:
                              this instead denotes the name of the  _impersonated_ user.
                              If authenticated using an API key, this is
                              the name of the API key owner.
-  `user.realm`         ::   Name of the realm to which the _effective_ user 
+  `user.realm`         ::   Name of the realm to which the _effective_ user
                             belongs. If authenticated using an API key, this is
                             the name of the realm to which the API key owner belongs.
   `user.run_by.name`   ::    This attribute is present only if the request is
@@ -233,320 +395,3 @@ that have been previously described:
                             This attribute is only provided for authentication using an API key.
   `api_key.name`       ::   API key name provided in the <<security-api-create-api-key,create API key>> request.
                             This attribute is only provided for authentication using an API key.
-
-
-[discrete]
-[[audit-event-attributes-deprecated-formats]]
-=== Audit event attributes for the deprecated formats
-
-The following table shows the common attributes that can be associated with
-every event, when it is output to the `<clustername>_access.log` file.
-
-.Common attributes
-[cols="2,7",options="header"]
-|======
-| Attribute           | Description
-| `@timestamp`        | When the event occurred.
-| `node_name`         | The name of the node.
-| `node_host_name`    | The hostname of the node.
-| `node_host_address` | The IP address of the node.
-| `layer`             | The layer from which this event originated: `rest`, `transport` or `ip_filter`
-| `event_type`        | The type of event that occurred: `anonymous_access_denied`,
-                        `authentication_failed`, `authentication_success`,
-                        `realm_authentication_failed`, `access_denied`, `access_granted`,
-                        `connection_denied`, `connection_granted`, `tampered_request`,
-                        `run_as_denied`, `run_as_granted`.
-|======
-
-These are positional attributes, which are printed at the beginning of each log line and
-are not adjoined by the attribute name.
-
-The attribute `origin_address` is also common to every audit event. It is always
-named, that is, it is not positional. It denotes the source IP address of the
-request associated with this event. This might be the address of the client, the
-address of another cluster node, or the local node's bound address (if the request
-originated locally). Unless the client connects directly to the cluster, the
-_client  address_ is the address of the first OSI layer 3 proxy in front of the
-cluster.
-
-In addition, every event might have the `opaque_id` attribute, with the value as
-it has been passed in by the `X-Opaque-Id` HTTP request header. This header can
-be used freely by the client to mark API calls, as it has no semantics in
-Elasticsearch. Every audit event, generated as part of handling a request thus
-marked, contains the `opaque_id` attribute.
-
-The following tables show the attributes that can are associated with each type
-of event, in addition to the common ones previously described:
-
-.REST anonymous_access_denied attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `uri`                | The REST endpoint URI.
-| `request_body`       | The body of the request, if enabled.
-|======
-
-.REST authentication_success attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `principal`          | The _effective_ (impersonated) username. Usually this is
-                         the same as the _authenticated_ username.
-| `run_by_principal`   | The _authenticated_ (impersonator) username.
-                         This attribute is present only if the request is
-                         using the <<run-as-privilege, run as authorization functionality>>.
-                         Otherwise, the _effective_ user is the same as the
-                         _authenticated_ one, which is indicated by the `principal`
-                         attribute.
-| `realm`              | The realm that authenticated the user.
-| `uri`                | The REST endpoint URI.
-| `params`             | The REST URI query parameters.
-| `request_body`       | The body of the request, if enabled.
-|======
-
-.REST authentication_failed attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `principal`          | The principal (username) that failed authentication.
-                         If the request's authentication token is invalid, this
-                         information might be missing.
-| `uri`                | The REST endpoint URI.
-| `request_body`       | The body of the request, if enabled.
-|======
-
-.REST realm_authentication_failed attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `realm`              | The realm that failed to authenticate the user.
-                         **A separate entry is logged for each consulted realm.**
-| `principal`          | The principal (username) that failed authentication.
-| `uri`                | The REST endpoint URI.
-| `request_body`       | The body of the request, if enabled.
-|======
-
-.REST tampered_request attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `uri`                | The REST endpoint URI.
-| `request_body`       | The body of the request, if enabled.
-|======
-
-.Transport anonymous_access_denied attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         pertains to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport authentication_success attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _effective_ (impersonated) username. Usually this is
-                         the same as the _authenticated_ username.
-| `run_by_principal`   | The _authenticated_ (impersonator) username.
-                         This attribute is present only if the request is
-                         using the <<run-as-privilege, run as authorization functionality>>.
-                         Otherwise, the _effective_ and the _authenticated_
-                         users are equivalent and are indicated by the
-                         `principal` attribute.
-| `realm`              | The realm that authenticated the user.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         pertains to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport authentication_failed attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _effective_ (impersonated) username. Usually this is
-                         the same as the _authenticated_ username. If the
-                         request's authentication token is invalid, this
-                         information might be missing.  
-| `run_by_principal`   | The _authenticated_ (impersonator) username.
-                         This attribute is present only if the request is
-                         using the <<run-as-privilege, run as authorization functionality>>.
-                         Otherwise, the _effective_ and the _authenticated_
-                         users are equivalent and are indicated by the
-                         `principal` attribute.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         pertains to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport realm_authentication_failed attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `realm`              | The realm that failed to authenticate the user.
-                         **A separate entry is logged for each consulted realm.**
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The principal (username) that failed authentication.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         pertains to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport access_granted attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _effective_ (impersonated) username for which
-                         authorization succeeded. Unless the request is using
-                         the <<run-as-privilege, run as authorization functionality>>,
-                         the _effective_ and _authenticated_ usernames are equivalent.
-| `realm`              | The realm name that `principal` belongs to.
-| `run_by_principal`   | The _authenticated_ (impersonator) username.
-                         This attribute is present only if the request is
-                         using the <<run-as-privilege, run as authorization functionality>>.
-                         Otherwise, the _effective_ and the _authenticated_
-                         usernames are equivalent and are indicated by the
-                         `principal` attribute.
-| `run_by_realm`       | The realm name that `run_by_principal` belongs to
-                         (when applicable).
-| `roles`              | The set of roles granting permissions.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         pertains to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport access_denied attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _effective_ (impersonated) username for which
-                         authorization failed. Unless the request is using
-                         the <<run-as-privilege, run as authorization functionality>>,
-                         the _effective_ and the _authenticated_ usernames are
-                         equivalent.
-| `realm`              | The realm name that `principal` belongs to.
-| `run_by_principal`   | The _authenticated_ (impersonator) username.
-                         This attribute is present only if the request is
-                         using the <<run-as-privilege, run as authorization functionality>>.
-                         Otherwise, the _effective_ and the _authenticated_
-                         usernames are equivalent and are indicated by the
-                         `principal` attribute.
-| `run_by_realm`       | The realm name that `run_by_principal` belongs to
-                         (when applicable).
-| `roles`              | The set of roles granting permissions.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         relates to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport run_as_granted attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _authenticated_ (_impersonator_) username for which
-                         the impersonation operation was granted.
-| `realm`              | The realm name that the _authenticated_ user belongs to.
-| `run_as_principal`   | The _impersonated_ username.
-| `run_as_realm`       | The realm name that the _impersonated_ username belongs to.
-| `roles`              | The set of roles granting permissions.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         relates to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport run_as_denied attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _authenticated_ (_impersonator_) username for which
-                         the impersonation operation was denied.
-| `realm`              | The realm name that the _authenticated_ user belongs to.
-| `run_as_principal`   | The _impersonated_ username.
-| `run_as_realm`       | The realm name that the _impersonated_ username belongs to.
-| `roles`              | The set of roles granting permissions.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         relates to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.Transport tampered_request attributes
-[cols="2,7",options="header"]
-|======
-| Attribute            | Description
-| `origin_type`        | Where the request originated: `rest` (request
-                         originated from a REST API request), `transport`
-                         (request was received on the transport channel), or
-                         `local_node` (the local node issued the request).
-| `principal`          | The _effective_ (impersonated) username. Unless the request
-                         is using the <<run-as-privilege, run as authorization functionality>>,
-                         the _effective_ and the _authenticated_ usernames are 
-                         equivalent. If the requests's authentication token is
-                         invalid, this information might be missing. 
-| `run_by_principal`   | The _authenticated_ (impersonator) username.
-                         This attribute is present only if the request is
-                         using the <<run-as-privilege, run as authorization functionality>>.
-                         Otherwise, the _effective_ and the _authenticated_ usernames
-                         are equivalent and are indicated by the `principal` attribute.
-| `action`             | The name of the action that was executed.
-| `indices`            | A comma-separated list of indices this request
-                         pertains to (when applicable).
-| `request`            | The type of request that was executed.
-|======
-
-.IP filter connection_granted attributes
-[cols="2,7",options="header"]
-|======
-| Attribute           | Description
-| `transport_profile` | The transport profile the request targeted.
-| `rule`              | The <<ip-filtering, IP filtering>> rule that granted
-                        the request.
-|======
-
-.IP filter connection_denied attributes
-[cols="2,7",options="header"]
-|======
-| Attribute           | Description
-| `transport_profile` | The transport profile the request targeted.
-| `rule`              | The <<ip-filtering, IP filtering>> rule that denied
-                        the request.
-|======

x-pack/docs/en/security/auditing/ignore-policy.asciidoc

@@ -0,0 +1,79 @@
+[role="xpack"]
+[[audit-log-ignore-policy]]
+=== Logfile audit events ignore policies
+
+The comprehensive audit trail is necessary to ensure accountability. It offers tremendous
+value during incident response and can even be required for demonstrating compliance.
+
+The drawback of an audited system is represented by the inevitable performance penalty incurred.
+In all truth, the audit trail spends _I/O ops_ that are not available anymore for the user's queries.
+Sometimes the verbosity of the audit trail may become a problem that the event type restrictions,
+<<audit-log-settings, defined by `include` and `exclude`>>, will not alleviate.
+
+*Audit events ignore policies* are a finer way to tune the verbosity of the audit trail.
+These policies define rules that match audit events which will be _ignored_ (read as: not printed).
+Rules match on the values of attributes of audit events and complement the `include` or `exclude` method.
+Imagine the corpus of audit events and the policies chopping off unwanted events.
+With a sole exception, all audit events are subject to the ignore policies.
+The exception are events of type `security_config_change`, which cannot be filtered out,
+unless excluded altogether.
+
+IMPORTANT: When utilizing audit events ignore policies you are acknowledging potential
+accountability gaps that could render illegitimate actions undetectable.
+Please take time to review these policies whenever your system architecture changes.
+
+A policy is a named set of filter rules. Each filter rule applies to a single event attribute,
+one of the `users`, `realms`, `roles` or `indices` attributes. The filter rule defines
+a list of <<regexp-syntax,Lucene regexp>>, *any* of which has to match the value of the audit
+event attribute for the rule to match.
+A policy matches an event if *all* the rules comprising it match the event.
+An audit event is ignored, therefore not printed, if it matches *any* policy. All other
+non-matching events are printed as usual.
+
+All policies are defined under the `xpack.security.audit.logfile.events.ignore_filters`
+settings namespace. For example, the following policy named _example1_ matches
+events from the _kibana_system_ or _admin_user_ principals that operate over indices of the
+wildcard form _app-logs*_:
+
+[source,yaml]
+----------------------------
+xpack.security.audit.logfile.events.ignore_filters:
+  example1:
+    users: ["kibana_system", "admin_user"]
+    indices: ["app-logs*"]
+----------------------------
+
+An audit event generated by the _kibana_system_ user and operating over multiple indices
+, some of which do not match the indices wildcard, will not match.
+As expected, operations generated by all other users (even operating only on indices that
+match the _indices_ filter) will not match this policy either.
+
+Audit events of different types may have <<audit-event-attributes, different attributes>>.
+If an event does not contain an attribute for which some policy defines filters, the
+event will not match the policy.
+For example, the following policy will never match `authentication_success` or
+`authentication_failed` events, irrespective of the user's roles, because these
+event schemas do not contain the `role` attribute:
+
+[source,yaml]
+----------------------------
+xpack.security.audit.logfile.events.ignore_filters:
+  example2:
+    roles: ["admin", "ops_admin_*"]
+----------------------------
+
+Likewise, any events of users with multiple roles, some of which do not match the
+regexps will not match this policy.
+
+For completeness, although practical use cases should be sparse, a filter can match
+a missing attribute of an event, using the empty string ("") or the empty list ([]).
+For example, the following policy will match events that do not have the `indices`
+attribute (`anonymous_access_denied`, `authentication_success` and other types) as well
+as events over the _next_ index.
+
+[source,yaml]
+----------------------------
+xpack.security.audit.logfile.events.ignore_filters:
+  example3:
+    indices: ["next", ""]
+----------------------------

x-pack/docs/en/security/auditing/index.asciidoc

@@ -4,4 +4,6 @@ include::event-types.asciidoc[]
 
 include::output-logfile.asciidoc[]
 
+include::ignore-policy.asciidoc[]
+
 include::auditing-search-queries.asciidoc[]

x-pack/docs/en/security/auditing/output-logfile.asciidoc

@@ -2,87 +2,39 @@
 [[audit-log-output]]
 === Logfile audit output
 
-The `logfile` audit output is the default output for auditing. It writes data to
-the `<clustername>_audit.json` file in the logs directory. To maintain
-compatibility with releases prior to 6.5.0, a `<clustername>_access.log` file
-is also generated. They differ in the output format but the contents
-are similar. For systems that are not ingesting the audit file for search or
-analytics it is strongly recommended to keep only the newer format.
-
-To turn off the deprecated output format, you can disable the logger in the
-`log4j2.properties` file:
-
-[source, properties]
---------------------------------------------------
-# change info to off
-# logger.xpack_security_audit_deprecated_logfile.level = info
-logger.xpack_security_audit_deprecated_logfile.level = off
---------------------------------------------------
-
-Alternatively, use the
-<<cluster-update-settings,cluster update settings API>> to dynamically
-configure the logger:
-
-[source,console]
---------------------------------------------------
-PUT /_cluster/settings
-{
-  "persistent": {
-    "logger.org.elasticsearch.xpack.security.audit.logfile.DeprecatedLoggingAuditTrail": "off"
-  }
-}
---------------------------------------------------
+The `logfile` audit output is the only output for auditing. It writes data to
+the `<clustername>_audit.json` file in the logs directory.
 
 NOTE: If you overwrite the `log4j2.properties` and do not specify appenders for
 any of the audit trails, audit events are forwarded to the root appender, which
 by default points to the `elasticsearch.log` file.
 
-
 [discrete]
 [[audit-log-entry-format]]
 === Log entry format
 
-The log entries in the `<clustername>_audit.json` file have the following format:
-
-- Each log entry is a one line JSON document and each one is printed on a separate line.
-- The fields of a log entry are ordered. However, if a field does not have a value it
-  will not be printed. The precise line pattern, together with the complete field
-  order, are specified in the `log4j2.properties` config file.
-- The log entry does not contain nested inner JSON objects, i.e. the doc is flat.
-- The field names follow a dotted notation to flatten inner objects.
-- A field's value can be a string, a number or an array of strings.
-- A field's value, a request body as well, will be escaped as per the JSON RFC 4627.
+The audit events are formatted as JSON documents, and each event is printed on a separate
+line in the `<clustername>_audit.json` file. The entries themselves do not contain the
+end-of-line delimiter.
+The audit event JSON format is somewhat particular, as *most* fields follow a dotted
+name syntax, are ordered, and contain non-null string values. This format creates a
+structured columnar aspect, similar to a CSV, that can be more easily inspected visually
+(compared to an equivalent nested JSON document).
+
+There are however a few attributes that are exceptions to the above format. The `put`,
+`delete`, `change`, `create` and `invalidate` attributes, which are only present for
+events with the `event.type: "security_config_change"` attribute, contain the *nested JSON*
+representation of the security change taking effect. The contents of the security config change
+are hence not displayed as top-level dot-named fields in the audit event document. That's because
+the fields are specific to the particular kind of security change and do not show up in
+any other audit events. The benefits of a columnar format are therefore much more limited; the
+space-saving benefits of the nested structure is the favoured trade-off in this case.
+
+When the `request.body` attribute is present (see <<auditing-search-queries>>), it contains a
+string value containing the full HTTP request body, escaped as per the JSON RFC 4677.
 
 There is a list of <<audit-event-types, audit event types>> specifying the
-set of fields for each sog entry type.
-
-[discrete]
-[[deprecated-audit-log-entry-format]]
-=== Deprecated log entry format
-
-The log entries in the `<clustername>_access.log` file have the following format:
-
-[source,txt]
-----------------------------------------------------------------------------
-[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>
-----------------------------------------------------------------------------
-
-`<timestamp>`       ::      When the event occurred. You can configure the
-                            timestamp format in `log4j2.properties`.
-`<local_node_info>` ::      Information about the local node that generated
-                            the log entry. You can control what node information
-                            is included by configuring the
-                            <<node-audit-settings,local node info settings>>.
-`<layer>`           ::      The layer from which this event originated:
-                            `rest`, `transport` or `ip_filter`.
-`<entry_type>`      ::       The type of event that occurred: `anonymous_access_denied`,
-                            `authentication_failed`, `access_denied`, `access_granted`,
-                            `connection_granted`, `connection_denied`.
-`<attribute_list>`  ::      A comma-separated list of key-value pairs that contain
-                            data pertaining to the event. Formatted as
-                            `attr1=[val1], attr2=[val2]`. See <<audit-event-attributes,
-                            Audit Entry Attributes>> for the attributes that can be included
-                            for each type of event.
+set of fields, as well as examples, for each entry type.
 
 [discrete]
 [[audit-log-settings]]
@@ -93,94 +45,14 @@ controlled using settings in the `elasticsearch.yml` file. See
 <<event-audit-settings>> and
 <<node-audit-settings>>.
 
-IMPORTANT: No filtering is performed when auditing, so sensitive data may be
-audited in plain text when including the request body in audit events.
+IMPORTANT: Be advised that *sensitive data may be audited in plain text* when including
+the request body in audit events, even though all the security APIs, such as those that
+change the user’s password, have the credentials filtered out when audited.
 
-[[logging-file]]
 You can also configure how the logfile is written in the `log4j2.properties`
-file located in `ES_PATH_CONF`. By default, audit information is appended to the
+file located in `ES_PATH_CONF` (or check out the relevant portion of
+the https://github.com/elastic/elasticsearch/blob/{branch}/x-pack/plugin/core/src/main/config/log4j2.properties[log4j2.properties in the sources]).
+By default, audit information is appended to the
 `<clustername>_audit.json` file located in the standard Elasticsearch `logs` directory
-(typically located at `$ES_HOME/logs`). The file rolls over on a daily basis.
-The deprecated logfile audit format (`<clustername>_access.log`) can be disabled
-from the same `log4j2.properties` file (hint: look for the comment
-instructing to set the log level to `off`). The deprecated format is a duplication
-of information that is in place to assure backwards compatibility. If you are
-not strict about the audit format it is strongly recommended to only use the
-`<clustername>_audit.json` log appender.
-
-[discrete]
-[[audit-log-ignore-policy]]
-=== Logfile audit events ignore policies
-
-The comprehensive audit trail is necessary to ensure accountability. It offers tremendous
-value during incident response and can even be required for demonstrating compliance.
-
-The drawback of an audited system is represented by the inevitable performance penalty incurred.
-In all truth, the audit trail spends _I/O ops_ that are not available anymore for the user's queries.
-Sometimes the verbosity of the audit trail may become a problem that the event type restrictions,
-<<audit-log-settings, defined by `include` and `exclude`>>, will not alleviate.
-
-*Audit events ignore policies* are a finer way to tune the verbosity of the audit trail.
-These policies define rules that match audit events which will be _ignored_ (read as: not printed).
-Rules match on the values of attributes of audit events and complement the <<audit-log-settings, include/exclude>> method.
-Imagine the corpus of audit events and the policies chopping off unwanted events.
-
-IMPORTANT: When utilizing audit events ignore policies you are acknowledging potential
-accountability gaps that could render illegitimate actions undetectable.
-Please take time to review these policies whenever your system architecture changes.
-
-A policy is a named set of filter rules. Each filter rule applies to a single event attribute,
-one of the `users`, `realms`, `roles` or `indices` attributes. The filter rule defines
-a list of <<regexp-syntax,Lucene regexp>>, *any* of which has to match the value of the audit
-event attribute for the rule to match.
-A policy matches an event if *all* the rules comprising it match the event.
-An audit event is ignored, therefore not printed, if it matches *any* policy. All other
-non-matching events are printed as usual.
-
-All policies are defined under the `xpack.security.audit.logfile.events.ignore_filters`
-settings namespace. For example, the following policy named _example1_ matches
-events from the _kibana_system_ or _admin_user_ principals **and** operating over indices of the
-wildcard form _app-logs*_:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.logfile.events.ignore_filters:
-  example1:
-    users: ["kibana_system", "admin_user"]
-    indices: ["app-logs*"]
-----------------------------
-
-An audit event generated by the _kibana_system_ user and operating over multiple indices
-, some of which do not match the indices wildcard, will not match.
-As expected, operations generated by all other users (even operating only on indices that
-match the _indices_ filter) will not match this policy either.
-
-Audit events of different types may have <<audit-event-attributes, different attributes>>.
-If an event does not contain an attribute for which some policy defines filters, the
-event will not match the policy.
-For example, the following policy named _example2_, will never match `authentication_success` or
-`authentication_failed` events, irrespective of the user's roles, because these
-event schemas do not contain the `role` attribute:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.logfile.events.ignore_filters:
-  example2:
-    roles: ["admin", "ops_admin_*"]
-----------------------------
-
-Likewise, any events of users with multiple roles, some of which do not match the
-regexps will not match this policy.
-
-For completeness, although practical use cases should be sparse, a filter can match
-a missing attribute of an event, using the empty string ("") or the empty list ([]).
-For example, the following policy will match events that do not have the `indices`
-attribute (`anonymous_access_denied`, `authentication_success` and other types) as well
-as events over the _next_ index.
-
-[source,yaml]
-----------------------------
-xpack.security.audit.logfile.events.ignore_filters:
-  example3:
-    indices: ["next", ""]
-----------------------------
+(typically located at `$ES_HOME/logs`).
+The file is also rotated and archived daily or upon reaching the 1GB file size limit.