[DOCS] Rework manual security docs for 8.0+ (#80285)

* [DOCS] Remove sentence about security being disabled by default

* Updating introduction

* Remove minimal security page

* Clarify configuring security before starting ES

* Clarifications

* Remove old file

* Add set passwords page

* Update change passwords page, clarify TLS adjustments, and other edits

* Update test

* Minor clarification to intro text

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

docs/reference/redirects.asciidoc

@@ -158,10 +158,21 @@ See <<grok,Grokking grok>>.
 
 // [START] Security redirects
 
+[role="exclude",id="security-minimal-setup"]
+=== Set up minimal security for {es}
+
+Refer to <<manually-configure-security,Manually configure security for the {stack}>>.
+
 [role="exclude",id="get-started-users"]
-=== Create users
+==== Create users
+
+Refer to the <<reset-password,`elasticsearch-reset-password`>> tool for resetting
+passwords of built-in users.
+
+[role="exclude",id="add-built-in-users"]
+==== Configure {kib} to connect to {es} with a password
 
-See <<security-create-builtin-users,Create passwords for built-in users>>.
+Refer to <<configuring-stack-security,Start the {stack} with security enabled>>.
 
 [role="exclude",id="encrypting-communications-certificates"]
 === Generate certificates

x-pack/docs/en/security/configuring-es.asciidoc

@@ -1,140 +0,0 @@
-[role="xpack"]
-[[configuring-security]]
-== Configuring security in {es}
-++++
-<titleabbrev>Configuring security</titleabbrev>
-++++
-
-The {es} {security-features} enable you to easily secure a cluster. You can
-password-protect your data as well as implement more advanced security measures
-such as encrypting communications, role-based access control, IP filtering, and
-auditing.
-
-. Verify that you are using a license that includes the specific
-{security-features} you want.
-+
---
-For more information, see https://www.elastic.co/subscriptions and
-{kibana-ref}/managing-licenses.html[License management].
---
-
-. Verify that the `xpack.security.enabled` setting is `true` on each node in
-your cluster. If you are using basic or trial licenses, the default value is `false`.
-For more information, see <<security-settings>>.
-
-. If you plan to run {es} in a Federal Information Processing Standard (FIPS)
-140-2 enabled JVM, see <<fips-140-compliance>>.
-
-. <<encrypt-internode-communication,Configure Transport Layer Security (TLS/SSL) for internode-communication>>.
-+
---
-NOTE: This requirement applies to clusters with more than one node and to
-clusters with a single node that listens on an external interface. Single-node
-clusters that use a loopback interface do not have this requirement. For more
-information, see
-<<configuring-stack-security>>.
-
---
-
-. If it is not already running, start {es}.
-
-. Set the passwords for all built-in users.
-+
---
-The {es} {security-features} provide
-<<built-in-users,built-in users>> to
-help you get up and running. The +elasticsearch-setup-passwords+ command is the
-simplest method to set the built-in users' passwords for the first time.
-
-For example, you can run the command in an "interactive" mode, which prompts you
-to enter new passwords for the built-in users:
-
-[source,shell]
---------------------------------------------------
-bin/elasticsearch-setup-passwords interactive
---------------------------------------------------
-
-For more information about the command options, see <<setup-passwords>>.
-
-IMPORTANT: The `elasticsearch-setup-passwords` command uses a transient bootstrap
-password that is no longer valid after the command runs successfully. You cannot
-run the `elasticsearch-setup-passwords` command a second time. Instead, you can
-update passwords from the **Management > Users** UI in {kib} or use the security
-user API.
-
---
-
-. Choose which types of realms you want to use to authenticate users.
-+
---
-TIP: The types of authentication realms that you can enable varies according to
-your subscription. For more information, see https://www.elastic.co/subscriptions.
-
---
-** <<active-directory-realm,Active Directory realms>>
-** <<file-realm,File realms>>
-** <<kerberos-realm,Kerberos realms>>
-** <<ldap-realm,LDAP realms>>
-** <<native-realm,Native realms>>
-** <<pki-realm,PKI realms>>
-** <<saml-realm,SAML realms>>
-
-. Set up roles and users to control access to {es}.
-+
---
-For example, to grant _John Doe_ full access to all indices that match
-the pattern `events*` and enable them to create visualizations and dashboards
-for those indices in {kib}, you could create an `events_admin` role
-and assign the role to a new `johndoe` user.
-
-[source,shell]
-----------------------------------------------------------
-curl -XPOST -u elastic 'localhost:9200/_security/role/events_admin' -H "Content-Type: application/json" -d '{
-  "indices" : [
-    {
-      "names" : [ "events*" ],
-      "privileges" : [ "all" ]
-    },
-    {
-      "names" : [ ".kibana*" ],
-      "privileges" : [ "manage", "read", "index" ]
-    }
-  ]
-}'
-
-curl -XPOST -u elastic 'localhost:9200/_security/user/johndoe' -H "Content-Type: application/json" -d '{
-  "password" : "userpassword",
-  "full_name" : "John Doe",
-  "email" : "john.doe@anony.mous",
-  "roles" : [ "events_admin" ]
-}'
-----------------------------------------------------------
-// NOTCONSOLE
---
-
-. [[enable-auditing]](Optional) Enable auditing to keep track of attempted and
-successful interactions with your {es} cluster:
-+
---
-TIP: Audit logging is available with specific subscriptions. For more
-information, see https://www.elastic.co/subscriptions.
-
-.. Add the following setting to `elasticsearch.yml` on all nodes in your cluster:
-+
-[source,yaml]
-----------------------------
-xpack.security.audit.enabled: true
-----------------------------
-+
-For more information, see <<enable-audit-logging>> and <<auditing-settings>>.
-
-.. Restart {es}.
-
-Events are logged to a dedicated `<clustername>_audit.json` file in
-`ES_HOME/logs`, on each cluster node.
---
-
-To walk through the configuration of {security-features} in {es}, {kib}, {ls}, and {metricbeat}, see <<security-getting-started>>.
-
-include::reference/files.asciidoc[]
-include::fips-140-compliance.asciidoc[]

x-pack/docs/en/security/securing-communications/change-passwords-native-users.asciidoc

@@ -0,0 +1,48 @@
+[[change-passwords-native-users]]
+=== Setting passwords for native and built-in users
+
+After you implement security, you might need or want to change passwords for
+different users. You can use the <<reset-password,`elasticsearch-reset-password`>>
+tool or the <<security-api-change-password,change passwords API>> to change
+passwords for native users and <<built-in-users,built-in users>>, such as the
+`elastic` or `kibana_system` users. 
+
+For example, the following command changes the password for a user with the
+username `user1` to an auto-generated value, and prints the new password
+to the terminal:
+
+[source,shell]
+----
+bin/elasticsearch-reset-password -u user1
+----
+
+To explicitly set a password for a user, include the `-i` parameter with the 
+intended password.
+
+[source,shell]
+----
+bin/elasticsearch-reset-password -u user1 -i <password>
+----
+
+If you're working in {kib} or don't have command-line access, you can use the 
+change passwords API to change a user's password:
+
+////
+[source,console]
+----
+POST _security/user/user1?refresh=true
+{
+  "password" : "l0ng-r4nd0m-p@ssw0rd",
+  "roles": ["admin"]
+}
+----
+////
+
+[source,console]
+----
+POST /_security/user/user1/_password
+{
+  "password" : "new-test-password"
+}
+----
+//TEST[continued]

x-pack/docs/en/security/securing-communications/security-basic-setup.asciidoc

@@ -4,9 +4,11 @@
 <titleabbrev>Set up basic security</titleabbrev>
 ++++
 
-After adding password protection in the <<security-minimal-setup,minimal security configuration>>, you'll need to configure Transport Layer Security
-(TLS). The transport layer handles all internal communication between nodes in
-your cluster.
+When you start {es} for the first time, passwords are generated for the `elastic`
+user and TLS is automatically configured for you. If you configure security 
+manually _before_ starting your {es} nodes, the auto-configuration process will 
+respect your security configuration. You can adjust your TLS configuration at 
+any time, such as <<update-node-certs,updating node certificates>>.
 
 IMPORTANT: If your cluster has multiple nodes, then you must configure
 TLS between nodes. <<dev-vs-prod-mode,Production mode>> clusters will not start
@@ -42,15 +44,6 @@ this scenario we're applying TLS to the {es} transport port. In
 HTTP port.
 ****
 
-[[basic-setup-prerequisites]]
-==== Prerequisites
-
-Complete the steps in <<security-minimal-setup,Minimal security for the Elastic Stack>> to enable {es} security features on every node in your cluster. You can
-then encrypt communications between your nodes with TLS.
-
-NOTE: You only need to create passwords for the built-in users one time for the
-entire cluster.
-
 [[generate-certificates]]
 ==== Generate the certificate authority
 
@@ -72,8 +65,8 @@ For the transport layer, we recommend using a separate, dedicated CA instead
 of an existing, possibly shared CA so that node membership is tightly controlled. Use the `elasticsearch-certutil` tool to
 generate a CA for your cluster.
 
-. On any single node, use the `elasticsearch-certutil` tool to generate a CA for
-your cluster.
+. Before starting {es}, use the `elasticsearch-certutil` tool on any single node
+to generate a CA for your cluster.
 +
 [source,shell]
 ----
@@ -181,7 +174,7 @@ matches the DNS or IP address. See the
 
 . Complete the previous steps for each node in your cluster.
 
-. On *every* node in your cluster, restart {es}. The method for
+. On *every* node in your cluster, start {es}. The method for
 <<starting-elasticsearch,starting>> and <<starting-elasticsearch,stopping>> {es}
 varies depending on how you installed it.
 +

x-pack/docs/en/security/securing-communications/security-minimal-setup.asciidoc

@@ -1,175 +0,0 @@
-[[security-minimal-setup]]
-=== Set up minimal security for {es}
-++++
-<titleabbrev>Set up minimal security</titleabbrev>
-++++
-
-You enable the Elasticsearch security features and then create
-passwords for built-in users. You can add more users later, but using the
-built-in users simplifies the process of enabling security for your
-cluster.
-
-include::../security-manual-configuration.asciidoc[tag=minimal-security-note]
-
-==== Prerequisites
-
-. Install and configure {es} and {kib}. See https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html[Getting started with the Elastic Stack].
-
-. Verify that you are using a license that includes the specific security
-features you want.
-+
-The basic license includes minimal security settings for the Elastic Stack, so
-you can just download the distribution and get to work. You can also enable a
-free trial license to access all features of the Elastic Stack. See https://www.elastic.co/subscriptions[subscriptions] and https://www.elastic.co/guide/en/kibana/current/managing-licenses.html[license management].
-
-==== Enable {es} security features
-
-When you use the basic license, the {es} security features are disabled by
-default. Enabling the {es} security features enables basic authentication so
-that you can run a local cluster with username and password authentication.
-
-. On *every* node in your cluster, stop both {kib} and {es} if they are running.
-
-. On *every* node in your cluster, add the `xpack.security.enabled` setting to
-the `$ES_PATH_CONF/elasticsearch.yml` file and set the value to `true`:
-+
-[source,yaml]
-----
-xpack.security.enabled: true
-----
-+
-NOTE: The `$ES_PATH_CONF` variable is the path for the {es}
-configuration files. If you installed {es} using archive distributions
-(`zip` or `tar.gz`), the variable defaults to `$ES_HOME/config`. If you used
-package distributions (Debian or RPM), the variable defaults to `/etc/elasticsearch`.
-
-. If your cluster has a single node, add the `discovery.type` setting in the
-`$ES_PATH_CONF/elasticsearch.yml` file and set the value to `single-node`. This
-setting ensures that your node does not inadvertently connect to other clusters
-that might be running on your network.
-+
-[source,yaml]
-----
-discovery.type: single-node
-----
-
-[[security-create-builtin-users]]
-==== Create passwords for built-in users
-
-To communicate with the cluster, you must configure a username for the built-in
-users. Unless you enable anonymous access, all requests that don’t include a
-user name and password are rejected.
-
-NOTE: You only need to set passwords for the `elastic` and `kibana_system` users
-when enabling minimal or basic security.
-
-. On *every* node in your cluster, start {es}. For example, if you installed 
-{es} with a `.tar.gz` package, run the following command from the `ES_HOME`
-directory:
-+
-[source,shell]
-----
-./bin/elasticsearch
-----
-
-. In another terminal window, set the passwords for the built-in users by
-running the <<setup-passwords,`elasticsearch-setup-passwords`>> utility.
-+
-IMPORTANT: You can run the `elasticsearch-setup-passwords` utility
-against any node in your cluster. However, you should only run this utility *one
-time* for the entire cluster.
-+
-Using the `auto` parameter
-outputs randomly-generated passwords to the console that you can change later
-if necessary:
-+
-[source,shell]
-----
-./bin/elasticsearch-setup-passwords auto
-----
-+
-If you want to use your own passwords, run the command with the
-`interactive` parameter instead of the `auto` parameter. Using this mode
-steps you through password configuration for all of the built-in users.
-+
-[source,shell]
-----
-./bin/elasticsearch-setup-passwords interactive
-----
-
-. Save the generated passwords. You'll need them to add the built-in user to
-{kib}.
-
-WARNING: After you set a password for the `elastic` user, you cannot run the
-`elasticsearch-setup-passwords` command a second time.
-
-*Next*: <<add-built-in-users,Configure {kib} to connect to {es} with a password>>
-
-[[add-built-in-users]]
-==== Configure {kib} to connect to {es} with a password
-
-When the {es} security features are enabled, users must log in to {kib} with a
-valid username and password.
-
-You'll configure {kib} to use the built-in `kibana_system` user and the password that
-you created earlier. {kib} performs some background tasks that require use of the
-`kibana_system` user.
-
-This account is not meant for individual users and does not have permission to log in
-to {kib} from a browser. Instead, you'll log in to {kib} as the `elastic` superuser. 
-
-. Add the `elasticsearch.username` setting to the `KIB_PATH_CONF/kibana.yml`
-file and set the value to the `kibana_system` user:
-+
-[source,yaml]
-----
-elasticsearch.username: "kibana_system"
-----
-+
-NOTE: The `KIB_PATH_CONF` variable is the path for the {kib}
-configuration files. If you installed {kib} using archive distributions
-(`zip` or `tar.gz`), the variable defaults to `KIB_HOME/config`. If you used
-package distributions (Debian or RPM), the variable defaults to `/etc/kibana`.
-
-. From the directory where you installed {kib}, run the following commands
-to create the {kib} keystore and add the secure settings:
-
-   a. Create the {kib} keystore:
-+
-[source,shell]
-----
-./bin/kibana-keystore create
-----
-
-   b. Add the password for the `kibana_system` user to the {kib} keystore:
-+
-[source,shell]
-----
-./bin/kibana-keystore add elasticsearch.password
-----
-+
-When prompted, enter the password for the `kibana_system` user.
-
-. Restart {kib}. For example, if you installed {kib} with a `.tar.gz` package, run the following command from the {kib} directory:
-+
-[source,shell]
-----
-./bin/kibana
-----
-
-. Log in to {kib} as the `elastic` user. Use this superuser account to 
-{kibana-ref}/tutorial-secure-access-to-kibana.html[manage spaces, create new users, and assign roles]. If you're running {kib} locally, go to `http://localhost:5601` to view the login page.
-
-[[minimal-security-whatsnext]]
-==== What's next?
-
-Congratulations! You enabled password protection for your local cluster to
-prevent unauthorized access. You can log in to {kib} securely as the `elastic`
-user and create additional users and roles. If you're running a <<single-node-discovery,single-node cluster>>, then you can stop here.
-
-If your cluster has multiple nodes, then you must configure Transport Layer
-Security (TLS) between nodes. <<dev-vs-prod-mode,Production mode>> clusters
-will not start if you do not enable TLS.
-
-<<security-basic-setup,Set up basic security for the Elastic Stack>> to
-secure all internal communication between nodes in your cluster.

x-pack/docs/en/security/security-manual-configuration.asciidoc

@@ -5,51 +5,30 @@
 ++++
 
 Security needs vary depending on whether you're developing locally on your
-laptop or securing all communications in a production environment. Because
-security needs vary, the following scenarios provide options for configuring
-the Elastic Stack.
+laptop or securing all communications in a production environment. Regardless
+of where you're deploying the {stack} ("ELK"), running a secure cluster is
+incredibly important to protect your data. That's why security is
+<<configuring-stack-security,enabled and configured by default>> in {es} 8.0
+and later.
 
-Each subsequent scenario builds on the previous one so that you can add
-additional security by building on the existing layer.
+If you want to use your own Certificate Authority (CA) or would rather manually
+configure security, the following scenarios provide steps for configuring TLS
+on the transport layer, plus securing HTTPS traffic if you want it.
 
-These scenarios don't cover every situation, but provide a framework for
-securing {es} and the Elastic Stack based on typical use cases.
-
-NOTE: If you want to enable security by default when you start {es} for the
-first time without manually configuring security, refer to
-<<configuring-stack-security,Starting the Elastic Stack with security enabled>>.
+If you configure security manually _before_ starting your {es} nodes, the
+auto-configuration process will respect your security configuration. You can
+adjust your TLS configuration at any time, such as 
+<<update-node-certs,updating node certificates>>.
 
 image::images/elastic-security-overview.png[Elastic Security layers]
 
-
-[discrete]
-[[security-minimal-overview]]
-=== Minimal security ({es} Development)
-
-If you want to set up {es} on your laptop and start developing, this scenario
-is for you. This configuration prevents unauthorized access to your local
-cluster by setting up passwords for the built-in users. You also configure
-password authentication for {kib}.
-
-// tag::minimal-security-note[]
-IMPORTANT: The minimal security scenario is not sufficient for
-<<dev-vs-prod-mode,production mode>> clusters. If your cluster has multiple
-nodes, you must enable minimal security and then
-<<security-basic-setup,configure Transport Layer Security (TLS)>> between
-nodes.
-
-// end::minimal-security-note[]
-
-<<security-minimal-setup,Set up minimal security>>
-
 [discrete]
 [[security-basic-overview]]
-=== Basic security ({es} Production)
+=== Basic security ({es} + {kib})
 
-This scenario builds on the minimal security requirements by adding transport
-Layer Security (TLS) for communication between nodes. This additional layer
-requires that nodes verify security certificates, which prevents unauthorized
-nodes from joining your {es} cluster.
+This scenario configures TLS for communication between
+nodes. This security layer requires that nodes verify security certificates,
+which prevents unauthorized nodes from joining your {es} cluster.
 
 Your external HTTP traffic between {es} and {kib} won't be encrypted, but
 internode communication will be secured.
@@ -58,7 +37,7 @@ internode communication will be secured.
 
 [discrete]
 [[security-basic-https-overview]]
-=== Basic security plus secured HTTPS traffic (Elastic Stack)
+=== Basic security plus secured HTTPS traffic ({stack})
 
 This scenario builds on the one for basic security and secures all HTTP
 traffic with TLS. In addition to configuring TLS on the transport interface of
@@ -75,12 +54,12 @@ cluster are secure.
 
 <<security-basic-setup-https,Set up basic security plus HTTPS traffic>>
 
-include::securing-communications/security-minimal-setup.asciidoc[]
-
 include::securing-communications/security-basic-setup.asciidoc[]
 
 include::securing-communications/security-basic-setup-https.asciidoc[]
 
+include::securing-communications/change-passwords-native-users.asciidoc[]
+
 include::securing-communications/enabling-cipher-suites.asciidoc[]
 
 include::securing-communications/tls-versions-jdk.asciidoc[]