Ingest download databases docs (#111688)

Co-authored-by: Joe Gallo <joegallo@gmail.com>

docs/reference/ingest/apis/delete-geoip-database.asciidoc

@@ -0,0 +1,55 @@
+[[delete-geoip-database-api]]
+=== Delete geoip database configuration  API
+++++
+<titleabbrev>Delete geoip database configuration</titleabbrev>
+++++
+
+Deletes a geoip database configuration.
+
+[source,console]
+----
+DELETE /_ingest/geoip/database/my-database-id
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
+
+
+[[delete-geoip-database-api-request]]
+==== {api-request-title}
+
+`DELETE /_ingest/geoip/database/<database>`
+
+[[delete-geoip-database-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the
+`manage` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+[[delete-geoip-database-api-path-params]]
+==== {api-path-parms-title}
+
+`<database>`::
++
+--
+(Required, string) Database configuration ID used to limit the request.
+
+--
+
+
+[[delete-geoip-database-api-query-params]]
+==== {api-query-parms-title}
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
+
+
+[[delete-geoip-database-api-example]]
+==== {api-examples-title}
+
+
+[[delete-geoip-database-api-specific-ex]]
+===== Delete a specific geoip database configuration
+
+[source,console]
+----
+DELETE /_ingest/geoip/database/example-database-id
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]

docs/reference/ingest/apis/delete-pipeline.asciidoc

@@ -62,7 +62,7 @@ use a value of `*`.
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
 
 
-[[delete-pipeline-api-api-example]]
+[[delete-pipeline-api-example]]
 ==== {api-examples-title}
 
 

docs/reference/ingest/apis/get-geoip-database.asciidoc

@@ -0,0 +1,80 @@
+[[get-geoip-database-api]]
+=== Get geoip database configuration API
+++++
+<titleabbrev>Get geoip database configuration</titleabbrev>
+++++
+
+Returns information about one or more geoip database configurations.
+
+[source,console]
+----
+GET /_ingest/geoip/database/my-database-id
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
+
+
+
+[[get-geoip-database-api-request]]
+==== {api-request-title}
+
+`GET /_ingest/geoip/database/<database>`
+
+`GET /_ingest/geoip/database`
+
+[[get-geoip-database-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the
+ `manage` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+[[get-geoip-database-api-path-params]]
+==== {api-path-parms-title}
+
+`<database>`::
+(Optional, string)
+Comma-separated list of database configuration IDs to retrieve. Wildcard (`*`) expressions are
+supported.
++
+To get all database configurations, omit this parameter or use `*`.
+
+
+[[get-geoip-database-api-query-params]]
+==== {api-query-parms-title}
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
+
+
+[[get-geoip-database-api-example]]
+==== {api-examples-title}
+
+
+[[get-geoip-database-api-specific-ex]]
+===== Get information for a specific geoip database configuration
+
+[source,console]
+----
+GET /_ingest/geoip/database/my-database-id
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
+
+The API returns the following response:
+
+[source,console-result]
+----
+{
+  "databases" : [
+    {
+      "id" : "my-database-id",
+      "version" : 1,
+      "modified_date_millis" : 1723040276114,
+      "database" : {
+        "name" : "GeoIP2-Domain",
+        "maxmind" : {
+          "account_id" : "1234567"
+        }
+      }
+    }
+  ]
+}
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]

docs/reference/ingest/apis/get-pipeline.asciidoc

@@ -65,7 +65,7 @@ To get all ingest pipelines, omit this parameter or use `*`.
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
 
-[[get-pipeline-api-api-example]]
+[[get-pipeline-api-example]]
 ==== {api-examples-title}
 
 

docs/reference/ingest/apis/index.asciidoc

@@ -13,7 +13,7 @@ Use the following APIs to create, manage, and test ingest pipelines:
 * <<put-pipeline-api>> to create or update a pipeline
 * <<get-pipeline-api>> to retrieve a pipeline configuration
 * <<delete-pipeline-api>> to delete a pipeline
-* <<simulate-pipeline-api>> to test a pipeline
+* <<simulate-pipeline-api>> and <<simulate-ingest-api>> to test ingest pipelines
 
 [discrete]
 [[ingest-stat-apis]]
@@ -21,12 +21,27 @@ Use the following APIs to create, manage, and test ingest pipelines:
 
 Use the following APIs to get statistics about ingest processing:
 
-* <<geoip-stats-api>> to get download statistics for GeoIP2 databases used with
+* <<geoip-stats-api>> to get download statistics for IP geolocation databases used with
 the <<geoip-processor,`geoip` processor>>.
 
+[discrete]
+[[ingest-geoip-database-apis]]
+=== Ingest GeoIP Database APIs
+
+preview::["The commercial IP geolocation database download management APIs are in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."]
+
+Use the following APIs to configure and manage commercial IP geolocation database downloads:
+
+* <<put-geoip-database-api>> to create or update a database configuration
+* <<get-geoip-database-api>> to retrieve a database configuration
+* <<delete-geoip-database-api>> to delete a database configuration
+
 include::put-pipeline.asciidoc[]
-include::delete-pipeline.asciidoc[]
-include::geoip-stats-api.asciidoc[]
 include::get-pipeline.asciidoc[]
+include::delete-pipeline.asciidoc[]
 include::simulate-pipeline.asciidoc[]
 include::simulate-ingest.asciidoc[]
+include::geoip-stats.asciidoc[]
+include::put-geoip-database.asciidoc[]
+include::get-geoip-database.asciidoc[]
+include::delete-geoip-database.asciidoc[]

docs/reference/ingest/apis/put-geoip-database.asciidoc

@@ -0,0 +1,72 @@
+[[put-geoip-database-api]]
+=== Create or update geoip database configuration API
+++++
+<titleabbrev>Create or update geoip database configuration</titleabbrev>
+++++
+
+Creates or updates an IP geolocation database configuration.
+
+IMPORTANT: The Maxmind `account_id` shown below requires a license key. Because the license key is sensitive information,
+it is stored as a <<secure-settings,secure setting>> in {es} named `ingest.geoip.downloader.maxmind.license_key`. Only
+one Maxmind license key is currently allowed per {es} cluster. A valid license key must be in the secure settings in order
+to download from Maxmind. The license key setting does not take effect until all nodes are restarted.
+
+[source,console]
+----
+PUT _ingest/geoip/database/my-database-id
+{
+  "name": "GeoIP2-Domain",
+  "maxmind": {
+    "account_id": "1025402"
+  }
+}
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
+
+[[put-geoip-database-api-request]]
+==== {api-request-title}
+
+`PUT /_ingest/geoip/database/<database>`
+
+[[put-geoip-database-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the
+`manage` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+
+[[put-geoip-database-api-path-params]]
+==== {api-path-parms-title}
+
+`<database>`::
++
+__
+(Required, string) ID of the database configuration to create or update.
+
+[[put-geoip-database-api-query-params]]
+==== {api-query-parms-title}
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
+
+[[put-geoip-database-api-request-body]]
+==== {api-request-body-title}
+
+// tag::geoip-database-object[]
+`name`::
+(Required, string)
+The provider-assigned name of the IP geolocation database to download.
+
+`<provider>`::
+(Required, a provider object and its associated configuration)
+The configuration necessary to identify which IP geolocation provider to use to download
+the database, as well as any provider-specific configuration necessary for such downloading.
++
+At present, the only supported provider is `maxmind`, and the maxmind provider
+requires that an `account_id` (string) is configured.
+// end::geoip-database-object[]
+
+[[geoip-database-configuration-licensing]]
+==== Licensing
+
+Downloading databases from third party providers is a commercial feature that requires an
+appropriate license. For more information, refer to https://www.elastic.co/subscriptions.

docs/reference/ingest/processors/geoip.asciidoc

@@ -24,6 +24,9 @@ stats API>>.
 If your cluster can't connect to the Elastic GeoIP endpoint or you want to
 manage your own updates, see <<manage-geoip-database-updates>>.
 
+If you would like to have {es} download database files directly from Maxmind using your own provided
+license key, see <<put-geoip-database-api>>.
+
 If {es} can't connect to the endpoint for 30 days all updated databases will become
 invalid. {es} will stop enriching documents with geoip data and will add `tags: ["_geoip_expired_database"]`
 field instead.
@@ -36,9 +39,9 @@ field instead.
 [options="header"]
 |======
 | Name                                     | Required  | Default                                                                            | Description
-| `field`                                  | yes       | -                                                                                  | The field to get the ip address from for the geographical lookup.
+| `field`                                  | yes       | -                                                                                  | The field to get the IP address from for the geographical lookup.
 | `target_field`                           | no        | geoip                                                                              | The field that will hold the geographical information looked up from the MaxMind database.
-| `database_file`                          | no        | GeoLite2-City.mmdb                                                                 | The database filename referring to one of the automatically downloaded GeoLite2 databases (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or the name of a supported database file in the `ingest-geoip` config directory.
+| `database_file`                          | no        | GeoLite2-City.mmdb                                                                 | The database filename referring to one of the automatically downloaded GeoLite2 databases (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb), or the name of a supported database file in the `ingest-geoip` config directory, or the name of a <<get-geoip-database-api, configured database>> (with the `.mmdb` suffix appended).
 | `properties`                             | no        | [`continent_name`, `country_iso_code`, `country_name`, `region_iso_code`, `region_name`, `city_name`, `location`] *   | Controls what properties are added to the `target_field` based on the geoip lookup.
 | `ignore_missing`                         | no        | `false`                                                                            | If `true` and `field` does not exist, the processor quietly exits without modifying the document
 | `first_only`                             | no        | `true`                                                                             | If `true` only first found geoip data will be returned, even if `field` contains array