Disable GeoIP downloader (#72716)

Due to problems discovered in #72572 we have to disable geoip downloader for now. We use ingest.geoip.downloader.enabled.default as feature flag.
This change also reverts changes to docs.

docs/reference/ingest/apis/geoip-stats-api.asciidoc

@@ -23,9 +23,6 @@ GET _ingest/geoip/stats
 * If the {es} {security-features} are enabled, you must have the `monitor` or
 `manage` <<privileges-list-cluster,cluster privilege>> to use this API.
 
-* If <<ingest-geoip-downloader-enabled,`ingest.geoip.downloader.enabled`>> is
-disabled, this API returns zero values and an empty `nodes` object.
-
 [role="child_attributes"]
 [[geoip-stats-api-response-body]]
 ==== {api-response-body-title}
@@ -90,4 +87,4 @@ Downloaded database files, including related license files. {es} stores these
 files in the node's <<es-tmpdir,temporary directory>>:
 `$ES_TMPDIR/geoip-databases/<node_id>`.
 =====
-====
+====

docs/reference/ingest/processors/geoip.asciidoc

@@ -4,20 +4,20 @@
 <titleabbrev>GeoIP</titleabbrev>
 ++++
 
-The `geoip` processor adds information about the geographical location of an
-IPv4 or IPv6 address.
+The `geoip` processor adds information about the geographical location of IP addresses, based on data from the Maxmind databases.
+This processor adds this information by default under the `geoip` field. The `geoip` processor can resolve both IPv4 and
+IPv6 addresses.
 
-[[geoip-automatic-updates]]
-By default, the processor uses the GeoLite2 City, GeoLite2 Country, and GeoLite2
-ASN GeoIP2 databases from
-http://dev.maxmind.com/geoip/geoip2/geolite2/[MaxMind], shared under the
-CCA-ShareAlike 4.0 license. {es} automatically downloads updates for
-these databases from the Elastic GeoIP endpoint:
-https://geoip.elastic.co/v1/database. To get download statistics for these
-updates, use the <<geoip-stats-api,GeoIP stats API>>.
+The `ingest-geoip` module ships by default with the GeoLite2 City, GeoLite2 Country and GeoLite2 ASN GeoIP2 databases from Maxmind made available
+under the CCA-ShareAlike 4.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/
 
-If your cluster can't connect to the Elastic GeoIP endpoint or you want to
-manage your own updates, see <<manage-geoip-database-updates>>.
+The `geoip` processor can run with other city, country and ASN GeoIP2 databases
+from Maxmind. The database files must be copied into the `ingest-geoip` config
+directory located at `$ES_CONFIG/ingest-geoip`. Custom database files must be
+stored uncompressed and the extension must be `-City.mmdb`, `-Country.mmdb`, or
+`-ASN.mmdb` to indicate the type of the database. The
+`database_file` processor option is used to specify the filename of the custom
+database to use for the processor.
 
 [[using-ingest-geoip]]
 ==== Using the `geoip` Processor in a Pipeline
@@ -28,7 +28,7 @@ manage your own updates, see <<manage-geoip-database-updates>>.
 |======
 | Name                   | Required  | Default                                                                            | Description
 | `field`                | yes       | -                                                                                  | The field to get the ip address from for the geographical lookup.
-| `target_field`         | no        | geoip                                                                              | The field that will hold the geographical information looked up from the MaxMind database.
+| `target_field`         | no        | geoip                                                                              | The field that will hold the geographical information looked up from the Maxmind database.
 | `database_file`        | no        | GeoLite2-City.mmdb                                                                 | The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the `ingest-geoip` config directory.
 | `properties`           | no        | [`continent_name`, `country_iso_code`, `country_name`, `region_iso_code`, `region_name`, `city_name`, `location`] *   | Controls what properties are added to the `target_field` based on the geoip lookup.
 | `ignore_missing`       | no        | `false`                                                                            | If `true` and `field` does not exist, the processor quietly exits without modifying the document
@@ -299,79 +299,6 @@ GET /my_ip_locations/_search
 // TESTRESPONSE[s/"took" : 3/"took" : $body.took/]
 ////
 
-[[manage-geoip-database-updates]]
-==== Manage your own GeoIP2 database updates
-
-If you can't <<geoip-automatic-updates,automatically update>> your GeoIP2
-databases from the Elastic endpoint, you have a few other options:
-
-* <<use-proxy-geoip-endpoint,Use a proxy endpoint>>
-* <<use-custom-geoip-endpoint,Use a custom endpoint>>
-* <<manually-update-geoip-databases,Manually update your GeoIP2 databases>>
-
-[[use-proxy-geoip-endpoint]]
-**Use a proxy endpoint**
-
-If you can't connect directly to the Elastic GeoIP endpoint, consider setting up
-a secure proxy. You can then specify the proxy endpoint URL in the
-<<ingest-geoip-downloader-endpoint,`ingest.geoip.downloader.endpoint`>> setting
-of each node’s `elasticsearch.yml` file.
-
-[[use-custom-geoip-endpoint]]
-**Use a custom endpoint**
-
-You can create a service that mimics the Elastic GeoIP endpoint. You can then
-get automatic updates from this service.
-
-. Download your `.mmdb` database files from the
-http://dev.maxmind.com/geoip/geoip2/geolite2[MaxMind site].
-
-. Copy your database files to a single directory.
-
-. From your {es} directory, run:
-+
-[source,sh]
-----
-./bin/elasticsearch-geoip -s my/source/dir [-t target/directory]
-----
-
-. Serve the static database files from your directory. For example, you can use
-Docker to serve the files from an nginx server:
-+
-[source,sh]
-----
-docker run -v my/source/dir:/usr/share/nginx/html:ro nginx
-----
-
-. Specify the service's endpoint URL in the
-<<ingest-geoip-downloader-endpoint,`ingest.geoip.downloader.endpoint`>> setting
-of each node’s `elasticsearch.yml` file.
-+
-By default, {es} checks the endpoint for updates every three days. To use
-another polling interval, use the <<cluster-update-settings,update cluster
-settings API>> to set
-<<ingest-geoip-downloader-poll-interval,`ingest.geoip.downloader.poll.interval`>>.
-
-[[manually-update-geoip-databases]]
-**Manually update your GeoIP2 databases**
-
-. Use the <<cluster-update-settings,update cluster settings API>> to set
-`ingest.geoip.downloader.enabled` to `false`. This disables automatic updates
-that may overwrite your database changes. This also deletes all downloaded
-databases.
-
-. Download your `.mmdb` database files from the
-http://dev.maxmind.com/geoip/geoip2/geolite2[MaxMind site].
-+
-You can also use custom city, country, and ASN `.mmdb` files. These files must
-be uncompressed and use the respective `-City.mmdb`, `-Country.mmdb`, or
-`-ASN.mmdb` extensions.
-
-. Copy the database files to `$ES_CONFIG/ingest-geoip`.
-
-. In your `geoip` processors, configure the `database_file` parameter to use a
-custom database file.
-
 [[ingest-geoip-settings]]
 ===== Node Settings
 
@@ -382,28 +309,3 @@ The `geoip` processor supports the following setting:
     The maximum number of results that should be cached. Defaults to `1000`.
 
 Note that these settings are node settings and apply to all `geoip` processors, i.e. there is one cache for all defined `geoip` processors.
-
-[[geoip-cluster-settings]]
-===== Cluster settings
-
-[[ingest-geoip-downloader-enabled]]
-`ingest.geoip.downloader.enabled`::
-(<<dynamic-cluster-setting,Dynamic>>, Boolean)
-If `true`, {es} automatically downloads and manages updates for GeoIP2 databases
-from the `ingest.geoip.downloader.endpoint`. If `false`, {es} does not download
-updates and deletes all downloaded databases. Defaults to `true`.
-
-[[ingest-geoip-downloader-endpoint]]
-`ingest.geoip.downloader.endpoint`::
-(<<static-cluster-setting,Static>>, string)
-Endpoint URL used to download updates for GeoIP2 databases. Defaults to
-`https://geoip.elastic.co/v1/database`. {es} stores downloaded database files in
-each node's <<es-tmpdir,temporary directory>> at
-`$ES_TMPDIR/geoip-databases/<node_id>`.
-
-[[ingest-geoip-downloader-poll-interval]]
-`ingest.geoip.downloader.poll.interval`::
-(<<dynamic-cluster-setting,Dynamic>>, <<time-units,time value>>)
-How often {es} checks for GeoIP2 database updates at the
-`ingest.geoip.downloader.endpoint`. Must be greater than `1d` (one day). Defaults
-to `3d` (three days).

modules/ingest-geoip/build.gradle

@@ -55,6 +55,7 @@ tasks.named("internalClusterTest").configure {
   if (useFixture) {
     nonInputProperties.systemProperty "geoip_endpoint", "${-> fixtureAddress()}"
   }
+  systemProperty "ingest.geoip.downloader.enabled.default", "true"
 }
 
 tasks.register("copyDefaultGeoIp2DatabaseFiles", Copy) {

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/GeoIpDownloaderTaskExecutor.java

@@ -37,7 +37,7 @@ import static org.elasticsearch.ingest.geoip.GeoIpDownloader.GEOIP_DOWNLOADER;
  */
 public final class GeoIpDownloaderTaskExecutor extends PersistentTasksExecutor<GeoIpTaskParams> implements ClusterStateListener {
 
-    private static final boolean ENABLED_DEFAULT = "false".equals(System.getProperty("ingest.geoip.downloader.enabled.default")) == false;
+    static final boolean ENABLED_DEFAULT = "true".equals(System.getProperty("ingest.geoip.downloader.enabled.default"));
     public static final Setting<Boolean> ENABLED_SETTING = Setting.boolSetting("ingest.geoip.downloader.enabled", ENABLED_DEFAULT,
         Setting.Property.Dynamic, Setting.Property.NodeScope);
 

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/IngestGeoIpPlugin.java

@@ -113,6 +113,10 @@ public class IngestGeoIpPlugin extends Plugin implements IngestPlugin, SystemInd
             throw new UncheckedIOException(e);
         }
 
+        if(GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false){
+            return List.of(databaseRegistry.get());
+        }
+
         geoIpDownloaderTaskExecutor = new GeoIpDownloaderTaskExecutor(client, new HttpClient(), clusterService, threadPool);
         return List.of(databaseRegistry.get(), geoIpDownloaderTaskExecutor);
     }
@@ -126,6 +130,9 @@ public class IngestGeoIpPlugin extends Plugin implements IngestPlugin, SystemInd
     public List<PersistentTasksExecutor<?>> getPersistentTasksExecutor(ClusterService clusterService, ThreadPool threadPool,
                                                                        Client client, SettingsModule settingsModule,
                                                                        IndexNameExpressionResolver expressionResolver) {
+        if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) {
+            return Collections.emptyList();
+        }
         return List.of(geoIpDownloaderTaskExecutor);
     }
 
@@ -158,6 +165,9 @@ public class IngestGeoIpPlugin extends Plugin implements IngestPlugin, SystemInd
 
     @Override
     public Collection<SystemIndexDescriptor> getSystemIndexDescriptors(Settings settings) {
+        if (GeoIpDownloaderTaskExecutor.ENABLED_DEFAULT == false) {
+            return Collections.emptyList();
+        }
         SystemIndexDescriptor geoipDatabasesIndex = SystemIndexDescriptor.builder()
             .setIndexPattern(DATABASES_INDEX)
             .setDescription("GeoIP databases")
@@ -166,7 +176,6 @@ public class IngestGeoIpPlugin extends Plugin implements IngestPlugin, SystemInd
                 .put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1)
                 .put(IndexMetadata.SETTING_NUMBER_OF_REPLICAS, 0)
                 .put(IndexMetadata.SETTING_AUTO_EXPAND_REPLICAS, "0-1")
-                .put(IndexMetadata.SETTING_INDEX_HIDDEN, true)
                 .build())
             .setOrigin("geoip")
             .setVersionMetaKey("version")

modules/ingest-geoip/src/main/java/org/elasticsearch/ingest/geoip/stats/GeoIpDownloaderStatsTransportAction.java

@@ -33,17 +33,20 @@ public class GeoIpDownloaderStatsTransportAction extends TransportNodesAction<Re
 
     private final TransportService transportService;
     private final DatabaseRegistry registry;
-    private final GeoIpDownloaderTaskExecutor geoIpDownloaderTaskExecutor;
+    private GeoIpDownloaderTaskExecutor geoIpDownloaderTaskExecutor;
 
     @Inject
     public GeoIpDownloaderStatsTransportAction(TransportService transportService, ClusterService clusterService,
-                                               ThreadPool threadPool, ActionFilters actionFilters, DatabaseRegistry registry,
-                                               GeoIpDownloaderTaskExecutor geoIpDownloaderTaskExecutor) {
+                                               ThreadPool threadPool, ActionFilters actionFilters, DatabaseRegistry registry) {
         super(GeoIpDownloaderStatsAction.NAME, threadPool, clusterService, transportService, actionFilters, Request::new,
             NodeRequest::new, ThreadPool.Names.MANAGEMENT, NodeResponse.class);
         this.transportService = transportService;
         this.registry = registry;
-        this.geoIpDownloaderTaskExecutor = geoIpDownloaderTaskExecutor;
+    }
+
+    @Inject(optional = true)
+    public void setTaskExecutor(GeoIpDownloaderTaskExecutor taskExecutor){
+        geoIpDownloaderTaskExecutor = taskExecutor;
     }
 
     @Override
@@ -63,7 +66,7 @@ public class GeoIpDownloaderStatsTransportAction extends TransportNodesAction<Re
 
     @Override
     protected NodeResponse nodeOperation(NodeRequest request, Task task) {
-        GeoIpDownloader geoIpTask = geoIpDownloaderTaskExecutor.getCurrentTask();
+        GeoIpDownloader geoIpTask = geoIpDownloaderTaskExecutor == null ? null : geoIpDownloaderTaskExecutor.getCurrentTask();
         GeoIpDownloaderStats stats = geoIpTask == null || geoIpTask.getStatus() == null ? null : geoIpTask.getStatus();
         return new NodeResponse(transportService.getLocalNode(), stats, registry.getAvailableDatabases(), registry.getFilesInTemp());
     }