Introduce hidden indices (#50452)

This change introduces a new feature for indices so that they can be
hidden from wildcard expansion. The feature is referred to as hidden
indices. An index can be marked hidden through the use of an index
setting, `index.hidden`, at creation time. One primary use case for
this feature is to have a construct that fits indices that are created
by the stack that contain data used for display to the user and/or
intended for querying by the user. The desire to keep them hidden is
to avoid confusing users when searching all of the data they have
indexed and getting results returned from indices created by the
system.

Hidden indices have the following properties:
* API calls for all indices (empty indices array, _all, or *) will not
  return hidden indices by default.
* Wildcard expansion will not return hidden indices by default unless
  the wildcard pattern begins with a `.`. This behavior is similar to
  shell expansion of wildcards.
* REST API calls can enable the expansion of wildcards to hidden
  indices with the `expand_wildcards` parameter. To expand wildcards
  to hidden indices, use the value `hidden` in conjunction with `open`
  and/or `closed`.
* Creation of a hidden index will ignore global index templates. A
  global index template is one with a match-all pattern.
* Index templates can make an index hidden, with the exception of a
  global index template.
* Accessing a hidden index directly requires no additional parameters.

Relates #50251

docs/reference/index-modules.asciidoc

@@ -79,6 +79,13 @@ indices.
     Indicates whether <<query-filter-context, cached filters>> are pre-loaded for
     nested queries. Possible values are `true` (default) and `false`.
 
+`index.hidden`::
+
+    Indicates whether the index should be hidden by default. Hidden indices are not
+    returned by default when using a wildcard expression. This behavior is controlled
+    per request through the use of the `expand_wildcards` parameter. Possible values are
+    `true` and `false` (default).
+
 [float]
 [[dynamic-index-settings]]
 === Dynamic index settings

docs/reference/rest-api/common-parms.asciidoc

@@ -53,7 +53,7 @@ Specifies what to do when the request:
 --
 * Contains wildcard expressions and there are no {transforms} that match.
 * Contains the `_all` string or no identifiers and there are no matches.
-* Contains wildcard expressions and there are only partial matches. 
+* Contains wildcard expressions and there are only partial matches.
 
 The default value is `true`, which returns an empty `transforms` array when
 there are no matches and the subset of results when there are partial matches.
@@ -69,7 +69,7 @@ Specifies what to do when the request:
 --
 * Contains wildcard expressions and there are no {transforms} that match.
 * Contains the `_all` string or no identifiers and there are no matches.
-* Contains wildcard expressions and there are only partial matches. 
+* Contains wildcard expressions and there are only partial matches.
 
 The default value is `true`, which returns a successful acknowledgement message
 when there are no matches. When there are only partial matches, the API stops
@@ -90,7 +90,7 @@ end::analyzer[]
 
 tag::analyze_wildcard[]
 `analyze_wildcard`::
-(Optional, boolean) If `true`, wildcard and prefix queries are 
+(Optional, boolean) If `true`, wildcard and prefix queries are
 analyzed. Defaults to `false`.
 end::analyze_wildcard[]
 
@@ -118,7 +118,7 @@ end::completion-fields[]
 
 tag::default_operator[]
 `default_operator`::
-(Optional, string) The default operator for query string query: AND or OR. 
+(Optional, string) The default operator for query string query: AND or OR.
 Defaults to `OR`.
 end::default_operator[]
 
@@ -144,7 +144,7 @@ end::detailed[]
 
 tag::df[]
 `df`::
-(Optional, string) Field to use as default where no field prefix is 
+(Optional, string) Field to use as default where no field prefix is
 given in the query string.
 end::df[]
 
@@ -181,6 +181,9 @@ Expand only to open indices.
 `closed`::
 Expand only to closed indices.
 
+`hidden`::
+Expansion of wildcards will include hidden indices.
+
 `none`::
 Wildcard expressions are not accepted.
 --
@@ -188,7 +191,7 @@ end::expand-wildcards[]
 
 tag::field_statistics[]
 `field_statistics`::
-(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies, 
+(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies,
 and sum of total term frequencies.
 Defaults to `true`.
 end::field_statistics[]
@@ -297,9 +300,9 @@ end::help[]
 
 tag::bulk-id[]
 `_id`::
-(Optional, string) 
+(Optional, string)
 The document ID.
-If no ID is specified, a document ID is automatically generated. 
+If no ID is specified, a document ID is automatically generated.
 end::bulk-id[]
 
 tag::if_primary_term[]
@@ -316,7 +319,7 @@ end::if_seq_no[]
 
 tag::ignore_throttled[]
 `ignore_throttled`::
-(Optional, boolean) If `true`, concrete, expanded or aliased indices are 
+(Optional, boolean) If `true`, concrete, expanded or aliased indices are
 ignored when throttled.
 end::ignore_throttled[]
 
@@ -362,7 +365,7 @@ end::index[]
 tag::bulk-index[]
 `_index`::
 (Optional, string)
-The name of the target index. 
+The name of the target index.
 Required if not specified as a path parameter.
 end::bulk-index[]
 
@@ -447,7 +450,7 @@ end::index-template[]
 
 tag::lenient[]
 `lenient`::
-(Optional, boolean) If `true`, format-based query failures (such as 
+(Optional, boolean) If `true`, format-based query failures (such as
 providing text to a numeric field) will be ignored. Defaults to `false`.
 end::lenient[]
 
@@ -618,13 +621,13 @@ end::search-q[]
 
 tag::query[]
 `query`::
-(Optional, <<query-dsl,query object>>) Defines the search definition using the 
+(Optional, <<query-dsl,query object>>) Defines the search definition using the
 <<query-dsl,Query DSL>>.
 end::query[]
 
 tag::realtime[]
 `realtime`::
-(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time. 
+(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time.
 Defaults to `true`. See <<realtime>>.
 end::realtime[]
 
@@ -638,7 +641,7 @@ end::refresh[]
 
 tag::request_cache[]
 `request_cache`::
-(Optional, boolean) If `true`, the request cache is used for this request. 
+(Optional, boolean) If `true`, the request cache is used for this request.
 Defaults to the index-level setting.
 end::request_cache[]
 
@@ -668,14 +671,14 @@ end::cat-s[]
 
 tag::scroll[]
 `scroll`::
-(Optional, <<time-units, time units>>) Specifies how long a consistent view of 
+(Optional, <<time-units, time units>>) Specifies how long a consistent view of
 the index should be maintained for scrolled search.
 end::scroll[]
 
 tag::scroll_size[]
 `scroll_size`::
-(Optional, integer) Size of the scroll request that powers the operation. 
-Defaults to 100. 
+(Optional, integer) Size of the scroll request that powers the operation.
+Defaults to 100.
 end::scroll_size[]
 
 tag::search_timeout[]
@@ -731,7 +734,7 @@ end::size-transforms[]
 
 tag::slices[]
 `slices`::
-(Optional, integer) The number of slices this task should be divided into. 
+(Optional, integer) The number of slices this task should be divided into.
 Defaults to 1 meaning the task isn't sliced into subtasks.
 end::slices[]
 
@@ -742,24 +745,24 @@ end::sort[]
 
 tag::source[]
 `_source`::
-(Optional, string) True or false to return the `_source` field or not, or a 
+(Optional, string) True or false to return the `_source` field or not, or a
 list of fields to return.
 end::source[]
 
 tag::source_excludes[]
 `_source_excludes`::
-(Optional, string) A list of fields to exclude from the returned `_source` 
+(Optional, string) A list of fields to exclude from the returned `_source`
 field.
 end::source_excludes[]
 
 tag::source_includes[]
 `_source_includes`::
-(Optional, string) A list of fields to extract and return from the `_source` 
+(Optional, string) A list of fields to extract and return from the `_source`
 field.
 end::source_includes[]
 
 tag::source-transforms[]
-The source of the data for the {transform}. 
+The source of the data for the {transform}.
 end::source-transforms[]
 
 tag::source-index-transforms[]
@@ -823,13 +826,13 @@ end::task-id[]
 
 tag::term_statistics[]
 `term_statistics`::
-(Optional, boolean) If `true`, the response includes term frequency and document frequency. 
+(Optional, boolean) If `true`, the response includes term frequency and document frequency.
 Defaults to `false`.
 end::term_statistics[]
 
 tag::terminate_after[]
 `terminate_after`::
-(Optional, integer) The maximum number of documents to collect for each shard, 
+(Optional, integer) The maximum number of documents to collect for each shard,
 upon reaching which the query execution will terminate early.
 end::terminate_after[]
 
@@ -871,7 +874,7 @@ end::transform-id-wildcard[]
 
 tag::cat-v[]
 `v`::
-(Optional, boolean) If `true`, the response includes column headings. 
+(Optional, boolean) If `true`, the response includes column headings.
 Defaults to `false`.
 end::cat-v[]
 
@@ -912,6 +915,6 @@ end::wait_for_active_shards[]
 
 tag::wait_for_completion[]
 `wait_for_completion`::
-(Optional, boolean) If `true`, the request blocks until the operation is complete. 
+(Optional, boolean) If `true`, the request blocks until the operation is complete.
 Defaults to `true`.
 end::wait_for_completion[]

server/src/main/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequest.java

@@ -39,7 +39,7 @@ import java.util.concurrent.TimeUnit;
 public class ClusterHealthRequest extends MasterNodeReadRequest<ClusterHealthRequest> implements IndicesRequest.Replaceable {
 
     private String[] indices;
-    private IndicesOptions indicesOptions = IndicesOptions.lenientExpand();
+    private IndicesOptions indicesOptions = IndicesOptions.lenientExpandHidden();
     private TimeValue timeout = new TimeValue(30, TimeUnit.SECONDS);
     private ClusterHealthStatus waitForStatus;
     private boolean waitForNoRelocatingShards = false;

server/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java

@@ -43,7 +43,6 @@ import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
 import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.cluster.metadata.MetaDataCreateIndexService;
 import org.elasticsearch.cluster.metadata.MetaDataIndexAliasesService;
-import org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService;
 import org.elasticsearch.cluster.service.ClusterService;
 import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.inject.Inject;
@@ -63,6 +62,8 @@ import java.util.Optional;
 import java.util.regex.Pattern;
 import java.util.stream.Collectors;
 
+import static org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.findTemplates;
+
 /**
  * Main class to swap the index pointed to by an alias, given some conditions
  */
@@ -122,7 +123,9 @@ public class TransportRolloverAction extends TransportMasterNodeAction<RolloverR
             : generateRolloverIndexName(sourceProvidedName, indexNameExpressionResolver);
         final String rolloverIndexName = indexNameExpressionResolver.resolveDateMathExpression(unresolvedName);
         MetaDataCreateIndexService.validateIndexName(rolloverIndexName, state); // will fail if the index already exists
-        checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias());
+        final Boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.exists(rolloverRequest.getCreateIndexRequest().settings()) ?
+            IndexMetaData.INDEX_HIDDEN_SETTING.get(rolloverRequest.getCreateIndexRequest().settings()) : null;
+        checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias(), isHidden);
         IndicesStatsRequest statsRequest = new IndicesStatsRequest().indices(rolloverRequest.getAlias())
             .clear()
             .indicesOptions(IndicesOptions.fromOptions(true, false, true, true))
@@ -291,8 +294,9 @@ public class TransportRolloverAction extends TransportMasterNodeAction<RolloverR
      * the rollover alias will point to multiple indices. This causes indexing requests to be rejected.
      * To avoid this, we make sure that there is no duplicated alias in index templates before creating a new index.
      */
-    static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias) {
-        final List<IndexTemplateMetaData> matchedTemplates = MetaDataIndexTemplateService.findTemplates(metaData, rolloverIndexName);
+    static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias,
+                                                      @Nullable Boolean isHidden) {
+        final List<IndexTemplateMetaData> matchedTemplates = findTemplates(metaData, rolloverIndexName, isHidden);
         for (IndexTemplateMetaData template : matchedTemplates) {
             if (template.aliases().containsKey(rolloverRequestAlias)) {
                 throw new IllegalArgumentException(String.format(Locale.ROOT,

server/src/main/java/org/elasticsearch/action/bulk/TransportBulkAction.java

@@ -307,7 +307,7 @@ public class TransportBulkAction extends HandledTransportAction<BulkRequest, Bul
                 }
             } else if (indexRequest.index() != null) {
                 // the index does not exist yet (and this is a valid request), so match index templates to look for pipelines
-                List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index());
+                List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index(), null);
                 assert (templates != null);
                 // order of templates are highest order first
                 for (final IndexTemplateMetaData template : templates) {

server/src/main/java/org/elasticsearch/action/search/MultiSearchRequest.java

@@ -23,6 +23,7 @@ import org.elasticsearch.action.ActionRequest;
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.CompositeIndicesRequest;
 import org.elasticsearch.action.support.IndicesOptions;
+import org.elasticsearch.action.support.IndicesOptions.WildcardStates;
 import org.elasticsearch.common.CheckedBiConsumer;
 import org.elasticsearch.common.bytes.BytesReference;
 import org.elasticsearch.common.io.stream.StreamInput;
@@ -304,15 +305,7 @@ public class MultiSearchRequest extends ActionRequest implements CompositeIndice
             xContentBuilder.field("index", request.indices());
         }
         if (request.indicesOptions() != null && request.indicesOptions() != SearchRequest.DEFAULT_INDICES_OPTIONS) {
-            if (request.indicesOptions().expandWildcardsOpen() && request.indicesOptions().expandWildcardsClosed()) {
-                xContentBuilder.field("expand_wildcards", "all");
-            } else if (request.indicesOptions().expandWildcardsOpen()) {
-                xContentBuilder.field("expand_wildcards", "open");
-            } else if (request.indicesOptions().expandWildcardsClosed()) {
-                xContentBuilder.field("expand_wildcards", "closed");
-            } else {
-                xContentBuilder.field("expand_wildcards", "none");
-            }
+            WildcardStates.toXContent(request.indicesOptions().getExpandWildcards(), xContentBuilder);
             xContentBuilder.field("ignore_unavailable", request.indicesOptions().ignoreUnavailable());
             xContentBuilder.field("allow_no_indices", request.indicesOptions().allowNoIndices());
         }

server/src/main/java/org/elasticsearch/action/support/IndicesOptions.java

@@ -19,6 +19,7 @@
 package org.elasticsearch.action.support;
 
 
+import org.elasticsearch.Version;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.common.xcontent.ToXContent;
@@ -29,10 +30,9 @@ import org.elasticsearch.rest.RestRequest;
 import java.io.IOException;
 import java.util.Collection;
 import java.util.EnumSet;
-import java.util.HashSet;
 import java.util.Locale;
 import java.util.Map;
-import java.util.Set;
+import java.util.stream.Collectors;
 
 import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue;
 import static org.elasticsearch.common.xcontent.support.XContentMapValues.nodeStringArrayValue;
@@ -45,7 +45,8 @@ public class IndicesOptions implements ToXContentFragment {
 
     public enum WildcardStates {
         OPEN,
-        CLOSED;
+        CLOSED,
+        HIDDEN;
 
         public static final EnumSet<WildcardStates> NONE = EnumSet.noneOf(WildcardStates.class);
 
@@ -54,24 +55,45 @@ public class IndicesOptions implements ToXContentFragment {
                 return defaultStates;
             }
 
-            Set<WildcardStates> states = new HashSet<>();
+            EnumSet<WildcardStates> states = EnumSet.noneOf(WildcardStates.class);
             String[] wildcards = nodeStringArrayValue(value);
+            // TODO why do we let patterns like "none,all" or "open,none,closed" get used. The location of 'none' in the array changes the
+            // meaning of the resulting value
             for (String wildcard : wildcards) {
-                if ("open".equals(wildcard)) {
-                    states.add(OPEN);
-                } else if ("closed".equals(wildcard)) {
-                    states.add(CLOSED);
-                } else if ("none".equals(wildcard)) {
-                    states.clear();
-                } else if ("all".equals(wildcard)) {
-                    states.add(OPEN);
-                    states.add(CLOSED);
-                } else {
-                    throw new IllegalArgumentException("No valid expand wildcard value [" + wildcard + "]");
+                switch (wildcard) {
+                    case "open":
+                        states.add(OPEN);
+                        break;
+                    case "closed":
+                        states.add(CLOSED);
+                        break;
+                    case "hidden":
+                        states.add(HIDDEN);
+                        break;
+                    case "none":
+                        states.clear();
+                        break;
+                    case "all":
+                        states = EnumSet.allOf(WildcardStates.class);
+                        break;
+                    default:
+                        throw new IllegalArgumentException("No valid expand wildcard value [" + wildcard + "]");
                 }
             }
 
-            return states.isEmpty() ? NONE : EnumSet.copyOf(states);
+            return states;
+        }
+
+        public static XContentBuilder toXContent(EnumSet<WildcardStates> states, XContentBuilder builder) throws IOException {
+            if (states.isEmpty()) {
+                builder.field("expand_wildcards", "none");
+            } else if (states.containsAll(EnumSet.allOf(WildcardStates.class))) {
+                builder.field("expand_wildcards", "all");
+            } else {
+                builder.field("expand_wildcards",
+                    states.stream().map(state -> state.toString().toLowerCase(Locale.ROOT)).collect(Collectors.joining(",")));
+            }
+            return builder;
         }
     }
 
@@ -91,9 +113,15 @@ public class IndicesOptions implements ToXContentFragment {
     public static final IndicesOptions LENIENT_EXPAND_OPEN =
         new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
             EnumSet.of(WildcardStates.OPEN));
+    public static final IndicesOptions LENIENT_EXPAND_OPEN_HIDDEN =
+        new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
+            EnumSet.of(WildcardStates.OPEN, WildcardStates.HIDDEN));
     public static final IndicesOptions LENIENT_EXPAND_OPEN_CLOSED =
         new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
             EnumSet.of(WildcardStates.OPEN, WildcardStates.CLOSED));
+    public static final IndicesOptions LENIENT_EXPAND_OPEN_CLOSED_HIDDEN =
+        new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE),
+            EnumSet.of(WildcardStates.OPEN, WildcardStates.CLOSED, WildcardStates.HIDDEN));
     public static final IndicesOptions STRICT_EXPAND_OPEN_CLOSED =
         new IndicesOptions(EnumSet.of(Option.ALLOW_NO_INDICES), EnumSet.of(WildcardStates.OPEN, WildcardStates.CLOSED));
     public static final IndicesOptions STRICT_EXPAND_OPEN_FORBID_CLOSED =
@@ -149,6 +177,13 @@ public class IndicesOptions implements ToXContentFragment {
         return expandWildcards.contains(WildcardStates.CLOSED);
     }
 
+    /**
+     * @return Whether wildcard expressions should get expanded to hidden indices
+     */
+    public boolean expandWildcardsHidden() {
+        return expandWildcards.contains(WildcardStates.HIDDEN);
+    }
+
     /**
      * @return Whether execution on closed indices is allowed.
      */
@@ -173,39 +208,71 @@ public class IndicesOptions implements ToXContentFragment {
     }
 
     /**
-     *
      * @return whether indices that are marked as throttled should be ignored
      */
     public boolean ignoreThrottled() {
         return options.contains(Option.IGNORE_THROTTLED);
     }
 
+    /**
+     * @return a copy of the {@link WildcardStates} that these indices options will expand to
+     */
+    public EnumSet<WildcardStates> getExpandWildcards() {
+        return EnumSet.copyOf(expandWildcards);
+    }
+
     public void writeIndicesOptions(StreamOutput out) throws IOException {
         out.writeEnumSet(options);
-        out.writeEnumSet(expandWildcards);
+        if (out.getVersion().before(Version.V_8_0_0) && expandWildcards.contains(WildcardStates.HIDDEN)) {
+            final EnumSet<WildcardStates> states = EnumSet.copyOf(expandWildcards);
+            states.remove(WildcardStates.HIDDEN);
+            out.writeEnumSet(states);
+        } else {
+            out.writeEnumSet(expandWildcards);
+        }
     }
 
     public static IndicesOptions readIndicesOptions(StreamInput in) throws IOException {
-        return new IndicesOptions(in.readEnumSet(Option.class), in.readEnumSet(WildcardStates.class));
+        EnumSet<Option> options = in.readEnumSet(Option.class);
+        EnumSet<WildcardStates> states = in.readEnumSet(WildcardStates.class);
+        if (in.getVersion().before(Version.V_8_0_0)) {
+            states.add(WildcardStates.HIDDEN);
+        }
+        return new IndicesOptions(options, states);
     }
 
     public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
                                              boolean expandToClosedIndices) {
-        return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, true, false, false, false);
+        return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, false);
+    }
+
+    public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
+                                             boolean expandToClosedIndices, boolean expandToHiddenIndices) {
+        return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, expandToHiddenIndices, true,
+            false, false, false);
     }
 
     public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
                                              boolean expandToClosedIndices, IndicesOptions defaultOptions) {
         return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices,
-            defaultOptions.allowAliasesToMultipleIndices(), defaultOptions.forbidClosedIndices(), defaultOptions.ignoreAliases(),
-            defaultOptions.ignoreThrottled());
+            defaultOptions.expandWildcardsHidden(), defaultOptions.allowAliasesToMultipleIndices(),
+            defaultOptions.forbidClosedIndices(), defaultOptions.ignoreAliases(), defaultOptions.ignoreThrottled());
+    }
+
+    public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
+                                             boolean expandToClosedIndices, boolean allowAliasesToMultipleIndices,
+                                             boolean forbidClosedIndices, boolean ignoreAliases,
+                                             boolean ignoreThrottled) {
+        return fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices, expandToClosedIndices, false,
+            allowAliasesToMultipleIndices, forbidClosedIndices, ignoreAliases, ignoreThrottled);
     }
 
     public static IndicesOptions fromOptions(boolean ignoreUnavailable, boolean allowNoIndices, boolean expandToOpenIndices,
-            boolean expandToClosedIndices, boolean allowAliasesToMultipleIndices, boolean forbidClosedIndices, boolean ignoreAliases,
+                                             boolean expandToClosedIndices, boolean expandToHiddenIndices,
+                                             boolean allowAliasesToMultipleIndices, boolean forbidClosedIndices, boolean ignoreAliases,
                                              boolean ignoreThrottled) {
-        final Set<Option> opts = new HashSet<>();
-        final Set<WildcardStates> wildcards = new HashSet<>();
+        final EnumSet<Option> opts = EnumSet.noneOf(Option.class);
+        final EnumSet<WildcardStates> wildcards = EnumSet.noneOf(WildcardStates.class);
 
         if (ignoreUnavailable) {
             opts.add(Option.IGNORE_UNAVAILABLE);
@@ -219,6 +286,9 @@ public class IndicesOptions implements ToXContentFragment {
         if (expandToClosedIndices) {
             wildcards.add(WildcardStates.CLOSED);
         }
+        if (expandToHiddenIndices) {
+            wildcards.add(WildcardStates.HIDDEN);
+        }
         if (allowAliasesToMultipleIndices == false) {
             opts.add(Option.FORBID_ALIASES_TO_MULTIPLE_INDICES);
         }
@@ -278,6 +348,7 @@ public class IndicesOptions implements ToXContentFragment {
                 nodeBooleanValue(allowNoIndicesString, "allow_no_indices", defaultSettings.allowNoIndices()),
                 wildcards.contains(WildcardStates.OPEN),
                 wildcards.contains(WildcardStates.CLOSED),
+                wildcards.contains(WildcardStates.HIDDEN),
                 defaultSettings.allowAliasesToMultipleIndices(),
                 defaultSettings.forbidClosedIndices(),
                 defaultSettings.ignoreAliases(),
@@ -348,6 +419,14 @@ public class IndicesOptions implements ToXContentFragment {
         return LENIENT_EXPAND_OPEN;
     }
 
+    /**
+     * @return indices options that ignores unavailable indices, expands wildcards to open and hidden indices, and
+     *         allows that no indices are resolved from wildcard expressions (not returning an error).
+     */
+    public static IndicesOptions lenientExpandOpenHidden() {
+        return LENIENT_EXPAND_OPEN_HIDDEN;
+    }
+
     /**
      * @return indices options that ignores unavailable indices,  expands wildcards to both open and closed
      * indices and allows that no indices are resolved from wildcard expressions (not returning an error).
@@ -356,6 +435,14 @@ public class IndicesOptions implements ToXContentFragment {
         return LENIENT_EXPAND_OPEN_CLOSED;
     }
 
+    /**
+     * @return indices options that ignores unavailable indices,  expands wildcards to all open and closed
+     * indices and allows that no indices are resolved from wildcard expressions (not returning an error).
+     */
+    public static IndicesOptions lenientExpandHidden() {
+        return LENIENT_EXPAND_OPEN_CLOSED_HIDDEN;
+    }
+
     @Override
     public boolean equals(Object obj) {
         if (obj == null) {
@@ -383,6 +470,7 @@ public class IndicesOptions implements ToXContentFragment {
                 ", allow_no_indices=" + allowNoIndices() +
                 ", expand_wildcards_open=" + expandWildcardsOpen() +
                 ", expand_wildcards_closed=" + expandWildcardsClosed() +
+                ", expand_wildcards_hidden=" + expandWildcardsHidden() +
                 ", allow_aliases_to_multiple_indices=" + allowAliasesToMultipleIndices() +
                 ", forbid_closed_indices=" + forbidClosedIndices() +
                 ", ignore_aliases=" + ignoreAliases() +

server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetaData.java

@@ -256,6 +256,13 @@ public class IndexMetaData implements Diffable<IndexMetaData>, ToXContentFragmen
                       Setting.Property.Dynamic,
                       Setting.Property.IndexScope);
 
+    /**
+     * Whether the index is considered hidden or not. A hidden index will not be resolved in
+     * normal wildcard searches unless explicitly allowed
+     */
+    public static final Setting<Boolean> INDEX_HIDDEN_SETTING =
+        Setting.boolSetting("index.hidden", false, Property.IndexScope, Property.Final);
+
     /**
      * an internal index format description, allowing us to find out if this index is upgraded or needs upgrading
      */

server/src/main/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolver.java

@@ -158,7 +158,7 @@ public class IndexNameExpressionResolver {
         for (ExpressionResolver expressionResolver : expressionResolvers) {
             expressions = expressionResolver.resolve(context, expressions);
         }
-        
+
         if (expressions.isEmpty()) {
             if (!options.allowNoIndices()) {
                 IndexNotFoundException infe;
@@ -187,7 +187,7 @@ public class IndexNameExpressionResolver {
                     if (expression.equals(MetaData.ALL)) {
                         infe = new IndexNotFoundException("no indices exist", expression);
                     } else {
-                        infe = new IndexNotFoundException(expression);  
+                        infe = new IndexNotFoundException(expression);
                     }
                     infe.setResources("index_expression", expression);
                     throw infe;
@@ -670,6 +670,8 @@ public class IndexNameExpressionResolver {
         public List<String> resolve(Context context, List<String> expressions) {
             IndicesOptions options = context.getOptions();
             MetaData metaData = context.getState().metaData();
+            // only check open/closed since if we do not expand to open or closed it doesn't make sense to
+            // expand to hidden
             if (options.expandWildcardsClosed() == false && options.expandWildcardsOpen() == false) {
                 return expressions;
             }
@@ -739,7 +741,7 @@ public class IndexNameExpressionResolver {
 
                 final IndexMetaData.State excludeState = excludeState(options);
                 final Map<String, AliasOrIndex> matches = matches(context, metaData, expression);
-                Set<String> expand = expand(context, excludeState, matches);
+                Set<String> expand = expand(context, excludeState, matches, expression, options.expandWildcardsHidden());
                 if (add) {
                     result.addAll(expand);
                 } else {
@@ -834,7 +836,8 @@ public class IndexNameExpressionResolver {
                 .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
         }
 
-        private static Set<String> expand(Context context, IndexMetaData.State excludeState, Map<String, AliasOrIndex> matches) {
+        private static Set<String> expand(Context context, IndexMetaData.State excludeState, Map<String, AliasOrIndex> matches,
+                                          String expression, boolean includeHidden) {
             Set<String> expand = new HashSet<>();
             for (Map.Entry<String, AliasOrIndex> entry : matches.entrySet()) {
                 AliasOrIndex aliasOrIndex = entry.getValue();
@@ -843,7 +846,14 @@ public class IndexNameExpressionResolver {
                 } else {
                     for (IndexMetaData meta : aliasOrIndex.getIndices()) {
                         if (excludeState == null || meta.getState() != excludeState) {
-                            expand.add(meta.getIndex().getName());
+                            if (includeHidden) {
+                                expand.add(meta.getIndex().getName());
+                            } else if (IndexMetaData.INDEX_HIDDEN_SETTING.get(meta.getSettings()) == false) {
+                                expand.add(meta.getIndex().getName());
+                            } else if (meta.getIndex().getName().startsWith(".") &&
+                                expression.startsWith(".") && Regex.isSimpleMatchPattern(expression)) {
+                                expand.add(meta.getIndex().getName());
+                            }
                         }
                     }
                 }
@@ -857,12 +867,18 @@ public class IndexNameExpressionResolver {
         }
 
         private static List<String> resolveEmptyOrTrivialWildcard(IndicesOptions options, MetaData metaData) {
-            if (options.expandWildcardsOpen() && options.expandWildcardsClosed()) {
+            if (options.expandWildcardsOpen() && options.expandWildcardsClosed() && options.expandWildcardsHidden()) {
                 return Arrays.asList(metaData.getConcreteAllIndices());
-            } else if (options.expandWildcardsOpen()) {
+            } else if (options.expandWildcardsOpen() && options.expandWildcardsClosed()) {
+                return Arrays.asList(metaData.getConcreteVisibleIndices());
+            } else if (options.expandWildcardsOpen() && options.expandWildcardsHidden()) {
                 return Arrays.asList(metaData.getConcreteAllOpenIndices());
-            } else if (options.expandWildcardsClosed()) {
+            } else if (options.expandWildcardsOpen()) {
+                return Arrays.asList(metaData.getConcreteVisibleOpenIndices());
+            } else if (options.expandWildcardsClosed() && options.expandWildcardsHidden()) {
                 return Arrays.asList(metaData.getConcreteAllClosedIndices());
+            } else if (options.expandWildcardsClosed()) {
+                return Arrays.asList(metaData.getConcreteVisibleClosedIndices());
             } else {
                 return Collections.emptyList();
             }

server/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java

@@ -177,15 +177,19 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
     private final int totalOpenIndexShards;
 
     private final String[] allIndices;
+    private final String[] visibleIndices;
     private final String[] allOpenIndices;
+    private final String[] visibleOpenIndices;
     private final String[] allClosedIndices;
+    private final String[] visibleClosedIndices;
 
     private final SortedMap<String, AliasOrIndex> aliasAndIndexLookup;
 
     MetaData(String clusterUUID, boolean clusterUUIDCommitted, long version, CoordinationMetaData coordinationMetaData,
              Settings transientSettings, Settings persistentSettings, DiffableStringMap hashesOfConsistentSettings,
              ImmutableOpenMap<String, IndexMetaData> indices, ImmutableOpenMap<String, IndexTemplateMetaData> templates,
-             ImmutableOpenMap<String, Custom> customs, String[] allIndices, String[] allOpenIndices, String[] allClosedIndices,
+             ImmutableOpenMap<String, Custom> customs, String[] allIndices, String[] visibleIndices, String[] allOpenIndices,
+             String[] visibleOpenIndices, String[] allClosedIndices, String[] visibleClosedIndices,
              SortedMap<String, AliasOrIndex> aliasAndIndexLookup) {
         this.clusterUUID = clusterUUID;
         this.clusterUUIDCommitted = clusterUUIDCommitted;
@@ -210,8 +214,11 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
         this.totalOpenIndexShards = totalOpenIndexShards;
 
         this.allIndices = allIndices;
+        this.visibleIndices = visibleIndices;
         this.allOpenIndices = allOpenIndices;
+        this.visibleOpenIndices = visibleOpenIndices;
         this.allClosedIndices = allClosedIndices;
+        this.visibleClosedIndices = visibleClosedIndices;
         this.aliasAndIndexLookup = aliasAndIndexLookup;
     }
 
@@ -479,14 +486,41 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
         return allIndices;
     }
 
+    /**
+     * Returns all the concrete indices that are not hidden.
+     */
+    public String[] getConcreteVisibleIndices() {
+        return visibleIndices;
+    }
+
+    /**
+     * Returns all of the concrete indices that are open.
+     */
     public String[] getConcreteAllOpenIndices() {
         return allOpenIndices;
     }
 
+    /**
+     * Returns all of the concrete indices that are open and not hidden.
+     */
+    public String[] getConcreteVisibleOpenIndices() {
+        return visibleOpenIndices;
+    }
+
+    /**
+     * Returns all of the concrete indices that are closed.
+     */
     public String[] getConcreteAllClosedIndices() {
         return allClosedIndices;
     }
 
+    /**
+     * Returns all of the concrete indices that are closed and not hidden.
+     */
+    public String[] getConcreteVisibleClosedIndices() {
+        return visibleClosedIndices;
+    }
+
     /**
      * Returns indexing routing for the given <code>aliasOrIndex</code>. Resolves routing from the alias metadata used
      * in the write index.
@@ -1147,18 +1181,31 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
             // 2) The aliasAndIndexLookup can be updated instead of rebuilding it all the time.
 
             final Set<String> allIndices = new HashSet<>(indices.size());
+            final List<String> visibleIndices = new ArrayList<>();
             final List<String> allOpenIndices = new ArrayList<>();
+            final List<String> visibleOpenIndices = new ArrayList<>();
             final List<String> allClosedIndices = new ArrayList<>();
+            final List<String> visibleClosedIndices = new ArrayList<>();
             final Set<String> duplicateAliasesIndices = new HashSet<>();
             for (ObjectCursor<IndexMetaData> cursor : indices.values()) {
                 final IndexMetaData indexMetaData = cursor.value;
                 final String name = indexMetaData.getIndex().getName();
                 boolean added = allIndices.add(name);
                 assert added : "double index named [" + name + "]";
+                final boolean visible = IndexMetaData.INDEX_HIDDEN_SETTING.get(indexMetaData.getSettings()) == false;
+                if (visible) {
+                    visibleIndices.add(name);
+                }
                 if (indexMetaData.getState() == IndexMetaData.State.OPEN) {
-                    allOpenIndices.add(indexMetaData.getIndex().getName());
+                    allOpenIndices.add(name);
+                    if (visible) {
+                        visibleOpenIndices.add(name);
+                    }
                 } else if (indexMetaData.getState() == IndexMetaData.State.CLOSE) {
-                    allClosedIndices.add(indexMetaData.getIndex().getName());
+                    allClosedIndices.add(name);
+                    if (visible) {
+                        visibleClosedIndices.add(name);
+                    }
                 }
                 indexMetaData.getAliases().keysIt().forEachRemaining(duplicateAliasesIndices::add);
             }
@@ -1186,13 +1233,16 @@ public class MetaData implements Iterable<IndexMetaData>, Diffable<MetaData>, To
             // TODO: I think we can remove these arrays. it isn't worth the effort, for operations on all indices.
             // When doing an operation across all indices, most of the time is spent on actually going to all shards and
             // do the required operations, the bottleneck isn't resolving expressions into concrete indices.
-            String[] allIndicesArray = allIndices.toArray(new String[allIndices.size()]);
-            String[] allOpenIndicesArray = allOpenIndices.toArray(new String[allOpenIndices.size()]);
-            String[] allClosedIndicesArray = allClosedIndices.toArray(new String[allClosedIndices.size()]);
+            String[] allIndicesArray = allIndices.toArray(Strings.EMPTY_ARRAY);
+            String[] visibleIndicesArray = visibleIndices.toArray(Strings.EMPTY_ARRAY);
+            String[] allOpenIndicesArray = allOpenIndices.toArray(Strings.EMPTY_ARRAY);
+            String[] visibleOpenIndicesArray = visibleOpenIndices.toArray(Strings.EMPTY_ARRAY);
+            String[] allClosedIndicesArray = allClosedIndices.toArray(Strings.EMPTY_ARRAY);
+            String[] visibleClosedIndicesArray = visibleClosedIndices.toArray(Strings.EMPTY_ARRAY);
 
             return new MetaData(clusterUUID, clusterUUIDCommitted, version, coordinationMetaData, transientSettings, persistentSettings,
-                    hashesOfConsistentSettings, indices.build(), templates.build(), customs.build(), allIndicesArray, allOpenIndicesArray,
-                    allClosedIndicesArray, aliasAndIndexLookup);
+                hashesOfConsistentSettings, indices.build(), templates.build(), customs.build(), allIndicesArray, visibleIndicesArray,
+                allOpenIndicesArray, visibleOpenIndicesArray, allClosedIndicesArray, visibleClosedIndicesArray, aliasAndIndexLookup);
         }
 
         private SortedMap<String, AliasOrIndex> buildAliasAndIndexLookup() {

server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java

@@ -278,8 +278,10 @@ public class MetaDataCreateIndexService {
 
         // we only find a template when its an API call (a new index)
         // find templates, highest order are better matching
+        final Boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.exists(request.settings()) ?
+            IndexMetaData.INDEX_HIDDEN_SETTING.get(request.settings()) : null;
         final List<IndexTemplateMetaData> templates = sourceMetaData == null ?
-            Collections.unmodifiableList(MetaDataIndexTemplateService.findTemplates(currentState.metaData(), request.index())) :
+            Collections.unmodifiableList(MetaDataIndexTemplateService.findTemplates(currentState.metaData(), request.index(), isHidden)) :
             List.of();
 
         final Map<String, Object> mappings = Collections.unmodifiableMap(parseMappings(request.mappings(), templates, xContentRegistry));

server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexTemplateService.java

@@ -28,6 +28,7 @@ import org.elasticsearch.action.support.master.MasterNodeRequest;
 import org.elasticsearch.cluster.ClusterState;
 import org.elasticsearch.cluster.ClusterStateUpdateTask;
 import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.Priority;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.UUIDs;
@@ -55,7 +56,9 @@ import java.util.HashSet;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.Optional;
 import java.util.Set;
+import java.util.function.Predicate;
 
 import static org.elasticsearch.indices.cluster.IndicesClusterStateService.AllocatedIndices.IndexRemovalReason.NO_LONGER_ASSIGNED;
 
@@ -196,19 +199,61 @@ public class MetaDataIndexTemplateService {
     }
 
     /**
-     * Finds index templates whose index pattern matched with the given index name.
-     * The result is sorted by {@link IndexTemplateMetaData#order} descending.
+     * Finds index templates whose index pattern matched with the given index name. In the case of
+     * hidden indices, a template with a match all pattern or global template will not be returned.
+     *
+     * @param metaData The {@link MetaData} containing all of the {@link IndexTemplateMetaData} values
+     * @param indexName The name of the index that templates are being found for
+     * @param isHidden Whether or not the index is known to be hidden. May be {@code null} if the index
+     *                 being hidden has not been explicitly requested. When {@code null} if the result
+     *                 of template application results in a hidden index, then global templates will
+     *                 not be returned
+     * @return a list of templates sorted by {@link IndexTemplateMetaData#order()} descending.
+     *
      */
-    public static List<IndexTemplateMetaData> findTemplates(MetaData metaData, String indexName) {
+    public static List<IndexTemplateMetaData> findTemplates(MetaData metaData, String indexName, @Nullable Boolean isHidden) {
+        final Predicate<String> patternMatchPredicate = pattern -> Regex.simpleMatch(pattern, indexName);
         final List<IndexTemplateMetaData> matchedTemplates = new ArrayList<>();
         for (ObjectCursor<IndexTemplateMetaData> cursor : metaData.templates().values()) {
             final IndexTemplateMetaData template = cursor.value;
-            final boolean matched = template.patterns().stream().anyMatch(pattern -> Regex.simpleMatch(pattern, indexName));
-            if (matched) {
-                matchedTemplates.add(template);
+            if (isHidden == null || isHidden == Boolean.FALSE) {
+                final boolean matched = template.patterns().stream().anyMatch(patternMatchPredicate);
+                if (matched) {
+                    matchedTemplates.add(template);
+                }
+            } else {
+                assert isHidden == Boolean.TRUE;
+                final boolean isNotMatchAllTemplate = template.patterns().stream().noneMatch(Regex::isMatchAllPattern);
+                if (isNotMatchAllTemplate) {
+                    if (template.patterns().stream().anyMatch(patternMatchPredicate)) {
+                        matchedTemplates.add(template);
+                    }
+                }
             }
         }
         CollectionUtil.timSort(matchedTemplates, Comparator.comparingInt(IndexTemplateMetaData::order).reversed());
+
+        // this is complex but if the index is not hidden in the create request but is hidden as the result of template application,
+        // then we need to exclude global templates
+        if (isHidden == null) {
+            final Optional<IndexTemplateMetaData> templateWithHiddenSetting = matchedTemplates.stream()
+                .filter(template -> IndexMetaData.INDEX_HIDDEN_SETTING.exists(template.settings())).findFirst();
+            if (templateWithHiddenSetting.isPresent()) {
+                final boolean templatedIsHidden = IndexMetaData.INDEX_HIDDEN_SETTING.get(templateWithHiddenSetting.get().settings());
+                if (templatedIsHidden) {
+                    // remove the global templates
+                    matchedTemplates.removeIf(current -> current.patterns().stream().anyMatch(Regex::isMatchAllPattern));
+                }
+                // validate that hidden didn't change
+                final Optional<IndexTemplateMetaData> templateWithHiddenSettingPostRemoval = matchedTemplates.stream()
+                    .filter(template -> IndexMetaData.INDEX_HIDDEN_SETTING.exists(template.settings())).findFirst();
+                if (templateWithHiddenSettingPostRemoval.isEmpty() ||
+                    templateWithHiddenSetting.get() != templateWithHiddenSettingPostRemoval.get()) {
+                    throw new IllegalStateException("A global index template [" + templateWithHiddenSetting.get().name() +
+                        "] defined the index hidden setting, which is not allowed");
+                }
+            }
+        }
         return matchedTemplates;
     }
 
@@ -307,6 +352,13 @@ public class MetaDataIndexTemplateService {
         }
         List<String> indexSettingsValidation = metaDataCreateIndexService.getIndexSettingsValidationErrors(request.settings, true);
         validationErrors.addAll(indexSettingsValidation);
+
+        if (request.indexPatterns.stream().anyMatch(Regex::isMatchAllPattern)) {
+            if (IndexMetaData.INDEX_HIDDEN_SETTING.exists(request.settings)) {
+                validationErrors.add("global templates may not specify the setting " + IndexMetaData.INDEX_HIDDEN_SETTING.getKey());
+            }
+        }
+
         if (!validationErrors.isEmpty()) {
             ValidationException validationException = new ValidationException();
             validationException.addValidationErrors(validationErrors);

server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java

@@ -77,6 +77,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings {
             IndexMetaData.INDEX_BLOCKS_READ_ONLY_ALLOW_DELETE_SETTING,
             IndexMetaData.INDEX_PRIORITY_SETTING,
             IndexMetaData.INDEX_DATA_PATH_SETTING,
+            IndexMetaData.INDEX_HIDDEN_SETTING,
             IndexMetaData.INDEX_FORMAT_SETTING,
             SearchSlowLog.INDEX_SEARCH_SLOWLOG_THRESHOLD_FETCH_DEBUG_SETTING,
             SearchSlowLog.INDEX_SEARCH_SLOWLOG_THRESHOLD_FETCH_WARN_SETTING,

server/src/test/java/org/elasticsearch/action/OriginalIndicesTests.java

@@ -19,6 +19,7 @@
 
 package org.elasticsearch.action;
 
+import org.elasticsearch.Version;
 import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.common.io.stream.BytesStreamOutput;
 import org.elasticsearch.common.io.stream.StreamInput;
@@ -26,7 +27,7 @@ import org.elasticsearch.test.ESTestCase;
 
 import java.io.IOException;
 
-import static org.elasticsearch.test.VersionUtils.randomVersion;
+import static org.elasticsearch.test.VersionUtils.randomCompatibleVersion;
 import static org.hamcrest.CoreMatchers.equalTo;
 
 public class OriginalIndicesTests extends ESTestCase {
@@ -41,7 +42,7 @@ public class OriginalIndicesTests extends ESTestCase {
             OriginalIndices originalIndices = randomOriginalIndices();
 
             BytesStreamOutput out = new BytesStreamOutput();
-            out.setVersion(randomVersion(random()));
+            out.setVersion(randomCompatibleVersion(random(), Version.CURRENT));
             OriginalIndices.writeOriginalIndices(originalIndices, out);
 
             StreamInput in = out.bytes().streamInput();
@@ -49,7 +50,15 @@ public class OriginalIndicesTests extends ESTestCase {
             OriginalIndices originalIndices2 = OriginalIndices.readOriginalIndices(in);
 
             assertThat(originalIndices2.indices(), equalTo(originalIndices.indices()));
-            assertThat(originalIndices2.indicesOptions(), equalTo(originalIndices.indicesOptions()));
+            // indices options are not equivalent when sent to an older version and re-read due
+            // to the addition of hidden indices as expand to hidden indices is always true when
+            // read from a prior version
+            // TODO update version on backport!
+            if (out.getVersion().onOrAfter(Version.V_8_0_0) || originalIndices.indicesOptions().expandWildcardsHidden()) {
+                assertThat(originalIndices2.indicesOptions(), equalTo(originalIndices.indicesOptions()));
+            } else if (originalIndices.indicesOptions().expandWildcardsHidden()) {
+                assertThat(originalIndices2.indicesOptions(), equalTo(originalIndices.indicesOptions()));
+            }
         }
     }
 

server/src/test/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequestTests.java

@@ -56,6 +56,11 @@ public class ClusterHealthRequestTests extends ESTestCase {
         assertThat(cloneRequest.indicesOptions(), equalTo(originalRequest.indicesOptions()));
     }
 
+    public void testRequestReturnsHiddenIndicesByDefault() {
+        final ClusterHealthRequest defaultRequest = new ClusterHealthRequest();
+        assertTrue(defaultRequest.indicesOptions().expandWildcardsHidden());
+    }
+
     public void testBwcSerialization() throws Exception {
         for (int runs = 0; runs < randomIntBetween(5, 20); runs++) {
             // Generate a random cluster health request in version < 7.2.0 and serializes it

server/src/test/java/org/elasticsearch/action/admin/cluster/shards/ClusterSearchShardsRequestTests.java

@@ -40,7 +40,7 @@ public class ClusterSearchShardsRequestTests extends ESTestCase {
         }
         if (randomBoolean()) {
             request.indicesOptions(
-                    IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
+                    IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
         }
         if (randomBoolean()) {
             request.preference(randomAlphaOfLengthBetween(3, 10));
@@ -54,7 +54,7 @@ public class ClusterSearchShardsRequestTests extends ESTestCase {
             request.routing(routings);
         }
 
-        Version version = VersionUtils.randomIndexCompatibleVersion(random());
+        Version version = VersionUtils.randomCompatibleVersion(random(), Version.CURRENT);
         try (BytesStreamOutput out = new BytesStreamOutput()) {
             out.setVersion(version);
             request.writeTo(out);
@@ -62,7 +62,12 @@ public class ClusterSearchShardsRequestTests extends ESTestCase {
                 in.setVersion(version);
                 ClusterSearchShardsRequest deserialized = new ClusterSearchShardsRequest(in);
                 assertArrayEquals(request.indices(), deserialized.indices());
-                assertEquals(request.indicesOptions(), deserialized.indicesOptions());
+                // indices options are not equivalent when sent to an older version and re-read due
+                // to the addition of hidden indices as expand to hidden indices is always true when
+                // read from a prior version
+                if (version.onOrAfter(Version.V_8_0_0) || request.indicesOptions().expandWildcardsHidden()) { // TODO change on backport!
+                    assertEquals(request.indicesOptions(), deserialized.indicesOptions());
+                }
                 assertEquals(request.routing(), deserialized.routing());
                 assertEquals(request.preference(), deserialized.preference());
             }

server/src/test/java/org/elasticsearch/action/admin/indices/close/CloseIndexRequestTests.java

@@ -52,40 +52,66 @@ public class CloseIndexRequestTests extends ESTestCase {
         {
             final CloseIndexRequest request = randomRequest();
             try (BytesStreamOutput out = new BytesStreamOutput()) {
-                out.setVersion(VersionUtils.randomVersionBetween(random(), VersionUtils.getFirstVersion(),
-                    VersionUtils.getPreviousVersion(Version.V_7_2_0)));
+                out.setVersion(VersionUtils.randomCompatibleVersion(random(), Version.CURRENT));
                 request.writeTo(out);
 
                 try (StreamInput in = out.bytes().streamInput()) {
+                    in.setVersion(out.getVersion());
                     assertEquals(request.getParentTask(), TaskId.readFromStream(in));
                     assertEquals(request.masterNodeTimeout(), in.readTimeValue());
                     assertEquals(request.timeout(), in.readTimeValue());
                     assertArrayEquals(request.indices(), in.readStringArray());
-                    assertEquals(request.indicesOptions(), IndicesOptions.readIndicesOptions(in));
+                    final IndicesOptions indicesOptions = IndicesOptions.readIndicesOptions(in);
+                    // indices options are not equivalent when sent to an older version and re-read due
+                    // to the addition of hidden indices as expand to hidden indices is always true when
+                    // read from a prior version
+                    // TODO update version on backport!
+                    if (out.getVersion().onOrAfter(Version.V_8_0_0) || request.indicesOptions().expandWildcardsHidden()) {
+                        assertEquals(request.indicesOptions(), indicesOptions);
+                    }
+                    if (in.getVersion().onOrAfter(Version.V_7_2_0)) {
+                        assertEquals(request.waitForActiveShards(), ActiveShardCount.readFrom(in));
+                    } else {
+                        assertEquals(0, in.available());
+                    }
                 }
             }
         }
         {
             final CloseIndexRequest sample = randomRequest();
+            final Version version = VersionUtils.randomCompatibleVersion(random(), Version.CURRENT);
             try (BytesStreamOutput out = new BytesStreamOutput()) {
+                out.setVersion(version);
                 sample.getParentTask().writeTo(out);
                 out.writeTimeValue(sample.masterNodeTimeout());
                 out.writeTimeValue(sample.timeout());
                 out.writeStringArray(sample.indices());
                 sample.indicesOptions().writeIndicesOptions(out);
+                if (out.getVersion().onOrAfter(Version.V_7_2_0)) {
+                    sample.waitForActiveShards().writeTo(out);
+                }
 
                 final CloseIndexRequest deserializedRequest;
                 try (StreamInput in = out.bytes().streamInput()) {
-                    in.setVersion(VersionUtils.randomVersionBetween(random(), VersionUtils.getFirstVersion(),
-                        VersionUtils.getPreviousVersion(Version.V_7_2_0)));
+                    in.setVersion(version);
                     deserializedRequest = new CloseIndexRequest(in);
                 }
                 assertEquals(sample.getParentTask(), deserializedRequest.getParentTask());
                 assertEquals(sample.masterNodeTimeout(), deserializedRequest.masterNodeTimeout());
                 assertEquals(sample.timeout(), deserializedRequest.timeout());
                 assertArrayEquals(sample.indices(), deserializedRequest.indices());
-                assertEquals(sample.indicesOptions(), deserializedRequest.indicesOptions());
-                assertEquals(ActiveShardCount.NONE, deserializedRequest.waitForActiveShards());
+                // indices options are not equivalent when sent to an older version and re-read due
+                // to the addition of hidden indices as expand to hidden indices is always true when
+                // read from a prior version
+                // TODO change version on backport
+                if (out.getVersion().onOrAfter(Version.V_8_0_0) || sample.indicesOptions().expandWildcardsHidden()) {
+                    assertEquals(sample.indicesOptions(), deserializedRequest.indicesOptions());
+                }
+                if (out.getVersion().onOrAfter(Version.V_7_2_0)) {
+                    assertEquals(sample.waitForActiveShards(), deserializedRequest.waitForActiveShards());
+                } else {
+                    assertEquals(ActiveShardCount.NONE, deserializedRequest.waitForActiveShards());
+                }
             }
         }
     }
@@ -94,7 +120,8 @@ public class CloseIndexRequestTests extends ESTestCase {
         CloseIndexRequest request = new CloseIndexRequest();
         request.indices(generateRandomStringArray(10, 5, false, false));
         if (randomBoolean()) {
-            request.indicesOptions(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
+            request.indicesOptions(
+                IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean()));
         }
         if (randomBoolean()) {
             request.timeout(randomPositiveTimeValue());

server/src/test/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverActionTests.java

@@ -79,6 +79,7 @@ import org.mockito.ArgumentCaptor;
 import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Locale;
@@ -359,7 +360,24 @@ public class TransportRolloverActionTests extends ESTestCase {
         String indexName = randomFrom("foo-123", "bar-xyz");
         String aliasName = randomFrom("foo-write", "bar-write");
         final IllegalArgumentException ex = expectThrows(IllegalArgumentException.class,
-            () -> TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName));
+            () -> TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName, randomBoolean()));
+        assertThat(ex.getMessage(), containsString("index template [test-template]"));
+    }
+
+    public void testHiddenAffectsResolvedTemplates() {
+        final IndexTemplateMetaData template = IndexTemplateMetaData.builder("test-template")
+            .patterns(Collections.singletonList("*"))
+            .putAlias(AliasMetaData.builder("foo-write")).putAlias(AliasMetaData.builder("bar-write").writeIndex(randomBoolean()))
+            .build();
+        final MetaData metaData = MetaData.builder().put(createMetaData(randomAlphaOfLengthBetween(5, 7)), false).put(template).build();
+        String indexName = randomFrom("foo-123", "bar-xyz");
+        String aliasName = randomFrom("foo-write", "bar-write");
+
+        // hidden shouldn't throw
+        TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName, Boolean.TRUE);
+        // not hidden will throw
+        final IllegalArgumentException ex = expectThrows(IllegalArgumentException.class, () ->
+            TransportRolloverAction.checkNoDuplicatedAliasInIndexTemplate(metaData, indexName, aliasName, randomFrom(Boolean.FALSE, null)));
         assertThat(ex.getMessage(), containsString("index template [test-template]"));
     }
 

server/src/test/java/org/elasticsearch/action/admin/indices/template/put/MetaDataIndexTemplateServiceTests.java

@@ -48,17 +48,20 @@ import java.util.Set;
 import java.util.concurrent.CountDownLatch;
 import java.util.stream.Collectors;
 
+import static java.util.Collections.singletonList;
 import static org.elasticsearch.common.settings.Settings.builder;
 import static org.hamcrest.CoreMatchers.containsString;
 import static org.hamcrest.CoreMatchers.equalTo;
 import static org.hamcrest.CoreMatchers.instanceOf;
 import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.is;
 
 public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
     public void testIndexTemplateInvalidNumberOfShards() {
         PutRequest request = new PutRequest("test", "test_shards");
-        request.patterns(Collections.singletonList("test_shards*"));
+        request.patterns(singletonList("test_shards*"));
 
         request.settings(builder()
             .put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, "0")
@@ -76,7 +79,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
 
     public void testIndexTemplateValidationAccumulatesValidationErrors() {
         PutRequest request = new PutRequest("test", "putTemplate shards");
-        request.patterns(Collections.singletonList("_test_shards*"));
+        request.patterns(singletonList("_test_shards*"));
         request.settings(builder().put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, "0").build());
 
         List<Throwable> throwables = putTemplate(xContentRegistry(), request);
@@ -101,7 +104,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
 
     public void testIndexTemplateWithValidateMapping() throws Exception {
         PutRequest request = new PutRequest("api", "validate_template");
-        request.patterns(Collections.singletonList("te*"));
+        request.patterns(singletonList("te*"));
         request.putMapping("type1", Strings.toString(XContentFactory.jsonBuilder().startObject().startObject("type1")
                         .startObject("properties").startObject("field2").field("type", "text").field("analyzer", "custom_1").endObject()
                         .endObject().endObject().endObject()));
@@ -114,7 +117,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
 
     public void testBrokenMapping() throws Exception {
         PutRequest request = new PutRequest("api", "broken_mapping");
-        request.patterns(Collections.singletonList("te*"));
+        request.patterns(singletonList("te*"));
         request.putMapping("type1", "abcde");
 
         List<Throwable> errors = putTemplateDetail(request);
@@ -126,7 +129,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
     public void testAliasInvalidFilterInvalidJson() throws Exception {
         //invalid json: put index template fails
         PutRequest request = new PutRequest("api", "blank_mapping");
-        request.patterns(Collections.singletonList("te*"));
+        request.patterns(singletonList("te*"));
         request.putMapping("type1", "{}");
         Set<Alias> aliases = new HashSet<>();
         aliases.add(new Alias("invalid_alias").filter("abcde"));
@@ -140,15 +143,63 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
 
     public void testFindTemplates() throws Exception {
         client().admin().indices().prepareDeleteTemplate("*").get(); // Delete all existing templates
-        putTemplateDetail(new PutRequest("test", "foo-1").patterns(Arrays.asList("foo-*")).order(1));
-        putTemplateDetail(new PutRequest("test", "foo-2").patterns(Arrays.asList("foo-*")).order(2));
-        putTemplateDetail(new PutRequest("test", "bar").patterns(Arrays.asList("bar-*")).order(between(0, 100)));
+        putTemplateDetail(new PutRequest("test", "foo-1").patterns(singletonList("foo-*")).order(1));
+        putTemplateDetail(new PutRequest("test", "foo-2").patterns(singletonList("foo-*")).order(2));
+        putTemplateDetail(new PutRequest("test", "bar").patterns(singletonList("bar-*")).order(between(0, 100)));
         final ClusterState state = client().admin().cluster().prepareState().get().getState();
-        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234").stream()
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", randomBoolean()).stream()
             .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1"));
-        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz").stream()
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", randomBoolean()).stream()
             .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar"));
-        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz"), empty());
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", randomBoolean()), empty());
+    }
+
+    public void testFindTemplatesWithHiddenIndices() throws Exception {
+        client().admin().indices().prepareDeleteTemplate("*").get(); // Delete all existing templates
+        putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "foo-1").patterns(singletonList("foo-*")).order(1));
+        putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "foo-2").patterns(singletonList("foo-*")).order(2));
+        putTemplateDetail(
+            new PutRequest("testFindTemplatesWithHiddenIndices", "bar").patterns(singletonList("bar-*")).order(between(0, 100)));
+        putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "global").patterns(singletonList("*")));
+        putTemplateDetail(new PutRequest("testFindTemplatesWithHiddenIndices", "sneaky-hidden")
+            .patterns(singletonList("sneaky*")).settings(Settings.builder().put("index.hidden", true).build()));
+        final ClusterState state = client().admin().cluster().prepareState().get().getState();
+
+        // hidden
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", true).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", true).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", true), empty());
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "sneaky1", true).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("sneaky-hidden"));
+
+        // not hidden
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", false).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1", "global"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", false).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar", "global"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", false).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("global"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "sneaky1", false).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), containsInAnyOrder("global", "sneaky-hidden"));
+
+        // unknown
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "foo-1234", null).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("foo-2", "foo-1", "global"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "bar-xyz", null).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("bar", "global"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "baz", null).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("global"));
+        assertThat(MetaDataIndexTemplateService.findTemplates(state.metaData(), "sneaky1", null).stream()
+            .map(IndexTemplateMetaData::name).collect(Collectors.toList()), contains("sneaky-hidden"));
+    }
+
+    public void testPutGlobalTemplateWithIndexHiddenSetting() throws Exception {
+        List<Throwable> errors = putTemplateDetail(new PutRequest("testPutGlobalTemplateWithIndexHiddenSetting", "sneaky-hidden")
+            .patterns(singletonList("*")).settings(Settings.builder().put("index.hidden", true).build()));
+        assertThat(errors.size(), is(1));
+        assertThat(errors.get(0).getMessage(), containsString("global templates may not specify the setting index.hidden"));
     }
 
     private static List<Throwable> putTemplate(NamedXContentRegistry xContentRegistry, PutRequest request) {
@@ -192,7 +243,7 @@ public class MetaDataIndexTemplateServiceTests extends ESSingleNodeTestCase {
                 null,
                 null,
                 new Environment(builder().put(Environment.PATH_HOME_SETTING.getKey(), createTempDir().toString()).build(), null),
-                null,
+                IndexScopedSettings.DEFAULT_SCOPED_SETTINGS,
                 null,
                 xContentRegistry(),
                 true);

server/src/test/java/org/elasticsearch/action/search/MultiSearchRequestTests.java

@@ -25,9 +25,13 @@ import org.elasticsearch.common.CheckedRunnable;
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.bytes.BytesArray;
+import org.elasticsearch.common.bytes.BytesReference;
 import org.elasticsearch.common.xcontent.NamedXContentRegistry;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.common.xcontent.XContentHelper;
 import org.elasticsearch.common.xcontent.XContentParser;
 import org.elasticsearch.common.xcontent.XContentType;
+import org.elasticsearch.common.xcontent.json.JsonXContent;
 import org.elasticsearch.index.query.MatchAllQueryBuilder;
 import org.elasticsearch.index.query.QueryBuilder;
 import org.elasticsearch.rest.RestRequest;
@@ -43,6 +47,7 @@ import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
 import static java.util.Collections.singletonList;
 import static org.elasticsearch.search.RandomSearchRequestGenerator.randomSearchRequest;
@@ -256,6 +261,37 @@ public class MultiSearchRequestTests extends ESTestCase {
         }
     }
 
+    public void testWritingExpandWildcards() throws IOException {
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, true, true, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "all");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, true, false, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "open,closed");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, false, true, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "open,hidden");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), true, false, false, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "open");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, true, true, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "closed,hidden");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, true, false, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "closed");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, false, true, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "hidden");
+        assertExpandWildcardsValue(IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), false, false, false, randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean()), "none");
+    }
+
+    private void assertExpandWildcardsValue(IndicesOptions options, String expectedValue) throws IOException {
+        SearchRequest request = new SearchRequest();
+        request.indicesOptions(options);
+        try (XContentBuilder builder = JsonXContent.contentBuilder()) {
+            MultiSearchRequest.writeSearchRequestParams(request, builder);
+            Map<String, Object> map =
+                XContentHelper.convertToMap(XContentType.JSON.xContent(), BytesReference.bytes(builder).streamInput(), false);
+            final String value = (String) map.get("expand_wildcards");
+            assertEquals(expectedValue, value);
+        }
+    }
+
     public void testEqualsAndHashcode() {
         checkEqualsAndHashCode(createMultiSearchRequest(), MultiSearchRequestTests::copyRequest, MultiSearchRequestTests::mutate);
     }

server/src/test/java/org/elasticsearch/action/support/IndicesOptionsTests.java

@@ -35,6 +35,7 @@ import org.elasticsearch.test.ESTestCase;
 import org.elasticsearch.test.EqualsHashCodeTestUtils;
 
 import java.io.IOException;
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
@@ -45,6 +46,7 @@ import java.util.Map;
 
 import static org.elasticsearch.test.VersionUtils.randomVersionBetween;
 import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.Matchers.is;
 
 public class IndicesOptionsTests extends ESTestCase {
 
@@ -54,7 +56,7 @@ public class IndicesOptionsTests extends ESTestCase {
             Version version = randomVersionBetween(random(), Version.V_7_0_0, null);
             IndicesOptions indicesOptions = IndicesOptions.fromOptions(
                 randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
-                randomBoolean());
+                randomBoolean(), randomBoolean());
 
             BytesStreamOutput output = new BytesStreamOutput();
             output.setVersion(version);
@@ -68,6 +70,11 @@ public class IndicesOptionsTests extends ESTestCase {
             assertThat(indicesOptions2.allowNoIndices(), equalTo(indicesOptions.allowNoIndices()));
             assertThat(indicesOptions2.expandWildcardsOpen(), equalTo(indicesOptions.expandWildcardsOpen()));
             assertThat(indicesOptions2.expandWildcardsClosed(), equalTo(indicesOptions.expandWildcardsClosed()));
+            if (version.before(Version.V_8_0_0)) {
+                assertThat(indicesOptions2.expandWildcardsHidden(), is(true));
+            } else {
+                assertThat(indicesOptions2.expandWildcardsHidden(), equalTo(indicesOptions.expandWildcardsHidden()));
+            }
 
             assertThat(indicesOptions2.forbidClosedIndices(), equalTo(indicesOptions.forbidClosedIndices()));
             assertThat(indicesOptions2.allowAliasesToMultipleIndices(), equalTo(indicesOptions.allowAliasesToMultipleIndices()));
@@ -77,27 +84,30 @@ public class IndicesOptionsTests extends ESTestCase {
     }
 
     public void testFromOptions() {
-        boolean ignoreUnavailable = randomBoolean();
-        boolean allowNoIndices = randomBoolean();
-        boolean expandToOpenIndices = randomBoolean();
-        boolean expandToClosedIndices = randomBoolean();
-        boolean allowAliasesToMultipleIndices = randomBoolean();
-        boolean forbidClosedIndices = randomBoolean();
-        boolean ignoreAliases = randomBoolean();
-        boolean ingoreThrottled = randomBoolean();
-
-        IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices,expandToOpenIndices,
-                expandToClosedIndices, allowAliasesToMultipleIndices, forbidClosedIndices, ignoreAliases, ingoreThrottled);
+        final boolean ignoreUnavailable = randomBoolean();
+        final boolean allowNoIndices = randomBoolean();
+        final boolean expandToOpenIndices = randomBoolean();
+        final boolean expandToClosedIndices = randomBoolean();
+        final boolean expandToHiddenIndices = randomBoolean();
+        final boolean allowAliasesToMultipleIndices = randomBoolean();
+        final boolean forbidClosedIndices = randomBoolean();
+        final boolean ignoreAliases = randomBoolean();
+        final boolean ignoreThrottled = randomBoolean();
+
+        IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices,
+            expandToClosedIndices, expandToHiddenIndices, allowAliasesToMultipleIndices, forbidClosedIndices, ignoreAliases,
+            ignoreThrottled);
 
         assertThat(indicesOptions.ignoreUnavailable(), equalTo(ignoreUnavailable));
         assertThat(indicesOptions.allowNoIndices(), equalTo(allowNoIndices));
         assertThat(indicesOptions.expandWildcardsOpen(), equalTo(expandToOpenIndices));
         assertThat(indicesOptions.expandWildcardsClosed(), equalTo(expandToClosedIndices));
+        assertThat(indicesOptions.expandWildcardsHidden(), equalTo(expandToHiddenIndices));
         assertThat(indicesOptions.allowAliasesToMultipleIndices(), equalTo(allowAliasesToMultipleIndices));
         assertThat(indicesOptions.allowAliasesToMultipleIndices(), equalTo(allowAliasesToMultipleIndices));
         assertThat(indicesOptions.forbidClosedIndices(), equalTo(forbidClosedIndices));
         assertEquals(ignoreAliases, indicesOptions.ignoreAliases());
-        assertEquals(ingoreThrottled, indicesOptions.ignoreThrottled());
+        assertEquals(ignoreThrottled, indicesOptions.ignoreThrottled());
     }
 
     public void testFromOptionsWithDefaultOptions() {
@@ -106,53 +116,64 @@ public class IndicesOptionsTests extends ESTestCase {
         boolean expandToOpenIndices = randomBoolean();
         boolean expandToClosedIndices = randomBoolean();
 
-        IndicesOptions defaultOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(),
+        IndicesOptions defaultOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
                 randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
 
-        IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices,expandToOpenIndices,
+        IndicesOptions indicesOptions = IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandToOpenIndices,
                 expandToClosedIndices, defaultOptions);
 
         assertEquals(ignoreUnavailable, indicesOptions.ignoreUnavailable());
         assertEquals(allowNoIndices, indicesOptions.allowNoIndices());
         assertEquals(expandToOpenIndices, indicesOptions.expandWildcardsOpen());
         assertEquals(expandToClosedIndices, indicesOptions.expandWildcardsClosed());
+        assertEquals(defaultOptions.expandWildcardsHidden(), indicesOptions.expandWildcardsHidden());
         assertEquals(defaultOptions.allowAliasesToMultipleIndices(), indicesOptions.allowAliasesToMultipleIndices());
         assertEquals(defaultOptions.forbidClosedIndices(), indicesOptions.forbidClosedIndices());
         assertEquals(defaultOptions.ignoreAliases(), indicesOptions.ignoreAliases());
     }
 
     public void testFromParameters() {
-        boolean expandWildcardsOpen = randomBoolean();
-        boolean expandWildcardsClosed = randomBoolean();
-        String expandWildcardsString;
-        if (expandWildcardsOpen && expandWildcardsClosed) {
+        final boolean expandWildcardsOpen = randomBoolean();
+        final boolean expandWildcardsClosed = randomBoolean();
+        final boolean expandWildcardsHidden = randomBoolean();
+        final String expandWildcardsString;
+        if (expandWildcardsOpen && expandWildcardsClosed && expandWildcardsHidden) {
             if (randomBoolean()) {
-                expandWildcardsString = "open,closed";
+                expandWildcardsString = "open,closed,hidden";
             } else {
                 expandWildcardsString = "all";
             }
-        } else if (expandWildcardsOpen) {
-            expandWildcardsString = "open";
-        } else if (expandWildcardsClosed) {
-            expandWildcardsString = "closed";
         } else {
-            expandWildcardsString = "none";
+            List<String> values = new ArrayList<>();
+            if (expandWildcardsOpen) {
+                values.add("open");
+            }
+            if (expandWildcardsClosed) {
+                values.add("closed");
+            }
+            if (expandWildcardsHidden) {
+                values.add("hidden");
+            }
+            if (values.isEmpty() && randomBoolean()) {
+                values.add("none");
+            }
+            expandWildcardsString = String.join(",", values);
         }
         boolean ignoreUnavailable = randomBoolean();
         String ignoreUnavailableString = Boolean.toString(ignoreUnavailable);
         boolean ignoreThrottled = randomBoolean();
-        String ignoreThrottledString = Boolean.toString(ignoreThrottled);
         boolean allowNoIndices = randomBoolean();
         String allowNoIndicesString = Boolean.toString(allowNoIndices);
 
         IndicesOptions defaultOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
-                randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
+                randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
 
         IndicesOptions updatedOptions = IndicesOptions.fromParameters(expandWildcardsString, ignoreUnavailableString,
                 allowNoIndicesString, ignoreThrottled, defaultOptions);
 
         assertEquals(expandWildcardsOpen, updatedOptions.expandWildcardsOpen());
         assertEquals(expandWildcardsClosed, updatedOptions.expandWildcardsClosed());
+        assertEquals(expandWildcardsHidden, updatedOptions.expandWildcardsHidden());
         assertEquals(ignoreUnavailable, updatedOptions.ignoreUnavailable());
         assertEquals(allowNoIndices, updatedOptions.allowNoIndices());
         assertEquals(defaultOptions.allowAliasesToMultipleIndices(), updatedOptions.allowAliasesToMultipleIndices());
@@ -161,20 +182,20 @@ public class IndicesOptionsTests extends ESTestCase {
     }
 
     public void testEqualityAndHashCode() {
-        IndicesOptions indicesOptions = IndicesOptions.fromOptions(
-            randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
-            randomBoolean());
+        IndicesOptions indicesOptions = IndicesOptions.fromOptions(randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(),
+            randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean(), randomBoolean());
 
         EqualsHashCodeTestUtils.checkEqualsAndHashCode(indicesOptions, opts -> {
             return IndicesOptions.fromOptions(opts.ignoreUnavailable(), opts.allowNoIndices(), opts.expandWildcardsOpen(),
-                opts.expandWildcardsClosed(), opts.allowAliasesToMultipleIndices(), opts.forbidClosedIndices(), opts.ignoreAliases(),
-                opts.ignoreThrottled());
+                opts.expandWildcardsClosed(), opts.expandWildcardsHidden(), opts.allowAliasesToMultipleIndices(),
+                opts.forbidClosedIndices(), opts.ignoreAliases(), opts.ignoreThrottled());
         }, opts -> {
             boolean mutated = false;
             boolean ignoreUnavailable = opts.ignoreUnavailable();
             boolean allowNoIndices = opts.allowNoIndices();
             boolean expandOpen = opts.expandWildcardsOpen();
             boolean expandClosed = opts.expandWildcardsClosed();
+            boolean expandHidden = opts.expandWildcardsHidden();
             boolean allowAliasesToMulti = opts.allowAliasesToMultipleIndices();
             boolean forbidClosed = opts.forbidClosedIndices();
             boolean ignoreAliases = opts.ignoreAliases();
@@ -196,6 +217,10 @@ public class IndicesOptionsTests extends ESTestCase {
                     expandClosed = !expandClosed;
                     mutated = true;
                 }
+                if (randomBoolean()) {
+                    expandHidden = !expandHidden;
+                    mutated = true;
+                }
                 if (randomBoolean()) {
                     allowAliasesToMulti = !allowAliasesToMulti;
                     mutated = true;
@@ -213,7 +238,7 @@ public class IndicesOptionsTests extends ESTestCase {
                     mutated = true;
                 }
             }
-            return IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandOpen, expandClosed,
+            return IndicesOptions.fromOptions(ignoreUnavailable, allowNoIndices, expandOpen, expandClosed, expandHidden,
                 allowAliasesToMulti, forbidClosed, ignoreAliases, ignoreThrottled);
         });
     }
@@ -221,7 +246,7 @@ public class IndicesOptionsTests extends ESTestCase {
     public void testFromMap() {
         IndicesOptions defaults = IndicesOptions.strictExpandOpen();
         Collection<String> wildcardStates = randomBoolean() ?
-                null : randomSubsetOf(Arrays.asList("open", "closed"));
+                null : randomSubsetOf(Arrays.asList("open", "closed", "hidden"));
         Boolean ignoreUnavailable = randomBoolean() ? null : randomBoolean();
         Boolean allowNoIndices = randomBoolean() ? null : randomBoolean();
         Boolean ignoreThrottled = randomBoolean() ? null : randomBoolean();
@@ -247,13 +272,15 @@ public class IndicesOptionsTests extends ESTestCase {
         IndicesOptions fromMap = IndicesOptions.fromMap(settings, defaults);
 
         boolean open = wildcardStates != null ? wildcardStates.contains("open") : defaults.expandWildcardsOpen();
-        assertEquals(fromMap.expandWildcardsOpen(), open);
+        assertEquals(open, fromMap.expandWildcardsOpen());
         boolean closed = wildcardStates != null ? wildcardStates.contains("closed") : defaults.expandWildcardsClosed();
-        assertEquals(fromMap.expandWildcardsClosed(), closed);
+        assertEquals(closed, fromMap.expandWildcardsClosed());
+        boolean hidden = wildcardStates != null ? wildcardStates.contains("hidden") : defaults.expandWildcardsHidden();
+        assertEquals(hidden, fromMap.expandWildcardsHidden());
 
-        assertEquals(fromMap.ignoreUnavailable(), ignoreUnavailable == null ? defaults.ignoreUnavailable() : ignoreUnavailable);
-        assertEquals(fromMap.allowNoIndices(), allowNoIndices == null ? defaults.allowNoIndices() : allowNoIndices);
-        assertEquals(fromMap.ignoreThrottled(), ignoreThrottled == null ? defaults.ignoreThrottled() : ignoreThrottled);
+        assertEquals(ignoreUnavailable == null ? defaults.ignoreUnavailable() : ignoreUnavailable, fromMap.ignoreUnavailable());
+        assertEquals(allowNoIndices == null ? defaults.allowNoIndices() : allowNoIndices, fromMap.allowNoIndices());
+        assertEquals(ignoreThrottled == null ? defaults.ignoreThrottled() : ignoreThrottled, fromMap.ignoreThrottled());
     }
 
     public void testToXContent() throws IOException {
@@ -284,6 +311,7 @@ public class IndicesOptionsTests extends ESTestCase {
         } else {
             assertFalse(((List<?>) map.get("expand_wildcards")).contains("closed"));
         }
+        assertEquals(wildcardStates.contains(WildcardStates.HIDDEN), ((List<?>) map.get("expand_wildcards")).contains("hidden"));
         assertEquals(map.get("ignore_unavailable"), options.contains(Option.IGNORE_UNAVAILABLE));
         assertEquals(map.get("allow_no_indices"), options.contains(Option.ALLOW_NO_INDICES));
         assertEquals(map.get("ignore_throttled"), options.contains(Option.IGNORE_THROTTLED));

server/src/test/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolverTests.java

@@ -272,11 +272,15 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
         MetaData.Builder mdBuilder = MetaData.builder()
                 .put(indexBuilder("foo").state(IndexMetaData.State.CLOSE))
                 .put(indexBuilder("bar"))
-                .put(indexBuilder("foobar").putAlias(AliasMetaData.builder("barbaz")));
+                .put(indexBuilder("foobar").putAlias(AliasMetaData.builder("barbaz")))
+                .put(indexBuilder("hidden", Settings.builder().put("index.hidden", true).build()))
+                .put(indexBuilder(".hidden", Settings.builder().put("index.hidden", true).build()))
+                .put(indexBuilder(".hidden-closed", Settings.builder().put("index.hidden", true).build()).state(IndexMetaData.State.CLOSE))
+                .put(indexBuilder("hidden-closed", Settings.builder().put("index.hidden", true).build()).state(IndexMetaData.State.CLOSE));
         ClusterState state = ClusterState.builder(new ClusterName("_name")).metaData(mdBuilder).build();
 
         // Only closed
-        IndicesOptions options = IndicesOptions.fromOptions(false, true, false, true);
+        IndicesOptions options = IndicesOptions.fromOptions(false, true, false, true, false);
         IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, options);
         String[] results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
         assertEquals(1, results.length);
@@ -291,8 +295,17 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
         assertEquals(1, results.length);
         assertEquals("bar", results[0]);
 
+        // implicit hidden for dot indices based on wildcard starting with .
+        results = indexNameExpressionResolver.concreteIndexNames(context, ".*");
+        assertEquals(1, results.length);
+        assertThat(results, arrayContainingInAnyOrder(".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, ".hidd*");
+        assertEquals(1, results.length);
+        assertThat(results, arrayContainingInAnyOrder(".hidden-closed"));
+
         // Only open
-        options = IndicesOptions.fromOptions(false, true, true, false);
+        options = IndicesOptions.fromOptions(false, true, true, false, false);
         context = new IndexNameExpressionResolver.Context(state, options);
         results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
         assertEquals(2, results.length);
@@ -306,8 +319,17 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
         assertEquals(1, results.length);
         assertEquals("bar", results[0]);
 
+        // implicit hidden for dot indices based on wildcard starting with .
+        results = indexNameExpressionResolver.concreteIndexNames(context, ".*");
+        assertEquals(1, results.length);
+        assertThat(results, arrayContainingInAnyOrder(".hidden"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, ".hidd*");
+        assertEquals(1, results.length);
+        assertThat(results, arrayContainingInAnyOrder(".hidden"));
+
         // Open and closed
-        options = IndicesOptions.fromOptions(false, true, true, true);
+        options = IndicesOptions.fromOptions(false, true, true, true, false);
         context = new IndexNameExpressionResolver.Context(state, options);
         results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
         assertEquals(3, results.length);
@@ -332,7 +354,111 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
         results = indexNameExpressionResolver.concreteIndexNames(context, "-*");
         assertEquals(0, results.length);
 
-        options = IndicesOptions.fromOptions(false, false, true, true);
+        // implicit hidden for dot indices based on wildcard starting with .
+        results = indexNameExpressionResolver.concreteIndexNames(context, ".*");
+        assertEquals(2, results.length);
+        assertThat(results, arrayContainingInAnyOrder(".hidden", ".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, ".hidd*");
+        assertEquals(2, results.length);
+        assertThat(results, arrayContainingInAnyOrder(".hidden", ".hidden-closed"));
+
+        // open closed and hidden
+        options = IndicesOptions.fromOptions(false, true, true, true, true);
+        context = new IndexNameExpressionResolver.Context(state, options);
+        results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
+        assertEquals(7, results.length);
+        assertThat(results, arrayContainingInAnyOrder("bar", "foobar", "foo", "hidden", "hidden-closed", ".hidden", ".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "foo*");
+        assertEquals(2, results.length);
+        assertThat(results, arrayContainingInAnyOrder("foobar", "foo"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "bar");
+        assertEquals(1, results.length);
+        assertEquals("bar", results[0]);
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "*", "-foo*");
+        assertEquals(5, results.length);
+        assertThat(results, arrayContainingInAnyOrder("bar", "hidden", "hidden-closed", ".hidden", ".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "*", "-foo", "-foobar");
+        assertEquals(5, results.length);
+        assertThat(results, arrayContainingInAnyOrder("bar", "hidden", "hidden-closed", ".hidden", ".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "*", "-foo", "-foobar", "-hidden*");
+        assertEquals(3, results.length);
+        assertThat(results, arrayContainingInAnyOrder("bar", ".hidden", ".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "hidden*");
+        assertEquals(2, results.length);
+        assertThat(results, arrayContainingInAnyOrder("hidden", "hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "hidden");
+        assertEquals(1, results.length);
+        assertThat(results, arrayContainingInAnyOrder("hidden"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "hidden-closed");
+        assertEquals(1, results.length);
+        assertThat(results, arrayContainingInAnyOrder("hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "-*");
+        assertEquals(0, results.length);
+
+        // open and hidden
+        options = IndicesOptions.fromOptions(false, true, true, false, true);
+        context = new IndexNameExpressionResolver.Context(state, options);
+        results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
+        assertEquals(4, results.length);
+        assertThat(results, arrayContainingInAnyOrder("bar", "foobar", "hidden", ".hidden"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "foo*");
+        assertEquals(1, results.length);
+        assertEquals("foobar", results[0]);
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "bar");
+        assertEquals(1, results.length);
+        assertEquals("bar", results[0]);
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "h*");
+        assertEquals(1, results.length);
+        assertEquals("hidden", results[0]);
+
+        // closed and hidden
+        options = IndicesOptions.fromOptions(false, true, false, true, true);
+        context = new IndexNameExpressionResolver.Context(state, options);
+        results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
+        assertEquals(3, results.length);
+        assertThat(results, arrayContainingInAnyOrder("foo", "hidden-closed", ".hidden-closed"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "foo*");
+        assertEquals(1, results.length);
+        assertEquals("foo", results[0]);
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "bar");
+        assertEquals(1, results.length);
+        assertEquals("bar", results[0]);
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "h*");
+        assertEquals(1, results.length);
+        assertEquals("hidden-closed", results[0]);
+
+        // only hidden
+        options = IndicesOptions.fromOptions(false, true, false, false, true);
+        context = new IndexNameExpressionResolver.Context(state, options);
+        results = indexNameExpressionResolver.concreteIndexNames(context, Strings.EMPTY_ARRAY);
+        assertThat(results, emptyArray());
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "h*");
+        assertThat(results, emptyArray());
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "hidden");
+        assertThat(results, arrayContainingInAnyOrder("hidden"));
+
+        results = indexNameExpressionResolver.concreteIndexNames(context, "hidden-closed");
+        assertThat(results, arrayContainingInAnyOrder("hidden-closed"));
+
+        options = IndicesOptions.fromOptions(false, false, true, true, true);
         IndexNameExpressionResolver.Context context2 = new IndexNameExpressionResolver.Context(state, options);
         IndexNotFoundException infe = expectThrows(IndexNotFoundException.class,
                 () -> indexNameExpressionResolver.concreteIndexNames(context2, "-*"));
@@ -349,7 +475,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
 
         //ignore unavailable and allow no indices
         {
-            IndicesOptions noExpandLenient = IndicesOptions.fromOptions(true, true, false, false);
+            IndicesOptions noExpandLenient = IndicesOptions.fromOptions(true, true, false, false, randomBoolean());
             IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandLenient);
             String[] results = indexNameExpressionResolver.concreteIndexNames(context, "baz*");
             assertThat(results, emptyArray());
@@ -371,7 +497,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
 
         //ignore unavailable but don't allow no indices
         {
-            IndicesOptions noExpandDisallowEmpty = IndicesOptions.fromOptions(true, false, false, false);
+            IndicesOptions noExpandDisallowEmpty = IndicesOptions.fromOptions(true, false, false, false, randomBoolean());
             IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandDisallowEmpty);
 
             {
@@ -396,7 +522,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
 
         //error on unavailable but allow no indices
         {
-            IndicesOptions noExpandErrorUnavailable = IndicesOptions.fromOptions(false, true, false, false);
+            IndicesOptions noExpandErrorUnavailable = IndicesOptions.fromOptions(false, true, false, false, randomBoolean());
             IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandErrorUnavailable);
             {
                 String[] results = indexNameExpressionResolver.concreteIndexNames(context, "baz*");
@@ -422,7 +548,7 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
 
         //error on both unavailable and no indices
         {
-            IndicesOptions noExpandStrict = IndicesOptions.fromOptions(false, false, false, false);
+            IndicesOptions noExpandStrict = IndicesOptions.fromOptions(false, false, false, false, randomBoolean());
             IndexNameExpressionResolver.Context context = new IndexNameExpressionResolver.Context(state, noExpandStrict);
             IndexNotFoundException infe = expectThrows(IndexNotFoundException.class,
                     () -> indexNameExpressionResolver.concreteIndexNames(context, "baz*"));
@@ -539,12 +665,17 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
     }
 
     private static IndexMetaData.Builder indexBuilder(String index) {
-        return IndexMetaData.builder(index).settings(settings());
+        return indexBuilder(index, Settings.EMPTY);
+    }
+
+    private static IndexMetaData.Builder indexBuilder(String index, Settings additionalSettings) {
+        return IndexMetaData.builder(index).settings(settings(additionalSettings));
     }
 
-    private static Settings.Builder settings() {
+    private static Settings.Builder settings(Settings additionalSettings) {
         return settings(Version.CURRENT).put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, 1)
-            .put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 0);
+            .put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 0)
+            .put(additionalSettings);
     }
 
     public void testConcreteIndicesIgnoreIndicesOneMissingIndex() {
@@ -1397,13 +1528,13 @@ public class IndexNameExpressionResolverTests extends ESTestCase {
 
     public void testIgnoreThrottled() {
         MetaData.Builder mdBuilder = MetaData.builder()
-            .put(indexBuilder("test-index").state(State.OPEN)
-                .settings(settings().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true))
+            .put(indexBuilder("test-index", Settings.builder().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true).build())
+                .state(State.OPEN)
                 .putAlias(AliasMetaData.builder("test-alias")))
             .put(indexBuilder("index").state(State.OPEN)
                 .putAlias(AliasMetaData.builder("test-alias2")))
-            .put(indexBuilder("index-closed").state(State.CLOSE)
-                .settings(settings().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true))
+            .put(indexBuilder("index-closed", Settings.builder().put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), true).build())
+                .state(State.CLOSE)
                 .putAlias(AliasMetaData.builder("test-alias-closed")));
         ClusterState state = ClusterState.builder(new ClusterName("_name")).metaData(mdBuilder).build();
         {

server/src/test/java/org/elasticsearch/index/HiddenIndexIT.java

@@ -0,0 +1,129 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.index;
+
+import org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse;
+import org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse;
+import org.elasticsearch.action.search.SearchResponse;
+import org.elasticsearch.action.support.IndicesOptions;
+import org.elasticsearch.action.support.WriteRequest.RefreshPolicy;
+import org.elasticsearch.cluster.metadata.MappingMetaData;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.index.query.QueryBuilders;
+import org.elasticsearch.indices.InvalidIndexTemplateException;
+import org.elasticsearch.test.ESIntegTestCase;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.is;
+
+public class HiddenIndexIT extends ESIntegTestCase {
+
+    public void testHiddenIndexSearch() {
+        assertAcked(client().admin().indices().prepareCreate("hidden-index")
+            .setSettings(Settings.builder().put("index.hidden", true).build())
+            .get());
+        client().prepareIndex("hidden-index").setSource("foo", "bar").setRefreshPolicy(RefreshPolicy.IMMEDIATE).get();
+
+        // default not visible to wildcard expansion
+        SearchResponse searchResponse =
+            client().prepareSearch(randomFrom("*", "_all", "h*", "*index")).setSize(1000).setQuery(QueryBuilders.matchAllQuery()).get();
+        boolean matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> "hidden-index".equals(hit.getIndex()));
+        assertFalse(matchedHidden);
+
+        // direct access allowed
+        searchResponse = client().prepareSearch("hidden-index").setSize(1000).setQuery(QueryBuilders.matchAllQuery()).get();
+        matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> "hidden-index".equals(hit.getIndex()));
+        assertTrue(matchedHidden);
+
+        // with indices option to include hidden
+        searchResponse = client().prepareSearch(randomFrom("*", "_all", "h*", "*index"))
+            .setSize(1000)
+            .setQuery(QueryBuilders.matchAllQuery())
+            .setIndicesOptions(IndicesOptions.LENIENT_EXPAND_OPEN_HIDDEN)
+            .get();
+        matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> "hidden-index".equals(hit.getIndex()));
+        assertTrue(matchedHidden);
+
+        // implicit based on use of pattern starting with . and a wildcard
+        assertAcked(client().admin().indices().prepareCreate(".hidden-index")
+            .setSettings(Settings.builder().put("index.hidden", true).build())
+            .get());
+        client().prepareIndex(".hidden-index").setSource("foo", "bar").setRefreshPolicy(RefreshPolicy.IMMEDIATE).get();
+        searchResponse = client().prepareSearch(randomFrom(".*", ".hidden-*"))
+            .setSize(1000)
+            .setQuery(QueryBuilders.matchAllQuery())
+            .get();
+        matchedHidden = Arrays.stream(searchResponse.getHits().getHits()).anyMatch(hit -> ".hidden-index".equals(hit.getIndex()));
+        assertTrue(matchedHidden);
+    }
+
+    public void testGlobalTemplatesDoNotApply() {
+        assertAcked(client().admin().indices().preparePutTemplate("a_global_template").setPatterns(List.of("*"))
+            .setMapping("foo", "type=text").get());
+        assertAcked(client().admin().indices().preparePutTemplate("not_global_template").setPatterns(List.of("a*"))
+            .setMapping("bar", "type=text").get());
+        assertAcked(client().admin().indices().preparePutTemplate("specific_template").setPatterns(List.of("a_hidden_index"))
+            .setMapping("baz", "type=text").get());
+        assertAcked(client().admin().indices().preparePutTemplate("unused_template").setPatterns(List.of("not_used"))
+            .setMapping("foobar", "type=text").get());
+
+        assertAcked(client().admin().indices().prepareCreate("a_hidden_index")
+            .setSettings(Settings.builder().put("index.hidden", true).build()).get());
+
+        GetMappingsResponse mappingsResponse = client().admin().indices().prepareGetMappings("a_hidden_index").get();
+        assertThat(mappingsResponse.mappings().size(), is(1));
+        MappingMetaData mappingMetaData = mappingsResponse.mappings().get("a_hidden_index");
+        assertNotNull(mappingMetaData);
+        Map<String, Object> propertiesMap = (Map<String, Object>) mappingMetaData.getSourceAsMap().get("properties");
+        assertNotNull(propertiesMap);
+        assertThat(propertiesMap.size(), is(2));
+        Map<String, Object> barMap = (Map<String, Object>) propertiesMap.get("bar");
+        assertNotNull(barMap);
+        assertThat(barMap.get("type"), is("text"));
+        Map<String, Object> bazMap = (Map<String, Object>) propertiesMap.get("baz");
+        assertNotNull(bazMap);
+        assertThat(bazMap.get("type"), is("text"));
+    }
+
+    public void testGlobalTemplateCannotMakeIndexHidden() {
+        InvalidIndexTemplateException invalidIndexTemplateException = expectThrows(InvalidIndexTemplateException.class,
+            () -> client().admin().indices().preparePutTemplate("a_global_template")
+                .setPatterns(List.of("*"))
+                .setSettings(Settings.builder().put("index.hidden", randomBoolean()).build())
+                .get());
+        assertThat(invalidIndexTemplateException.getMessage(), containsString("global templates may not specify the setting index.hidden"));
+    }
+
+    public void testNonGlobalTemplateCanMakeIndexHidden() {
+        assertAcked(client().admin().indices().preparePutTemplate("a_global_template")
+            .setPatterns(List.of("my_hidden_pattern*"))
+            .setMapping("foo", "type=text")
+            .setSettings(Settings.builder().put("index.hidden", true).build())
+            .get());
+        assertAcked(client().admin().indices().prepareCreate("my_hidden_pattern1").get());
+        GetSettingsResponse getSettingsResponse = client().admin().indices().prepareGetSettings("my_hidden_pattern1").get();
+        assertThat(getSettingsResponse.getSetting("my_hidden_pattern1", "index.hidden"), is("true"));
+    }
+}

test/framework/src/main/java/org/elasticsearch/test/TestCluster.java

@@ -25,6 +25,7 @@ import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse;
 import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
+import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.cluster.metadata.IndexMetaData;
 import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
@@ -134,7 +135,9 @@ public abstract class TestCluster implements Closeable {
         assert indices != null && indices.length > 0;
         if (size() > 0) {
             try {
-                assertAcked(client().admin().indices().prepareDelete(indices));
+                // include wiping hidden indices!
+                assertAcked(client().admin().indices().prepareDelete(indices)
+                    .setIndicesOptions(IndicesOptions.fromOptions(false, true, true, true, true, false, false, true, false)));
             } catch (IndexNotFoundException e) {
                 // ignore
             } catch (IllegalArgumentException e) {

x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/job/persistence/JobResultsProviderTests.java

@@ -160,6 +160,7 @@ public class JobResultsProviderTests extends ESTestCase {
 
         ImmutableOpenMap<String, AliasMetaData> aliases = ImmutableOpenMap.of();
         when(indexMetaData.getAliases()).thenReturn(aliases);
+        when(indexMetaData.getSettings()).thenReturn(Settings.EMPTY);
 
         ImmutableOpenMap<String, IndexMetaData> indexMap = ImmutableOpenMap.<String, IndexMetaData>builder()
                 .fPut(AnomalyDetectorsIndex.jobResultsAliasedName("foo"), indexMetaData).build();

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

@@ -16,6 +16,7 @@ import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.cluster.metadata.AliasMetaData;
 import org.elasticsearch.cluster.metadata.AliasOrIndex;
 import org.elasticsearch.cluster.metadata.IndexMetaData;
+import org.elasticsearch.cluster.metadata.IndexMetaData.State;
 import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
 import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.cluster.service.ClusterService;
@@ -120,16 +121,15 @@ class IndicesAndAliasesResolver {
                     : "indices are: " + Arrays.toString(indicesRequest.indices()); // Arrays.toString() can handle null values - all good
             resolvedIndicesBuilder.addLocal(getPutMappingIndexOrAlias((PutMappingRequest) indicesRequest, authorizedIndices, metaData));
         } else if (indicesRequest instanceof IndicesRequest.Replaceable) {
-            IndicesRequest.Replaceable replaceable = (IndicesRequest.Replaceable) indicesRequest;
-            final boolean replaceWildcards = indicesRequest.indicesOptions().expandWildcardsOpen()
-                    || indicesRequest.indicesOptions().expandWildcardsClosed();
-            IndicesOptions indicesOptions = indicesRequest.indicesOptions();
+            final IndicesRequest.Replaceable replaceable = (IndicesRequest.Replaceable) indicesRequest;
+            final IndicesOptions indicesOptions = indicesRequest.indicesOptions();
+            final boolean replaceWildcards = indicesOptions.expandWildcardsOpen() || indicesOptions.expandWildcardsClosed();
 
             // check for all and return list of authorized indices
             if (IndexNameExpressionResolver.isAllIndices(indicesList(indicesRequest.indices()))) {
                 if (replaceWildcards) {
                     for (String authorizedIndex : authorizedIndices) {
-                        if (isIndexVisible(authorizedIndex, indicesOptions, metaData)) {
+                        if (isIndexVisible("*", authorizedIndex, indicesOptions, metaData)) {
                             resolvedIndicesBuilder.addLocal(authorizedIndex);
                         }
                     }
@@ -356,7 +356,8 @@ class IndicesAndAliasesResolver {
                 if (replaceWildcards && Regex.isSimpleMatchPattern(dateMathName)) {
                     // continue
                     aliasOrIndex = dateMathName;
-                } else if (authorizedIndices.contains(dateMathName) && isIndexVisible(dateMathName, indicesOptions, metaData, true)) {
+                } else if (authorizedIndices.contains(dateMathName) &&
+                    isIndexVisible(aliasOrIndex, dateMathName, indicesOptions, metaData, true)) {
                     if (minus) {
                         finalIndices.remove(dateMathName);
                     } else {
@@ -373,7 +374,8 @@ class IndicesAndAliasesResolver {
                 wildcardSeen = true;
                 Set<String> resolvedIndices = new HashSet<>();
                 for (String authorizedIndex : authorizedIndices) {
-                    if (Regex.simpleMatch(aliasOrIndex, authorizedIndex) && isIndexVisible(authorizedIndex, indicesOptions, metaData)) {
+                    if (Regex.simpleMatch(aliasOrIndex, authorizedIndex) &&
+                        isIndexVisible(aliasOrIndex, authorizedIndex, indicesOptions, metaData)) {
                         resolvedIndices.add(authorizedIndex);
                     }
                 }
@@ -408,11 +410,12 @@ class IndicesAndAliasesResolver {
         return finalIndices;
     }
 
-    private static boolean isIndexVisible(String index, IndicesOptions indicesOptions, MetaData metaData) {
-        return isIndexVisible(index, indicesOptions, metaData, false);
+    private static boolean isIndexVisible(String expression, String index, IndicesOptions indicesOptions, MetaData metaData) {
+        return isIndexVisible(expression, index, indicesOptions, metaData, false);
     }
 
-    private static boolean isIndexVisible(String index, IndicesOptions indicesOptions, MetaData metaData, boolean dateMathExpression) {
+    private static boolean isIndexVisible(String expression, String index, IndicesOptions indicesOptions, MetaData metaData,
+                                          boolean dateMathExpression) {
         AliasOrIndex aliasOrIndex = metaData.getAliasAndIndexLookup().get(index);
         if (aliasOrIndex.isAlias()) {
             //it's an alias, ignore expandWildcardsOpen and expandWildcardsClosed.
@@ -422,15 +425,29 @@ class IndicesAndAliasesResolver {
         }
         assert aliasOrIndex.getIndices().size() == 1 : "concrete index must point to a single index";
         IndexMetaData indexMetaData = aliasOrIndex.getIndices().get(0);
-        if (indexMetaData.getState() == IndexMetaData.State.CLOSE && (indicesOptions.expandWildcardsClosed() || dateMathExpression)) {
+        final boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.get(indexMetaData.getSettings());
+        if (isHidden && indicesOptions.expandWildcardsHidden() == false && isVisibleDueToImplicitHidden(expression, index) == false) {
+            return false;
+        }
+
+        // the index is not hidden and since it is a date math expression, we consider it visible regardless of open/closed
+        if (dateMathExpression) {
+            assert State.values().length == 2 : "a new IndexMetaData.State value may need to be handled!";
+            return true;
+        }
+        if (indexMetaData.getState() == IndexMetaData.State.CLOSE && indicesOptions.expandWildcardsClosed()) {
             return true;
         }
-        if (indexMetaData.getState() == IndexMetaData.State.OPEN && (indicesOptions.expandWildcardsOpen() || dateMathExpression)) {
+        if (indexMetaData.getState() == IndexMetaData.State.OPEN && indicesOptions.expandWildcardsOpen()) {
             return true;
         }
         return false;
     }
 
+    private static boolean isVisibleDueToImplicitHidden(String expression, String index) {
+        return index.startsWith(".") && expression.startsWith(".") && Regex.isSimpleMatchPattern(expression);
+    }
+
     private static List<String> indicesList(String[] list) {
         return (list == null) ? null : Arrays.asList(list);
     }

x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java

@@ -33,6 +33,7 @@ import org.elasticsearch.action.support.PlainActionFuture;
 import org.elasticsearch.action.termvectors.MultiTermVectorsRequest;
 import org.elasticsearch.cluster.metadata.AliasMetaData;
 import org.elasticsearch.cluster.metadata.IndexMetaData;
+import org.elasticsearch.cluster.metadata.IndexMetaData.State;
 import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
 import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.cluster.service.ClusterService;
@@ -123,13 +124,13 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
                         .putAlias(AliasMetaData.builder("foounauthorized")).settings(settings))
                 .put(indexBuilder("foobar").putAlias(AliasMetaData.builder("foofoobar"))
                         .putAlias(AliasMetaData.builder("foobarfoo")).settings(settings))
-                .put(indexBuilder("closed").state(IndexMetaData.State.CLOSE)
+                .put(indexBuilder("closed").state(State.CLOSE)
                         .putAlias(AliasMetaData.builder("foofoobar")).settings(settings))
-                .put(indexBuilder("foofoo-closed").state(IndexMetaData.State.CLOSE).settings(settings))
-                .put(indexBuilder("foobar-closed").state(IndexMetaData.State.CLOSE).settings(settings))
+                .put(indexBuilder("foofoo-closed").state(State.CLOSE).settings(settings))
+                .put(indexBuilder("foobar-closed").state(State.CLOSE).settings(settings))
                 .put(indexBuilder("foofoo").putAlias(AliasMetaData.builder("barbaz")).settings(settings))
                 .put(indexBuilder("bar").settings(settings))
-                .put(indexBuilder("bar-closed").state(IndexMetaData.State.CLOSE).settings(settings))
+                .put(indexBuilder("bar-closed").state(State.CLOSE).settings(settings))
                 .put(indexBuilder("bar2").settings(settings))
                 .put(indexBuilder(indexNameExpressionResolver.resolveDateMathExpression("<datetime-{now/M}>")).settings(settings))
                 .put(indexBuilder("-index10").settings(settings))
@@ -139,6 +140,12 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
                 .put(indexBuilder("logs-00001").putAlias(AliasMetaData.builder("logs-alias").writeIndex(false)).settings(settings))
                 .put(indexBuilder("logs-00002").putAlias(AliasMetaData.builder("logs-alias").writeIndex(false)).settings(settings))
                 .put(indexBuilder("logs-00003").putAlias(AliasMetaData.builder("logs-alias").writeIndex(true)).settings(settings))
+                .put(indexBuilder("hidden-open").settings(Settings.builder().put(settings).put("index.hidden", true).build()))
+                .put(indexBuilder(".hidden-open").settings(Settings.builder().put(settings).put("index.hidden", true).build()))
+                .put(indexBuilder(".hidden-closed").state(State.CLOSE)
+                    .settings(Settings.builder().put(settings).put("index.hidden", true).build()))
+                .put(indexBuilder("hidden-closed").state(State.CLOSE)
+                    .settings(Settings.builder().put(settings).put("index.hidden", true).build()))
                 .put(indexBuilder(securityIndexName).settings(settings)).build();
 
         if (withAlias) {
@@ -150,7 +157,8 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
         userDashIndices = new User("dash", "dash");
         userNoIndices = new User("test", "test");
         rolesStore = mock(CompositeRolesStore.class);
-        String[] authorizedIndices = new String[] { "bar", "bar-closed", "foofoobar", "foobarfoo", "foofoo", "missing", "foofoo-closed"};
+        String[] authorizedIndices = new String[] { "bar", "bar-closed", "foofoobar", "foobarfoo", "foofoo", "missing", "foofoo-closed",
+            "hidden-open", "hidden-closed", ".hidden-open", ".hidden-closed"};
         String[] dashIndices = new String[]{"-index10", "-index11", "-index20", "-index21"};
         roleMap = new HashMap<>();
         roleMap.put("role", new RoleDescriptor("role", null,
@@ -1368,8 +1376,56 @@ public class IndicesAndAliasesResolverTests extends ESTestCase {
         assertEquals(message, index, putMappingIndexOrAlias);
     }
 
-    // TODO with the removal of DeleteByQuery is there another way to test resolving a write action?
-
+    public void testHiddenIndicesResolution() {
+        SearchRequest searchRequest = new SearchRequest();
+        searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, true, true, true));
+        List<String> authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
+        ResolvedIndices resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
+        assertThat(resolvedIndices.getLocal(), containsInAnyOrder("bar", "bar-closed", "foofoobar", "foobarfoo", "foofoo", "foofoo-closed",
+            "hidden-open", "hidden-closed", ".hidden-open", ".hidden-closed"));
+        assertThat(resolvedIndices.getRemote(), emptyIterable());
+
+        // open + hidden
+        searchRequest = new SearchRequest();
+        searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, true, false, true));
+        authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
+        resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
+        assertThat(resolvedIndices.getLocal(),
+            containsInAnyOrder("bar", "foofoobar", "foobarfoo", "foofoo", "hidden-open", ".hidden-open"));
+        assertThat(resolvedIndices.getRemote(), emptyIterable());
+
+        // open + implicit hidden for . indices
+        searchRequest = new SearchRequest(randomFrom(".*", ".hid*"));
+        searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, true, false, false));
+        authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
+        resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
+        assertThat(resolvedIndices.getLocal(), containsInAnyOrder(".hidden-open"));
+        assertThat(resolvedIndices.getRemote(), emptyIterable());
+
+        // closed + hidden, ignore aliases
+        searchRequest = new SearchRequest();
+        searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, false, true, true, true, false, true, false));
+        authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
+        resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
+        assertThat(resolvedIndices.getLocal(), containsInAnyOrder("bar-closed", "foofoo-closed", "hidden-closed", ".hidden-closed"));
+        assertThat(resolvedIndices.getRemote(), emptyIterable());
+
+        // closed + implicit hidden for . indices
+        searchRequest = new SearchRequest(randomFrom(".*", ".hid*"));
+        searchRequest.indicesOptions(IndicesOptions.fromOptions(false, false, false, true, false));
+        authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
+        resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
+        assertThat(resolvedIndices.getLocal(), containsInAnyOrder(".hidden-closed"));
+        assertThat(resolvedIndices.getRemote(), emptyIterable());
+
+        // allow no indices, do not expand to open or closed, expand hidden, ignore aliases
+        searchRequest = new SearchRequest();
+        searchRequest.indicesOptions(IndicesOptions.fromOptions(false, true, false, false, false, true, false, true, false));
+        authorizedIndices = buildAuthorizedIndices(user, SearchAction.NAME);
+        resolvedIndices = defaultIndicesResolver.resolveIndicesAndAliases(searchRequest, metaData, authorizedIndices);
+        assertThat(resolvedIndices.getLocal(), contains("-*"));
+        assertThat(resolvedIndices.getRemote(), emptyIterable());
+    }
 
     private List<String> buildAuthorizedIndices(User user, String action) {
         PlainActionFuture<Role> rolesListener = new PlainActionFuture<>();