Rename component templates and pipelines according to the new naming conventions (#99975)

- Creates a new StackTemplateRegistry that uses the new names
- The new registry only respects stack.templates.enabled for index templates
- Renames the old registry to LegacyStackTemplateRegistry
- Component templates are not duplicated but registered under two different names
- Documents the new naming convention
- Index templates are not renamed, at least for now, as there are some challenges with it
  See 7fd0423 for more details.

docs/changelog/99975.yaml

@@ -0,0 +1,5 @@
+pr: 99975
+summary: Rename component templates and pipelines according to the new naming conventions
+area: Indices APIs
+type: enhancement
+issues: []

docs/reference/ilm/apis/put-lifecycle.asciidoc

@@ -37,6 +37,11 @@ previous versions.
 
 `<policy_id>`::
   (Required, string) Identifier for the policy.
++
+[IMPORTANT]
+====
+To avoid naming collisions with built-in and Fleet-managed ILM policies, avoid using `@` as part of the id of your own ILM policies.
+====
 
 [[ilm-put-lifecycle-query-params]]
 ==== {api-query-parms-title}

docs/reference/indices/index-templates.asciidoc

@@ -28,7 +28,7 @@ applied.
 template, the settings from the <<indices-create-index,create index>> request
 take precedence over settings specified in the index template and its component
 templates.
-* Settings specified in the index template itself take precedence over the settings 
+* Settings specified in the index template itself take precedence over the settings
 in its component templates.
 * If a new data stream or index matches more than one index template, the index
 template with the highest priority is used.
@@ -65,6 +65,8 @@ For example, if you don't use {fleet} or {agent} and want to create a template
 for the `logs-*` index pattern, assign your template a priority of `500`. This
 ensures your template is applied instead of the built-in template for
 `logs-*-*`.
+
+- To avoid naming collisions with built-in and Fleet-managed index templates, avoid using `@` as part of the name of your own index templates.
 ****
 
 [discrete]

docs/reference/indices/put-component-template.asciidoc

@@ -91,23 +91,28 @@ Name of the component template to create.
 {es} includes the following built-in component templates:
 
 // tag::built-in-component-templates[]
-- `logs-mappings`
-- `logs-settings`
-- `metrics-mappings`
-- `metrics-settings`
-- `metrics-tsdb-settings`
-- `synthetics-mapping`
-- `synthetics-settings`
+- `logs@mappings`
+- `logs@settings`
+- `metrics@mappings`
+- `metrics@settings`
+- `metrics@tsdb-settings`
+- `synthetics@mapping`
+- `synthetics@settings`
 // end::built-in-component-templates[]
 
 {fleet-guide}/fleet-overview.html[{agent}] uses these templates to configure
-backing indices for its data streams. If you use {agent} and want to overwrite
-one of these templates, set the `version` for your replacement template higher
-than the current version.
-
-If you don't use {agent} and want to disable all built-in component and index
-templates, set <<stack-templates-enabled,`stack.templates.enabled`>> to `false`
-using the <<cluster-update-settings,cluster update settings API>>.
+backing indices for its data streams.
+If you want to customize these templates, don't override them as they may be reset after an update.
+Instead, look for a `*@custom` component template in the `composed_of` section of the managed index template.
+These custom component templates allow you to customize the mappings of managed index templates,
+without having to override managed index templates or component templates.
+Note that the custom component templates may not exist yet.
+After you create them using the <<indices-component-template>>, they'll be picked up by the index template.
+See <<data-streams-change-mappings-and-settings>> on how to apply the changes to the corresponding data stream.
+
+To avoid naming collisions with built-in and Fleet-managed component templates,
+avoid using `@` as part of your own component template names.
+The exception of that rule are the `*@custom` component templates that let you safely customize managed index templates.
 ====
 
 [[put-component-template-api-query-params]]

docs/reference/ingest.asciidoc

@@ -307,13 +307,13 @@ matches these templates to your {fleet} data streams based on the
 {fleet-guide}/data-streams.html#data-streams-naming-scheme[stream's naming
 scheme].
 
-Each default integration pipeline calls a nonexistent, unversioned `@custom` ingest pipeline.
+Each default integration pipeline calls a nonexistent, unversioned `*@custom` ingest pipeline.
 If unaltered, this pipeline call has no effect on your data. However, you can modify this call to
 create custom pipelines for integrations that persist across upgrades.
 Refer to {fleet-guide}/data-streams-pipeline-tutorial.html[Tutorial: Transform data with custom ingest pipelines] to learn more.
 
 {fleet} doesn't provide a default ingest pipeline for the **Custom logs** integration,
-but you can specify a pipeline for this integration using an 
+but you can specify a pipeline for this integration using an
 <<pipeline-custom-logs-index-template,index template>> or a
 <<pipeline-custom-logs-configuration,custom configuration>>.
 

docs/reference/ingest/apis/put-pipeline.asciidoc

@@ -43,7 +43,13 @@ PUT _ingest/pipeline/my-pipeline-id
 
 `<pipeline>`::
 (Required, string) ID of the ingest pipeline to create or update.
-
++
+[IMPORTANT]
+====
+To avoid naming collisions with built-in and Fleet-managed ingest pipelines, avoid using `@` as part of your own ingest pipelines names.
+The exception of that rule are the `*@custom` ingest pipelines that let you safely add a custom pipeline to managed pipelines.
+See also <<pipelines-for-fleet-elastic-agent>>.
+====
 
 [[put-pipeline-api-query-params]]
 ==== {api-query-parms-title}

modules/data-streams/src/javaRestTest/java/org/elasticsearch/datastreams/DataStreamUpgradeRestIT.java

@@ -65,7 +65,7 @@ public class DataStreamUpgradeRestIT extends DisabledSecurityDataStreamTestCase
             {
               "index_patterns": [ "logs-mysql-*" ],
               "priority": 200,
-              "composed_of": [ "logs-mappings", "logs-settings" ],
+              "composed_of": [ "logs@mappings", "logs@settings" ],
               "data_stream": {},
               "template": {
                 "mappings": {
@@ -103,7 +103,7 @@ public class DataStreamUpgradeRestIT extends DisabledSecurityDataStreamTestCase
             {
               "index_patterns": [ "logs-mysql-*" ],
               "priority": 200,
-              "composed_of": [ "logs-mappings", "logs-settings" ],
+              "composed_of": [ "logs@mappings", "logs@settings" ],
               "data_stream": {},
               "template": {
                 "mappings": {
@@ -168,7 +168,7 @@ public class DataStreamUpgradeRestIT extends DisabledSecurityDataStreamTestCase
             {
               "index_patterns": [ "logs-mysql-*" ],
               "priority": 200,
-              "composed_of": [ "logs-mappings", "logs-settings" ],
+              "composed_of": [ "logs@mappings", "logs@settings" ],
               "data_stream": {},
               "template": {
                 "mappings": {
@@ -205,7 +205,7 @@ public class DataStreamUpgradeRestIT extends DisabledSecurityDataStreamTestCase
             {
               "index_patterns": [ "logs-mysql-*" ],
               "priority": 200,
-              "composed_of": [ "logs-mappings", "logs-settings" ],
+              "composed_of": [ "logs@mappings", "logs@settings" ],
               "data_stream": {},
               "template": {
                 "mappings": {
@@ -285,7 +285,7 @@ public class DataStreamUpgradeRestIT extends DisabledSecurityDataStreamTestCase
     private void waitForLogsComponentTemplateInitialization() throws Exception {
         assertBusy(() -> {
             try {
-                Request logsComponentTemplateRequest = new Request("GET", "/_component_template/logs-*");
+                Request logsComponentTemplateRequest = new Request("GET", "/_component_template/logs@*");
                 Response response = client().performRequest(logsComponentTemplateRequest);
                 assertThat(response.getStatusLine().getStatusCode(), equalTo(200));
 
@@ -297,7 +297,7 @@ public class DataStreamUpgradeRestIT extends DisabledSecurityDataStreamTestCase
                 List<?> componentTemplates = (List<?>) responseBody.get("component_templates");
                 assertThat(componentTemplates.size(), equalTo(2));
                 Set<String> names = componentTemplates.stream().map(m -> ((Map<String, String>) m).get("name")).collect(Collectors.toSet());
-                assertThat(names, containsInAnyOrder("logs-mappings", "logs-settings"));
+                assertThat(names, containsInAnyOrder("logs@mappings", "logs@settings"));
             } catch (ResponseException responseException) {
                 // Retry in case of a 404, maybe they haven't been initialized yet.
                 if (responseException.getResponse().getStatusLine().getStatusCode() == 404) {

modules/data-streams/src/javaRestTest/java/org/elasticsearch/datastreams/EcsLogsDataStreamIT.java

@@ -47,7 +47,7 @@ public class EcsLogsDataStreamIT extends DisabledSecurityDataStreamTestCase {
                       "processors": [
                         {
                           "pipeline" : {
-                            "name": "logs@json-message",
+                            "name": "logs@json-pipeline",
                             "description": "A pipeline that automatically parses JSON log events into top-level fields if they are such"
                           }
                         }

modules/data-streams/src/javaRestTest/java/org/elasticsearch/datastreams/LogsDataStreamIT.java

@@ -294,7 +294,7 @@ public class LogsDataStreamIT extends DisabledSecurityDataStreamTestCase {
                       "processors": [
                         {
                           "pipeline" : {
-                            "name": "logs@json-message",
+                            "name": "logs@json-pipeline",
                             "description": "A pipeline that automatically parses JSON log events into top-level fields if they are such"
                           }
                         }
@@ -452,7 +452,7 @@ public class LogsDataStreamIT extends DisabledSecurityDataStreamTestCase {
                   "priority": 200,
                   "data_stream": {},
                   "index_patterns": ["logs-*-*"],
-                  "composed_of": ["logs-test-subobjects-mappings", "ecs@dynamic_templates"]
+                  "composed_of": ["logs-test-subobjects-mappings", "ecs@mappings"]
                 }
                 """);
             assertOK(client.performRequest(request));

test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java

@@ -642,14 +642,23 @@ public abstract class ESRestTestCase extends ESTestCase {
             "watch-history-ilm-policy-16",
             "ml-size-based-ilm-policy",
             "logs",
+            "logs@lifecycle",
             "metrics",
+            "metrics@lifecycle",
             "profiling",
+            "profiling@lifecycle",
             "synthetics",
+            "synthetics@lifecycle",
             "7-days-default",
+            "7-days@lifecycle",
             "30-days-default",
+            "30-days@lifecycle",
             "90-days-default",
+            "90-days@lifecycle",
             "180-days-default",
+            "180-days@lifecycle",
             "365-days-default",
+            "365-days@lifecycle",
             ".fleet-files-ilm-policy",
             ".fleet-file-data-ilm-policy",
             ".fleet-actions-results-ilm-policy",
@@ -1869,6 +1878,10 @@ public abstract class ESRestTestCase extends ESTestCase {
         if (name.startsWith("elastic-connectors")) {
             return true;
         }
+        if (name.contains("@")) {
+            // We have a naming convention that internal component templates contain `@`. See also index-templates.asciidoc.
+            return true;
+        }
         switch (name) {
             case ".watches":
             case "security_audit_log":
@@ -1891,7 +1904,6 @@ public abstract class ESRestTestCase extends ESTestCase {
             case "logstash-index-template":
             case "security-index-template":
             case "data-streams-mappings":
-            case "ecs@dynamic_templates":
             case "search-acl-filter":
             case ".kibana-reporting":
                 return true;

x-pack/plugin/core/template-resources/src/main/resources/logs-settings.json → x-pack/plugin/core/template-resources/src/main/resources/logs@settings.json

@@ -12,7 +12,7 @@
         "mapping": {
           "ignore_malformed": true
         },
-        "default_pipeline": "logs-default-pipeline"
+        "default_pipeline": "logs@default-pipeline"
       }
     }
   },

x-pack/plugin/core/template-resources/src/main/resources/logs-template.json → x-pack/plugin/core/template-resources/src/main/resources/logs@template.json

@@ -3,10 +3,10 @@
   "priority": 100,
   "data_stream": {},
   "composed_of": [
-    "logs-mappings",
-    "logs-settings",
+    "logs@mappings",
+    "logs@settings",
     "logs@custom",
-    "ecs@dynamic_templates"
+    "ecs@mappings"
   ],
   "ignore_missing_component_templates": ["logs@custom"],
   "allow_auto_create": true,

x-pack/plugin/core/template-resources/src/main/resources/metrics-template.json → x-pack/plugin/core/template-resources/src/main/resources/metrics@template.json

@@ -3,9 +3,9 @@
   "priority": 100,
   "data_stream": {},
   "composed_of": [
-    "metrics-mappings",
-    "data-streams-mappings",
-    "metrics-settings"
+    "metrics@mappings",
+    "data-streams@mappings",
+    "metrics@settings"
   ],
   "allow_auto_create": true,
   "_meta": {

x-pack/plugin/core/template-resources/src/main/resources/synthetics-template.json → x-pack/plugin/core/template-resources/src/main/resources/synthetics@template.json

@@ -3,9 +3,9 @@
   "priority": 100,
   "data_stream": {},
   "composed_of": [
-    "synthetics-mappings",
-    "data-streams-mappings",
-    "synthetics-settings"
+    "synthetics@mappings",
+    "data-streams@mappings",
+    "synthetics@settings"
   ],
   "allow_auto_create": true,
   "_meta": {

x-pack/plugin/ml/qa/native-multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/MlNativeIntegTestCase.java

@@ -63,6 +63,7 @@ import org.elasticsearch.xpack.core.ilm.LifecycleAction;
 import org.elasticsearch.xpack.core.ilm.LifecycleSettings;
 import org.elasticsearch.xpack.core.ilm.LifecycleType;
 import org.elasticsearch.xpack.core.ilm.RolloverAction;
+import org.elasticsearch.xpack.core.ilm.ShrinkAction;
 import org.elasticsearch.xpack.core.ilm.TimeseriesLifecycleType;
 import org.elasticsearch.xpack.core.ml.MachineLearningField;
 import org.elasticsearch.xpack.core.ml.MlMetaIndex;
@@ -319,6 +320,7 @@ abstract class MlNativeIntegTestCase extends ESIntegTestCase {
             entries.add(new NamedWriteableRegistry.Entry(LifecycleAction.class, DeleteAction.NAME, DeleteAction::readFrom));
             entries.add(new NamedWriteableRegistry.Entry(LifecycleAction.class, ForceMergeAction.NAME, ForceMergeAction::new));
             entries.add(new NamedWriteableRegistry.Entry(LifecycleAction.class, RolloverAction.NAME, RolloverAction::read));
+            entries.add(new NamedWriteableRegistry.Entry(LifecycleAction.class, ShrinkAction.NAME, ShrinkAction::new));
             entries.add(
                 new NamedWriteableRegistry.Entry(
                     PersistentTaskParams.class,

x-pack/plugin/stack/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/stack/10_basic.yml

@@ -9,34 +9,58 @@ setup:
   - do:
       ilm.get_lifecycle:
         policy: "logs"
+  - do:
+      ilm.get_lifecycle:
+        policy: "logs@lifecycle"
 
   - do:
       ilm.get_lifecycle:
         policy: "metrics"
+  - do:
+      ilm.get_lifecycle:
+        policy: "metrics@lifecycle"
 
   - do:
       cluster.get_component_template:
         name: data-streams-mappings
+  - do:
+      cluster.get_component_template:
+        name: data-streams@mappings
 
   - do:
       cluster.get_component_template:
         name: logs-mappings
+  - do:
+      cluster.get_component_template:
+        name: logs@mappings
 
   - do:
       cluster.get_component_template:
         name: logs-settings
+  - do:
+      cluster.get_component_template:
+        name: logs@settings
 
   - do:
       cluster.get_component_template:
         name: metrics-mappings
+  - do:
+      cluster.get_component_template:
+        name: metrics@mappings
 
   - do:
       cluster.get_component_template:
         name: metrics-settings
+  - do:
+      cluster.get_component_template:
+        name: metrics@settings
 
   - do:
       cluster.get_component_template:
         name: metrics-tsdb-settings
+  - do:
+      cluster.get_component_template:
+        name: metrics@tsdb-settings
 
   - do:
       indices.get_index_template:

x-pack/plugin/stack/src/main/java/org/elasticsearch/xpack/stack/LegacyStackTemplateRegistry.java

@@ -0,0 +1,267 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+package org.elasticsearch.xpack.stack;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.elasticsearch.Version;
+import org.elasticsearch.client.internal.Client;
+import org.elasticsearch.cluster.ClusterChangedEvent;
+import org.elasticsearch.cluster.metadata.ComponentTemplate;
+import org.elasticsearch.cluster.metadata.ComposableIndexTemplate;
+import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.threadpool.ThreadPool;
+import org.elasticsearch.xcontent.NamedXContentRegistry;
+import org.elasticsearch.xcontent.XContentParserConfiguration;
+import org.elasticsearch.xcontent.json.JsonXContent;
+import org.elasticsearch.xpack.core.ClientHelper;
+import org.elasticsearch.xpack.core.ilm.LifecyclePolicy;
+import org.elasticsearch.xpack.core.template.IndexTemplateConfig;
+import org.elasticsearch.xpack.core.template.IndexTemplateRegistry;
+import org.elasticsearch.xpack.core.template.IngestPipelineConfig;
+import org.elasticsearch.xpack.core.template.LifecyclePolicyConfig;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.elasticsearch.xpack.stack.StackTemplateRegistry.STACK_TEMPLATES_ENABLED;
+
+@Deprecated(since = "8.12.0", forRemoval = true)
+public class LegacyStackTemplateRegistry extends IndexTemplateRegistry {
+    private static final Logger logger = LogManager.getLogger(LegacyStackTemplateRegistry.class);
+
+    // Current version of the registry requires all nodes to be at least 8.9.0.
+    public static final Version MIN_NODE_VERSION = Version.V_8_9_0;
+
+    // The stack template registry version. This number must be incremented when we make changes
+    // to built-in templates.
+    public static final int REGISTRY_VERSION = 3;
+
+    public static final String TEMPLATE_VERSION_VARIABLE = "xpack.stack.template.version";
+
+    private final ClusterService clusterService;
+    private volatile boolean stackTemplateEnabled;
+
+    // General mappings conventions for any data that ends up in a data stream
+    public static final String DATA_STREAMS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "data-streams-mappings";
+
+    // ECS dynamic mappings
+    public static final String ECS_DYNAMIC_MAPPINGS_COMPONENT_TEMPLATE_NAME = "ecs@dynamic_templates";
+
+    //////////////////////////////////////////////////////////
+    // Built in ILM policies for users to use
+    //////////////////////////////////////////////////////////
+    public static final String ILM_7_DAYS_POLICY_NAME = "7-days-default";
+    public static final String ILM_30_DAYS_POLICY_NAME = "30-days-default";
+    public static final String ILM_90_DAYS_POLICY_NAME = "90-days-default";
+    public static final String ILM_180_DAYS_POLICY_NAME = "180-days-default";
+    public static final String ILM_365_DAYS_POLICY_NAME = "365-days-default";
+
+    //////////////////////////////////////////////////////////
+    // Logs components (for matching logs-*-* indices)
+    //////////////////////////////////////////////////////////
+    public static final String LOGS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "logs-mappings";
+    public static final String LOGS_SETTINGS_COMPONENT_TEMPLATE_NAME = "logs-settings";
+    public static final String LOGS_ILM_POLICY_NAME = "logs";
+
+    //////////////////////////////////////////////////////////
+    // Metrics components (for matching metric-*-* indices)
+    //////////////////////////////////////////////////////////
+    public static final String METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "metrics-mappings";
+    public static final String METRICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "metrics-settings";
+    public static final String METRICS_TSDB_SETTINGS_COMPONENT_TEMPLATE_NAME = "metrics-tsdb-settings";
+    public static final String METRICS_ILM_POLICY_NAME = "metrics";
+
+    //////////////////////////////////////////////////////////
+    // Synthetics components (for matching synthetics-*-* indices)
+    //////////////////////////////////////////////////////////
+    public static final String SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "synthetics-mappings";
+    public static final String SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "synthetics-settings";
+    public static final String SYNTHETICS_ILM_POLICY_NAME = "synthetics";
+
+    public LegacyStackTemplateRegistry(
+        Settings nodeSettings,
+        ClusterService clusterService,
+        ThreadPool threadPool,
+        Client client,
+        NamedXContentRegistry xContentRegistry
+    ) {
+        super(nodeSettings, clusterService, threadPool, client, xContentRegistry);
+        this.clusterService = clusterService;
+        this.stackTemplateEnabled = STACK_TEMPLATES_ENABLED.get(nodeSettings);
+    }
+
+    @Override
+    public void initialize() {
+        super.initialize();
+        clusterService.getClusterSettings().addSettingsUpdateConsumer(STACK_TEMPLATES_ENABLED, this::updateEnabledSetting);
+    }
+
+    private void updateEnabledSetting(boolean newValue) {
+        if (newValue) {
+            this.stackTemplateEnabled = true;
+        } else {
+            logger.info(
+                "stack composable templates [{}] and component templates [{}] will not be installed or reinstalled",
+                String.join(",", getComposableTemplateConfigs().keySet()),
+                String.join(",", getComponentTemplateConfigs().keySet())
+            );
+            this.stackTemplateEnabled = false;
+        }
+    }
+
+    private static final List<LifecyclePolicyConfig> LIFECYCLE_POLICY_CONFIGS = List.of(
+        new LifecyclePolicyConfig(LOGS_ILM_POLICY_NAME, "/logs@lifecycle.json"),
+        new LifecyclePolicyConfig(METRICS_ILM_POLICY_NAME, "/metrics@lifecycle.json"),
+        new LifecyclePolicyConfig(SYNTHETICS_ILM_POLICY_NAME, "/synthetics@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_7_DAYS_POLICY_NAME, "/7-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_30_DAYS_POLICY_NAME, "/30-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_90_DAYS_POLICY_NAME, "/90-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_180_DAYS_POLICY_NAME, "/180-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_365_DAYS_POLICY_NAME, "/365-days@lifecycle.json")
+    );
+
+    @Override
+    protected List<LifecyclePolicyConfig> getLifecycleConfigs() {
+        return LIFECYCLE_POLICY_CONFIGS;
+    }
+
+    @Override
+    protected List<LifecyclePolicy> getLifecyclePolicies() {
+        if (stackTemplateEnabled) {
+            return lifecyclePolicies;
+        } else {
+            return Collections.emptyList();
+        }
+    }
+
+    private static final Map<String, ComponentTemplate> COMPONENT_TEMPLATE_CONFIGS;
+
+    static {
+        final Map<String, ComponentTemplate> componentTemplates = new HashMap<>();
+        for (IndexTemplateConfig config : List.of(
+            new IndexTemplateConfig(
+                DATA_STREAMS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
+                "/data-streams@mappings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                LOGS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
+                "/logs@mappings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                ECS_DYNAMIC_MAPPINGS_COMPONENT_TEMPLATE_NAME,
+                "/ecs@mappings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                LOGS_SETTINGS_COMPONENT_TEMPLATE_NAME,
+                "/logs@settings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
+                "/metrics@mappings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                METRICS_SETTINGS_COMPONENT_TEMPLATE_NAME,
+                "/metrics@settings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                METRICS_TSDB_SETTINGS_COMPONENT_TEMPLATE_NAME,
+                "/metrics@tsdb-settings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
+                "/synthetics@mappings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            ),
+            new IndexTemplateConfig(
+                SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME,
+                "/synthetics@settings.json",
+                REGISTRY_VERSION,
+                TEMPLATE_VERSION_VARIABLE
+            )
+        )) {
+            try {
+                componentTemplates.put(
+                    config.getTemplateName(),
+                    ComponentTemplate.parse(JsonXContent.jsonXContent.createParser(XContentParserConfiguration.EMPTY, config.loadBytes()))
+                );
+            } catch (IOException e) {
+                throw new AssertionError(e);
+            }
+        }
+        COMPONENT_TEMPLATE_CONFIGS = Map.copyOf(componentTemplates);
+    }
+
+    @Override
+    protected Map<String, ComponentTemplate> getComponentTemplateConfigs() {
+        if (stackTemplateEnabled) {
+            return COMPONENT_TEMPLATE_CONFIGS;
+        } else {
+            return Map.of();
+        }
+    }
+
+    @Override
+    protected Map<String, ComposableIndexTemplate> getComposableTemplateConfigs() {
+        return Map.of();
+    }
+
+    private static final List<IngestPipelineConfig> INGEST_PIPELINE_CONFIGS = List.of(
+        new IngestPipelineConfig("logs@json-message", "/logs@json-pipeline.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
+        new IngestPipelineConfig("logs-default-pipeline", "/logs@default-pipeline.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE)
+    );
+
+    @Override
+    protected List<IngestPipelineConfig> getIngestPipelines() {
+        return INGEST_PIPELINE_CONFIGS;
+    }
+
+    @Override
+    protected String getOrigin() {
+        return ClientHelper.STACK_ORIGIN;
+    }
+
+    @Override
+    protected boolean requiresMasterNode() {
+        // Stack templates use the composable index template and component APIs,
+        // these APIs aren't supported in 7.7 and earlier and in mixed cluster
+        // environments this can cause a lot of ActionNotFoundTransportException
+        // errors in the logs during rolling upgrades. If these templates
+        // are only installed via elected master node then the APIs are always
+        // there and the ActionNotFoundTransportException errors are then prevented.
+        return true;
+    }
+
+    @Override
+    protected boolean isClusterReady(ClusterChangedEvent event) {
+        // Ensure current version of the components are installed only once all nodes are updated to 8.9.0.
+        // This is necessary to prevent an error caused nby the usage of the ignore_missing_pipeline property
+        // in the pipeline processor, which has been introduced only in 8.9.0
+        Version minNodeVersion = event.state().nodes().getMinNodeVersion();
+        return minNodeVersion.onOrAfter(MIN_NODE_VERSION);
+    }
+}

x-pack/plugin/stack/src/main/java/org/elasticsearch/xpack/stack/StackPlugin.java

@@ -59,8 +59,22 @@ public class StackPlugin extends Plugin implements ActionPlugin {
         AllocationService allocationService,
         IndicesService indicesService
     ) {
-        StackTemplateRegistry templateRegistry = new StackTemplateRegistry(settings, clusterService, threadPool, client, xContentRegistry);
-        templateRegistry.initialize();
-        return Collections.singleton(templateRegistry);
+        LegacyStackTemplateRegistry legacyStackTemplateRegistry = new LegacyStackTemplateRegistry(
+            settings,
+            clusterService,
+            threadPool,
+            client,
+            xContentRegistry
+        );
+        legacyStackTemplateRegistry.initialize();
+        StackTemplateRegistry stackTemplateRegistry = new StackTemplateRegistry(
+            settings,
+            clusterService,
+            threadPool,
+            client,
+            xContentRegistry
+        );
+        stackTemplateRegistry.initialize();
+        return List.of(legacyStackTemplateRegistry, stackTemplateRegistry);
     }
 }

x-pack/plugin/stack/src/main/java/org/elasticsearch/xpack/stack/StackTemplateRegistry.java

@@ -29,7 +29,6 @@ import org.elasticsearch.xpack.core.template.IngestPipelineConfig;
 import org.elasticsearch.xpack.core.template.LifecyclePolicyConfig;
 
 import java.io.IOException;
-import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -56,43 +55,43 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
     private volatile boolean stackTemplateEnabled;
 
     // General mappings conventions for any data that ends up in a data stream
-    public static final String DATA_STREAMS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "data-streams-mappings";
+    public static final String DATA_STREAMS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "data-streams@mappings";
 
     // ECS dynamic mappings
-    public static final String ECS_DYNAMIC_MAPPINGS_COMPONENT_TEMPLATE_NAME = "ecs@dynamic_templates";
+    public static final String ECS_DYNAMIC_MAPPINGS_COMPONENT_TEMPLATE_NAME = "ecs@mappings";
 
     //////////////////////////////////////////////////////////
     // Built in ILM policies for users to use
     //////////////////////////////////////////////////////////
-    public static final String ILM_7_DAYS_POLICY_NAME = "7-days-default";
-    public static final String ILM_30_DAYS_POLICY_NAME = "30-days-default";
-    public static final String ILM_90_DAYS_POLICY_NAME = "90-days-default";
-    public static final String ILM_180_DAYS_POLICY_NAME = "180-days-default";
-    public static final String ILM_365_DAYS_POLICY_NAME = "365-days-default";
+    public static final String ILM_7_DAYS_POLICY_NAME = "7-days@lifecycle";
+    public static final String ILM_30_DAYS_POLICY_NAME = "30-days@lifecycle";
+    public static final String ILM_90_DAYS_POLICY_NAME = "90-days@lifecycle";
+    public static final String ILM_180_DAYS_POLICY_NAME = "180-days@lifecycle";
+    public static final String ILM_365_DAYS_POLICY_NAME = "365-days@lifecycle";
 
     //////////////////////////////////////////////////////////
     // Logs components (for matching logs-*-* indices)
     //////////////////////////////////////////////////////////
-    public static final String LOGS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "logs-mappings";
-    public static final String LOGS_SETTINGS_COMPONENT_TEMPLATE_NAME = "logs-settings";
-    public static final String LOGS_ILM_POLICY_NAME = "logs";
+    public static final String LOGS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "logs@mappings";
+    public static final String LOGS_SETTINGS_COMPONENT_TEMPLATE_NAME = "logs@settings";
+    public static final String LOGS_ILM_POLICY_NAME = "logs@lifecycle";
     public static final String LOGS_INDEX_TEMPLATE_NAME = "logs";
 
     //////////////////////////////////////////////////////////
     // Metrics components (for matching metric-*-* indices)
     //////////////////////////////////////////////////////////
-    public static final String METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "metrics-mappings";
-    public static final String METRICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "metrics-settings";
-    public static final String METRICS_TSDB_SETTINGS_COMPONENT_TEMPLATE_NAME = "metrics-tsdb-settings";
-    public static final String METRICS_ILM_POLICY_NAME = "metrics";
+    public static final String METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "metrics@mappings";
+    public static final String METRICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "metrics@settings";
+    public static final String METRICS_TSDB_SETTINGS_COMPONENT_TEMPLATE_NAME = "metrics@tsdb-settings";
+    public static final String METRICS_ILM_POLICY_NAME = "metrics@lifecycle";
     public static final String METRICS_INDEX_TEMPLATE_NAME = "metrics";
 
     //////////////////////////////////////////////////////////
     // Synthetics components (for matching synthetics-*-* indices)
     //////////////////////////////////////////////////////////
-    public static final String SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "synthetics-mappings";
-    public static final String SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "synthetics-settings";
-    public static final String SYNTHETICS_ILM_POLICY_NAME = "synthetics";
+    public static final String SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "synthetics@mappings";
+    public static final String SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "synthetics@settings";
+    public static final String SYNTHETICS_ILM_POLICY_NAME = "synthetics@lifecycle";
     public static final String SYNTHETICS_INDEX_TEMPLATE_NAME = "synthetics";
 
     ///////////////////////////////////
@@ -132,14 +131,14 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
     }
 
     private static final List<LifecyclePolicyConfig> LIFECYCLE_POLICY_CONFIGS = List.of(
-        new LifecyclePolicyConfig(LOGS_ILM_POLICY_NAME, "/logs-policy.json"),
-        new LifecyclePolicyConfig(METRICS_ILM_POLICY_NAME, "/metrics-policy.json"),
-        new LifecyclePolicyConfig(SYNTHETICS_ILM_POLICY_NAME, "/synthetics-policy.json"),
-        new LifecyclePolicyConfig(ILM_7_DAYS_POLICY_NAME, "/" + ILM_7_DAYS_POLICY_NAME + ".json"),
-        new LifecyclePolicyConfig(ILM_30_DAYS_POLICY_NAME, "/" + ILM_30_DAYS_POLICY_NAME + ".json"),
-        new LifecyclePolicyConfig(ILM_90_DAYS_POLICY_NAME, "/" + ILM_90_DAYS_POLICY_NAME + ".json"),
-        new LifecyclePolicyConfig(ILM_180_DAYS_POLICY_NAME, "/" + ILM_180_DAYS_POLICY_NAME + ".json"),
-        new LifecyclePolicyConfig(ILM_365_DAYS_POLICY_NAME, "/" + ILM_365_DAYS_POLICY_NAME + ".json")
+        new LifecyclePolicyConfig(LOGS_ILM_POLICY_NAME, "/logs@lifecycle.json"),
+        new LifecyclePolicyConfig(METRICS_ILM_POLICY_NAME, "/metrics@lifecycle.json"),
+        new LifecyclePolicyConfig(SYNTHETICS_ILM_POLICY_NAME, "/synthetics@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_7_DAYS_POLICY_NAME, "/7-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_30_DAYS_POLICY_NAME, "/30-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_90_DAYS_POLICY_NAME, "/90-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_180_DAYS_POLICY_NAME, "/180-days@lifecycle.json"),
+        new LifecyclePolicyConfig(ILM_365_DAYS_POLICY_NAME, "/365-days@lifecycle.json")
     );
 
     @Override
@@ -149,11 +148,7 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
 
     @Override
     protected List<LifecyclePolicy> getLifecyclePolicies() {
-        if (stackTemplateEnabled) {
-            return lifecyclePolicies;
-        } else {
-            return Collections.emptyList();
-        }
+        return lifecyclePolicies;
     }
 
     private static final Map<String, ComponentTemplate> COMPONENT_TEMPLATE_CONFIGS;
@@ -163,55 +158,55 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
         for (IndexTemplateConfig config : List.of(
             new IndexTemplateConfig(
                 DATA_STREAMS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
-                "/data-streams-mappings.json",
+                "/data-streams@mappings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 LOGS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
-                "/logs-mappings.json",
+                "/logs@mappings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 ECS_DYNAMIC_MAPPINGS_COMPONENT_TEMPLATE_NAME,
-                "/ecs-dynamic-mappings.json",
+                "/ecs@mappings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 LOGS_SETTINGS_COMPONENT_TEMPLATE_NAME,
-                "/logs-settings.json",
+                "/logs@settings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
-                "/metrics-mappings.json",
+                "/metrics@mappings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 METRICS_SETTINGS_COMPONENT_TEMPLATE_NAME,
-                "/metrics-settings.json",
+                "/metrics@settings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 METRICS_TSDB_SETTINGS_COMPONENT_TEMPLATE_NAME,
-                "/metrics-tsdb-settings.json",
+                "/metrics@tsdb-settings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
-                "/synthetics-mappings.json",
+                "/synthetics@mappings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             ),
             new IndexTemplateConfig(
                 SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME,
-                "/synthetics-settings.json",
+                "/synthetics@settings.json",
                 REGISTRY_VERSION,
                 TEMPLATE_VERSION_VARIABLE
             )
@@ -230,20 +225,16 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
 
     @Override
     protected Map<String, ComponentTemplate> getComponentTemplateConfigs() {
-        if (stackTemplateEnabled) {
-            return COMPONENT_TEMPLATE_CONFIGS;
-        } else {
-            return Map.of();
-        }
+        return COMPONENT_TEMPLATE_CONFIGS;
     }
 
     private static final Map<String, ComposableIndexTemplate> COMPOSABLE_INDEX_TEMPLATE_CONFIGS = parseComposableTemplates(
-        new IndexTemplateConfig(LOGS_INDEX_TEMPLATE_NAME, "/logs-template.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
-        new IndexTemplateConfig(METRICS_INDEX_TEMPLATE_NAME, "/metrics-template.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
-        new IndexTemplateConfig(SYNTHETICS_INDEX_TEMPLATE_NAME, "/synthetics-template.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
+        new IndexTemplateConfig(LOGS_INDEX_TEMPLATE_NAME, "/logs@template.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
+        new IndexTemplateConfig(METRICS_INDEX_TEMPLATE_NAME, "/metrics@template.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
+        new IndexTemplateConfig(SYNTHETICS_INDEX_TEMPLATE_NAME, "/synthetics@template.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
         new IndexTemplateConfig(
             KIBANA_REPORTING_INDEX_TEMPLATE_NAME,
-            "/kibana-reporting-template.json",
+            "/kibana-reporting@template.json",
             REGISTRY_VERSION,
             TEMPLATE_VERSION_VARIABLE
         )
@@ -259,14 +250,8 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
     }
 
     private static final List<IngestPipelineConfig> INGEST_PIPELINE_CONFIGS = List.of(
-        new IngestPipelineConfig("logs@json-message", "/logs-json-message-pipeline.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
-        new IngestPipelineConfig(
-            "logs-default-pipeline",
-            "/logs-default-pipeline.json",
-            REGISTRY_VERSION,
-            TEMPLATE_VERSION_VARIABLE,
-            Collections.singletonList("logs@json-message")
-        )
+        new IngestPipelineConfig("logs@json-pipeline", "/logs@json-pipeline.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE),
+        new IngestPipelineConfig("logs@default-pipeline", "/logs@default-pipeline.json", REGISTRY_VERSION, TEMPLATE_VERSION_VARIABLE)
     );
 
     @Override

x-pack/plugin/stack/src/test/java/org/elasticsearch/xpack/stack/StackTemplateRegistryTests.java

@@ -63,10 +63,12 @@ import java.util.stream.Collectors;
 
 import static org.hamcrest.Matchers.anEmptyMap;
 import static org.hamcrest.Matchers.anyOf;
+import static org.hamcrest.Matchers.empty;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.hasSize;
 import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.not;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.spy;
 import static org.mockito.Mockito.when;
@@ -92,7 +94,7 @@ public class StackTemplateRegistryTests extends ESTestCase {
         threadPool.shutdownNow();
     }
 
-    public void testDisabledDoesNotAddTemplates() {
+    public void testDisabledDoesNotAddIndexTemplates() {
         Settings settings = Settings.builder().put(StackTemplateRegistry.STACK_TEMPLATES_ENABLED.getKey(), false).build();
         StackTemplateRegistry disabledRegistry = new StackTemplateRegistry(
             settings,
@@ -101,9 +103,34 @@ public class StackTemplateRegistryTests extends ESTestCase {
             client,
             NamedXContentRegistry.EMPTY
         );
-        assertThat(disabledRegistry.getComponentTemplateConfigs(), anEmptyMap());
         assertThat(disabledRegistry.getComposableTemplateConfigs(), anEmptyMap());
-        assertThat(disabledRegistry.getLifecyclePolicies(), hasSize(0));
+    }
+
+    public void testDisabledStillAddsComponentTemplatesAndIlmPolicies() {
+        Settings settings = Settings.builder().put(StackTemplateRegistry.STACK_TEMPLATES_ENABLED.getKey(), false).build();
+        StackTemplateRegistry disabledRegistry = new StackTemplateRegistry(
+            settings,
+            clusterService,
+            threadPool,
+            client,
+            NamedXContentRegistry.EMPTY
+        );
+        assertThat(disabledRegistry.getComponentTemplateConfigs(), not(anEmptyMap()));
+        assertThat(
+            disabledRegistry.getComponentTemplateConfigs()
+                .keySet()
+                .stream()
+                // We have a naming convention that internal component templates contain `@`. See also put-component-template.asciidoc.
+                .filter(t -> t.contains("@") == false)
+                .collect(Collectors.toSet()),
+            empty()
+        );
+        assertThat(disabledRegistry.getLifecyclePolicies(), not(empty()));
+        assertThat(
+            // We have a naming convention that internal ILM policies contain `@`. See also put-lifecycle.asciidoc.
+            disabledRegistry.getLifecyclePolicies().stream().filter(p -> p.getName().contains("@") == false).collect(Collectors.toSet()),
+            empty()
+        );
     }
 
     public void testThatNonExistingTemplatesAreAddedImmediately() throws Exception {
@@ -356,7 +383,7 @@ public class StackTemplateRegistryTests extends ESTestCase {
                 assertThat(putComposableTemplateRequest.name(), equalTo("syslog"));
                 ComposableIndexTemplate composableIndexTemplate = putComposableTemplateRequest.indexTemplate();
                 assertThat(composableIndexTemplate.composedOf(), hasSize(2));
-                assertThat(composableIndexTemplate.composedOf().get(0), equalTo("logs-settings"));
+                assertThat(composableIndexTemplate.composedOf().get(0), equalTo("logs@settings"));
                 assertThat(composableIndexTemplate.composedOf().get(1), equalTo("syslog@custom"));
                 assertThat(composableIndexTemplate.getIgnoreMissingComponentTemplates(), hasSize(1));
                 assertThat(composableIndexTemplate.getIgnoreMissingComponentTemplates().get(0), equalTo("syslog@custom"));

x-pack/plugin/stack/src/test/resources/non-required-template.json

@@ -3,7 +3,7 @@
   "priority": 100,
   "data_stream": {},
   "composed_of": [
-    "logs-settings",
+    "logs@settings",
     "syslog@custom"
   ],
   "ignore_missing_component_templates": ["syslog@custom"],