Use deprecation logger for CLDR date format specifiers (#112917)

The addition of the logger requires several updates to tests to deal with the possible warning, or muting if there is not way to specify an allowed (but not mandatory) warning

docs/reference/cat/component-templates.asciidoc

@@ -6,8 +6,8 @@
 
 [IMPORTANT]
 ====
-cat APIs are only intended for human consumption using the command line or {kib} 
-console. They are _not_ intended for use by applications. For application 
+cat APIs are only intended for human consumption using the command line or {kib}
+console. They are _not_ intended for use by applications. For application
 consumption, use the <<getting-component-templates,get component template API>>.
 ====
 
@@ -82,8 +82,7 @@ PUT _component_template/my-template-2
           "type": "keyword"
         },
         "created_at": {
-          "type": "date",
-          "format": "EEE MMM dd HH:mm:ss Z yyyy"
+          "type": "date"
         }
       }
     }

docs/reference/esql/esql-rest.asciidoc

@@ -233,6 +233,7 @@ POST /_query
 }
 ----
 // TEST[setup:library]
+// TEST[skip:This can output a warning, and asciidoc doesn't support allowed_warnings]
 
 [discrete]
 [[esql-rest-params]]

docs/reference/indices/index-templates.asciidoc

@@ -134,8 +134,7 @@ PUT _index_template/template_1
           "type": "keyword"
         },
         "created_at": {
-          "type": "date",
-          "format": "EEE MMM dd HH:mm:ss Z yyyy"
+          "type": "date"
         }
       }
     },

docs/reference/indices/put-component-template.asciidoc

@@ -26,8 +26,7 @@ PUT _component_template/template_1
           "type": "keyword"
         },
         "created_at": {
-          "type": "date",
-          "format": "EEE MMM dd HH:mm:ss Z yyyy"
+          "type": "date"
         }
       }
     }

docs/reference/indices/put-index-template-v1.asciidoc

@@ -4,8 +4,8 @@
 <titleabbrev>Create or update index template (legacy)</titleabbrev>
 ++++
 
-IMPORTANT: This documentation is about legacy index templates, 
-which are deprecated and will be replaced by the composable templates introduced in {es} 7.8. 
+IMPORTANT: This documentation is about legacy index templates,
+which are deprecated and will be replaced by the composable templates introduced in {es} 7.8.
 For information about composable templates, see <<index-templates>>.
 
 Creates or updates an index template.
@@ -27,8 +27,7 @@ PUT _template/template_1
         "type": "keyword"
       },
       "created_at": {
-        "type": "date",
-        "format": "EEE MMM dd HH:mm:ss Z yyyy"
+        "type": "date"
       }
     }
   }
@@ -67,8 +66,8 @@ that you can automatically apply when creating new indices.
 {es} applies templates to new indices
 based on an index pattern that matches the index name.
 
-NOTE: Composable templates always take precedence over legacy templates. 
-If no composable template matches a new index, 
+NOTE: Composable templates always take precedence over legacy templates.
+If no composable template matches a new index,
 matching legacy templates are applied according to their order.
 
 Index templates are only applied during index creation.
@@ -263,4 +262,4 @@ The API returns the following response:
     "version" : 123
   }
 }
---------------------------------------------------
+--------------------------------------------------

docs/reference/indices/simulate-template.asciidoc

@@ -26,8 +26,7 @@ PUT _index_template/template_1
           "type": "keyword"
         },
         "created_at": {
-          "type": "date",
-          "format": "EEE MMM dd HH:mm:ss Z yyyy"
+          "type": "date"
         }
       }
     },
@@ -74,19 +73,19 @@ privilege>> to use this API.
 `<index-template>`::
 (Optional, string)
 Name of the index template to simulate.
-To test a template configuration before you add it to the cluster, 
+To test a template configuration before you add it to the cluster,
 omit this parameter and specify the template configuration in the request body.
 
 [[simulate-template-api-query-params]]
 ==== {api-query-parms-title}
 ////
 `cause`::
-(Optional, string) The reason for using the specified template for the simulation. 
+(Optional, string) The reason for using the specified template for the simulation.
 ////
 
 `create`::
-(Optional, Boolean) If `true`, the template passed in the body is 
-only used if no existing templates match the same index patterns. 
+(Optional, Boolean) If `true`, the template passed in the body is
+only used if no existing templates match the same index patterns.
 If `false`, the simulation uses the template with the highest priority.
 Note that the template is not permanently added or updated in either case;
 it is only used for the simulation.
@@ -114,7 +113,7 @@ include::{es-ref-dir}/indices/put-index-template.asciidoc[tag=index-template-api
 .Properties of `overlapping`
 [%collapsible%open]
 ====
-`index_patterns`:: 
+`index_patterns`::
 (array) Index patterns that the superseded template applies to.
 
 `name`::
@@ -123,7 +122,7 @@ include::{es-ref-dir}/indices/put-index-template.asciidoc[tag=index-template-api
 
 `template`::
 (object)
-The settings, mappings, and aliases that would be applied to matching indices. 
+The settings, mappings, and aliases that would be applied to matching indices.
 +
 .Properties of `template`
 [%collapsible%open]
@@ -227,8 +226,8 @@ The response shows the index settings, mappings, and aliases applied by the `fin
 [[simulate-template-config-ex]]
 ===== Simulating an arbitrary template configuration
 
-To see what settings will be applied by a template before you add it to the cluster, 
-you can pass a template configuration in the request body. 
+To see what settings will be applied by a template before you add it to the cluster,
+you can pass a template configuration in the request body.
 The specified template is used for the simulation if it has a higher priority than existing templates.
 
 [source,console]
@@ -266,8 +265,8 @@ The response shows any overlapping templates with a lower priority.
       }
     },
     "mappings" : {
-      "properties" : { 
-        "@timestamp" : {           
+      "properties" : {
+        "@timestamp" : {
           "type" : "date"
         }
       }

docs/reference/ingest/common-log-format-example.asciidoc

@@ -28,7 +28,7 @@ Pipelines**.
 image::images/ingest/ingest-pipeline-list.png[Kibana's Ingest Pipelines list view,align="center"]
 
 . Click **Create pipeline > New pipeline**.
-. Set **Name** to `my-pipeline` and optionally add a description for the 
+. Set **Name** to `my-pipeline` and optionally add a description for the
 pipeline.
 . Add a <<grok-processor,grok processor>> to parse the log message:
 
@@ -128,6 +128,7 @@ PUT _ingest/pipeline/my-pipeline
 // end::common-log-pipeline[]
 }
 ----
+// TEST[skip:This can output a warning, and asciidoc doesn't support allowed_warnings]
 ////
 --
 

modules/ingest-common/build.gradle

@@ -51,6 +51,20 @@ tasks.named("thirdPartyAudit").configure {
   )
 }
 
+tasks.named("yamlRestTest").configure {
+  systemProperty 'tests.rest.blacklist', [
+    // for some reason, allowed_warnings on the test isn't working here
+    'ingest/30_date_processor/Test date processor with no timezone configured',
+  ].join(',')
+}
+
+tasks.named("yamlRestTestV7CompatTest").configure {
+  systemProperty 'tests.rest.blacklist', [
+    // for some reason, allowed_warnings on the test isn't working here
+    'ingest/30_date_processor/Test date processor with no timezone configured',
+  ].join(',')
+}
+
 tasks.named("yamlRestTestV7CompatTransform").configure { task ->
   task.addAllowedWarningRegex("\\[types removal\\].*")
 }

modules/ingest-common/src/yamlRestTest/resources/rest-api-spec/test/ingest/30_date_processor.yml

@@ -1,3 +1,6 @@
+setup:
+  - requires:
+      test_runner_features: allowed_warnings
 ---
 teardown:
   - do:
@@ -97,6 +100,8 @@ teardown:
 "Test date processor with no timezone configured":
 
   - do:
+      allowed_warnings:
+        - 'Date format [dd/MMM/yyyy:H:m:s Z] contains textual field specifiers that could change in JDK 23'
       ingest.put_pipeline:
         id: "my_pipeline"
         # sample formats from beats, featuring mongodb, icinga, apache
@@ -165,6 +170,8 @@ teardown:
   - match: { acknowledged: true }
 
   - do:
+      allowed_warnings:
+        - 'Date format [dd/MMM/yyyy:H:m:s Z] contains textual field specifiers that could change in JDK 23'
       index:
         index: test
         id: "1"

qa/smoke-test-ingest-with-all-dependencies/src/yamlRestTest/resources/rest-api-spec/test/ingest/20_combine_processors.yml

@@ -1,6 +1,11 @@
+setup:
+  - requires:
+      test_runner_features: allowed_warnings
 ---
 "Test with date processor":
   - do:
+      allowed_warnings:
+        - 'Date format [dd/MMM/yyyy:HH:mm:ss xx] contains textual field specifiers that could change in JDK 23'
       ingest.put_pipeline:
         id: "_id"
         body:  >
@@ -41,6 +46,8 @@
   - match: { acknowledged: true }
 
   - do:
+      allowed_warnings:
+        - 'Date format [dd/MMM/yyyy:HH:mm:ss xx] contains textual field specifiers that could change in JDK 23'
       index:
         index: test
         id: "1"
@@ -70,6 +77,8 @@
 ---
 "Test with date processor and ECS-v1":
   - do:
+      allowed_warnings:
+        - 'Date format [dd/MMM/yyyy:HH:mm:ss xx] contains textual field specifiers that could change in JDK 23'
       ingest.put_pipeline:
         id: "_id"
         body:  >
@@ -99,6 +108,8 @@
   - match: { acknowledged: true }
 
   - do:
+      allowed_warnings:
+        - 'Date format [dd/MMM/yyyy:HH:mm:ss xx] contains textual field specifiers that could change in JDK 23'
       index:
         index: test
         id: "1"

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/180_locale_dependent_mapping.yml

@@ -1,6 +1,11 @@
+setup:
+  - requires:
+      test_runner_features: allowed_warnings
 ---
 "Test Index and Search locale dependent mappings / dates":
   - do:
+      allowed_warnings:
+        - 'Date format [E, d MMM yyyy HH:mm:ss Z] contains textual field specifiers that could change in JDK 23'
       indices.create:
           index: test_index
           body:

server/src/internalClusterTest/java/org/elasticsearch/reservedstate/service/ComponentTemplatesFileSettingsIT.java

@@ -129,8 +129,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                                 "type": "keyword"
                               },
                               "created_at": {
-                                "type": "date",
-                                "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                                "type": "date"
                               }
                             }
                           },
@@ -160,8 +159,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                                 "type": "keyword"
                               },
                               "created_at": {
-                                "type": "date",
-                                "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                                "type": "date"
                               }
                             }
                           },
@@ -191,8 +189,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                                 "type": "keyword"
                               },
                               "created_at": {
-                                "type": "date",
-                                "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                                "type": "date"
                               }
                             }
                           },
@@ -260,8 +257,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                                 "type": "keyword"
                               },
                               "created_at": {
-                                "type": "date",
-                                "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                                "type": "date"
                               }
                             }
                           },
@@ -291,8 +287,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                                 "type": "keyword"
                               },
                               "created_at": {
-                                "type": "date",
-                                "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                                "type": "date"
                               }
                             }
                           },
@@ -336,8 +331,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                                 "type": "keyword"
                               },
                               "created_at": {
-                                "type": "date",
-                                "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                                "type": "date"
                               }
                             }
                           },
@@ -666,8 +660,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                         "type": "keyword"
                       },
                       "created_at": {
-                        "type": "date",
-                        "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                        "type": "date"
                       }
                     }
                   },
@@ -708,8 +701,7 @@ public class ComponentTemplatesFileSettingsIT extends ESIntegTestCase {
                         "type": "keyword"
                       },
                       "created_at": {
-                        "type": "date",
-                        "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                        "type": "date"
                       }
                     }
                   },

server/src/main/java/org/elasticsearch/common/time/DateUtils.java

@@ -13,7 +13,6 @@ import org.elasticsearch.common.logging.DeprecationCategory;
 import org.elasticsearch.common.logging.DeprecationLogger;
 import org.elasticsearch.core.Predicates;
 import org.elasticsearch.core.UpdateForV9;
-import org.elasticsearch.logging.LogManager;
 
 import java.time.Clock;
 import java.time.Duration;
@@ -392,13 +391,17 @@ public class DateUtils {
     // check for all textual fields, and localized zone offset
     // the weird thing with Z is to ONLY match 4 in a row, with no Z before or after (but those groups can also be empty)
     private static final Predicate<String> CONTAINS_CHANGING_TEXT_SPECIFIERS = System.getProperty("java.locale.providers", "")
-        .contains("COMPAT") ? Pattern.compile("[BEGOavz]|LLL|MMM|QQQ|ccc|eee|(?<!Z)Z{4}(?!Z)").asPredicate() : Predicates.never();
+        .contains("COMPAT") ? Pattern.compile("[BEGOavz]|LLL|MMM|QQQ|qqq|ccc|eee|(?<!Z)Z{4}(?!Z)").asPredicate() : Predicates.never();
 
     @UpdateForV9    // this can be removed, we will only use CLDR on v9
     static void checkTextualDateFormats(String format) {
         if (CONTAINS_CHANGING_TEXT_SPECIFIERS.test(format)) {
-            LogManager.getLogger(DateFormatter.class)
-                .warn("Date format [{}] contains textual field specifiers that could change in JDK 23", format);
+            deprecationLogger.warn(
+                DeprecationCategory.PARSING,
+                "cldr_date_formats_" + format,
+                "Date format [{}] contains textual field specifiers that could change in JDK 23",
+                format
+            );
         }
     }
 }

server/src/test/java/org/elasticsearch/action/admin/indices/template/reservedstate/ReservedComposableIndexTemplateActionTests.java

@@ -171,8 +171,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                           "type": "keyword"
                         },
                         "created_at": {
-                          "type": "date",
-                          "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                          "type": "date"
                         }
                       }
                     },
@@ -213,8 +212,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                           "type": "keyword"
                         },
                         "created_at": {
-                          "type": "date",
-                          "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                          "type": "date"
                         }
                       }
                     },
@@ -367,8 +365,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -398,8 +395,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -471,8 +467,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -536,8 +531,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -566,8 +560,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -611,8 +604,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -652,8 +644,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -792,8 +783,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },
@@ -857,8 +847,7 @@ public class ReservedComposableIndexTemplateActionTests extends ESTestCase {
                             "type": "keyword"
                           },
                           "created_at": {
-                            "type": "date",
-                            "format": "EEE MMM dd HH:mm:ss Z yyyy"
+                            "type": "date"
                           }
                         }
                       },

x-pack/plugin/esql/qa/testFixtures/src/main/resources/date.csv-spec

@@ -481,6 +481,7 @@ dateFormatLocale
 from employees | where emp_no == 10049 or emp_no == 10050 | sort emp_no 
 | eval birth_month = date_format("MMMM", birth_date) | keep emp_no, birth_date, birth_month;
 ignoreOrder:true
+warningRegex:Date format \[MMMM] contains textual field specifiers that could change in JDK 23
 
 emp_no:integer  |  birth_date:datetime       | birth_month:keyword
 10049           |  null                      | null

x-pack/plugin/sql/qa/server/src/main/java/org/elasticsearch/xpack/sql/qa/CustomDateFormatTestCase.java

@@ -35,8 +35,8 @@ public abstract class CustomDateFormatTestCase extends BaseRestSqlTestCase {
         "HH:mm yyyy-MM-dd",
         "HH:mm:ss yyyy-dd-MM",
         "HH:mm:ss VV",
-        "HH:mm:ss VV z",
-        "yyyy-MM-dd'T'HH:mm:ss'T'VV'T'z" };
+        "HH:mm:ss VV Z",
+        "yyyy-MM-dd'T'HH:mm:ss'T'VV'T'Z" };
     private static String[] nowFunctions = new String[] { "NOW()", "CURRENT_DATE()", "CURRENT_TIME()", "CURRENT_TIMESTAMP()" };
     private static String[] operators = new String[] { " < ", " > ", " <= ", " >= ", " = ", " != " };
 

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/70_locale.yml

@@ -29,6 +29,7 @@ setup:
   - do:
       allowed_warnings_regex:
         - "No limit defined, adding default limit of \\[.*\\]"
+        - "Date format \\[MMMM] contains textual field specifiers that could change in JDK 23"
       esql.query:
         body:
           query: 'FROM events | eval fixed_format = date_format("MMMM", @timestamp), variable_format = date_format(format, @timestamp) | sort @timestamp | keep @timestamp, fixed_format, variable_format'
@@ -50,6 +51,7 @@ setup:
   - do:
       allowed_warnings_regex:
         - "No limit defined, adding default limit of \\[.*\\]"
+        - "Date format \\[MMMM] contains textual field specifiers that could change in JDK 23"
       esql.query:
         body:
           query: 'FROM events | eval fixed_format = date_format("MMMM", @timestamp), variable_format = date_format(format, @timestamp) | sort @timestamp | keep @timestamp, fixed_format, variable_format'