Browse Source

[doc] joda migration guide to version 8 (#60506)

indices created in v6 and not migrated earlier to java-time have to be
updated before upgrading to v8
closes #60374
Przemyslaw Gomulka 5 years ago
parent
commit
a3a7ae77fa

+ 2 - 1
docs/reference/migration/migrate_8_0.asciidoc

@@ -49,7 +49,7 @@ Elasticsearch 8.0 node will not start in the presence of indices created in a
 version of Elasticsearch before 7.0.
 
 *Impact* +
-Reindex indices created in {es} 6.x or before with {es} 7.x if they need to be carried forward to  {es} 8.x. 
+Reindex indices created in {es} 6.x or before with {es} 7.x if they need to be carried forward to  {es} 8.x.
 ====
 
 .REST API endpoints containing `_xpack` have been removed.
@@ -92,3 +92,4 @@ include::migrate_8_0/settings.asciidoc[]
 include::migrate_8_0/snapshots.asciidoc[]
 include::migrate_8_0/threadpool.asciidoc[]
 include::migrate_8_0/transport.asciidoc[]
+include::migrate_8_0/migrate_to_java_time.asciidoc[]

+ 18 - 1
docs/reference/migration/migrate_8_0/mappings.asciidoc

@@ -59,6 +59,23 @@ The setting has been deprecated with 7.5 and is no longer supported on new indic
 Mappings for older indices will continue to work but emit a deprecation warning.
 
 *Impact* +
-The `enabled` setting for `_field_names` should be removed from templates and mappings. 
+The `enabled` setting for `_field_names` should be removed from templates and mappings.
 Disabling _field_names is not necessary because it no longer carries a large index overhead.
 ====
+
+//tag::notable-breaking-changes[]
+.Java-time date formats replace joda-time formats
+[%collapsible]
+====
+*Details* +
+In 7.0, {es} switched from joda time to java time for date-related parsing,
+formatting, and calculations. Indices created in 7.0 and later versions are
+already required to use mappings with java-time date formats. However,
+earlier indices using joda-time formats must be reindexed to use
+mappings with java-time formats.
+
+*Impact* +
+For a detailed migration guide, see the {ref}/migrate-to-java-time.html[Java
+time migration guide].
+====
+// end::notable-breaking-changes[]

+ 318 - 0
docs/reference/migration/migrate_8_0/migrate_to_java_time.asciidoc

@@ -0,0 +1,318 @@
+[[migrate-to-java-time]]
+=== Java time migration guide
+
+With 7.0, {es} switched from joda time to java time for date-related parsing,
+formatting, and calculations. This guide is designed to help you determine
+if your cluster is impacted and, if so, prepare for the upgrade.
+
+
+[discrete]
+[[java-time-convert-date-formats]]
+=== Convert date formats
+
+To upgrade to {es} 8, you'll need to convert any joda-time date formats
+to their java-time equivalents.
+
+[discrete]
+[[java-time-migration-impacted-features]]
+=== Impacted features
+The switch to java time only impacts custom <<date,`date`>> and
+<<date_nanos,`date_nanos`>> formats.
+
+These formats are commonly used in:
+
+* <<mapping,Index mappings>>
+* <<indices-templates,Index templates>>
+* <<pipeline,Ingest pipelines>>
+
+If you don't use custom date formats, you can skip the rest of this guide.
+Most custom date formats are compatible. However, several require
+an update.
+
+To see if your date format is impacted, use the <<migration-api-deprecation,deprecation info API>>
+or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant].
+
+[discrete]
+[[java-time-migration-incompatible-date-formats]]
+=== Incompatible date formats
+Custom date formats containing the following joda-time literals should be
+migrated.
+
+`Y` (Year of era)::
++
+--
+Replace with `y`.
+
+*Example:*
+`YYYY-MM-dd` should become `yyyy-MM-dd`.
+
+In java time, `Y` is used for
+https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[week-based year].
+Using `Y` in place of `y` could result in off-by-one errors in year calculation.
+
+For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z`  will give `2019-01`
+For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z`  will give `2019-01` (counter-intuitive) because there is >4 days of that week in 2019
+--
+
+`y` (Year)::
++
+--
+Replace with `u`.
+
+*Example:*
+`yyyy-MM-dd` should become `uuuu-MM-dd`.
+
+In java time, `y` is used for year of era. `u` can contain non-positive
+values while `y` cannot. `y` can also be associated with an era field.
+--
+
+
+`C` (Century of era)::
++
+--
+Century of era is not supported in java time.
+There is no replacement. Instead, we recommend you preprocess your input.
+--
+
+`x` (Week year)::
++
+--
+Replace with `Y`.
+
+In java time, `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset].
+
+[WARNING]
+====
+Failure to properly convert `x` (Week year) to `Y` could result in data loss.
+====
+--
+
+`Z` (Zone offset/id)::
++
+--
+Replace with multiple `X`'s.
+
+`Z` has a similar meaning in java time. However, java time expects different
+numbers of literals to parse different forms.
+
+Consider migrating to `X`, which gives you more control over how time is parsed.
+For example, the joda-time format `YYYY-MM-dd'T'hh:mm:ssZZ` accepts the following dates:
+
+```
+2010-01-01T01:02:03Z
+2010-01-01T01:02:03+01
+2010-01-01T01:02:03+01:02
+2010-01-01T01:02:03+01:02:03
+```
+
+In java time, you cannot parse all these dates using a single format
+Instead, you must specify 3 separate formats:
+
+```
+2010-01-01T01:02:03Z
+2010-01-01T01:02:03+01
+both parsed with yyyy-MM-dd'T'hh:mm:ssX
+
+2010-01-01T01:02:03+01:02
+yyyy-MM-dd'T'hh:mm:ssXXX
+
+2010-01-01T01:02:03+01:02:03
+yyyy-MM-dd'T'hh:mm:ssXXXXX
+```
+
+
+The formats must then be delimited using `||`:
+[source,txt]
+--------------------------------------------------
+yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX
+--------------------------------------------------
+
+The same applies if you expect your pattern to occur without a colon (`:`):
+For example, the `YYYY-MM-dd'T'hh:mm:ssZ` format accepts the following date forms:
+```
+2010-01-01T01:02:03Z
+2010-01-01T01:02:03+01
+2010-01-01T01:02:03+0102
+2010-01-01T01:02:03+010203
+```
+To accept all these forms in java time, you must use the `||` delimiter:
+[source,txt]
+--------------------------------------------------
+yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX
+--------------------------------------------------
+--
+
+`d` (Day)::
++
+--
+In java time, `d` is still interpreted as "day" but is less flexible.
+
+For example, the joda-time date format `YYYY-MM-dd` accepts `2010-01-01` or
+`2010-01-1`.
+
+In java time, you must use the `||` delimiter to provide specify each format:
+
+[source,txt]
+--------------------------------------------------
+yyyy-MM-dd||yyyy-MM-d
+--------------------------------------------------
+
+In java time, `d` also does not accept more than 2 digits. To accept days with more
+than two digits, you must include a text literal in your java-time date format.
+For example, to parse `2010-01-00001`, you must use the following java-time date format:
+
+[source,txt]
+--------------------------------------------------
+yyyy-MM-'000'dd
+--------------------------------------------------
+--
+
+`e` (Name of day)::
++
+--
+In java time, `e` is still interpreted as "name of day" but does not parse
+short- or full-text forms.
+
+For example, the joda-time date format `EEE YYYY-MM` accepts both
+`Wed 2020-01` and `Wednesday 2020-01`.
+
+To accept both of these dates in java time, you must specify each format using
+the `||` delimiter:
+
+[source,txt]
+--------------------------------------------------
+cccc yyyy-MM||ccc yyyy-MM
+--------------------------------------------------
+
+The joda-time literal `E` is interpreted as "day of week."
+The java-time literal `c` is interpreted as "localized day of week."
+`E` does not accept full-text day formats, such as `Wednesday`.
+--
+
+`EEEE` and similar text forms::
++
+--
+Support for full-text forms depends on the locale data provided with your Java
+Development Kit (JDK) and other implementation details. We recommend you
+test formats containing these patterns carefully before upgrading.
+--
+
+`z` (Time zone text)::
++
+--
+In java time, `z` outputs 'Z' for Zulu when given a UTC timezone.
+--
+
+[discrete]
+[[java-time-migration-test]]
+=== Test with your data
+
+We strongly recommend you test any date format changes using real data before
+deploying in production.
+
+For help with date debugging, consider using
+https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.]
+
+[discrete]
+[[java-time-migrate-update-mappings]]
+=== Update index mappings
+To update joda-time date formats in index mappings, you must create a new index
+with an updated mapping and reindex your data to it.
+
+The following `my-index-000001` index contains a mapping for the `datetime` field, a
+`date` field with a custom joda-time date format.
+////
+[source,console]
+--------------------------------------------------
+PUT my-index-000001
+{
+  "mappings": {
+    "properties": {
+      "datetime": {
+        "type": "date",
+        "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
+      }
+    }
+  }
+}
+--------------------------------------------------
+////
+
+[source,console]
+--------------------------------------------------
+GET my-index-000001/_mapping
+--------------------------------------------------
+// TEST[continued]
+
+[source,console-result]
+--------------------------------------------------
+{
+  "my-index-000001" : {
+    "mappings" : {
+      "properties" : {
+         "datetime": {
+           "type": "date",
+           "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
+         }
+      }
+    }
+  }
+}
+--------------------------------------------------
+
+
+To change the date format for the `datetime` field, create a separate index
+containing an updated mapping and date format.
+
+For example, the following `my-index-000002` index changes the `datetime` field's
+date format to `uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis`.
+
+[source,console]
+--------------------------------------------------
+PUT my-index-000002
+{
+  "mappings": {
+    "properties": {
+      "datetime": {
+        "type": "date",
+        "format": "uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis"
+      }
+    }
+  }
+}
+--------------------------------------------------
+// TEST[continued]
+
+Next, reindex data from the old index to the new index.
+
+The following <<docs-reindex,reindex>> API request reindexes data from
+`my-index-000001` to `my-index-000002`.
+
+[source,console]
+--------------------------------------------------
+POST _reindex
+{
+  "source": {
+    "index": "my-index-000001"
+  },
+  "dest": {
+    "index": "my-index-000002"
+  }
+}
+--------------------------------------------------
+// TEST[continued]
+
+If you use index aliases, update them to point to the new index.
+
+[source,console]
+--------------------------------------------------
+POST /_aliases
+{
+  "actions" : [
+    { "remove" : { "index" : "my-index-000001", "alias" : "my-index" } },
+    { "add" : { "index" : "my-index-000002", "alias" : "my-index" } }
+  ]
+}
+--------------------------------------------------
+// TEST[continued]
+