Browse Source

Forward port release notes for v8.13.0 (#106783)

elasticsearchmachine 1 year ago
parent
commit
1c60a8dd8f

+ 115 - 1
docs/reference/migration/migrate_8_13.asciidoc

@@ -16,5 +16,119 @@ coming::[8.13.0]
 [[breaking-changes-8.13]]
 === Breaking changes
 
-There are no breaking changes in {es} 8.13.
+The following changes in {es} 8.13 might affect your applications
+and prevent them from operating normally.
+Before upgrading to 8.13, review these changes and take the described steps
+to mitigate the impact.
+
+
+There are no notable breaking changes in {es} 8.13.
+But there are some less critical breaking changes.
+
+[discrete]
+[[breaking_813_index_setting_changes]]
+==== Index setting changes
+
+[[change_index_look_ahead_time_index_settings_default_value_from_2_hours_to_30_minutes]]
+.Change `index.look_ahead_time` index setting's default value from 2 hours to 30 minutes.
+[%collapsible]
+====
+*Details* +
+Lower the `index.look_ahead_time` index setting's max value from 2 hours to 30 minutes.
+
+*Impact* +
+Documents with @timestamp of 30 minutes or more in the future will be rejected. Before documents with @timestamp of 2 hours or more in the future were rejected. If the previous behaviour should be kept, then update the `index.look_ahead_time` setting to two hours before performing the upgrade.
+====
+
+[[lower_look_ahead_time_index_settings_max_value]]
+.Lower the `look_ahead_time` index setting's max value
+[%collapsible]
+====
+*Details* +
+Lower the `look_ahead_time` index setting's max value from 7 days to 2 hours.
+
+*Impact* +
+Any value between 2 hours and 7 days will be as a look ahead time  of 2 hours is defined
+====
+
+[discrete]
+[[breaking_813_rest_api_changes]]
+==== REST API changes
+
+[[esql_grammar_from_metadata_no_longer_requires]]
+.ESQL: Grammar - FROM METADATA no longer requires []
+[%collapsible]
+====
+*Details* +
+Remove [ ] for METADATA option inside FROM command statements
+
+*Impact* +
+Previously to return metadata fields, one had to use square brackets: (eg. 'FROM index [METADATA _index]'). This is no longer needed: the [ ] are dropped and do not have to be specified, thus simplifying the command above to:'FROM index METADATA _index'.
+====
+
+[[es_ql_remove_project_keyword_from_grammar]]
+.ES|QL: remove PROJECT keyword from the grammar
+[%collapsible]
+====
+*Details* +
+Removes the PROJECT keyword (an alias for KEEP) from ES|QL grammar
+
+*Impact* +
+Before this change, users could use PROJECT as an alias for KEEP in ESQL queries, (eg. 'FROM idx | PROJECT name, surname') the parser replaced PROJECT with KEEP, emitted a warning: 'PROJECT command is no longer supported, please use KEEP instead' and the query was executed normally. With this change, PROJECT command is no longer recognized by the query parser; queries using PROJECT command now return a parsing exception.
+====
+
+[[esql_remove_nan_finite_infinite]]
+.[ESQL] Remove is_nan, is_finite, and `is_infinite`
+[%collapsible]
+====
+*Details* +
+Removes the functions `is_nan`, `is_finite`, and `is_infinite`.
+
+*Impact* +
+Attempting to use the above functions will now be a planner time error.  These functions are no longer supported.
+====
+
+
+[discrete]
+[[deprecated-8.13]]
+=== Deprecations
+
+The following functionality has been deprecated in {es} 8.13
+and will be removed in a future version.
+While this won't have an immediate impact on your applications,
+we strongly encourage you to take the described steps to update your code
+after upgrading to 8.13.
+
+To find out if you are using any deprecated functionality,
+enable <<deprecation-logging, deprecation logging>>.
+
+[discrete]
+[[deprecations_813_cluster_and_node_setting]]
+==== Cluster and node setting deprecations
+
+[[deprecate_client_type]]
+.Deprecate `client.type`
+[%collapsible]
+====
+*Details* +
+The node setting `client.type` has been ignored since the node client was removed in 8.0. The setting is now deprecated and will be removed in a future release.
+
+*Impact* +
+Remove the `client.type` setting from `elasticsearch.yml`
+====
+
+[discrete]
+[[deprecations_813_rest_api]]
+==== REST API deprecations
+
+[[desirednode_deprecate_node_version_field_make_it_optional_for_current_version]]
+.`DesiredNode:` deprecate `node_version` field and make it optional for the current version
+[%collapsible]
+====
+*Details* +
+The desired_node API includes a `node_version` field to perform validation on the new node version required. This kind of check is too broad, and it's better done by external logic, so it has been removed, making the `node_version` field not necessary. The field will be removed in a later version.
+
+*Impact* +
+Users should update their usages of `desired_node` to not include the `node_version` field anymore.
+====
 

+ 437 - 2
docs/reference/release-notes/8.13.0.asciidoc

@@ -1,8 +1,443 @@
 [[release-notes-8.13.0]]
 == {es} version 8.13.0
 
-coming[8.13.0]
-
 Also see <<breaking-changes-8.13,Breaking changes in 8.13>>.
 
+[[breaking-8.13.0]]
+[float]
+=== Breaking changes
+
+ES|QL::
+* ESQL: Grammar - FROM METADATA no longer requires [] {es-pull}105221[#105221]
+* ES|QL: remove PROJECT keyword from the grammar {es-pull}105064[#105064]
+* [ESQL] Remove is_nan, is_finite, and `is_infinite` {es-pull}104091[#104091]
+
+TSDB::
+* Change `index.look_ahead_time` index setting's default value from 2 hours to 30 minutes. {es-pull}103898[#103898]
+* Lower the `look_ahead_time` index setting's max value from 7 days to 2 hours. {es-pull}103434[#103434]
+
+[[bug-8.13.0]]
+[float]
+=== Bug fixes
+
+Aggregations::
+* Disable parallel collection for terms aggregation with `min_doc_count` equals to 0 {es-pull}106156[#106156]
+* `GlobalOrdCardinalityAggregator` should use `HyperLogLogPlusPlus` instead of `HyperLogLogPlusPlusSparse` {es-pull}105546[#105546]
+
+Allocation::
+* Fix disk computation when initializing new shards {es-pull}102879[#102879]
+* Fix disk computation when initializing unassigned shards in desired balance computation {es-pull}102207[#102207]
+
+Application::
+* Fix Search Applications bug where deleting an alias before deleting an application intermittently caused errors {es-pull}106329[#106329]
+* Use search to determine if cluster contains data {es-pull}103920[#103920]
+* [Connector API] Bugfix: support list type in filtering advenced snippet value {es-pull}105633[#105633]
+* [Connector API] Fix default ordering in `SyncJob` list endpoint {es-pull}105945[#105945]
+* [Connector API] Fix serialisation of script params in connector index service {es-pull}106060[#106060]
+
+Authentication::
+* Execute SAML authentication on the generic threadpool {es-pull}105232[#105232] (issue: {es-issue}104962[#104962])
+
+Authorization::
+* Adjust interception of requests for specific shard IDs {es-pull}101656[#101656]
+
+Client::
+* Validate settings in `ReloadSecureSettings` API {es-pull}103176[#103176]
+
+Data streams::
+* Apm-data: fix `@custom` component templates {es-pull}104182[#104182]
+* Avoid false-positive matches on intermediate objects in `ecs@mappings` {es-pull}105440[#105440] (issue: {es-issue}102794[#102794])
+* Execute lazy rollover with an internal dedicated user #104732 {es-pull}104905[#104905] (issue: {es-issue}104732[#104732])
+* Fix write index resolution when an alias is pointing to a TSDS {es-pull}104440[#104440] (issue: {es-issue}104189[#104189])
+* x-pack/plugin/core: add `match_mapping_type` to `ecs@mappings` dynamic templates {es-pull}103035[#103035]
+
+Distributed::
+* Fix logger Strings.format calls {es-pull}104573[#104573]
+* Request indexing memory pressure in APM node metrics publisher {es-pull}103520[#103520]
+
+ES|QL::
+* ESQL: Add single value checks on LIKE/RLIKE pushdown {es-pull}103807[#103807] (issue: {es-issue}103806[#103806])
+* ESQL: Correct out-of-range filter pushdowns {es-pull}99961[#99961] (issue: {es-issue}99960[#99960])
+* ESQL: Fix Analyzer to not interpret escaped * as a pattern {es-pull}105325[#105325] (issue: {es-issue}104955[#104955])
+* ESQL: Fix a bug loading unindexed text fields {es-pull}104553[#104553]
+* ESQL: Fix bug in grammar that allowed spaces inside id pattern {es-pull}105476[#105476] (issue: {es-issue}105441[#105441])
+* ESQL: Fix replacement of nested expressions in aggs with multiple parameters {es-pull}104718[#104718] (issue: {es-issue}104706[#104706])
+* ESQL: Fix wrong attribute shadowing in pushdown rules {es-pull}105650[#105650] (issue: {es-issue}105434[#105434])
+* ESQL: Improve pushdown of certain filters {es-pull}103538[#103538] (issue: {es-issue}103536[#103536])
+* ESQL: allow `null` in date math {es-pull}103610[#103610] (issue: {es-issue}103085[#103085])
+* ESQL: make `cidr_match` foldable {es-pull}105403[#105403] (issue: {es-issue}105376[#105376])
+* ES|QL: Disable optimizations that rely on Expression.nullable() {es-pull}105691[#105691]
+* ES|QL: Improve type validation in aggs for UNSIGNED_LONG better support for VERSION {es-pull}104911[#104911] (issue: {es-issue}102961[#102961])
+* ES|QL: better management of exact subfields for TEXT fields {es-pull}103510[#103510] (issue: {es-issue}99899[#99899])
+* Fix error on sorting unsortable `geo_point` and `cartesian_point` {es-pull}106351[#106351] (issue: {es-issue}106007[#106007])
+* For empty mappings use a `LocalRelation` {es-pull}105081[#105081] (issue: {es-issue}104809[#104809])
+* Resume driver when failing to fetch pages {es-pull}106392[#106392] (issue: {es-issue}106262[#106262])
+* Review KEEP logic to prevent duplicate column names {es-pull}103316[#103316]
+* `ProjectOperator` should not retain references to released blocks {es-pull}105848[#105848]
+
+Engine::
+* Consider currently refreshing data in the memory usage of refresh {es-pull}104122[#104122]
+* Release `TranslogSnapshot` buffer after iteration {es-pull}106398[#106398] (issue: {es-issue}106390[#106390])
+
+Health::
+* Make Health API more resilient to multi-version clusters {es-pull}105789[#105789] (issue: {es-issue}90183[#90183])
+* Stop the periodic health logger when es is stopping {es-pull}105272[#105272]
+
+ILM+SLM::
+* Remove `hashCode` and `equals` from `OperationModeUpdateTask` {es-pull}104265[#104265] (issue: {es-issue}100871[#100871])
+* [ILM] Delete step deletes data stream with only one index {es-pull}105772[#105772]
+
+Indices APIs::
+* Fix `require_alias` implicit true value on presence {es-pull}104099[#104099] (issue: {es-issue}103945[#103945])
+
+Infra/CLI::
+* Fix server cli to always pass through exit code {es-pull}104943[#104943]
+
+Infra/Core::
+* Do not enable APM agent 'instrument', it's not required for manual tracing {es-pull}105055[#105055]
+* Fix bogus assertion tripped by force-executed tasks {es-pull}104581[#104581] (issue: {es-issue}104580[#104580])
+* Metrics: Allow `AsyncCounters` to switch providers {es-pull}103025[#103025]
+* Metrics: Handle null observations in observers {es-pull}103091[#103091]
+
+Infra/Node Lifecycle::
+* Close rather than stop `HttpServerTransport` on shutdown {es-pull}102759[#102759] (issue: {es-issue}102501[#102501])
+
+Ingest Node::
+* Add stable `ThreadPool` constructor to `LogstashInternalBridge` {es-pull}105163[#105163]
+* Adding `executedPipelines` to the `IngestDocument` copy constructor {es-pull}105427[#105427]
+* Revert "x-pack/plugin/apm-data: download geoip DB on pipeline creation" {es-pull}104505[#104505]
+* X-pack/plugin/apm-data: fix `@custom` pipeline support {es-pull}104113[#104113]
+
+Machine Learning::
+* Allow GET inference models by user a with read only permission {es-pull}105346[#105346]
+* Avoid computing `currentInferenceProcessors` on every cluster state {es-pull}106057[#106057]
+* Catch all the potential exceptions in the ingest processor code {es-pull}105391[#105391]
+* Changed system auditor to use levels {es-pull}105429[#105429]
+* During ML maintenance, reset jobs in the reset state without a corresponding task {es-pull}106062[#106062]
+* Fix `categorize_text` aggregation nested under empty buckets {es-pull}105987[#105987] (issue: {es-issue}105836[#105836])
+* Fix resetting a job if the original reset task no longer exists. {es-pull}106020[#106020]
+* Retry updates to model snapshot ID on job config {es-pull}104077[#104077]
+* The OpenAI model parameter should be in service settings not task settings. Move the configuration field to service settings {es-pull}105458[#105458]
+* Undeploy elser when inference model deleted {es-pull}104230[#104230]
+
+Mapping::
+* Fix parsing of flattened fields within subobjects: false {es-pull}105373[#105373]
+
+Network::
+* Fix use-after-free at event-loop shutdown {es-pull}105486[#105486]
+
+Search::
+* Correct profiled rewrite time for knn with a pre-filter {es-pull}104150[#104150]
+* Force execution of `SearchService.Reaper` {es-pull}106544[#106544] (issue: {es-issue}106543[#106543])
+* Move `TransportTermsEnumAction` coordination off transport threads {es-pull}104408[#104408]
+* Remove `SearchException` usages without a proper status code {es-pull}105150[#105150]
+* Require the name field for `inner_hits` for collapse {es-pull}104666[#104666]
+* add validation on _id field when upsert new doc {es-pull}103399[#103399] (issue: {es-issue}102981[#102981])
+
+Security::
+* Revert "Validate settings in `ReloadSecureSettings` API" {es-pull}103310[#103310]
+
+Snapshot/Restore::
+* Do not record s3 http request time when it is not available {es-pull}105103[#105103]
+* `URLRepository` should not block shutdown {es-pull}105588[#105588]
+
+TLS::
+* Respect --pass option in certutil csr mode {es-pull}106105[#106105]
+
+Transform::
+* Fix `_reset` API when called with `force=true` on a failed transform {es-pull}106574[#106574] (issue: {es-issue}106573[#106573])
+* Fix a bug where destination index aliases are not set up for an unattended transform {es-pull}105499[#105499]
+* Remove duplicate checkpoint audits {es-pull}105164[#105164] (issue: {es-issue}105106[#105106])
+* Return results in order {es-pull}105089[#105089] (issue: {es-issue}104847[#104847])
+* Use deduced mappings for determining proper fields' format even if `deduce_mappings==false` {es-pull}103682[#103682] (issue: {es-issue}103115[#103115])
+
+Vector Search::
+* Fix bug when nested knn pre-filter might match nested docs {es-pull}105994[#105994]
+
+Watcher::
+* Handling exceptions on watcher reload {es-pull}105442[#105442] (issue: {es-issue}69842[#69842])
+
+[[deprecation-8.13.0]]
+[float]
+=== Deprecations
+
+Distributed::
+* `DesiredNode:` deprecate `node_version` field and make it optional (unused) in current parser {es-pull}104209[#104209]
+
+Infra/Core::
+* Deprecate `client.type` {es-pull}104574[#104574]
+
+[[enhancement-8.13.0]]
+[float]
+=== Enhancements
+
+Aggregations::
+* Add index mapping parameter for `counted_keyword` {es-pull}103646[#103646]
+* Introduce an `AggregatorReducer` to reduce the footprint of aggregations in the coordinating node {es-pull}105207[#105207]
+* Release resources in `BestBucketsDeferringCollector` earlier {es-pull}104893[#104893]
+* Support sampling in `counted_terms` aggregation {es-pull}103846[#103846]
+
+Allocation::
+* Account for reserved disk size {es-pull}103903[#103903]
+* Derive expected replica size from primary {es-pull}102078[#102078]
+
+Application::
+* Add serverless scopes for Connector APIs {es-pull}104063[#104063]
+* [Connector API] Change required privileges to indices:data/read(write) {es-pull}105289[#105289]
+* [Connector API] Implement update `index_name` action {es-pull}104648[#104648]
+* [Connector API] Support filtering by name, index name in list action {es-pull}105131[#105131]
+* [Connector API] Support filtering connectors by service type and a query {es-pull}105178[#105178]
+* [Connector API] Support updating configuration values only {es-pull}105249[#105249]
+* [Connectors API] Add new field `api_key_secret_id` to Connector {es-pull}104982[#104982]
+* [Connectors API] Implement connector status update action {es-pull}104750[#104750]
+* [Connectors API] Implement update native action endpoint {es-pull}104654[#104654]
+* [Connectors API] Implement update service type action {es-pull}104643[#104643]
+* [Connectors API] Relax strict response parsing for get/list operations {es-pull}104909[#104909]
+* [Profiling] Extract properties faster from source {es-pull}104356[#104356]
+* [Profiling] Mark all templates as managed {es-pull}103783[#103783]
+* [Profiling] Speed up processing of stacktraces {es-pull}104674[#104674]
+* [Profiling] Support downsampling of generic events {es-pull}104730[#104730]
+* [Profiling] Use shard request cache consistently {es-pull}103643[#103643]
+
+Authentication::
+* Expose API key authentication metrics {es-pull}103178[#103178]
+* Expose realms authentication metrics {es-pull}104200[#104200]
+* Expose service account authentication metrics {es-pull}104043[#104043]
+* Expose token authentication metrics {es-pull}104142[#104142]
+* Hot-reloadable LDAP bind password {es-pull}104320[#104320]
+* Support of `match` for the Query API Key API {es-pull}104594[#104594]
+
+Authorization::
+* [Security Solution] Allow write permission for `kibana_system` role on endpoint response index {es-pull}103555[#103555]
+
+CRUD::
+* Avoid wrapping searchers multiple times in mget {es-pull}104227[#104227] (issue: {es-issue}85069[#85069])
+
+Client::
+* Add rest spec for Query User API {es-pull}104529[#104529]
+
+Cluster Coordination::
+* Add troubleshooting docs link to `PeerFinder` logs {es-pull}104787[#104787]
+* Report current master in `PeerFinder` {es-pull}104396[#104396]
+
+Data streams::
+* Introduce lazy rollover for mapping updates in data streams {es-pull}103309[#103309] (issue: {es-issue}89346[#89346])
+* Use new `ignore_dynamic_beyond_limit` in logs and metric data streams {es-pull}105180[#105180]
+* X-pack/plugin/apm-data: add dynamic setting for enabling template registry {es-pull}104386[#104386] (issue: {es-issue}104385[#104385])
+* X-pack/plugin/core: rename `double_metrics` template {es-pull}103033[#103033]
+* x-pack/plugin/apm-data: Add a new field transaction.profiler_stack_trace_ids to traces-apm@mappings.yaml {es-pull}105223[#105223]
+* x-pack/plugin/apm-data: Map some APM fields as flattened and fix error.grouping_name script {es-pull}103032[#103032]
+* x-pack/plugin/core: make automatic rollovers lazy {es-pull}105273[#105273] (issue: {es-issue}104083[#104083])
+
+Discovery-Plugins::
+* Set read timeout for fetching IMDSv2 token {es-pull}104407[#104407] (issue: {es-issue}104244[#104244])
+
+Downsampling::
+* Support patch transport version from 8.12 {es-pull}104406[#104406]
+
+ES|QL::
+* Add ES|QL async delete API {es-pull}103628[#103628]
+* Avoid humongous blocks {es-pull}103340[#103340]
+* ESQL: Add TO_UPPER and TO_LOWER functions {es-pull}104309[#104309]
+* ESQL: Add option to drop null fields {es-pull}102428[#102428]
+* ESQL: Add plan consistency verification after each optimizer {es-pull}105371[#105371]
+* ESQL: Check field exists before load from `_source` {es-pull}103632[#103632]
+* ESQL: Delay finding field load infrastructure {es-pull}103821[#103821]
+* ESQL: Expand shallow copy with vecs {es-pull}103681[#103681] (issue: {es-issue}100528[#100528])
+* ESQL: Extend STATS command to support aggregate expressions {es-pull}104958[#104958]
+* ESQL: Infer not null for aggregated fields {es-pull}103673[#103673] (issue: {es-issue}102787[#102787])
+* ESQL: Nested expressions inside stats command {es-pull}104387[#104387] (issue: {es-issue}99828[#99828])
+* ESQL: Pre-allocate rows in TopNOperator {es-pull}104796[#104796]
+* ESQL: Referencing expressions that contain backticks requires <<esql-identifiers,escaping those backticks>>. {es-pull}100740[#100740] (issue: {es-issue}100312[#100312])
+* ESQL: Simpify IS NULL/IS NOT NULL evaluation {es-pull}103099[#103099] (issue: {es-issue}103097[#103097])
+* ESQL: Speed up reading many nulls {es-pull}105088[#105088]
+* ESQL: Support loading shapes from source into WKB blocks {es-pull}104269[#104269]
+* ESQL: Track the rest of `DocVector` {es-pull}103727[#103727]
+* ESQL: `MV_FIRST` and `MV_LAST` {es-pull}103928[#103928]
+* ESQL: add `date_diff` function {es-pull}104118[#104118] (issue: {es-issue}101942[#101942])
+* ESQL: push down "[text_field] is not null" {es-pull}105593[#105593]
+* ES|QL Async Query API {es-pull}103398[#103398]
+* Prepare enrich plan to support multi clusters {es-pull}104355[#104355]
+* Reading points from source to reduce precision loss {es-pull}103698[#103698]
+* Remove deprecated Block APIs {es-pull}103592[#103592]
+* Reserve bytes before serializing page {es-pull}105269[#105269]
+* Support ST_CENTROID over spatial points {es-pull}104218[#104218] (issue: {es-issue}104656[#104656])
+* Support cross clusters query in ESQL {es-pull}101640[#101640]
+* Support enrich ANY mode in cross clusters query {es-pull}104840[#104840]
+* Support enrich coordinator mode {es-pull}104936[#104936]
+* Support enrich remote mode {es-pull}104993[#104993]
+
+Geo::
+* Add support for Well Known Binary (WKB) in the fields API for spatial fields {es-pull}103461[#103461]
+* Add the possibility to transform WKT to WKB directly {es-pull}104030[#104030]
+
+Health::
+* Add APM metrics to `HealthPeriodicLogger` {es-pull}102765[#102765]
+* Extend `repository_integrity` health indicator for unknown and invalid repos {es-pull}104614[#104614] (issue: {es-issue}103784[#103784])
+
+ILM+SLM::
+* Add "step":"ERROR" to ILM explain response for missing policy {es-pull}103720[#103720] (issue: {es-issue}99030[#99030])
+* Add default rollover conditions to ILM explain API response {es-pull}104721[#104721] (issue: {es-issue}103395[#103395])
+* ILM/SLM history policies forcemerge in hot and dsl configuration {es-pull}103190[#103190]
+
+Infra/CLI::
+* Add replay diagnostic dir to system jvm options {es-pull}103535[#103535]
+
+Infra/Circuit Breakers::
+* Lower G1 minimum full GC interval {es-pull}105259[#105259]
+
+Infra/Core::
+* Adding threadpool metrics {es-pull}102371[#102371]
+* ES - document observing with rejections {es-pull}104859[#104859]
+* Thread pool metrics {es-pull}104500[#104500]
+
+Infra/Metrics::
+* Modify name of threadpool metric for rejected {es-pull}105015[#105015]
+
+Infra/Node Lifecycle::
+* Wait for async searches to finish when shutting down {es-pull}101487[#101487]
+
+Infra/Transport API::
+* Make `ParentTaskAssigningClient.getRemoteClusterClient` method also return `ParentTaskAssigningClient` {es-pull}100813[#100813]
+
+Ingest Node::
+* Adding `ActionRequestLazyBuilder` implementation of `RequestBuilder` {es-pull}104927[#104927]
+* Adding a `RequestBuilder` interface {es-pull}104778[#104778]
+* Adding a custom exception for problems with the graph of pipelines to be applied to a document {es-pull}105196[#105196]
+* Improving the performance of the ingest simulate verbose API {es-pull}105265[#105265]
+* Ingest geoip processor cache 'no results' from the database {es-pull}104092[#104092]
+* Limiting the number of nested pipelines that can be executed {es-pull}105428[#105428]
+* Modifying request builders {es-pull}104636[#104636]
+
+Java Low Level REST Client::
+* Set thread name used by REST client {es-pull}103160[#103160]
+
+Machine Learning::
+* Add optional pruning configuration (weighted terms scoring) to text expansion query {es-pull}102862[#102862]
+* Add text_embedding inference service with multilingual-e5 and custom eland models {es-pull}104949[#104949]
+* Add 3 automatic restarts for `pytorch_inference` processes that stop unexpectedly {es-pull}104433[#104433]
+* Add support for Cohere inference service {es-pull}104559[#104559]
+* Always test for spikes and dips as well as changes in the change point aggregation {es-pull}103922[#103922]
+* Apply windowing and chunking to long documents {es-pull}104363[#104363]
+* Automatically download the ELSER model when PUT in `_inference` {es-pull}104334[#104334]
+* Better handling of number of allocations in pytorch_inference in the case that hardware_concurrency fails {ml-pull}2607[#2607]
+* Change detection aggregation improvements {es-pull}102824[#102824]
+* Conditionally send the dimensions field as part of the openai requests {es-pull}105299[#105299] (issue: {es-issue}105005[#105005])
+* Endpoint to find positions of Grok pattern matches {es-pull}104394[#104394]
+* Ensure unique IDs between inference models and trained model deployments {es-pull}103996[#103996]
+* Expose some ML metrics via APM {es-pull}102584[#102584]
+* Make `task_type` optional in `_inference` APIs {es-pull}104483[#104483]
+* Update `missingTrainedModel` message to include: you may need to create it {es-pull}104155[#104155]
+* Upgrade MKL to version 2024.0 on Linux x86_64 {ml-pull}2619[#2619]
+* Upgrade PyTorch to version 2.1.2. {ml-pull}2588[#2588]
+* Upgrade zlib to version 1.2.13 on Windows {ml-pull}2588[#2588]
+* Use Boost.JSON for JSON processing {ml-pull}2614[#2614]
+* Validate inference model ids {es-pull}103669[#103669]
+
+
+Mapping::
+* Add `index.mapping.total_fields.ignore_dynamic_beyond_limit` setting to ignore dynamic fields when field limit is reached {es-pull}96235[#96235]
+* Make field limit more predictable {es-pull}102885[#102885]
+
+Network::
+* Prune unnecessary information from TransportNodesStatsAction.NodeStatsRequest {es-pull}102559[#102559] (issue: {es-issue}100878[#100878])
+
+Percolator::
+* Return `matched_queries` in Percolator {es-pull}103084[#103084] (issue: {es-issue}10163[#10163])
+
+Query Languages::
+* Introduce Alias.unwrap method {es-pull}104575[#104575]
+
+Search::
+* Dyamically adjust node metrics cache expire {es-pull}104460[#104460]
+* Enhancement: Metrics for Search Took Times using Action Listeners {es-pull}104996[#104996]
+* Field caps performance pt2 {es-pull}105941[#105941]
+* Field-caps field has value lookup use map instead of looping array {es-pull}105770[#105770]
+* Flag in `_field_caps` to return only fields with values in index {es-pull}103651[#103651]
+* Include better output in profiling & `toString` for automaton based queries {es-pull}105468[#105468]
+* Metrics for search latencies {es-pull}102557[#102557]
+* Ref count search response bytes {es-pull}103763[#103763] (issue: {es-issue}102657[#102657])
+* Remove leniency in msearch parsing {es-pull}103232[#103232]
+* Resolve Cluster API {es-pull}102726[#102726]
+* Reuse number field mapper tests in other modules {es-pull}99142[#99142] (issue: {es-issue}92947[#92947])
+* S3 first byte latency metric {es-pull}102435[#102435]
+* Update s3 latency metric to use micros {es-pull}103633[#103633]
+* Upgrade to Lucene 9.10.0 {es-pull}105578[#105578]
+
+Security::
+* Add Query Users API {es-pull}104033[#104033]
+* Add `ApiKey` expiration time to audit log {es-pull}103959[#103959]
+* Add expiration time to update api key api {es-pull}103453[#103453]
+* Add stricter validation for api key expiration time {es-pull}103973[#103973]
+* Add support for the `simple_query_string` to the Query API Key API {es-pull}104132[#104132]
+* Add support for the `type` parameter, for sorting, to the Query API Key API {es-pull}104625[#104625]
+* Aggs support for Query API Key Information API {es-pull}104895[#104895]
+* Hot-reloadable remote cluster credentials {es-pull}102798[#102798]
+
+Snapshot/Restore::
+* Add s3 `HeadObject` request to request stats {es-pull}105105[#105105]
+* Expose `OperationPurpose` via `CustomQueryParameter` to s3 logs {es-pull}105044[#105044]
+* Fix blob cache race, decay, time dependency {es-pull}104784[#104784]
+* Pause shard snapshots on graceful shutdown {es-pull}101717[#101717]
+* Retry indefinitely for s3 indices blob read errors {es-pull}103300[#103300]
+
+Store::
+* List hidden shard stores by default {es-pull}103710[#103710]
+
+TLS::
+* 'elasticsearch-certutil cert' now verifies the issuing chain of the generated certificate {es-pull}103948[#103948]
+
+TSDB::
+* Improve storage efficiency for non-metric fields in TSDB {es-pull}99747[#99747]
+* Introduce experimental pass-through field type {es-pull}103648[#103648]
+* Nest pass-through objects within objects {es-pull}105062[#105062]
+* Restrict usage of certain aggregations when in sort order execution is required {es-pull}104665[#104665]
+* Small time series agg improvement {es-pull}106288[#106288]
+
+Transform::
+* Allow transforms to use PIT with remote clusters again {es-pull}105192[#105192] (issue: {es-issue}104518[#104518])
+* Transforms: Adding basic stats API param {es-pull}104878[#104878]
+
+Vector Search::
+* Add new int8_flat and flat vector index types {es-pull}104872[#104872]
+* Add support for more than one `inner_hit` when searching nested vectors {es-pull}104006[#104006]
+* Making `k` and `num_candidates` optional for knn search {es-pull}101209[#101209] (issue: {es-issue}97533[#97533])
+
+[[feature-8.13.0]]
+[float]
+=== New features
+
+Data streams::
+* Add `require_data_stream` parameter to indexing requests to enforce indexing operations target a data stream {es-pull}101872[#101872] (issue: {es-issue}97032[#97032])
+* Redirect failed ingest node operations to a failure store when available {es-pull}103481[#103481]
+
+ES|QL::
+* ESQL: Introduce mode setting for ENRICH {es-pull}103949[#103949]
+* ESQL: add =~ operator (case insensitive equality) {es-pull}103656[#103656]
+
+Health::
+* Create a DSL health indicator as part of the health API {es-pull}103130[#103130]
+
+Infra/Core::
+* Add gradle tasks and code to modify and access mappings between version ids and release versions {es-pull}103627[#103627]
+
+Mapping::
+* Add `unmatch_mapping_type`, and support array of types {es-pull}103171[#103171] (issues: {es-issue}102807[#102807], {es-issue}102795[#102795])
+
+Search::
+* Added Duplicate Word Check Feature to Analysis Nori {es-pull}103325[#103325] (issue: {es-issue}103321[#103321])
+* [Synonyms] Mark Synonyms as GA {es-pull}103223[#103223]
+
+[[upgrade-8.13.0]]
+[float]
+=== Upgrades
+
+Query Languages::
+* Upgrade ANTLR4 to 4.13.1 {es-pull}105334[#105334] (issue: {es-issue}102953[#102953])
+
+Search::
+* Upgrade to Lucene 9.9.0 {es-pull}102782[#102782]
+* Upgrade to Lucene 9.9.1 {es-pull}103387[#103387]
+* Upgrade to Lucene 9.9.2 {es-pull}104753[#104753]
+
 

+ 10 - 0
docs/reference/release-notes/highlights.asciidoc

@@ -62,6 +62,16 @@ fields that don't have a value. This can be done through the newly added
 
 {es-pull}103651[#103651]
 
+[discrete]
+[[new_lucene_9_10_release]]
+=== New Lucene 9.10 release
+- https://github.com/apache/lucene/pull/13090: Prevent humongous allocations in ScalarQuantizer when building quantiles.
+- https://github.com/apache/lucene/pull/12962: Speedup concurrent multi-segment HNSW graph search
+- https://github.com/apache/lucene/pull/13033: Range queries on numeric/date/ip fields now exit earlier on segments whose values don't intersect with the query range. This should especially help when there are other required clauses in the `bool` query and when the range filter is narrow, e.g. filtering on the last 5 minutes.
+- https://github.com/apache/lucene/pull/13026: `bool` queries that mix `filter` and `should` clauses will now propagate minimum competitive scores through the `should` clauses. This should yield speedups when sorting by descending score.
+
+{es-pull}105578[#105578]
+
 // end::notable-highlights[]