rejected-requests.asciidoc 2.5 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980
  1. [[rejected-requests]]
  2. === Rejected requests
  3. When {es} rejects a request, it stops the operation and returns an error with a
  4. `429` response code. Rejected requests are commonly caused by:
  5. * A <<high-cpu-usage,depleted thread pool>>. A depleted `search` or `write`
  6. thread pool returns a `TOO_MANY_REQUESTS` error message.
  7. * A <<circuit-breaker-errors,circuit breaker error>>.
  8. * High <<index-modules-indexing-pressure,indexing pressure>> that exceeds the
  9. <<memory-limits,`indexing_pressure.memory.limit`>>.
  10. [discrete]
  11. [[check-rejected-tasks]]
  12. ==== Check rejected tasks
  13. To check the number of rejected tasks for each thread pool, use the
  14. <<cat-thread-pool,cat thread pool API>>. A high ratio of `rejected` to
  15. `completed` tasks, particularly in the `search` and `write` thread pools, means
  16. {es} regularly rejects requests.
  17. [source,console]
  18. ----
  19. GET /_cat/thread_pool?v=true&h=id,name,queue,active,rejected,completed
  20. ----
  21. `write` thread pool rejections frequently appear in the erring API and
  22. correlating log as `EsRejectedExecutionException` with either
  23. `QueueResizingEsThreadPoolExecutor` or `queue capacity`.
  24. These errors are often related to <<task-queue-backlog,backlogged tasks>>.
  25. [discrete]
  26. [[check-circuit-breakers]]
  27. ==== Check circuit breakers
  28. To check the number of tripped <<circuit-breaker,circuit breakers>>, use the
  29. <<cluster-nodes-stats,node stats API>>.
  30. [source,console]
  31. ----
  32. GET /_nodes/stats/breaker
  33. ----
  34. These statistics are cumulative from node startup. For more information, see
  35. <<circuit-breaker,circuit breaker errors>>.
  36. [discrete]
  37. [[check-indexing-pressure]]
  38. ==== Check indexing pressure
  39. To check the number of <<index-modules-indexing-pressure,indexing pressure>>
  40. rejections, use the <<cluster-nodes-stats,node stats API>>.
  41. [source,console]
  42. ----
  43. GET _nodes/stats?human&filter_path=nodes.*.indexing_pressure
  44. ----
  45. These stats are cumulative from node startup.
  46. Indexing pressure rejections appear as an
  47. `EsRejectedExecutionException`, and indicate that they were rejected due
  48. to `combined_coordinating_and_primary`, `coordinating`, `primary`, or `replica`.
  49. These errors are often related to <<task-queue-backlog,backlogged tasks>>,
  50. <<docs-bulk,bulk index>> sizing, or the ingest target's
  51. <<index-modules,`refresh_interval` setting>>.
  52. [discrete]
  53. [[prevent-rejected-requests]]
  54. ==== Prevent rejected requests
  55. **Fix high CPU and memory usage**
  56. If {es} regularly rejects requests and other tasks, your cluster likely has high
  57. CPU usage or high JVM memory pressure. For tips, see <<high-cpu-usage>> and
  58. <<high-jvm-memory-pressure>>.