circuit-breaker-errors.asciidoc 3.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102
  1. [[circuit-breaker-errors]]
  2. === Circuit breaker errors
  3. {es} uses <<circuit-breaker,circuit breakers>> to prevent nodes from running out
  4. of JVM heap memory. If Elasticsearch estimates an operation would exceed a
  5. circuit breaker, it stops the operation and returns an error.
  6. By default, the <<parent-circuit-breaker,parent circuit breaker>> triggers at
  7. 95% JVM memory usage. To prevent errors, we recommend taking steps to reduce
  8. memory pressure if usage consistently exceeds 85%.
  9. See https://www.youtube.com/watch?v=k3wYlRVbMSw[this video] for a walkthrough
  10. of diagnosing circuit breaker errors.
  11. ****
  12. If you're using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to https://www.elastic.co/guide/en/cloud/current/ec-autoops.html[Monitor with AutoOps].
  13. ****
  14. [discrete]
  15. [[diagnose-circuit-breaker-errors]]
  16. ==== Diagnose circuit breaker errors
  17. **Error messages**
  18. If a request triggers a circuit breaker, {es} returns an error with a `429` HTTP
  19. status code.
  20. [source,js]
  21. ----
  22. {
  23. 'error': {
  24. 'type': 'circuit_breaking_exception',
  25. 'reason': '[parent] Data too large, data for [<http_request>] would be [123848638/118.1mb], which is larger than the limit of [123273216/117.5mb], real usage: [120182112/114.6mb], new bytes reserved: [3666526/3.4mb]',
  26. 'bytes_wanted': 123848638,
  27. 'bytes_limit': 123273216,
  28. 'durability': 'TRANSIENT'
  29. },
  30. 'status': 429
  31. }
  32. ----
  33. // NOTCONSOLE
  34. {es} also writes circuit breaker errors to <<logging,`elasticsearch.log`>>. This
  35. is helpful when automated processes, such as allocation, trigger a circuit
  36. breaker.
  37. [source,txt]
  38. ----
  39. Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [num/numGB], which is larger than the limit of [num/numGB], usages [request=0/0b, fielddata=num/numKB, in_flight_requests=num/numGB, accounting=num/numGB]
  40. ----
  41. **Check JVM memory usage**
  42. If you've enabled Stack Monitoring, you can view JVM memory usage in {kib}. In
  43. the main menu, click **Stack Monitoring**. On the Stack Monitoring **Overview**
  44. page, click **Nodes**. The **JVM Heap** column lists the current memory usage
  45. for each node.
  46. You can also use the <<cat-nodes,cat nodes API>> to get the current
  47. `heap.percent` for each node.
  48. [source,console]
  49. ----
  50. GET _cat/nodes?v=true&h=name,node*,heap*
  51. ----
  52. To get the JVM memory usage for each circuit breaker, use the
  53. <<cluster-nodes-stats,node stats API>>.
  54. [source,console]
  55. ----
  56. GET _nodes/stats/breaker
  57. ----
  58. [discrete]
  59. [[prevent-circuit-breaker-errors]]
  60. ==== Prevent circuit breaker errors
  61. **Reduce JVM memory pressure**
  62. High JVM memory pressure often causes circuit breaker errors. See
  63. <<high-jvm-memory-pressure>>.
  64. **Avoid using fielddata on `text` fields**
  65. For high-cardinality `text` fields, fielddata can use a large amount of JVM
  66. memory. To avoid this, {es} disables fielddata on `text` fields by default. If
  67. you've enabled fielddata and triggered the <<fielddata-circuit-breaker,fielddata
  68. circuit breaker>>, consider disabling it and using a `keyword` field instead.
  69. See <<fielddata-mapping-param>>.
  70. **Clear the fielddata cache**
  71. If you've triggered the fielddata circuit breaker and can't disable fielddata,
  72. use the <<indices-clearcache,clear cache API>> to clear the fielddata cache.
  73. This may disrupt any in-flight searches that use fielddata.
  74. [source,console]
  75. ----
  76. POST _cache/clear?fielddata=true
  77. ----
  78. // TEST[s/^/PUT my-index\n/]