increase-other-node-capacity.asciidoc 3.2 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394
  1. // tag::cloud[]
  2. . Log in to the {ess-console}[{ecloud} console].
  3. +
  4. . On the **Elasticsearch Service** panel, click the gear under the `Manage deployment` column that corresponds to the
  5. name of your deployment.
  6. +
  7. . Go to `Actions > Edit deployment` and then go to the `Coordinating instances` or the `Machine Learning instances`
  8. section depending on the roles listed in the diagnosis:
  9. +
  10. [role="screenshot"]
  11. image::images/troubleshooting/disk/increase-disk-capacity-other-node.png[Increase disk capacity of other nodes,align="center"]
  12. . Choose a larger than the pre-selected capacity configuration from the drop-down menu and click `save`. Wait for
  13. the plan to be applied and the problem should be resolved.
  14. // end::cloud[]
  15. // tag::self-managed[]
  16. In order to increase the disk capacity of any other node, you will need to replace the instance that has run out of
  17. space with one of higher disk capacity.
  18. . First, retrieve the disk threshold that will indicate how much disk space is needed. The relevant threshold is
  19. the <<cluster-routing-watermark-high, high watermark>> and can be retrieved via the following command:
  20. +
  21. [source,console]
  22. ----
  23. GET _cluster/settings?include_defaults&filter_path=*.cluster.routing.allocation.disk.watermark.high*
  24. ----
  25. +
  26. The response will look like this:
  27. +
  28. [source,console-result]
  29. ----
  30. {
  31. "defaults": {
  32. "cluster": {
  33. "routing": {
  34. "allocation": {
  35. "disk": {
  36. "watermark": {
  37. "high": "90%",
  38. "high.max_headroom": "150GB"
  39. }
  40. }
  41. }
  42. }
  43. }
  44. }
  45. ----
  46. // TEST[skip:illustration purposes only]
  47. +
  48. The above means that in order to resolve the disk shortage we need to either drop our disk usage below the 90% or have
  49. more than 150GB available, read more how this threshold works <<cluster-routing-watermark-high, here>>.
  50. . The next step is to find out the current disk usage, this will allow to calculate how much extra space is needed.
  51. In the following example, we show only a machine learning node for readability purposes:
  52. +
  53. [source,console]
  54. ----
  55. GET /_cat/nodes?v&h=name,node.role,disk.used_percent,disk.used,disk.avail,disk.total
  56. ----
  57. +
  58. The response will look like this:
  59. +
  60. [source,console-result]
  61. ----
  62. name node.role disk.used_percent disk.used disk.avail disk.total
  63. instance-0000000000 l 85.31 3.4gb 500mb 4gb
  64. ----
  65. // TEST[skip:illustration purposes only]
  66. . The desired situation is to drop the disk usage below the relevant threshold, in our example 90%. Consider adding
  67. some padding, so it will not go over the threshold soon. Assuming you have the new node ready, add this node to the
  68. cluster.
  69. . Verify that the new node has joined the cluster:
  70. +
  71. [source,console]
  72. ----
  73. GET /_cat/nodes?v&h=name,node.role,disk.used_percent,disk.used,disk.avail,disk.total
  74. ----
  75. +
  76. The response will look like this:
  77. +
  78. [source,console-result]
  79. ----
  80. name node.role disk.used_percent disk.used disk.avail disk.total
  81. instance-0000000000 l 85.31 3.4gb 500mb 4gb
  82. instance-0000000001 l 41.31 3.4gb 4.5gb 8gb
  83. ----
  84. // TEST[skip:illustration purposes only]
  85. . Now you can remove the out of disk space instance.
  86. // end::self-managed[]