elasticsearch/docs/reference/troubleshooting/common-issues/disk-usage-exceeded.asciidoc

[[disk-usage-exceeded]]
=== Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

This error indicates a data node is critically low on disk space and has reached
the <<cluster-routing-flood-stage,flood-stage disk usage watermark>>. To prevent
a full disk, when a node reaches this watermark, {es} blocks writes to any index
with a shard on the node. If the block affects related system indices, {kib} and
other {stack} features may become unavailable.

{es} will automatically remove the write block when the affected node's disk
usage goes below the <<cluster-routing-watermark-high,high disk watermark>>. To
achieve this, {es} automatically moves some of the affected node's shards to
other nodes in the same data tier.

To verify that shards are moving off the affected node, use the <<cat-shards,cat
shards API>>.

[source,console]
----
GET _cat/shards?v=true
----

If shards remain on the node, use the <<cluster-allocation-explain,cluster
allocation explanation API>> to get an explanation for their allocation status.

[source,console]
----
GET _cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": false,
  "current_node": "my-node"
}
----
// TEST[s/^/PUT my-index\n/]
// TEST[s/"primary": false,/"primary": false/]
// TEST[s/"current_node": "my-node"//]

To immediately restore write operations, you can temporarily increase the disk
watermarks and remove the write block.

[source,console]
----
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}

PUT */_settings?expand_wildcards=all
{
  "index.blocks.read_only_allow_delete": null
}
----
// TEST[s/^/PUT my-index\n/]

As a long-term solution, we recommend you add nodes to the affected data tiers
or upgrade existing nodes to increase disk space. To free up additional disk
space, you can delete unneeded indices using the <<indices-delete-index,delete
index API>>.

[source,console]
----
DELETE my-index
----
// TEST[s/^/PUT my-index\n/]

When a long-term solution is in place, reset or reconfigure the disk watermarks.

[source,console]
----
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null
  }
}
----