Before you begin reading this guide, we recommend you run Elasticsearch Error Check-Up which can resolve issues that cause many errors.
Advanced users might want to skip right to the common problems section in each concept or try running the Check-Up which analyses ES to pinpoint the cause of many errors and provides suitable actionable recommendations how to resolve them (free tool that requires no installation).
“
Quick Summary
The cause of this error is low disk space on a data node, and as a preventive measure, Elasticsearch throws this log message and takes some preventive measures explained further.
Explanation
Elasticsearch considers the available disk space before deciding whether to allocate new shards, relocate shards away or put all indices on read mode based on a different threshold of this error. The reason is Elasticsearch indices consists of different shards which are persisted on data nodes and low disk space can cause issues.
Relevant settings related to log:
cluster.routing.allocation.disk.watermark – have three thresholds of low, high, and flood_stage and can be changed dynamically, accepting absolute values as well as percentage values.
Permanent fixes
a). Delete unused indices.
b) Merge segments to reduce the size of the shard on the affected node, more info on opster’s Elasticsearch expert’s STOF answer
c) Attach external disk or increase the disk used by the data node.
Temp hack/fixes
a) Changed settings values to a higher threshold by dynamically update settings using update cluster API:
PUT _cluster/settings
{
""transient"": {
""cluster.routing.allocation.b) Disable disk check by hitting below cluster update API
{
""transient"": {
""cluster.routing.allocation.C) Even After all these fixes, Elasticsearch won’t bring indices in write mode for that this API needs to be hit.
PUT _all/_settings
{
""index.blocks.read_only_”
Overview
There are various “watermark” thresholds on your Elasticsearch cluster. As the disk fills up on a node, the first threshold to be crossed will be the “low disk watermark”. Once this threshold is crossed, the Elasticsearch cluster will stop allocating shards to that node. This means that your cluster may become yellow.
The second threshold will then be the “high disk watermark threshold”. Finally, the “disk flood stage” will be reached. Once this threshold is passed, the cluster will then block writing to ALL indices that have one shard (primary or replica) on the node which has passed the watermark. Reads (searches) will still be possible.
Watch 2 min video for quick troubleshooting steps to resolve low disk watermark in Elasticsearch:
How to resolve it
Passing this threshold is a warning and you should not delay in taking action before the higher thresholds are reached. Here are possible actions you can take to resolve the issue:
- Delete old indices
- Remove documents from existing indices
- Increase disk space on the node
- Add new nodes to the cluster
You can see the settings you have applied with this command:
GET _cluster/settings
If they are not appropriate, you can modify them using a command such as below:
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "85%",
"cluster.routing.allocation.disk.watermark.high": "90%",
"cluster.routing.allocation.disk.watermark.flood_stage": "95%",
"cluster.info.update.interval": "1m"
}
}How to avoid it
There are various mechanisms to automatically delete stale data.
How to automatically delete stale data:
- Apply ISM (Index State management)
Using ISM you can get OpenSearch to automatically delete an index when your current index size reaches a given age.
- Use date-based indices
If your application uses date-based indices, then it is easy to delete old indices using a script.
- Use snapshots to store data offline
It may be appropriate to store snapshotted data offline and restore it in the event that the archived data needs to be reviewed or studied.
- Automate / simplify process to add new data nodes
Use automation tools such as terraform to automate the addition of new nodes to the cluster. If this is not possible, at the very least ensure you have a clearly documented process to create new nodes, add TLS certificates and configuration and bring them into the OpenSearch cluster in a short and predictable time frame.
Log Context
Log “Low disk watermark [{}] exceeded on {}; replicas will not be assigned to this node” classname is DiskThresholdMonitor.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
diskThresholdSettings.getFreeBytesThresholdFloodStage(); usage);
} else if (usage.getFreeBytes()
