Elasticsearch Search is Slow in nodesNames

Opster Team

Last updated: Feb 13, 2022

| 2 min read

If you’re suffering from search related issues or poor search performance, you should run Opster’s free Search Log Analyzer to optimize your searches.

With Opster’s Analyzer, you can easily locate slow searches and understand what led to them adding additional load to your system. You’ll receive customized recommendations for how to handle rejected searches and improve your search performance. The tool is free and takes just 2 minutes to run.

What it means

Slow searches might become a bottleneck and may cause a waiting queue to build.

There are a number of possible causes for slow search on particular nodes:

• Your application is not load balancing properly across all of the data nodes.
• Search and/or indexing operations are concentrated on specific nodes because of the way shards are allocated.
• The queries running on certain indices (concentrated on the nodes in question) are slow and need optimization.
• There are other processes (such as merges or snapshots) running on the nodes in question which are using resources on the slow nodes.

How to resolve

Look at the monitoring data to determine which indices are receiving most search / indexing operations. Often you will see that certain nodes have a high CPU usage, and that there are certain “hot” index shards that exist on those nodes. For example, imagine the following scenario:

• Index A has a high search rate,  
• Index A has  shards only on node 3 and 4
• Nodes 3 and 4 have higher cpu usage than the other data nodes.

index A is a strong candidate to be one of the  causes of the problem.

Check to see whether you have any of the issues described in loaded data nodes.

Activate slow logging

If you have slow logging activated, look to see which logs are taking the most time. In particular look for aggregations with large “size” values or nested queries which are the most common culprits of slow queries.  

To activate slow logging for a given index you can run the command below. For full instructions, see this guide.

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
   "index.search.slowlog.level": "info"
}

To save resources it is best to leave slow logging deactivated when you have finished using the value of -1.

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "-1",
  "index.search.slowlog.threshold.query.info": "-1",
  "index.search.slowlog.level": "info"
}

Once you have activated slow logging you can see the queries which are taking a long time.  Look for the issues mentioned in the article 10 Important Tips to Improve Search in Elasticsearch. 

Further reading on slow searches

Elasticsearch Slow Log Search Queries – A Complete Guide.

Elasticsearch Slow Indexing in Nodes.