Elasticsearch Shards Too Large – Shard Sizes

Opster Team

Last updated: Feb 19, 2023

| 2 min read

Overview

Watch how to resolve shards too large - 2 min

It is a best practice that Elasticsearch shard size should not go above 50GB for a single shard.  

The limit for shard size is not directly enforced by Elasticsearch. However, if you go above this limit you can find that Elasticsearch is unable to relocate or recover index shards (with the consequence of possible loss of data) or you may reach the lucene hard limit of 2 ³¹ documents per index.

How to resolve this issue

If your shards are too large, then you have 3 options:

1. Delete records from the index

If appropriate for your application, you may consider permanently deleting records from your index (for example old logs or other unnecessary records).

POST /my-index/_delete_by_query
{
     "query": {
        "range" : {
            "@timestamp" : {
          
                 "lte" : now-5d

            }
        }
    }
}

2. Re-index into several small indices

The following would reindex into one index for each event_type.

POST _reindex
{

  "source": {
	"index": "logs-all-events"
  },
  "dest": {
	"index": "logs-2-"
  },
  "script": {
	"lang": "painless",
	"source": "ctx._index = 'logs-2-' + (ctx._source.event_type)"
  }
}

3. Re-index into another single index but increase the number of shards

PUT /my_new_index/_settings
{
    "index" : {
        "number_of_shards" : 2
    }
}

POST _reindex
{
  "source": {
    "index": "my_old_index"
  },
  "dest": {
    "index": "my_new_index"
  }
}

Care points for reindexing

Notice duplicate data during the reindex process

During the reindex process, if your application uses wildcards in the index name (eg. logs-*) then results may be duplicated during the reindex process since you will have two copies of the data until the process completes. This can be avoided through the use of aliases.

Notice modifications and updates during the reindex process

If data is susceptible to updates during the reindex process, then a procedure needs to be identified to capture them and ensure that the new indices reflect all modifications that are made during the transition reindexing period. For more information please refer to Opster tips on reindexing  

How to avoid the issue

There are various mechanisms to optimize shard size.

Apply ILM (index lifecycle management)

Using ILM you can get Elasticsearch to automatically create a new index when your current index size reaches a given maximum size or age.  ILM is suitable for documents which require no updates, but is not a viable option when documents require frequent updating, since to do so you need to know which of the previous index volumes contains the document you want to update.

Application strategy

You can design your application to ensure that shards are created of a sensible size:

Date based approach

myindex-yyyy.MM.dd

or myindex-yyyy.MM

This has the advantage that it is easy to delete old data simply by deleting older indices.

Attribute based approach

myindex-<clientID>

This has the advantage of allowing you to know exactly which index to search/update if you know the client ID. 

The disadvantage is that different clients may produce very different volumes of documents (and hence different shard sizes). This can be mitigated to some extent by adjusting the number of shards per client, but you may create the opposite problem (oversharding) if you have a large number of clients.  

ID range based approach

This may be possible where documents carry an ID coming from another system or application (such as a sql ID, or client ID) which enables us to distribute documents evenly.

eg. myindex-<last_digit_of_ID>

The above would give us 10 indices which will probably be evenly distributed, and also deterministic (we know which index we need to update).