In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.The Elasticsearch Check-Up is free and requires no installation.
To manage all aspects of your OpenSearch operation, you can use Opster’s Management Console (OMC). The OMC makes it easy to orchestrate and manage OpenSearch in any environment. Using the OMC you can deploy multiple clusters, configure node roles, scale cluster resources, manage certificates and more – all from a single interface, for free. Check it out here.
Quick links
Introduction
Elasticsearch and OpenSearch may have different opinions and priorities, but something that all of us agree on is that older data should be searchable, but also cheaper to store than recent data.
Object storage is a cheap way to store data, but the challenge is retrieving this data to make it searchable in an acceptable amount of time. Luckily, the evidence is proving that object storage services like AWS S3 can be very performant on this task, and both Elasticsearch and OpenSearch are using this service as an ally for the same purpose: reducing costs.
AWS UltraWarm/Cold
AWS UltraWarm is an AWS OpenSearch service feature that provides a cost-effective way to store large amounts of time-based data (immutable) using smart caching technologies to improve performance. UltraWarm instances will not have the SO disk overhead, or replicas disk need (as S3 has high availability features under the hood). This means that every GB you get, you can use for storing primary shards.
AWS OpenSearch service also offers a Cold Storage feature, backed on S3 to make the oldest data available on demand to make it searchable. Cold storage doesn’t have resources like CPU or RAM associated, only S3 disk space, so if you want to search or visualize your data, you must mount it to a Warm instance.
Elasticsearch Searchable snapshots
Elasticsearch leverages snapshots, making those searchable by partially or fully mounting them from an object storage (e.g: S3) to a local cluster.
For cold tier, the indices are saved in the object storage and fully mounted in the local node storage, allowing us to remove the replicas from the local nodes, thereby reducing storage.
For frozen tier, the indices are partially mounted in a local node cache, and the rest of the data resides in the object storage. That local node storage is a Least Recently Used (LRU) cache based on search activity, and contains a subset of the shards that are mapped to that node in the cluster state routing table.
You can learn more about searchable snapshots in our articles:
- Elasticsearch Multi-Tier Architecture | Hot, Warm, Cold & Frozen (opster.com)
- Elastic Pricing Calculator – How to Use the Pricing Calculator (opster.com)
- Elasticsearch Index Lifecycle Policy and Management – Important Tips (opster.com)
Summary comparison table
As you see, the strategies are different: OpenSearch allows you to move indices to special UltraWarm instances, and Elasticsearch mount snapshots from the object storage. The final purpose though is the same: Detach storage from computing, keeping storage expense as low as possible at acceptable response times.
The following table summarizes the differences between AWS UltraWarm/Cold and Elasticsearch Searchable Snapshots:
| Topic | AWS UltraWarm/Cold | Elastic Cold/Frozen tiers |
|---|---|---|
| How does it work? | Backed by Object Storage (S3), indices are moved to UltraWarm instances | Backed by Object Storage (S3, GCP, Azure, etc..) indices are mounted from snapshots |
| How can you manage it automatically? | Index State Management (ISM) | Index Lifecycle Management (ILM) |
| Can it be used on-prem/with a managed service? | Only available in AWS OpenSearch service (Also available for Elasticsearch 7.9+) | Available in Elastic Cloud and Elasticsearch on prem (Enterprise License required) |
| How is it billed in managed services? | Billed by hour + gb/month + data transfer | Billed by hour + data transfer |
| Long term data approach | Cold Storage. Data needs to be moved back to UltraWarm nodes to be searchable | Frozen tier. Partially cached data in local node, rest of the data in Object Repository |
Conclusion
Both Elasticsearch and OpenSearch offer alternatives to save costs by putting the older data into cheaper machines.
OpenSearch leverages the AWS UltraWarm feature, and Elasticsearch leverages the Searchable snapshots, moving some of the cluster’s data to object storage instead of having it in a regular disk.
In version 2.4.0 OpenSearch is releasing its own version of the Searchable Snapshot feature. It is experimental right now, but improvements are expected in the future (upd. Dec 22).
Find & fix Elasticsearch problems
