Elasticsearch How to Upgrade Elasticsearch Versions

Opster Team

Last updated: Dec 27, 2022

| 6 min read

In addition to reading this guide, we recommend you run the Elasticsearch Health Check-Up. It will detect issues and improve your Elasticsearch performance by analyzing your shard sizes, threadpools, memory, snapshots, disk watermarks and more.

The Elasticsearch Check-Up is free and requires no installation.

This guide will cover how to upgrade versions in Elasticsearch. If you’d like to perform this automatically, you can use Opster’s Management Console. The OMC makes it easy to orchestrate and manage your search deployment in any environment.

Using the OMC you can deploy multiple clusters, upgrade versions, configure node roles, scale cluster resources, manage certificates and more – all from a single interface, for free. Get access.

How to Upgrade Version with the OMC

 

Quick links

• Introduction
  • Elastic product EOL (End Of Life) dates
  • Compatibility matrix
  • Master Nodes or Non-Master Nodes? Which should be upgraded first?
  • Preparing to upgrade Elasticsearch nodes
    • Check the deprecation log
    • Upgrade Assistant in Kibana
    • Review the breaking changes
    • Check the Elasticsearch plugins compatibility
    • Set up a test environment
    • Take a backup and snapshots from your data
• Full Cluster Restart Upgrade (Offline Upgrade)
• Rolling Restart Upgrade (Online upgrade)
  • How to upgrade nodes in a rolling upgrade

Introduction 

Updating distributed systems, including Elasticsearch, can be a complex process due to the large volume of data, the number of nodes involved and the different configurations you may have in your cluster. In this article, we explain how to minimize risks when upgrading your Elasticsearch cluster.

There are two approaches to upgrading Elasticsearch service versions: full cluster restarts and rolling restarts. In the rest of the article we will discuss each in turn.

Always keep in mind that any changes to your system may result in data loss if you do not follow the instructions correctly. Always test and plan your upgrade carefully and take a backup of your data before performing any upgrades.

We’ll start with a review of important considerations, but you can jump to the instructions for full cluster restart upgrades and rolling restart upgrades anytime. 

Elastic product EOL (End Of Life) dates 

Each Elasticsearch version has its own end of life, and Elasticsearch explicitly mentions that each major version (release) of products is supported for 18 months after the release.

Read more

Note: In general it is recommended to always upgrade to the highest version possible, subject to compatibility / availability of any client libraries or plugins you are using. 

Compatibility matrix

Note: Elasticsearch version 7.17 is the last version that works with JDK 1.8 . 

Master nodes or non-master Nodes? Which should be upgraded first? 

Before starting an upgrade, you need to divide your cluster into: 

• Master-eligible nodes
• Master-ineligible nodes

The reason for this categorization is that higher version Elasticsearch master-ineligible nodes can always  join a  cluster with a lower version master node, but lower version master-ineligible nodes cannot join a higher version cluster.

This means that if you upgrade your master-eligible nodes before master-ineligible nodes, there is the possibility that some of the master-ineligible nodes leave the cluster and cannot rejoin. For this reason it is necessary to first upgrade all master-ineligible nodes. 

Preparing to upgrade Elasticsearch nodes

Elasticsearch nodes cannot be downgraded after upgrading. Before starting the upgrade process you should review the following: 

Check the deprecation log

You should read and resolve any issues highlighted in the deprecation log. These logs are usually located in: 

/var/log/elasticsearch/Your-Cluster-Name_deprecation.log

Upgrade Assistant in Kibana 

In the last minor version of each major version, eg. 6.8.23 or 7.17.5, you will find an upgrade assistant to help you ensure that your cluster settings and configurations are compatible with the next version. Use the menu in Kibana to navigate to the assistant. You must clear all issues using the assistant before attempting to upgrade to the next major version.

Review the breaking changes

With each new version, breaking changes documentation is published to make you aware of any functionality which may change or disappear. You should always check this to ensure that none of these settings, configurations or mappings are being used in your setup. The main things that could be affected are:

• Node configuration (elasticsearch.yml)
• Index mappings and templates
• Cluster settings

Check the Elasticsearch plugins compatibility

If you are using any Elasticsearch plugins, you should check the availability and compatibility of the plugin with the new version.

Set up a test environment

You should test the upgrade process in a test or staging environment before upgrading your production cluster in order to check and resolve all issues before upgrading production.

Take a backup and snapshots from your data

Remember that it is not possible to downgrade an Elasticsearch node, and the only way to practically reverse a failed upgrade will be to create a new cluster with the old version and recover your data from snapshots. Therefore it is essential to take snapshots of all Elasticsearch indices  before starting the upgrade process.

Full cluster restart upgrade (offline upgrade)

A full cluster restart upgrade consists of shutting down all the Elasticsearch nodes at once, upgrading all of them and then restarting each node. This type of upgrade will inevitably require your Elasticsearch cluster to be down for the duration of the upgrading process.  

Offline upgrades are usually easier than online upgrades since you do not have to manage a cluster with different versions of nodes at the same time. 

The steps are:

1. Disable shard allocation 
2. Stop all Elasticsearch nodes and upgrade them 
3. Upgrade any plugins
4. Start the Elasticsearch cluster
5. Re-enable shard allocation 
6. Upgrade client libraries to new version
7. Restart master eligible nodes
8. Restart non-master eligible nodes

Note that in a full cluster restart the master nodes must be started before the non-master nodes. This is to ensure that the master nodes can form the cluster in order to enable other nodes to join, and is contrary to a rolling upstart where the non master nodes should be upgraded before the master nodes.

Rolling restart upgrade (online upgrade)

A rolling restart upgrade will upgrade a cluster without any downtime. In this case, each node is upgraded and restarted in turn without ever stopping the entire Elasticsearch cluster.

It is NOT possible to carry out rolling restart upgrades involving a change in MAJOR versions, with the exception of the following:

• Upgrading Elasticsearch version 5.6.16 to version 6.x.x
• Upgrading Elasticsearch version 6.8.23 to version 7.x.x
• Upgrading Elasticsearch version 7.17.5 to version 8.x.x

For this reason, if you want to carry out a rolling restart upgrade between major versions, you should ALWAYS use the latest minor version as a stepping stone to upgrade to the next major version. For example, if you are using Elasticsearch 5.x.x you can upgrade to  5.6.16 and then to 6.8.23.

How to upgrade nodes in a rolling upgrade

To disable shard allocation, run:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

To re-enable shard allocation, run:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

Get cluster status and see how many nodes are in the cluster using:

GET _cluster/health