Elasticsearch How to Secure an Elasticsearch Cluster: TLS, SSL & CERTUTIL Certificates

Opster Team

Last updated: Mar 8, 2023

| 5 min read

Opster Team

Last updated: Mar 8, 2023

| 5 min read

Elasticsearch Cluster Security

Securing an Elasticsearch cluster and creating TLS certificates will almost inevitably require some downtime on your cluster, since the cluster will not be available until all nodes have their certificates installed.   

What is TLS and why do we need it?

TLS (Transport Layer Security) certificates are necessary to provide encryption keys to enable the nodes to encrypt their communications. Furthermore, each certificate must be created with the hostname and IP address of each node to enable client applications to use hostname verification, this in order to avoid “man in the middle” attacks. Lastly, to ensure that the certificate presented is genuine, each certificate must be signed by a “trusted” certificate authority (CA).  

HTTP and Transport Certificates

Inside the Elasticsearch configuration there are two sets of SSL configurations: HTTP and Transport. HTTP refers to the communication between clients and the Elasticsearch cluster, while Transport refers to the communication between different nodes within the cluster.  Occasionally it may be necessary to use different certificates for both sets of configurations, however the following guide assumes, for simplicity’s sake, that we will just use the same set of certificates for both HTTP and Transport layers.

Keystore or PEM?

It is possible to use both Keystore and PEM formats to generate and store certificates. In this guide we are using PEM format. If you want to use keystore format, then you would need to use a different command to generate the certificates and use the “keystore” parameters instead of “certificate” parameters in your Elasticsearch and kibana.yml files. 

Self-signed or Third Party Certificate Authority?

The certificate authority may be a Third Party Certificate company or institution, or you may create your own certificate authority certificates. This latter case is known as “self-signed” certificates. The process described below will create such “self-signed certificates”. 

Using self-signed certificates makes the installation process easier, but has the disadvantage that you will need to add your CA certificate to all of your client applications so that they know that they can trust the certificates presented.  

If you use Third Party Trusted CA certificates,  then you will need to generate certificate signing requests and use them to obtain certificates from your certificate authority, which is a more complex process, but the certificates will automatically be trusted by client applications and you will not have to install CA certificates on them.

Generation of SSL Certificates

A certificate generation tool is available under the installation directory of each node. The executable is normally here:  /usr/share/elasticsearch/bin/elasticsearch-certutil. However, all of the following will be run from the installation directory itself.

There are many ways of using the certificate generation tool, depending upon your use case.  The following approach is file-based using the “certs” mode. Alternatively you can use “HTTP” mode for a more interactive approach where the application will ask questions to guide you through the generation of your certificates. There is also a “CSR” mode which will enable you to generate certificate signing requests if your organization requires you to use a specific third party certificate authority.

cd /usr/share/elasticsearch

Create a file called “instances.yml” where you define the hostnames and IP addresses for each of the nodes in the Elasticsearch cluster.

instances:
  - name: data01
    dns:
      - data01 
      - localhost
    ip:
      - 127.0.0.1
      - 172.0.0.1

  - name: data02
    dns:
      - data02
      - localhost
    ip:
      - 127.0.0.1
      - 172.0.0.2

  - name: kibana
    dns:
      - kibanahost
      - localhost
    ip:
      - 127.0.0.1
      - 172.0.0.3
      - 54.10.01.02

Take care with both DNS and IP addresses used on the certificates – these should perfectly match the hostname and IP addresses that your nodes and clients will use to interface with the cluster. If you want to be able to use localhost or 127.0.0.1, these must also be added to the certificate (as shown).  

Bear in mind that external IP addresses may be dynamic, so it is generally better to use internal IP addresses and avoid exposing your cluster outside the network. If you need to do so, it is usually recommended to use a proxy or load balancer rather than expose the IP addresses directly.

In the example above:

• We have 2 nodes
• The hostname is localhost, data01 and data02
• We include the local network addresses (172.x.x.x)
• We have included the localhost DNS and 127.0.0.1

We have also included certificates for Kibana. If you want to use Kibana from outside the internal network, you may want to take the time to add a fixed IP or hostname accessible from outside your network to this IP and add the names to your SSL certificates.

Add the instances.yml file to one of the Elasticsearch nodes inside the /usr/share/elasticsearch/config/certificates/  directory (you may need to create the directory).

The following command will generate one CA certificate (certificate authority) and node certificates for each node that you have defined in “instances.yml”.

bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip

You can now unzip the certificates bundle:

unzip /certs/bundle.zip

Here you will find:

• ca.crt – This is the file that you will need to install on all nodes and client apps.

• ca.key – You will need this if you need to generate further certificates in the future for new nodes.

You will also find copies of each of the nodes’ certificates and their keys.

Take note of the expiry dates on your certificates. By default they will expire in 3 years, and you will need to replace them before that point. You don’t want your cluster to go down just because your certificates have expired.

Installation of SSL Certificates on the nodes

Now it is necessary to copy to each node:

The CA certificate  ca.crt  (NOT the key ca.key)

The certificate and key corresponding to each node eg. data01.crt data01.key 

On each node, copy these certificates to: 

/etc/elasticsearch/certs

Then change ownership of the certificates to Elasticsearch:

chown -R elasticsearch:elasticsearch /etc/elasticsearch/certs

Now add the paths to the certificates to elasticsearch.yml:

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/data01.key 
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/data01.crt 
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.http.ssl.verification_mode: full
xpack.security.http.ssl.key: /etc/elasticsearch/certs/data01.key 
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/data01.crt 
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]

Restart the node, and check the logs for any errors. Typically errors are very verbose, but may be caused by incorrect paths or permissions issues preventing the files being read by the Elasticsearch user.

Initial generation of passwords

You can automatically generate passwords for the default Elasticsearch users by running the following command:

From /usr/share/elasticsearch

bin/elasticsearch-setup-passwords auto

This command will generate an initial set of passwords for the default users in Elasticsearch.  Take note of these passwords because it will not be possible to obtain them again.

Kibana

In order to use Kibana, you will need to add the Elasticsearch CA certificate along with elasticsearch user and password to your kibana.yml file. You should also add the Kibana certificates generated above for SSL encryption.

Install the following certificates on your Kibana server:

• The CA certificate  ca.crt  (NOT the key ca.key)
• The certificate and key corresponding to kibana eg. kibana.crt kibana.key 

Copy these certificates to /etc/kibana/certs. 

Then change ownership of the certificates to Elasticsearch:

chown -R kibana:kibana /etc/kibana/certs

Edit kibana.yml:

elasticsearch.hosts: ["https://<your_elasticsearch_host>.com:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibanapassword"
elasticsearch.ssl.certificateAuthorities: ["/path/to/elasticsearch-ca.pem"]

server.ssl.certificate: "/etc/kibana/certs/kibana.crt"
server.ssl.key: "/etc/kibana/certs/kibana.key"
server.ssl.enabled: true

The Elasticsearch settings are required to allow Kibana to communicate with the Elasticsearch cluster.

The Kibana settings are required to make Kibana serve web pages using https.

Restart Kibana.

Other client applications

All other client applications will now require installation of a copy of the CA certificate (not key) along with the elasticsearch user and password so that the application knows to trust the Elasticsearch certificates.

Eg. CURL 

curl -XGET https://127.0.0.1:9200 -u elastic:mypassword --cacert /etc/elasticsearch/certs/ca.crt

Advanced topics

Use of CERTUTIL certificate creation tool

There are different ways to create certificates or certificate signing requests, protect them with passwords, change your certificate formats to keystore type and more. See this guide for further reading.

Creation of API Keys

If you want to avoid using basic authentication for Elasticsearch clients you can use API keys.

Creation of users and roles

Once you have set up security, you will need to create users and roles to give your applications the appropriate permissions. See this guide for further reading.

Storing passwords in the keystore

If you decide to generate the node private keys or the Kibana private key and protect them with a password, you can learn more about storing those passwords in the secure keystore by reading this guide.