How to Delete Older Elasticsearch indices using Curator

In this tutorial, we’ll explain how to delete older Elasticsearch indices using curator, there was a requirement in one of our project to have an opensource tool which will do log aggregation and monitoring and we got the best tool i.e., ELK stack (Elasticsearch Logstash Kibana) and it is Opensource.

If you are new to this ELK stack then check this ELK stack tutorial

How to Delete Older Elasticsearch indices using Curator

Install pip command:

#For Ubuntu
$ sudo apt-get install python-pip
Or

#For Amazon Linux or CentOS
$ sudo yum install python-pip

Install Curator on Linux using pip command?

$ sudo pip install Elasticsearch-curator

If this command didn’t work then add yum repo in your “/etc/yum.repos.d/” with filename .repo suffix

Yum repo for CentOS 6/RHEL/Amazon Linux:

[curator-5]
name=CentOS/RHEL 6 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/6
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Yum repo for CentOS 7/RHEL/Amazon Linux 2:

[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Verify Curator installed properly:

$ which curator
/usr/local/bin/curator

$ /usr/local/bin/curator --version

A good practice is to install on Elasticsearch machine itself.

If your log size is more and you want to keep old data for 5days as per your requirement then you need to delete old Elasticsearch indices where all logs get stored and these results free up some disk space for newly generated logs. And you will be knowing Logstash creates a new index every day this is the default configuration.

If you don’t want to delete old indices then simply increase your disk space of Elasticsearch cluster.

Steps to delete old data/indices from Elasticsearch

This is very simple to do, follow mention steps:

Step 1: Install Curator and configure it to delete indices x days old with a specific pattern.

$ pip install Elasticsearch-curator

Step 2: Now, Configure Curator

$ vi curator.yml

---

client:

  hosts:

    - 127.0.0.1

  port: 9200

  url_prefix:

  use_ssl: False

  certificate:

  client_cert:

  client_key:

  ssl_no_validate: False

  http_auth:

  timeout: 30

  master_only: False



logging:

  loglevel: INFO

  logfile:

  logformat: default

  blacklist: ['elasticsearch', 'urllib3']

Step 3: Now we need to tell the curator what action needs to be done. Some of the action is mention below:

• Alias
• Allocation
• Close
• Cluster Routing
• Create Index
• Delete Indices
• Delete Snapshots
• Open
• forceMerge
• Replicas
• Restore
• Snapshot

In this tutorial, we will use delete indices action.

Sample Action file delete-indices.yml which will delete indices older than 5Days

$ vi delete-indices.yml

---

actions:

  1:

    action: delete-indices

    description: >-

      Delete indices older than 5 days (based on index name), for logstash-

      prefixed indices. Ignore the error if the filter does not result in an

      actionable list of indices (ignore_empty_list) and If you want to change the retention Days then goto unit_count:<enter no of day>.

    options:

      ignore_empty_list: True

      timeout_override:

      continue_if_exception: False

      disable_action: False

    filters:

    - filtertype: pattern

      kind: prefix

      value: logstash-

      exclude:

    - filtertype: age

      source: name

      direction: older

      timestring: '%Y.%m.%d'

      unit: days

      unit_count: 5

      exclude:

Command to check what pattern are the indices using?

$ curl -XGET 'localhost:9200/_cat/shards?pretty'

Step 4: Now, Goto the location where you have created the “delete-indices.yml” action file and run this action file with mention curator command.

Check which all indices are going to delete with the dry-run option. Dry-run option is used to test action file it will not delete the index

$ curator ./delete_index.yml --config ./curator.yml --dry-run
2018-01-07 17:27:46,075 INFO      Preparing Action ID: 1, "delete_indices"
2018-01-07 17:27:46,080 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018-01-07 17:27:46,538 INFO      DRY-RUN MODE.  No changes will be made.
2018-01-07 17:27:46,538 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2018-01-07 17:27:46,538 INFO      Action ID: 1, "delete_indices" completed.
2018-01-07 17:27:46,538 INFO      Job completed.

Curator command to delete old index

To cleanup old indices run below command:

$ curator ./delete_index.yml --config ./curator.yml

You can also configure this in cronjob using crontab –e.

$ crontab –e

$ 0 12 * * * root curator /path/delete-indices.yml --config /path/curator.yml >> /tmp/curator-index-delete.log 2>&1

If you have sudo permission then use this crontab entry:

$ 0 12 * * * sudo curator /path/delete-indices.yml --config /path/curator.yml >> /tmp/curator-index-delete.log 2>&1

This cronjob run at 12:00 you can change the time as per your requirement.

Tips to change time check Screenshot:

This is the end of the tutorial, we have explained how to delete older Elasticsearch indices using Curator.

Comment us below if you have any queries. If you like these tutorials please share it with your friends.

---

Check Other Articles

Compare ELK vs Splunk

ELK STACK ARCHITECTURE

Getting Started with Unix.

Benefits of the ELK Stack