Elasticsearch -Creating index and putting data of csv file

Question

I have a CSV file

I want to create separate index for each category of animal eg. all dogs should be in 1 index named 'dogs', cats should be in another index named 'cats' etc. While parsing the data if there is already an index of the animal then I add the entry in that index, else I want to create a separate index and add that entry to that index. At the end I should have 3 index with the following data: Dog - Jerry, Thera Cat - Lily, Melo Rabbit - Bunny

I want to know how can this be done using python. I am trying but not able to parse the csv and not able to create new index for each category.

Answer 1

According to my solution, you can just parse the CSV file in python and use ingest pipeline while indexing the documents.

To read the CSV file, please check the following script and insert the data to Elasticsearch with the pipeline parameter :

from elasticsearch import Elasticsearch, helpers
from csv import reader

es = Elasticsearch(host = "localhost", port = 9200)

with open('stockerbot-export.csv') as f:
    reader = csv.DictReader(f)
    helpers.bulk(es, reader, index='someindexname', pipeline='index-name-change-pipeline')

Before executing the command, you need to create a pipeline for our logic with the following request on kibana of Elasticsearch :

PUT _ingest/pipeline/index-name-change-pipeline
{
  "processors": [
    {
      "set": {
        "field": "_index",
        "value": "someprefix-{{{animal}}}"
      }
    }
  ]
}

You can use some request client instead of kibana also. I could not test python code end to end, but I see the result with the following request on Kibana screen :

POST test/_doc?pipeline=index-name-change-pipeline
{
  "id": 1,
  "animal": "dog",
  "name": "asb"
}

As you see there, the index name is test while indexing the data. But after the indexing is successful, the response will be this :

{
  "_index" : "someprefix-dog",
  "_id" : "5HEJyIAB4KMK-wcc6Rtb",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

So, our pipeline is working correctly as you see _index metadata.