What does "Limit of total fields [1000] in index [] has been exceeded" means in Elasticsearch

Question

I have Elasticsearch index created which has approx 350 fields (Including nested fields) , I have defined mapping only for few of them. While calling _update API I am getting below exception with 400 Bad request.

Limit of total fields [1000] in index [test_index] has been exceeded
I want to understand the exact reason for this exception since my number of fields and fields for which mapping is defined both are less than 1000.
Note: I have nested fields and dynamic mapping enabled.

Regards,
Sandeep

Answer 1

You can fix the issue by increasing the value of index.mapping.total_fields.limit (default 1000)

index.mapping.total_fields.limit

The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The default value is 1000.

To increase total fields limit to 2000, try this

PUT test_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

The reason to limit the number of fields is :

Defining too many fields in an index is a condition that can lead to a mapping explosion, which can cause out of memory errors and difficult situations to recover from. This is quite common with dynamic mappings. Every time a document contains new fields, those will end up in the index’s mappings

Related Resources :
1. Get the number of fields on an index
2. Settings to prevent mappings explosion

Answer 2

Another approach to handle this is to carefully design the mappings as the OP seems to be doing and to turn off dynamic mapping by setting dynamic = false (or even dynamic = strict)

Note that this approach can be applied to the entire mapping or to other properties/nested objects within, which enables quite a bit good degree of flexibility.

Reference to Dynamic mapping in the docs

Answer 3

I landed here via google. This worked for me (from the shell):

curl -s -XPUT http://localhost:9200/test-index/_settings  -H 'Content-Type: application/json' -d '{"index.mapping.total_fields.limit": 2000}'

Answer 4

Another possible solution, esp. when working in Groovy context: check that your data is a (well-formatted) String instance. In my case, I had been transforming my mapped data to Json using myMappedData as JSON in Groovy. The solution was to use (myMappedData as JSON).toString(). See https://stackoverflow.com/a/71938642/4420271.