Aggregating all fields for an object in a search query, without manually specifying the fields

Question

I have an index products which has an internal object attributes which looks like:

{
  properties: {
    id: {...},
    name: {...},
    colors: {...},
   // remaining fields
}
}

I'm trying to produce a search query with this form and I need to figure out how to write the aggs object.

{ query: {...}, aggs: {...} }

I can write this out manually for two fields to get the desired result, however the object contains 50+ fields so I need it to be able to handle it automatically

  "aggs": {
    "attributes.color_group.id": {
      "terms": {
        "field": "attributes.color_group.id.keyword"
      }
    },
    "attributes.product_type.id": {
      "terms": {
        "field": "attributes.product_type.id.keyword"
      }
    }
  }

Gives me the result:

  "aggregations" : {
    "attributes.product_type.id" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 34,
      "buckets" : [
        {
          "key" : "374",
          "doc_count" : 203
        },
        {
          "key" : "439",
          "doc_count" : 79
        },
        {
          "key" : "460",
          "doc_count" : 28
        },
        {
          "key" : "451",
          "doc_count" : 24
        },
        {
          "key" : "558",
          "doc_count" : 18
        },
        {
          "key" : "500",
          "doc_count" : 10
        },
        {
          "key" : "1559",
          "doc_count" : 9
        },
        {
          "key" : "1560",
          "doc_count" : 9
        },
        {
          "key" : "455",
          "doc_count" : 7
        },
        {
          "key" : "501",
          "doc_count" : 6
        }
      ]
    },
    "attributes.color_group.id" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 35,
      "buckets" : [
        {
          "key" : "12",
          "doc_count" : 98
        },
        {
          "key" : "54",
          "doc_count" : 48
        },
        {
          "key" : "118",
          "doc_count" : 43
        },
        {
          "key" : "110",
          "doc_count" : 41
        },
        {
          "key" : "111",
          "doc_count" : 35
        },
        {
          "key" : "71",
          "doc_count" : 35
        },
        {
          "key" : "119",
          "doc_count" : 24
        },
        {
          "key" : "62",
          "doc_count" : 21
        },
        {
          "key" : "115",
          "doc_count" : 20
        },
        {
          "key" : "113",
          "doc_count" : 15
        }
      ]
    }
  }

Which is exactly what I want. After some research I found that you can use query_string which would allow me to find everything starting with attributes., however it does not seem to work inside aggregations.

Answer 1

As I know what you are asking is not possible with inbuild functionality of elasticsearch. But there are some work around you can do like:

Use Search Template:

Below is Example for Search Template, where you will provide list of field as array and it will create the aggregation for all provided fields. you can store search template using Script API and use id of search template while calling search request.

POST dyagg/_search/template
{
  "source": """{
    "query": {
      "match_all": {}
    },
    "aggs": {
      {{#filter}}
      "{{.}}": {
        "terms": {
          "field": "{{.}}",
          "size": 10
        }
      }, {{/filter}}
      "name": {
        "terms": {
          "field": "name",
          "size": 10
        }
      }
    }
  }""",
  "params": {
    "filter":["lastname","firstname","city","country"]
  }
}

Response:

"aggregations" : {
    "country" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "India",
          "doc_count" : 4
        }
      ]
    },
    "firstname" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "Rajan",
          "doc_count" : 1
        },
        {
          "key" : "Sagar",
          "doc_count" : 1
        },
        {
          "key" : "Sajan",
          "doc_count" : 1
        },
        {
          "key" : "Sunny",
          "doc_count" : 1
        }
      ]
    },
    "city" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "Mumbai",
          "doc_count" : 2
        },
        {
          "key" : "Pune",
          "doc_count" : 2
        }
      ]
    },
    "name" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "Rajan Desai",
          "doc_count" : 1
        },
        {
          "key" : "Sagar Patel",
          "doc_count" : 1
        },
        {
          "key" : "Sajan Patel",
          "doc_count" : 1
        },
        {
          "key" : "Sunny Desai",
          "doc_count" : 1
        }
      ]
    },
    "lastname" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "Desai",
          "doc_count" : 2
        },
        {
          "key" : "Patel",
          "doc_count" : 2
        }
      ]
    }
  }

Second way is using programming. Please check this stackoverflow answer where they have mentioned about how to do in PHP so same you can follow for other language.

NOTE: If you noticed search template, I have added one static aggregation for name field and reason for adding is to avoid extra comma in the end of for loop complete. If you not add then you will get json_parse_exception.