BULK API : Malformed action/metadata line [3], expected START_OBJECT but found [VALUE_STRING]

Question

Using Elasticsearch 5.5,getting the following error while posting this bulk request, unable to figure out what is wrong with the request.

"type": "illegal_argument_exception",
"reason": "Malformed action/metadata line [3], expected START_OBJECT but found [VALUE_STRING]"

POST http://localhost:9200/access_log_index/access_log/_bulk

{ "index":{ "_id":11} }
{  
   "id":11,
   "tenant_id":682,
   "tenant_name":"kcc",
   "user.user_name":"k0772251",
   "access_date":"20170821",
   "access_time":"02:41:44.123+01:30",
   "operation_type":"launch_service",
   "remote_host":"qlsso.quicklaunchsso.com",
   "user_agent":"Mozilla/5.0 (Linux; Android 7.0; LGLS775 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Mobile Safari/537.36",
   "browser":"",
   "device":"",
   "application.application_id":1846,
   "application.application_name":"Desire2Learn",
   "geoip.ip":"192.95.18.163",
   "geoip.country_code":"US",
   "geoip.country_name":"United States",
   "geoip.region_code":"NJ",
   "geoip.region_name":"New Jersey",
   "geoip.city":"Newark",
   "geoip.zip_code":7102,
   "geoip.time_zone":"America/New_York",
   "geoip.latitude":40.7355,
   "geoip.longitude":-74.1741,
   "geoip.metro_code":501
}
{ "index":{"_id":12} }
{  
   "id":12,
   "tenant_id":682,
   "tenant_name":"kcc",
   "user.user_name":"k0772251",
   "access_date":"20170821",
   "access_time":"02:50:44.123+01:30",
   "operation_type":"launch_service",
   "remote_host":"qlsso.quicklaunchsso.com",
   "user_agent":"Mozilla/5.0 (Linux; Android 7.0; LGLS775 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Mobile Safari/537.36",
   "browser":"",
   "device":"",
   "application.application_id":2341,
   "application.application_name":"Gmail",
   "geoip.ip":"192.95.18.163",
   "geoip.country_code":"US",
   "geoip.country_name":"United States",
   "geoip.region_code":"NJ",
   "geoip.region_name":"New Jersey",
   "geoip.city":"Newark",
   "geoip.zip_code":7102,
   "geoip.time_zone":"America/New_York",
   "geoip.latitude":40.7355,
   "geoip.longitude":-74.1741,
   "geoip.metro_code":501
}

Answer 1

Your resource objects have to be specified on a single line like this

post /test322/type/_bulk
{ "index": {} }
{ "name": "Test1", "data": "This is my test data" }
{ "index": {} }
{ "name": "Test2", "data": "This is my test data2" }

Which seems really stupid and unintuitive I know since resources don't have to be on a single line when you create them using PUT or POST for non-bulk operations.

Answer 2

The following lines' format worked for me very well: Action, metadata, resource

Note: Action should be CREATE to add a resource to the dataset and resource should be written inline, NOT new line.

  POST http://localhost:9200/access_log_index/access_log/_bulk
  { "create" : { "_index" : "test", "_type" : "_doc", "_id" : "11" } }
  {  "id":11, "tenant_id":682 , ... }

Answer 3

You need to follow bulk format to successfully execute this. it expects the following JSON structure:

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n

For further reference, see this link https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-bulk.html

Answer 4

I was getting below error when I was manipulating the data set before pushing it to the elastic search(sometimes in serverless environment you can not be certain on order of the events received)

 {
      "type": "illegal_argument_exception",
      "reason": "Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_BOOLEAN]"
 }

This happens when you modify the sequence of the events that elastic search bulk API expects. Like, when you use 'doc_as_upsert': true , the you should have the data in b

{
    "update": {
        "_index": "index_name",
        "_id": "id1234"
    }
}
,
{
    "doc_as_upsert": true,
    "doc": {
        "id": "id1234",
        ...otherFields
    }
}

You can not skip the update object in this case.

I hope this will be helpful for those who manipulate the dataset before it goes to ES