'How to extract a json and write them to a specific file using fluentd?

I have an application that writes logs to the container log files. I want to use fluentd to filter these logs based on a string and write them to a file.

A single line from the container logs look like:

{
  "stream": "stdout",
  "time": "2022-01-21T07:54:18.14263612Z",
  "log": "{\"MATCH_TEXT\":{\"id\":\"001\",\"timestamp\":1642751532000,\"Details\":{\"key1\":{\"key2\":\"val2\",\"key-3\":\"val-3\",\"status\":\"done\"}}}}",
  "docker": {
    "container_id": "container_id"
  },
  "kubernetes": {
    "container_name": "test-container",
    "namespace_name": "test-ns",
    "pod_name": "test-pod",
    "container_image": "test-container_image:tag",
    "pod_id": "POD_ID",
    "pod_ip": "1.1.1.1",
    "host": "host"
  }
}

I am trying to find all logs containing the text MATCH_TEXT and log them to a file without the key MATCH_TEXT.

So my expected output becomes:

{\"id\":\"001\",\"timestamp\":1642751532000,\"Details\":{\"key1\":{\"key2\":\"val2\",\"key-3\":\"val-3\",\"status\":\"done\"}}}

So far I was able to add filters to bring it down to this.

# this filter matches logs containing the text MATCH_TEXT and tags them to new tag required_logs
 <match logs>
  @type copy
  <store>
    @type rewrite_tag_filter
    <rule>
      key $.log
      pattern MATCH_TEXT
      tag required_events
    </rule>
  </store>
</match> 

#this filter removes all the fields expect log field.
<filter required_events>
  @type record_transformer
  renew_record true
  keep_keys log
</filter>

#this filter removes the log key from the message
<filter required_events>
  @type parser
  key_name $.log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

# now were left with message:
# {"MATCH_TEXT":{"id":"001","timestamp":1642751532000,"Details":{"key1":{"key2":"val2","key-3":"val-3","status":"done"}}}}

This is where I start getting problems. I tried to further filter the message using filter:

<filter required_events>
  @type parser
  key_name $.MATCH_TEXT
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

But this makes fluentd give me an error:

2022-01-24 08:35:20 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '{\"MATCH_TEXT\":{\"id\":\"001\",\"timestamp\":1642751532000,\"Details\":{\"key1\":{\"key2\":\"val2\",\"key-3\":\"val-3\",\"status\":\"done\"}}}}"'" location=nil tag="required_events" time=2022-01-24 08:35:20.955622000 +0000 record={"MATCH_TEXT"=>{"id":"001","timestamp":1642751532000,"Details":{"key1":{"key2":"val2","key-3":"val-3","status":"done"}}}}

I am new to fluentd. Please help me figure out what I'm doing wrong here. I would like to do it without installing any new plugin if possible. Thanks!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source