'How to extract a json and write them to a specific file using fluentd?
I have an application that writes logs to the container log files. I want to use fluentd to filter these logs based on a string and write them to a file.
A single line from the container logs look like:
{
"stream": "stdout",
"time": "2022-01-21T07:54:18.14263612Z",
"log": "{\"MATCH_TEXT\":{\"id\":\"001\",\"timestamp\":1642751532000,\"Details\":{\"key1\":{\"key2\":\"val2\",\"key-3\":\"val-3\",\"status\":\"done\"}}}}",
"docker": {
"container_id": "container_id"
},
"kubernetes": {
"container_name": "test-container",
"namespace_name": "test-ns",
"pod_name": "test-pod",
"container_image": "test-container_image:tag",
"pod_id": "POD_ID",
"pod_ip": "1.1.1.1",
"host": "host"
}
}
I am trying to find all logs containing the text MATCH_TEXT and log them to a file without the key MATCH_TEXT.
So my expected output becomes:
{\"id\":\"001\",\"timestamp\":1642751532000,\"Details\":{\"key1\":{\"key2\":\"val2\",\"key-3\":\"val-3\",\"status\":\"done\"}}}
So far I was able to add filters to bring it down to this.
# this filter matches logs containing the text MATCH_TEXT and tags them to new tag required_logs
<match logs>
@type copy
<store>
@type rewrite_tag_filter
<rule>
key $.log
pattern MATCH_TEXT
tag required_events
</rule>
</store>
</match>
#this filter removes all the fields expect log field.
<filter required_events>
@type record_transformer
renew_record true
keep_keys log
</filter>
#this filter removes the log key from the message
<filter required_events>
@type parser
key_name $.log
reserve_data true
remove_key_name_field true
<parse>
@type json
</parse>
</filter>
# now were left with message:
# {"MATCH_TEXT":{"id":"001","timestamp":1642751532000,"Details":{"key1":{"key2":"val2","key-3":"val-3","status":"done"}}}}
This is where I start getting problems. I tried to further filter the message using filter:
<filter required_events>
@type parser
key_name $.MATCH_TEXT
reserve_data true
remove_key_name_field true
<parse>
@type json
</parse>
</filter>
But this makes fluentd give me an error:
2022-01-24 08:35:20 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '{\"MATCH_TEXT\":{\"id\":\"001\",\"timestamp\":1642751532000,\"Details\":{\"key1\":{\"key2\":\"val2\",\"key-3\":\"val-3\",\"status\":\"done\"}}}}"'" location=nil tag="required_events" time=2022-01-24 08:35:20.955622000 +0000 record={"MATCH_TEXT"=>{"id":"001","timestamp":1642751532000,"Details":{"key1":{"key2":"val2","key-3":"val-3","status":"done"}}}}
I am new to fluentd. Please help me figure out what I'm doing wrong here. I would like to do it without installing any new plugin if possible. Thanks!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
