How to correctly handle and parse multiline logs from rsyslog

Question

Overview of the situation

Let's assume I have a file with logs from different services. This file contains many single lines. Let's suppose I have lines like this:

• <service 1> msg: "stack trace 1",
• <service 2> msg: "stack trace 2",
• <service 1> msg: "continuation of stack trace 1",
• <service 3> msg: "beggining of stack trace 3"
• <service 2> msg: "continuation of stack trace 2"

How I want it to work

I want to have output file, or a module, where those multilines will be correctly parsed into single line like this:

• <service 1> msg: "stack trace 1 continuation of stack trace 1",
• <service 2> msg: "stack trace 2 continuation of stack trace 2",
• <service 3> msg: "beggining of stack trace 3".
  So basically I want the rsyslog to correctly handle multiline messages that are mixed with another messages during logging.

How does it work

Currently, by using the imfile module, I can define a regex for a file to watch and then, catch some patterns like for example, java stack trace pattern or python stack trace pattern. The thing is, I can't define more input(type="imfile" ruleset="infiles" tag="some-tag" file="some-path-to-file" startmsg.regex="some-regex") regex (this line is included in rsyslog.conf) on a single file to catch simultaneously for example java stack traces and docker stack traces. So it's basically a no solution for me.

Answer 1

Rsyslog does not handle the case where a single log entry is split into multiple pieces and intermingled with pieces of other logs. If you have that situation, it's unlikely that the logs will be nicely split only at the line break, it's more likely that they get split based on byte count (on linux by default writes are buffered and written in 4k chunks for example)

If you are mixing different styles of multi-line logs in the same file, and can't define a regex that matches the beginning of every line (for example, are all your multi-line log continuing lines indented? if so, your regex could be "^[0-9a-zA-Z]" so any line that's not indented is the beginning of a log), then rsyslog does not support it.

Why can't you write your docker dumps and your java dumps to different files? having multiple programs write to the same file is just begging for OS buffering to intermingle the logs in undetectable ways.