Logstash Beats Input - multiple multiline codec

Question

I am trying to send multiple types of logs with beats and parse them on the logstash server.

I have beats configured and working properly and almost have logstash working correctly.

Where I am having issues is that other-log.log has entries that start with a different format string.

In an ideal world I would like to be able to apply a different multiline codec depending on the type of entry.

I have tried

if [type] == "server.log" {
    codec => multiline {
        pattern => "^\d{2}:\d{2}:\d{2},\d+"
        negate => true
        what => "previous"
    }
}

However that causes logstash to fail, my guess is that if is not allowed in the input block.

I have also tried to use the multiline filter plugin but it results in

"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: LoadError"

Does anyone have an idea as to how to make this work?

filebeat.yml

- input_type: log
  paths:
    - /application/server.log
  document_type: server.log
- input_type: log
  paths:
    - /tmp/other-log.log
  document_type: other.log

pipeline.conf

input {
    beats {
        host => "0.0.0.0"
        port => "5044"
        codec => multiline {
                pattern => "^\d{2}:\d{2}:\d{2},\d+"
                negate => true
                what => "previous"
        }
    }
}
filter {
    if [type] == "server.log" {
        grok {
            match => { "message" => "(?<date>^\d{2}:\d{2}:\d{2},\d+)\s(?<level>[A-Z]+)\s+\[(?<class>.*?)\]\s+(?<message>(?m).*)" }
            overwrite => ["message"]
            add_tag =>  [ "server.log" ]
        }
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
    elasticsearch { hosts => ["localhost:9200"] }
}

Answer 1

I moved the multiline to filebeat.yml and that solved my issues :)

Answer 2

Configuration for moving multiline to filebeat.yaml is here. I captured multiline logs using the following configuration. This is my filebeat.yaml configuration:

# ============================== Filebeat inputs ===============================
 filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /var/log/nginx/access.log
        - /var/log/nginx/error.log
      multiline.pattern: '^[[:space:]]' // these
      multiline.negate: false           // three
      multiline.match: after            // lines are important for capturing multiline logs
------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
  index: "my-index-name"

You can also take reference from this blog