Logstash Conf | Extracting Filename from Path

Question

I am trying to setup Logstash to feed Elasticsearch. In course, I've created the following conf file that seem to work nicely:

input {
  beats {
    port => 5044
  }
  
  file {
    path => "C:/f1/f2/Logs/f3/LocalHost#base#iway_2022-03-28T10_45_15.log"
  }
}

filter {
  grok {
    match => {
      "message" => [
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .(W.)%{DATA:thread}.%{INT:thread_pool}. %{GREEDYDATA:msgbody}",
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .%{DATA:thread}. %{GREEDYDATA:msgbody}"      
      ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "iway_logs"
    user => "elastic"
    password => "something"
    cacert => "C:\f1\f2\logstash-8.1.3\config\cert\elasticsearch_http_ca.crt"
  }
}

I have been trying to add two new fields but unsuccessful so far. Following is the current version of the conf file after several revises.

input {
  beats {
    port => 5044
  }
  
  file {
    path => "C:/f1/f2/Logs/f3/LocalHost#base#iway_2022-03-28T10_45_15.log"
  }
}

filter {
  grok {
    match => {
      "message" => [
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .(W.)%{DATA:thread}.%{INT:thread_pool}. %{GREEDYDATA:msgbody}",
        ".%{TIMESTAMP_ISO8601:timeStamp}. %{LOGLEVEL:loglevel} .%{DATA:thread}. %{GREEDYDATA:msgbody}"      
      ]
    }
  }
  grok {
        match => { 
            "path" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"
            }
  }
  mutate {
        split => { "filename" => "#" }
        add_field => { "serverName" => "%{[filename][0]}" }
        add_field => { "configName" => "%{[filename][1]}" }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "iway_logs"
    user => "elastic"
    password => "something"
    cacert => "C:\f1\f2\logstash-8.1.3\config\cert\elasticsearch_http_ca.crt"
  }
}

The result of new fields namely, serverName and configName, always reports the raw expression as opposed to an evaluated output. Could someone help? TIA.

Answer 1

You should probably leverage the dissect filter for that, like this:

filter {
  if [path] {
    dissect {
      mapping => {
       "path" => "C:/f1/f2/Logs/f3/%{serverName}#%{configName}#%{?ignore}.log"
      }
    }
  }
}

If you have ECS compatibility enabled, the path field is called [log][file][path], so your config should be this one instead:

filter {
  if [log][file][path] {
    dissect {
      mapping => {
       "[log][file][path]" => "C:/f1/f2/Logs/f3/%{serverName}#%{configName}#%{?ignore}.log"
      }
    }
  }
}