'How to parse C# inner exceptions from unstructured logs in Datadog with grok processor

So I have an unstructured log that I want to parse in datadog. Specifically I am trying to parse out the inner exceptions from a logged exception:

My.Exception.Type.CustomException: This is my exception message. ---> An.Inner.Exception.InnerException: This is an inner exception  message  ---> An.Inner.Exception.AnotherInnerException: This is another inner exception message
   at Stack.Trace.Goes.Here

I have made several attempts at solving this problem. The method that I would have thought would work, was this:

exception_rule %{notSpace:error.kind}:\s+%{data:error.message}\s*%{inner_exception}*\s+%{regex("   at.*"):error.stack}

And the helper rule defined as:

inner_exception (\s*--->\s+%{notSpace:error.inner.kind}:\s+%{data:error.inner.message})

The actual output:

{
  "error": {
    "kind": "My.Exception.Type.CustomException",
    "message": "This is my exception message.",
    "stack": "   at Stack.Trace.Goes.Here",
    "inner": {
      "kind": "An.Inner.Exception.AnotherInnerException",
      "message": "This is another inner exception message"
    }
  }
}

My expected output:

{
  "error": {
    "kind": "My.Exception.Type.CustomException",
    "message": "This is my exception message.",
    "stack": "   at Stack.Trace.Goes.Here",
    "inner": [
      {
        "kind": "An.Inner.Exception.InnerException",
        "message": "This is an inner exception message"
      },
      {
        "kind": "An.Inner.Exception.AnotherInnerException",
        "message": "This is another inner exception message"
      }
    ]
  }
}

I can get close to what I want if I use the array type:

exception_rule %{notSpace:error.kind}:\s+%{data:error.message}(\s*--->\s*(%{data:error.inner:array(""," ---> ")}))%{regex("   at.*"):error.stack}

Which yields the following output:

{
  "error": {
    "kind": "My.Exception.Type.CustomException",
    "message": "This is my exception message.",
    "stack": "   at Stack.Trace.Goes.Here",
    "inner": [
      "An.Inner.Exception.InnerException: This is an inner exception  message  ",
      "An.Inner.Exception.AnotherInnerException: This is another inner exception message\n"
    ]
  }
}

But here the inner exceptions aren't properly tagged and are treated as raw strings



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source