'How to dissect a log file with Filebeat that has multiple patterns?

I have trouble dissecting my log file due to it having a mixed structure therefore I'm unable to extract meaningful data.

A few example lines from my log:

2021.04.21 00:00:00.843  INF  getBaseData: UserName = 'some username', Password = 'some pass', HTTPS=0
2021.04.21 00:00:00.843  INF  getBaseData: UserName = 'some username', Password = 'some pass', HTTPS=0
2021.04.21 00:00:00.843  INF  getBaseData: UserName = 'some username', Password = 'some pass', HTTPS=0
2021.04.21 00:00:00.858  INF  *** BEGINNING OF ARCCore.performARCTask ***
2021.04.21 00:00:00.858  INF  *** BEGINNING OF ARCCore.ProcessTask ***
2021.04.21 00:00:01.266  INF  ARCCore.DCI4ARCSyncLogin: login successfully executed. - No error - DCI4ARCSync-CurrSessions/MaxSessions=17/400 CurrProcesses/MaxProcesses=16/250
2021.04.21 00:00:01.297  INF  ARCCore.DCI4ARCSyncLogin: login successfully executed. - No error - DCI4ARCSync-CurrSessions/MaxSessions=7/400 CurrProcesses/MaxProcesses=7/250
2021.04.21 00:00:08.165  INF  ***  BEGINNING OF SYNC ARC TO DC  ***--->bIsExternal:0
2021.04.21 00:00:08.434  INF  BOC login successfully executed.  - No Error - DCI4ARC-CurrSessions/MaxSessions=24/400 CurrProcesses/MaxProcesses=15/250
2021.04.21 00:00:08.635  INF  BOCVersionNr ==> V16.1.00.00
2021.04.21 00:00:08.804  INF  setEntitySnapshot successfully executed
2021.04.21 00:00:09.453  INF  getSnapshotList successfully executed
2021.04.21 00:00:09.461  INF  getARCVersion: ARCVersionNr ==> V16.0.00.06

I wrote a tokenizer with which I successfully dissected the first three lines of my log due to them matching the pattern but fail to read the rest.

My tokenizer pattern:

%{+timestamp} %{+timestamp} %{type} %{msg}: UserName = %{userName}, Password = %{password}, HTTPS=%{https}

the lines that get read successfully:

2021.04.21 00:00:00.843 INF getBaseData: UserName = 'some username', Password = 'some password', HTTPS=0 2021.04.21 00:00:00.843 INF getBaseData: UserName = 'some username', Password = 'some password', HTTPS=0 2021.04.21 00:00:00.843 INF getBaseData: UserName = 'some username ', Password = 'some password', HTTPS=0

The result:

{
  "https": "0",
  "msg": "getBaseData",
  "password": "'20213197'",
  "timestamp": "2021.04.21 00:00:00.843",
  "type": "INF",
  "userName": "'ARC_412_028_01_V01_2021042100000082'"
}
{
  "https": "0",
  "msg": "getBaseData",
  "password": "'20213205'",
  "timestamp": "2021.04.21 00:00:00.843",
  "type": "INF",
  "userName": "'ARC_412_028_01_V01_2021042100000084'"
}
{
  "https": "0",
  "msg": "getBaseData",
  "password": "'20213205'",
  "timestamp": "2021.04.21 00:00:00.843",
  "type": "INF",
  "userName": "'ARC_412_028_01_V01_2021042100000084'"
}


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source