'Cloud Operations - Ops Agent parsing error when retrieving DBT log timestamp

I've setup a custom logging service to retrieve my dbt logs to Cloud Logging. I've setup DBT to output the logs in a json format example:

{"code": "A001", "data": {"v": "=1.0.3"}, "invocation_id": "adf569c0-3a88-44b1-b538-3e6136bbe74a", "level": "info", "log_version": 1, "msg": "Running with dbt=1.0.3", "node_info": {}, "pid": 7168, "thread_name": "MainThread", "ts": "2022-03-10T14:36:10.859111Z", "type": "log_line"}

my agent's config.yaml is as follows:


logging:
  receivers:
    dbt_receiver:
      type: files

      include_paths: [/home/user/bigquery-oreilly-book/blogs/dbt_load/college-scorecard/logs/*.json]
      exclude_paths: [/home/user/bigquery-oreilly-book/blogs/dbt_load/college-scorecard/logs/*.log.legacy]

  processors:
    dbt_processor:
      type: parse_json

      field: message
      time_key: ts
      time_format: "%Y-%m-%dT%H:%M:%S.%6N%Z"


  service:
    pipelines:
      custom_pipeline:
        receivers:
        - dbt_receiver
        processors:
        - dbt_processor

I'm having a hard time getting the right format for the time_format with time_format: "%Y-%m-%dT%H:%M:%S.%6N%Z" I'm getting a

"[2022/03/10 15:04:50] [ warn] [parser:custom_pipeline.dbt_receiver.0] invalid time format %Y-%m-%dT%H:%M:%S.%6N%Z for '2022-03-10T16:04:08.669441Z'" error raised in Cloud Logging



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source