'Cloud Operations - Ops Agent parsing error when retrieving DBT log timestamp
I've setup a custom logging service to retrieve my dbt logs to Cloud Logging. I've setup DBT to output the logs in a json format example:
{"code": "A001", "data": {"v": "=1.0.3"}, "invocation_id": "adf569c0-3a88-44b1-b538-3e6136bbe74a", "level": "info", "log_version": 1, "msg": "Running with dbt=1.0.3", "node_info": {}, "pid": 7168, "thread_name": "MainThread", "ts": "2022-03-10T14:36:10.859111Z", "type": "log_line"}
my agent's config.yaml is as follows:
logging:
receivers:
dbt_receiver:
type: files
include_paths: [/home/user/bigquery-oreilly-book/blogs/dbt_load/college-scorecard/logs/*.json]
exclude_paths: [/home/user/bigquery-oreilly-book/blogs/dbt_load/college-scorecard/logs/*.log.legacy]
processors:
dbt_processor:
type: parse_json
field: message
time_key: ts
time_format: "%Y-%m-%dT%H:%M:%S.%6N%Z"
service:
pipelines:
custom_pipeline:
receivers:
- dbt_receiver
processors:
- dbt_processor
I'm having a hard time getting the right format for the time_format with time_format: "%Y-%m-%dT%H:%M:%S.%6N%Z" I'm getting a
"[2022/03/10 15:04:50] [ warn] [parser:custom_pipeline.dbt_receiver.0] invalid time format %Y-%m-%dT%H:%M:%S.%6N%Z for '2022-03-10T16:04:08.669441Z'" error raised in Cloud Logging
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
