Doing the equivalent of log_struct in python logger

Question

In the google example, it gives the following:

logger.log_struct({
    'message': 'My second entry',
    'weather': 'partly cloudy',
})

How would I do the equivalent in python's logger. For example:

import logging
log.info(
    msg='My second entry', 
    extra = {'weather': "partly cloudy"}
)

When I view this in stackdriver, the extra fields aren't getting parsed properly:

2018-11-12 15:41:12.366 PST
My second entry

Expand all | Collapse all 

{
 insertId:  "1de1tqqft3x3ri"  
 jsonPayload: {
  message:  "My second entry"   
  python_logger:  "Xhdoo8x"   
 }
 logName:  "projects/Xhdoo8x/logs/python"  
 receiveTimestamp:  "2018-11-12T23:41:12.366883466Z"  
 resource: {…}  
 severity:  "INFO"  
 timestamp:  "2018-11-12T23:41:12.366883466Z"  
}

How would I do that?

The closest I'm able to do now is:

log.handlers[-1].client.logger('').log_struct("...")

But this still requires a second call...

Answer 1

This is not currently possible, see this open feature request on google-cloud-python for more details.

Answer 2

Official docs: Setting Up Cloud Logging for Python

You can write logs to Logging from Python applications

1. by using the Python logging handler included with the Logging client library, or
2. by using Cloud Logging API Cloud client library for Python directly.

I did not get the Python logging module to export jsonPayload, but the cloud logging library for Python works:

google-cloud-logging >= v.3.0.0 can do it

No need for Python logging workaround anymore, the only thing you need is to install Python >= 3.6 and

pip install google-cloud-logging

Then, you can use Python logging with

import os

# have the environment variable ready:
# GOOGLE_APPLICATION_CREDENTIALS
# Then:
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
client = google.cloud.logging.Client.from_service_account_json(
    GOOGLE_APPLICATION_CREDENTIALS
)
log_name = "test"
gcloud_logger = client.logger(log_name)

# jsonPayloads
gcloud_logger.log_struct(entry)
# messages
gcloud_logger.log_text('hello')
# generic (can find out whether it is jsonPayload or message)!
gcloud_logger.log(entry or 'hello')

You run these commands in a Python file outside of GCP and reach GCP with more or less a one-liner, you only need the credentials.

You can use the gcloud logger even for printing and logging in one go, taken from Writing structured logs, untested.

Python logging to log jsonPayload into GCP logs (TL/DR)

I could not get this to run!

You can also use the built-in logging module of Python with the workaround mentioned in the other answer, but I did not get it to run. It will not work if you pass a dictionary or its json.dumps() directly as a parameter, since then, you get a string output of the whole dictionary which you cannot read as a json tree. But it also did not work for me when I used the logger.info() to log the jsonPayload / json.dumps in an example parameter called extras.

import json
import os

#...

# https://googleapis.dev/python/logging/latest/stdlib-usage.html
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
client = google.cloud.logging.Client.from_service_account_json(
    GOOGLE_APPLICATION_CREDENTIALS
)
log_name = "test"
handler = CloudLoggingHandler(client, name=log_name)
setup_logging(handler)
logger = logging.getLogger()
logger.setLevel(logging.INFO)  # Set default level.

#...

    # Complete a structured log entry.
    entry = dict(
        severity="NOTICE",
        message="This is the default display field.",
        # Log viewer accesses 'component' as jsonPayload.component'.
        component="arbitrary-property"
    )

    # Python logging to log jsonPayload into GCP logs
    logger.info('hello', extras=json.dumps(entry))

I also tried the google-structlog solution of the other answer, that only threw the error:

google_structlog.setup(log_name=log_name) TypeError: setup() got an unexpected keyword argument 'log_name'

I used Python v3.10.2 and

google-auth==1.35.0
google-cloud-core==1.7.2
google-cloud-logging==1.15.0
googleapis-common-protos==1.54.0

Research steps gcloud logging (TL/DR)

Following the fixed issue (see the merge and close in the end) on github at googleapis / python-logging: Logging: support sending structured logs to stackdriver via stdlib 'logging'. #13

you find feat!: support json logs #316 :

This PR adds full support for JSON logs, along with standard text logs. Now, users can call logging.error({'a':'b'}), and they will get a JsonPayload in Cloud Logging, Or call logging.error('test') to receive a TextPayload

As part of this change, I added a generic logger.log() function, which serves as a generic entry-point instead of logger.log_text or logger.log_struct. It will infer which log function is meant based on the input type

Previously, the library would attach the python logger name as part of the JSON payload for each log. Now, that information will be attached as a label instead, giving users full control of the log payload fields

Fixes #186, #263, #13

With the main new version listing the new feature:

chore(main): release 3.0.0 #473