how to hide "py4j.java_gateway:Received command c on object id p0"?

Question

Once logging is started in INFO level I keep getting bunch of py4j.java_gateway:Received command c on object id p0 on your logs. How can I hide it?

Answer 1

using the logging module run:

logging.getLogger("py4j").setLevel(logging.ERROR)

Answer 2

None of these answers have worked for me. Even after using the above solutions, I was still getting the logging errors.

After a long search I came across the following that solved my issue.

import logging
logger = spark._jvm.org.apache.log4j
logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)

I found this solution in the Databricks Knowledge base article

Answer 3

The best way to control pyspark and py4j logging is by setting the following snippet:

import logging
logging.getLogger("py4j").setLevel(<pyspark-level>)
logging.getLogger('pyspark').setLevel(<py4j-level>)
logger = logging.getLogger('pyspark')

For your case you should write:

import logging
logging.getLogger("py4j").setLevel(logging.INFO)
logging.getLogger('pyspark').setLevel(logging.ERROR)
logger = logging.getLogger('pyspark')