'How to setup logger in pyspark for multiple modules?
I have been trying to setup pyspark logger for multiple modules. For base script, I am able to set it up and use logger but unable to find any way of bringing that logger to other modules. Need some help on that aspect.
spark.sparkContext.setLogLevel("INFO")
log_4j = spark.sparkContext._jvm.org.apache.log4j
logger = log_4j.LogManager.getLogger(__name__)
Used this code to setup the logger in base script.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|