'Can spark data source API get query plan informations?
If a spark user call dataframe.count, is there a way for the data source API to know?
We have a custom implementation of spark 3's data source api. Knowing which function is being called would allow us to do some optimization to the data source.
I'm relatively new to spark so I'm not sure if I missed something or which is the best approach.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|