'How can I use GPUTree (the shap explainer) on AWS databricks?
AWS databricks GPU instances seem only to offer the CPU version of the shap library. Following the documentation, I can replace it with the GPU version (finishes without errors). Unfortunately when using it on a single dummy sample, it throws the unhelpful
ConnectException: Connection refused (Connection refused) Error while obtaining a new communication channel
which supposedly is an OOM error - besides that there should be plenty of space and ganglia shows so. I think I am at a dead end here. Any way to get a more useful error message? Other baby steps to try? Or a different approach that would accomplish the same thing (running GPUTree in databricks-AWS)?
Solution 1:[1]
Like the Tree explainer, the GPUTree explainer is specifically designed for tree-based machine learning models, but it is designed to accelerate the computations using NVIDIA GPUs. source
To run it, one needs:
shappackage installed on a machine with a GPU- Machine having a GPU at runtime.
Otherwise, try KernelExplainer, which you may find a bit slow
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |