How to run spark yarn jobs on a Hadoop cluster that is external to the K8S cluster

Question

I am running a on-prem k8s cluster and am running juypterhub on it.

I can successfully submit the job to an yarn queue, however the job will fail because users notebook pod IP is not resolvable and therefore it can’t talk back to the spark driver running on said pod and I get an error like:

Caused by: java.io.IOException: Failed to connect to podIP:33630 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:287) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.UnknownHostException: pod-ip

I believe I’m missing something in my setup that will allow yarn to talk back to the spawned notebook pods on the kubernetes cluster.

Any help or hints are greatly appreciated .

For now I am passing the spark driver the internal Kubernetes Pod IP of juypterhub by setting: "spark.driver.host" to str(socket.gethostbyname(socket.gethostname())). Could I change this to something else in the notebook I am running? I am not too sure what to change it to.

Thanks!