Where is my main method runs when using in yarn-cluster and detached mode

Question

I am new to flink and reading Flink 1.8 source code(https://github.com/apache/flink/tree/release-1.8) to understand how flink works with YARN.

I know there are detached mode and non-detached mode for the per-job cluster mode.

For the non-detached mode, such as the following command:

flink run -m yarn-cluster -c my.HelloFlink -yn 2 -ys 1 ./my.jar

After the yarn cluster is deployed,then the client process starts to run my main method(my.HelloFlink#main), and the client process doesn't terminate until my main method finished.

For the detached mode, such as the following command:

flink run -d -m yarn-cluster -c my.HelloFlink -yn 2 -ys 1 ./my.jar

After the yarn cluster is deployed,then the client process terminates soon, but I didn't find where(in which process) my main method gets to run(my.HelloFlink#main), could some one help me out here and help on where my main method runs?

Thanks, I have struggled on this question for days, thanks very much!

Answer 1

When you use flink run ... you are running a bash script which ends with this line

exec $JAVA_RUN $JVM_ARGS $FLINK_ENV_JAVA_OPTS "${log_setting[@]}" -classpath "`manglePathList "$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.client.cli.CliFrontend "$@"

which fires up a JVM running the CliFrontend. Your main method runs there. What your main method does is to construct a job graph and submit it to the yarn cluster, along with its dependencies. If you run in detached mode, this CliFrontend process simply exits after submitting the job, as it is no longer useful.

By the way, Flink 1.11 has added a new flink run-application deployment target that runs the main method in the Job Manager instead. This has significant advantages in some situations; see Application Deployment in Flink: Current State and the new Application Mode for details.