I'm trying to submit a Spark app from local machine Terminal to my Cluster.I'm using --master yarn-cluster
. I need to run the driver program on my Cluster too, not on the machine I do submit the application i.e my local machine
When I provide the path to application jar which is in my local machine, would spark-submit automatically upload it to my Cluster?
I'm using
bin/spark-submit --class com.my.application.XApp --master yarn-cluster --executor-memory 100m --num-executors 50 /Users/nish1013/proj1/target/x-service-1.0.0-201512141101-assembly.jar 1000
and getting error
Diagnostics: java.io.FileNotFoundException: File file:/Users/nish1013/proj1/target/x-service-1.0.0-201512141101- does not exist
In Documentation ,http://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit
Advanced Dependency Management When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster.
But seems like it does not !