Quantcast
Channel: Spark submit does automatically upload the jar to cluster? - Stack Overflow
Viewing all articles
Browse latest Browse all 4

Spark submit does automatically upload the jar to cluster?

$
0
0

I'm trying to submit a Spark app from local machine Terminal to my Cluster.I'm using --master yarn-cluster. I need to run the driver program on my Cluster too, not on the machine I do submit the application i.e my local machine

When I provide the path to application jar which is in my local machine, would spark-submit automatically upload it to my Cluster?

I'm using

    bin/spark-submit --class com.my.application.XApp --master yarn-cluster --executor-memory 100m --num-executors 50 /Users/nish1013/proj1/target/x-service-1.0.0-201512141101-assembly.jar 1000

and getting error

Diagnostics: java.io.FileNotFoundException: File file:/Users/nish1013/proj1/target/x-service-1.0.0-201512141101- does not exist

In Documentation ,http://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit

Advanced Dependency Management When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster.

But seems like it does not !


Viewing all articles
Browse latest Browse all 4

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>