Spark not able to run in yarn cluster mode

user3648294

I am trying to execute my code on a yarn cluster

The command which I am using is

$SPARK_HOME/bin/spark-submit \
    --class "MyApp" \
    target/scala-2.10/my-application_2.10-1.0.jar \
    --master yarn-cluster \
    --num-executors 3 \
    --driver-memory 6g \
    --executor-memory 7g \
    <outputPath>

But, I can see that this program is running only on the localhost.

Its able to read the file from hdfs.

I have tried this in standalone mode and it works fine.

Please suggest where is it going wrong.

user3648294

I am using Hadoop2.4 with Spark 1.1.0 . I was able to get it running in the cluster mode.

To solve it we simply removed all the configuration files from all the slave nodes. Earlier we were running in the standalone mode and that lead to duplicating the configuration on all the slaves. Once that was done it ran as expected in cluster mode. Although performance is not up to the standalone mode.

Thanks.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Executing Spark on yarn cluster mode AccessControlException

submitting PySpark app to spark on YARN in cluster mode

Can Spark streaming and Spark applications be run within the same YARN cluster?

Why ExceptionInInitializerError when submitting Spark application in YARN cluster mode?

Spark Streaming Kafka integration in CDH 5.8.3 in yarn-cluster mode

Spark Map partiton not working in yarn-cluster mode

SparkConf settings not used when running Spark app in cluster mode on YARN

Making spark use /etc/hosts file for binding in YARN cluster mode

emr spark master node runs out of memory in yarn cluster mode

For spark applications running on YARN, which deploy mode is better - client or cluster

How YARN knows data locality in Apache spark in cluster mode

Apache Spark in yarn-cluster mode is throwing Hadoop FileAlreadyExistsException

Unable to read Hbase data with spark in yarn cluster mode

setup/run spark (spark-shell) on yarn client mode

How to run simple Spark app with ZeroMQ on a YARN cluster?

Yarn Cluster optimization for Spark

Running Spark on YARN in yarn-cluster mode: Where does the console output go?

Resources/Documentation on how does the failover process work for the Spark Driver (and its YARN Container) in yarn-cluster mode

Spark (Yarn) applications started by Zeppelin in Yarn Cluster Mode aren't killed after zeppein is stopped

Pyspark on yarn-cluster mode

Error (application returned with exitcode 1) when running Spark in yarn-cluster mode

Missing hive-site when using spark-submit YARN cluster mode

Why does Spark on YARN in cluster mode fail with "Exception in thread "Driver" java.lang.NullPointerException"?

How to fetch Spark Streaming job statistics using REST calls when running in yarn-cluster mode

spark-submit --proxy-user do not work in yarn cluster mode

Why does spark-submit in YARN cluster mode not find python packages on executors?

spark-submit works for yarn-cluster mode but SparkLauncher doesn't, with same params

Spark File Logger in Yarn Mode

Spark: hdfs cluster mode