Spark : java.lang.NoClassDefFoundError: scala/collection/mutable/ArraySeq$ofRef

user1575148

I am trying to run a simple word count program with spark-submit and getting an exception.

Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/collection/mutable/ArraySeq$ofRef at SparkWordCount$.main(SparkWordCount.scala:18)

The code, starting with line 18 is

val count = input.flatMap(line ⇒ line.split(" "))
    .map(word ⇒ (word, 1))
    .reduceByKey(_ + _)

My environment:

  • Windows 10
  • java version "1.8.0_221"
  • spark-shell shows : Spark version 2.4.4 (Using Scala version 2.11.12)
  • scala -version command shows Scala code runner version 2.13.1
Vitaly Olegovitch

As stated in the comments, the solution is to use for development the same version of Scala that you will use on the cluster.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

KafkaUtils java.lang.NoClassDefFoundError Spark Streaming

java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession

java.lang.NoClassDefFoundError: org/apache/spark/Logging

java.lang.NoClassDefFoundError: org/apache/spark/TaskOutputFileAlreadyExistException

spark throws java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition

java.lang.NoClassDefFoundError when running a spark jar application

Apache Spark - Quick Start "java.lang.NoClassDefFoundError: scala/Serializable"

java.lang.NoClassDefFoundError

"java.lang.NoClassDefFoundError"

Spark and Cassandra Java application: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset

spark kafka streaming Error - " java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils

oozie spark 2.0 action giving exception: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

Spark error : java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/StreamWriteSupport

Spark: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging

Problem starting trivial standalone spark app: java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/StaticSQLConf$

Spark Cassandra Connector Error: java.lang.NoClassDefFoundError: com/datastax/spark/connector/TableRef

java.lang.NoClassDefFoundError in scalatest

Android: java.lang.NoClassDefFoundError

"java.lang.NoClassDefFoundError: Result"

java.lang.NoClassDefFoundError on OSGi

java.lang.NoClassDefFoundError in junit

Classloader java.lang.NoClassDefFoundError?

The dreaded java.lang.NoClassDefFoundError

Java.lang.NoClassDefFoundError in android

PowerMockito: java.lang.NoClassDefFoundError

Android java lang NoClassDefFoundError Caldroid

java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer for spark streaming

IntelliJ: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/types/DataType

Apache Spark - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream