mapGroupsWithState引发错误原因:java.lang.NoClassDefFoundError:无法初始化

on_the_shores_of_linux_sea

我正在尝试使用mapGroupsWithState读取csv并获取事件状态,并将其写入kafka。如果我注释掉mapGroupsWithState peice,则下面的代码有效。使用Spark版本2.3.1

val event = spark.read.option("header","true").csv(path)
val eventSession = imsi.orderBy("event_timestamp")
                   .groupByKey(_.key)
                   .mapGroupsWithState(GroupStateTimeout.NoTimeout())(updateAcrossEvents)
eventSession.toJSON.write.format("kafka")
        .option("kafka.bootstrap.servers", brokers)
        .option("topic", outputTopic).save 

错误

User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 85 in stage 11.0 failed 8 times, most recent failure: Lost task 85.7 in stage 11.0 (TID 53, XXX, executor 2): java.lang.NoClassDefFoundError: Could not initialize class xxxx$
at xxx.imsiProcessor$$anonfun$run$1$$anonfun$3.apply(xx.scala:86)
at xxx.imsiProcessor$$anonfun$run$1$$anonfun$3.apply(xx.scala:86)
at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$3.apply(KeyValueGroupedDataset.scala:279)
at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$3.apply(KeyValueGroupedDataset.scala:279)
at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$12.apply(objects.scala:361)
at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$12.apply(objects.scala:360)
at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$10$$anonfun$apply$4.apply(objects.scala:337)
at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$10$$anonfun$apply$4.apply(objects.scala:336)
Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:367)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:933)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:924)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:924)
    at com.telstra.elbrus.core.imsiProcessor$.spark$lzycompute(ImsiProcessor.scala:38)
on_the_shores_of_linux_sea

通过消除一些扩展,我能够使代码运行。裸露的代码开始运行。

本文收集自互联网,转载请注明来源。

如有侵权,请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章

java.lang.NoClassDefFoundError:无法初始化类org.powermock.modules.junit4.rule.PowerMockRule

java.lang.NoClassDefFoundError:无法初始化类javax.media.jai.JAI

java.lang.NoClassDefFoundError:无法初始化类org.bytedeco.javacpp.avutil

原因:java.lang.IllegalStateException:ParsePlugins已经初始化

java.lang.NoClassDefFoundError:无法初始化类java.awt.Toolkit

java.lang.NoClassDefFoundError:无法初始化类XXX

VM初始化期间发生错误(java / lang / NoClassDefFoundError:java / lang / Object)

原因:java.lang.NoClassDefFoundError:无法初始化类com.jayway.restassured.RestAssured

拒绝对先前失败的类java.lang.Class <nt>进行重新初始化:java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError:无法初始化类xxx.xxx.xxx.HibernateUtil

春季:java.lang.NoClassDefFoundError:无法初始化类

java.lang.NoClassDefFoundError:无法初始化com.datastax.driver.core.Cluster类

ARM64计算机上的Cassandra启动失败(java.lang.NoClassDefFoundError:无法初始化com.sun.jna.Native类)

Zeppelin java.lang.NoClassDefFoundError:无法初始化类org.apache.spark.rdd.RDDOperationScope $

关于java.lang.NoClassDefFoundError:无法初始化类org.xerial.snappy.Snappy

java.lang.IllegalStateException:无法初始化插件:MockMaker

java.lang.NoClassDefFoundError:无法初始化com.googlecode.objectify.ObjectifyService类

带有Java 9 java.lang.NoClassDefFoundError的Oracle JDBC驱动程序:无法初始化类oracle.jdbc.OracleDriver

Corda-无法初始化主类胶囊(引起原因:java.lang.NoClassDefFoundError:java / lang / instrument / Instrumentation)

获取java.lang.NoClassDefFoundError:升级到POI 4.0.0后无法初始化类org.apache.poi.ooxml.util.DocumentHelper

Android:Espresso:按意图进行的初始化活动-java.lang.RuntimeException:无法启动活动

原因:java.lang.NoClassDefFoundError:无法初始化类javassist.util.proxy.DefineClassHelper

java.lang.IllegalStateException:CameraX尚未初始化

java.lang.NoClassDefFoundError:无法初始化类org.springframework.mail.javamail.SmartMimeMessage

java.lang.NoClassDefFoundError:无法初始化类sun.nio.ch.FileChannelImpl

Grails域类单元测试,约束设置问题-java.lang.NoClassDefFoundError:无法初始化

java.lang.NoClassDefFoundError:无法初始化类org.apache.commons.logging.LogFactory

java.lang.IllegalStateException:ServletConfig尚未初始化

线程“主”中的异常java.lang.NoClassDefFoundError无法初始化类org.bytedeco.javacpp.avutil