搜索
搜索

Apache Spark MultilayerPerceptronClassifier失败，并出现ArrayIndexOutOfBoundsException

blue-sky 发表于 Dev
31
蓝天

我正在使用此代码来尝试预测： 

import org.apache.spark.sql.functions.col
import org.apache.spark.Logging
import org.apache.spark.graphx._
import org.apache.spark.{ SparkConf, SparkContext }
import org.apache.spark.SparkContext._
import org.apache.spark.sql.SQLContext._
import org.apache.log4j.Logger
import org.apache.log4j.Level
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.feature.VectorAssembler

object NN extends App {

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

val sc = new SparkContext(new SparkConf().setMaster("local[2]")
.setAppName("cs"))

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

val df = sc.parallelize(Seq(

("3", "1", "1"),

("2", "1", "1"),

("2", "3", "3"),

("3", "3", "3"),

("0", "1", "0")))

.toDF("label", "feature1", "feature2")

val numeric = df
.select(df.columns.map(c => col(c).cast("double").alias(c)): _*)

val assembler = new VectorAssembler()
.setInputCols(Array("feature1", "feature2"))
.setOutputCol("features")

val data = assembler.transform(numeric)

import org.apache.spark.ml.classification.MultilayerPerceptronClassifier

val layers = Array[Int](2, 3, 5, 4) // Note 2 neurons in the input layer
val trainer = new MultilayerPerceptronClassifier()
.setLayers(layers)
.setBlockSize(128)
.setSeed(1234L)
.setMaxIter(100)

val model = trainer.fit(data)
model.transform(data).show


}

对于数据框（df），如果我使用("4", "1", "1")而不是("3", "1", "1")我收到错误：

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Set current project to spark-applications1458853926-master (in build file:/C:/Users/Desktop/spark-applications1458853926-master/)
[info] Compiling 1 Scala source to C:\Users\Desktop\spark-applications1458853926-master\target\scala-2.11\classes...
[info] Running NN
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/04/06 12:42:11 INFO Remoting: Starting remoting
16/04/06 12:42:11 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:64056]
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ArrayIndexOutOfBoundsException: 4
[error]         at org.apache.spark.ml.classification.LabelConverter$.encodeLabeledPoint(MultilayerPerceptronClassifier.scala:85)
[error]         at org.apache.spark.ml.classification.MultilayerPerceptronClassifier$$anonfun$2.apply(MultilayerPerceptronClassifier.scala:165)
[error]         at org.apache.spark.ml.classification.MultilayerPerceptronClassifier$$anonfun$2.apply(MultilayerPerceptronClassifier.scala:165)
[error]         at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
[error]         at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:934)
[error]         at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:949)
[error]         at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:986)
[error]         at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:990)
[error]         at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
[error]         at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)
[error]         at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
[error]         at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
[error]         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
[error]         at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:89)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error]         at java.lang.Thread.run(Thread.java:745)
[error]
[error] Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ArrayIndexOutOfBoundsException: 4
        at org.apache.spark.ml.classification.LabelConverter$.encodeLabeledPoint(MultilayerPerceptronClassifier.scala:85)
        at org.apache.spark.ml.classification.MultilayerPerceptronClassifier$$anonfun$2.apply(MultilayerPerceptronClassifier.scala:165)
        at org.apache.spark.ml.classification.MultilayerPerceptronClassifier$$anonfun$2.apply(MultilayerPerceptronClassifier.scala:165)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
        at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:934)
        at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:949)
        at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:986)
        at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:990)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
        at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
        at org.apache.spark.mllib.optimization.LBFGS$.runLBFGS(LBFGS.scala:170)
        at org.apache.spark.mllib.optimization.LBFGS.optimize(LBFGS.scala:117)
        at org.apache.spark.ml.ann.FeedForwardTrainer.train(Layer.scala:878)
        at org.apache.spark.ml.classification.MultilayerPerceptronClassifier.train(MultilayerPerceptronClassifier.scala:170)
        at org.apache.spark.ml.classification.MultilayerPerceptronClassifier.train(MultilayerPerceptronClassifier.scala:110)
        at org.apache.spark.ml.Predictor.fit(Predictor.scala:90)
        at NN$.delayedEndpoint$NN$1(NN.scala:56)
        at NN$delayedInit$body.apply(NN.scala:15)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
        at scala.App$class.main(App.scala:76)
        at NN$.main(NN.scala:15)
        at NN.main(NN.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
        at org.apache.spark.ml.classification.LabelConverter$.encodeLabeledPoint(MultilayerPerceptronClassifier.scala:85)
        at org.apache.spark.ml.classification.MultilayerPerceptronClassifier$$anonfun$2.apply(MultilayerPerceptronClassifier.scala:165)
        at org.apache.spark.ml.classification.MultilayerPerceptronClassifier$$anonfun$2.apply(MultilayerPerceptronClassifier.scala:165)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
        at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:934)
        at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:949)
        at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:986)
        at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:990)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:369)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1595)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1143)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 19 s, completed 06-Apr-2016 12:42:20

为什么会收到ArrayIndexOutOfBoundsException，我没有正确设置标签？标签仅仅是标签就不能具有任何价值吗？在此示例中，看起来它们必须在0-3范围内。

马修·格雷夫斯

输出层使用一键编码；也就是说，标签“ 3”将转换为（0,0,0,1），其中“第三个”元素为1，其余元素为0。当您有4个输出节点且标签为4时，LabelConverter函数（在此处可见其源）将失败。（labelCount是4，labeledPoint.label.toInt是4，因此是您的错误。）

val output = Array.fill(labelCount)(0.0)
output(labeledPoint.label.toInt) = 1.0
(labeledPoint.features, Vectors.dense(output))

因此更改此行：

val layers = Array[Int](2, 3, 5, 4) // Note 2 neurons in the input layer

对此：

val layers = Array[Int](2, 3, 5, 5) // Note 2 neurons in the input layer and 5 neurons in the output layer

我希望它能工作。

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章

Apache Spark MultilayerPerceptronClassifier设置功能

Apache Spark任务失败

Apache Spark：任务在com.ning.compress.lzf.impl.UnsafeChunkDecoder.copyOverlappingLong处出现java.lang.ArrayIndexOutOfBoundsException失败

Apache Spark Python UDF 失败

Apache Spark流的集成测试失败

Apache Spark与Apache Spark 2

Apache Spark与Apache Ignite

是否在Apache Spark中重新提交了失败的任务？

org.apache.spark.SparkException：由于阶段失败，作业中止了：

为什么 Apache Spark 不重新提交失败的任务？

为什么Spark失败并出现java.lang.OutOfMemoryError：超出了GC开销限制？

无法部署本地Spark作业，工作程序失败并出现EndPointAssociationError

Spark SQL失败并出现java.lang.NoClassDefFoundError：org / codehaus / commons / compiler / UncheckedCompileException

在 Apache Spark 上运行 JPMML 时出现 UnmarshalException

Apache Spark：从IDE远程运行作业时出现无限循环

为什么Spark会失败并显示“值rdf不是org.apache.spark.sql.SparkSession的成员”？

apache spark agg（）函数

Autobatching in Apache Spark

Apache Spark和gRPC

Spark Apache源代码

Apache Spark RDD拆分“ |”

Apache Spark与Python

Apache Spark ALS建议

Apache Spark：map与mapPartitions？

Apache Spark中的Printschema（）

Apache Spark与Akka

Apache Spark与Kafka的集成

Apache Drill与Spark

Apache Spark Stats集合

TOP 榜单

  1. 1

    UITableView的项目向下滚动后更改颜色，然后快速备份

  2. 2

    Linux的官方Adobe Flash存储库是否已过时？

  3. 3

    用日期数据透视表和日期顺序查询

  4. 4

    应用发明者仅从列表中选择一个随机项一次

  5. 5

    Mac OS X更新后的GRUB 2问题

  6. 6

    验证REST API参数

  7. 7

    Java Eclipse中的错误13，如何解决？

  8. 8

    带有错误“ where”条件的查询如何返回结果？

  9. 9

    ggplot：对齐多个分面图-所有大小不同的分面

  10. 10

    尝试反复更改屏幕上按钮的位置 - kotlin android studio

  11. 11

    如何从视图一次更新多行（ASP.NET - Core）

  12. 12

    计算数据帧中每行的NA

  13. 13

    蓝屏死机没有修复解决方案

  14. 14

    在 Python 2.7 中。如何从文件中读取特定文本并分配给变量

  15. 15

    离子动态工具栏背景色

  16. 16

    VB.net将2条特定行导出到DataGridView

  17. 17

    通过 Git 在运行 Jenkins 作业时获取 ClassNotFoundException

  18. 18

    在Windows 7中无法删除文件（2）

  19. 19

    python中的boto3文件上传

  20. 20

    当我尝试下载 StanfordNLP en 模型时，出现错误

  21. 21

    Node.js中未捕获的异常错误，发生调用

热门标签

归档