java.lang.IllegalStateException：找不到任何构建目录

Rana 发表于 Dev

拉那

我想在Intellij中运行spark master和worker。我已经成功启动了火花大师和工人。工人也可以连接到主机，没有任何问题。我可以通过查看日志和启动Web UI来确认这一点。但是，当我尝试在此设置上运行任何应用程序时，问题就开始了。当前，我正在尝试在此设置上运行spark-shell，但是当它尝试掌握时，我在master上收到一些错误消息，在worker上收到IlligalStateException。

主日志（成功启动）

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/09/21 16:42:33 INFO Master: Started daemon with process name: 22237@rana-pc
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for TERM
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for HUP
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for INT
16/09/21 16:42:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/21 16:42:34 INFO SecurityManager: Changing view acls to: rana
16/09/21 16:42:34 INFO SecurityManager: Changing modify acls to: rana
16/09/21 16:42:34 INFO SecurityManager: Changing view acls groups to: 
16/09/21 16:42:34 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 16:42:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(rana); groups with view permissions: Set(); users  with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 16:42:35 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
16/09/21 16:42:35 INFO Master: Starting Spark master at spark://MASTER_IP:7077
16/09/21 16:42:35 INFO Master: Running Spark version 2.0.0
16/09/21 16:42:35 INFO log: Logging initialized @3645ms
16/09/21 16:42:36 INFO Server: jetty-9.2.16.v20160414
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@12fb5c3b{/app,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@48b5405c{/app/json,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6bf42129{/,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@17c4d9fc{/json,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5f1655bb{/static,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3dc349a5{/app/kill,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@7774da5d{/driver/kill,null,AVAILABLE}
16/09/21 16:42:36 INFO ServerConnector: Started ServerConnector@678fff3{HTTP/1.1}{0.0.0.0:8080}
16/09/21 16:42:36 INFO Server: Started @3999ms
16/09/21 16:42:36 INFO Utils: Successfully started service 'MasterUI' on port 8080.
16/09/21 16:42:36 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://MASTER_IP:8080
16/09/21 16:42:36 INFO Server: jetty-9.2.16.v20160414
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@49c1de92{/,null,AVAILABLE}
16/09/21 16:42:36 INFO ServerConnector: Started ServerConnector@3e8a8bfb{HTTP/1.1}{MASTER_IP:6066}
16/09/21 16:42:36 INFO Server: Started @4050ms
16/09/21 16:42:36 INFO Utils: Successfully started service on port 6066.
16/09/21 16:42:36 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@185eae98{/metrics/master/json,null,AVAILABLE}
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6f634fcf{/metrics/applications/json,null,AVAILABLE}
16/09/21 16:42:36 INFO Master: I have been elected leader! New state: ALIVE
16/09/21 16:42:46 INFO Master: Registering worker MASTER_IP:42839 with 6 cores, 6.7 GB RAM

工作日志（成功启动）

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/09/21 17:24:08 INFO Worker: Started daemon with process name: 24007@rana-pc
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for TERM
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for HUP
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for INT
16/09/21 17:24:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/21 17:24:08 INFO SecurityManager: Changing view acls to: rana
16/09/21 17:24:08 INFO SecurityManager: Changing modify acls to: rana
16/09/21 17:24:08 INFO SecurityManager: Changing view acls groups to: 
16/09/21 17:24:08 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 17:24:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(rana); groups with view permissions: Set(); users  with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 17:24:09 INFO Utils: Successfully started service 'sparkWorker' on port 50059.
16/09/21 17:24:09 INFO Worker: Starting Spark worker MASTER_IP:50059 with 6 cores, 6.7 GB RAM
16/09/21 17:24:09 INFO Worker: Running Spark version 2.0.0
16/09/21 17:24:09 INFO Worker: Spark home: /usr/local/spark
16/09/21 17:24:09 INFO log: Logging initialized @2328ms
16/09/21 17:24:09 INFO Server: jetty-9.2.16.v20160414
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@251c286b{/logPage,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@c5c7b5f{/logPage/json,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4aa264e6{/,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@1f240248{/json,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5b3f3eb7{/static,null,AVAILABLE}
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@34d6200c{/log,null,AVAILABLE}
16/09/21 17:24:09 INFO ServerConnector: Started ServerConnector@4eaf8f4d{HTTP/1.1}{0.0.0.0:8081}
16/09/21 17:24:09 INFO Server: Started @2440ms
16/09/21 17:24:09 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
16/09/21 17:24:09 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://MASTER_IP:8081
16/09/21 17:24:09 INFO Worker: Connecting to master MASTER_IP:7077...
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@270d60ee{/metrics/json,null,AVAILABLE}
16/09/21 17:24:09 INFO TransportClientFactory: Successfully created connection to /MASTER_IP:7077 after 35 ms (0 ms spent in bootstraps)
16/09/21 17:24:10 INFO Worker: Successfully registered with master spark://MASTER_IP:7077

主日志（尝试运行spark-shell时）

16/09/21 17:26:08 INFO Master: Registering app Spark shell
16/09/21 17:26:08 INFO Master: Registered app Spark shell with ID app-20160921172608-0000
16/09/21 17:26:08 INFO Master: Launching executor app-20160921172608-0000/0 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/0 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/1 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/1 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/2 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/2 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/3 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/3 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/4 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/4 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/5 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/5 because it is FAILED
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/6 on worker worker-20160921172409-MASTER_IP-50059
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/6 because it is FAILED

工作者日志（尝试运行spark-shell时）

16/09/21 17:26:09 INFO Worker: Asked to launch executor app-20160921172608-0000/0 for Spark shell
16/09/21 17:26:09 INFO SecurityManager: Changing view acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing view acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(rana); groups with view permissions: Set(); users  with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 17:26:09 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
    at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
    at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
    at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
    at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47)
    at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
    at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
    at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)
    at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
16/09/21 17:26:09 INFO Worker: Executor app-20160921172608-0000/0 finished with state FAILED message java.lang.IllegalStateException: Cannot find any build directories.
16/09/21 17:26:09 INFO Worker: Asked to launch executor app-20160921172608-0000/1 for Spark shell
16/09/21 17:26:09 INFO SecurityManager: Changing view acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls to: rana
16/09/21 17:26:09 INFO SecurityManager: Changing view acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(rana); groups with view permissions: Set(); users  with modify permissions: Set(rana); groups with modify permissions: Set()
16/09/21 17:26:09 ERROR ExecutorRunner: Error running executor
java.lang.IllegalStateException: Cannot find any build directories.
    at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
    at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
    at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
    at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39)
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47)
    at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63)
    at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51)
    at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145)

有人可以帮我解决这个问题吗？

拉那

我已经解决了这个问题。实际上，我收到此错误是因为主服务器无法找到该应用程序的启动器（spark-shell）。主服务器正在寻找目录$ SPARK_HOME / launcher / target / scala-2.11，但是找不到它。当我在IntelliJ中运行spark时，我的$ SPARK_HOME应该已经是启动器文件夹所在的项目目录。因此，我对此进行了更改，使其开始完美运行。现在，我可以在此设置上运行spark-shell应用程序。

如果您不想更改现有的设置，那么也可以通过对AbstractCommandBuilder.java文件本身中的路径进行硬编码来完成。

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。

编辑于 2020-11-3

我来说两句

0 条评论

登录后参与评论

上一篇：Angular 2形式：将html select绑定到对象集合

TOP 榜单

文章

java.lang.IllegalStateException：找不到任何构建目录

java.lang.IllegalStateException：找不到任何构建目录

Android Studio Kotlin：提取为常量

计算数据帧R中的字符串频率

如何使用Redux-Toolkit重置Redux Store

http：// localhost：3000 /＃！/为什么我在localhost链接中得到“＃！/”。

如何使用tweepy流式传输来自指定用户的推文（仅在该用户发布推文时流式传输）

TreeMap中的自定义排序

TYPO3：将 Formhandler 添加到新闻扩展

遍历元素数组以每X秒在浏览器上显示

在Ubuntu和Windows中，触摸板有时会滞后。硬件问题？

警告消息：在matrix（unlist（drop.item），ncol = 10，byrow = TRUE）中：数据长度[16]不是列数的倍数[10]>？

无法连接网络并在Ubuntu 14.04中找到eth0

将辅助轴原点与主要轴对齐

我可以ping IPv6但不能ping IPv4

在Jenkins服务器中使用Selenium和Ruby进行的黄瓜测试失败，但在本地计算机中通过

提交html表单时为空

使用C ++ 11将数组设置为零

如果从DB接收到的值为空，则JMeter JDBC调用将返回该值作为参数名称

尝试在Dell XPS13 9360上安装Windows 7时出错

如何在R中转置数据

无法使用 envoy 访问 .ssh/config

未捕获的SyntaxError：带有Ajax帖子的意外令牌u