在已部署的服务器上运行AWS EMR客户端时,此错误突然出现问题。这不会在本地发生,并且可以正常运行。基本上,我有一个EMR客户端,可用于构建和执行以下步骤:
class EMRClient(emrClusterId:String) extends LazyLogging{
val accessKey = ...// access key
val secretKey = ...//secret key
val credentials = new BasicAWSCredentials(accessKey, secretKey)
val REGION = <my region>
println(">>>>>>>>>>>>>>>>>>>>Initializing EMR client for clusterId " + emrClusterId + " . The region is " + REGION)
val emr = AmazonElasticMapReduceClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withRegion(REGION)
.build()
def executeHQLStep(s3ScriptPath:String, stepName:String, args:String = ""): AddJobFlowStepsResult= {
val hqlScriptStep = buildHQLStep(hqlScriptPath, stepName, args)
val stepSet = new java.util.HashSet[StepConfig]()
//stepSet.add(enableDebugging)
stepSet.add(hqlScriptStep)
executeJobFlowSteps(stepSet)
}
/**
* Builds a StepConfig to be executed in a job flow for a given .hql file from S3
* @param hqlScriptPath the location in S3 of the script file containing the script to run
* @param args optional field for arguments for hive script.
* @param stepName An identifier to give to EMR to name your Step
* @return
*/
private def buildHQLStep(hqlScriptPath:String, stepName:String, args:String= ""): StepConfig = {
new StepConfig()
.withName(stepName)
.withActionOnFailure(ActionOnFailure.CANCEL_AND_WAIT)
.withHadoopJarStep(stepFactory.newRunHiveScriptStep(hqlScriptPath, args))
}
private def executeJobFlowSteps(steps: java.util.Set[StepConfig]): AddJobFlowStepsResult = {
emr.addJobFlowSteps(new AddJobFlowStepsRequest()
.withJobFlowId(emrClusterId)
.withSteps(steps)) // where the error is thrown
}
}
但是,当在服务器上实例化此类println
时,顶部的所有语句都不可见,executeJobFlowSteps
将调用我的方法并引发此错误:
java.lang.NoSuchFieldError:com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.executeAddJobFlowSteps(AmazonElasticMapReduceClient.java:439)处的SIGNING_REGION com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.addJobFlowSteps(AmazonElasticMapReduceClient.addJobFlowSteps)在emrservices.EMRClient.executeHQLStep(EMRClient.scala:44)上的executeJobFlowSteps(EMRClient.scala:64)
这个项目由几个项目组成,我遇到的类似问题都说这与它们的AWS依赖关系有关,但是所有项目在build.sbt的库依赖项中都有: "com.amazonaws" % "aws-java-sdk" % "1.11.286"
对这个问题有什么想法吗?
看来您正在将1.11.286版本的aws-java-sdk与旧版本(1.11.149)的aws-java-sdk-core混合在一起。较新的客户端正在使用添加到核心模块的新字段,但是由于核心模块已过期,因此您会看到没有此类字段错误。您可以确保所有依赖项彼此同步吗?
本文收集自互联网,转载请注明来源。
如有侵权,请联系 [email protected] 删除。
我来说两句