Why do I get a type error in model.predictOnValues when I try the official example of Streaming Kmeans Clustering of Apache Spark?

Emre Sevinç

I'm trying the Streaming Clustering example code at the end of the official guide, but I get a type error. Here is my code:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.clustering.StreamingKMeans

object Kmeans {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local[2]").setAppName("kmeans")
    val ssc = new StreamingContext(conf, Seconds(3))

    val trainingData = ssc.textFileStream("training").map(Vectors.parse)
    val testData = ssc.textFileStream("test").map(LabeledPoint.parse)

    val numDimensions = 3
    val numClusters = 2
    val model = new StreamingKMeans()
      .setK(numClusters)
      .setDecayFactor(1.0)
      .setRandomCenters(numDimensions, 0.0)

    model.trainOn(trainingData)
    model.predictOnValues(testData).print()

    ssc.start()
    ssc.awaitTermination()
  }
}

But when I run

sbt package

I get the following error:

[error]  found   : org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
[error]  required: org.apache.spark.streaming.dstream.DStream[(?, org.apache.spark.mllib.linalg.Vector)]
[error]     model.predictOnValues(testData).print()
[error]                           ^
[error] one error found
[error] (compile:compile) Compilation failed
Jean Logeart

You need to map testData: DStream[LabeledPoint] to a DStream[(K, Vector)]:

model.predictOnValues(testData.map(lp => (lp.label, lp.features))).print()

You can find the complete example here: StreamingKMeansExample.scala

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Why do I get an error when I try to save mongoose model?

Why do I get a compilation error when I try to have two methods with the same name and parameter type?

Why do I get an error for "__CrtGetFileInformationByHandleEx " when I try to compile

Why do I get error when try to convert Carbon to DateTime?

Why Do I get an error when I try to get tkinter slider value?

Why do I get a NoClassDefFound error when I try to save my test plan?

Why do I get error when I try to alert the length of the passed array to the function?

Why do I get an error when I try to see my Text using useState?

Why do I get 'undefined' error when I try to read session atrribute from Controller

Why do I get an error when I try to add a dependent task to one created by the C plugin?

Why do I get an "invalid values" error when I try to access the network settings on Eclipse Mars?

Gitlab CI: why do I get an error when I try to define the pages stage?

Why do I see `503 Backend Error` when I try to get a Cloud Storage service account?

Why do i get this error when I try installing autopep8 using pip?

why do i get exception error when i try output array

Why do I get the error when I try to dereference a pointer to an array?

Why do i get this error when I try to perform some logical operation on dataframes?

Why do I get this error when I try to load the component in VueJS?

Why do I get this error in my logcat when I try to post JSON data?

Why do I get an error (Notice: Undefined index) when I try to echo a value from my database?

Why do I get a syntax error when I try to print a nested hash that has keys containing colons?

Why do I get an error message of NoneType object is not callable when I try to call this function?

Why do I get an Error when I try to rename a table after FROM?

why do i get an error when i try to use my models?

Why I get error when I try to declare local variable?

Why I get error when I try to create stored procedure?

Why I get error when I try to install pyaudio?

Why I get an error when I try to upload a picture

Why do I get a type inference error in this example and how to do it better?

TOP Ranking

  1. 1

    Failed to listen on localhost:8000 (reason: Cannot assign requested address)

  2. 2

    pump.io port in URL

  3. 3

    How to import an asset in swift using Bundle.main.path() in a react-native native module

  4. 4

    Loopback Error: connect ECONNREFUSED 127.0.0.1:3306 (MAMP)

  5. 5

    Compiler error CS0246 (type or namespace not found) on using Ninject in ASP.NET vNext

  6. 6

    BigQuery - concatenate ignoring NULL

  7. 7

    Spring Boot JPA PostgreSQL Web App - Internal Authentication Error

  8. 8

    ggplotly no applicable method for 'plotly_build' applied to an object of class "NULL" if statements

  9. 9

    ngClass error (Can't bind ngClass since it isn't a known property of div) in Angular 11.0.3

  10. 10

    How to remove the extra space from right in a webview?

  11. 11

    Change dd-mm-yyyy date format of dataframe date column to yyyy-mm-dd

  12. 12

    Jquery different data trapped from direct mousedown event and simulation via $(this).trigger('mousedown');

  13. 13

    maven-jaxb2-plugin cannot generate classes due to two declarations cause a collision in ObjectFactory class

  14. 14

    java.lang.NullPointerException: Cannot read the array length because "<local3>" is null

  15. 15

    How to use merge windows unallocated space into Ubuntu using GParted?

  16. 16

    flutter: dropdown item programmatically unselect problem

  17. 17

    Pandas - check if dataframe has negative value in any column

  18. 18

    Nuget add packages gives access denied errors

  19. 19

    Can't pre-populate phone number and message body in SMS link on iPhones when SMS app is not running in the background

  20. 20

    Generate random UUIDv4 with Elm

  21. 21

    Client secret not provided in request error with Keycloak

HotTag

Archive