如何在Apache Spark中执行LabelEncoding或分类值

我们正在开发sparkit-learn，旨在在PySpark上提供scikit-learn功能和API。您可以通过以下方式使用SparkLabelEncoder：

$ pip install sparkit-learn

>>> from splearn.preprocessing import SparkLabelEncoder
>>> from splearn import BlockRDD
>>>
>>> data = ["paris", "paris", "tokyo", "amsterdam"]
>>> y = BlockRDD(sc.parallelize(data))
>>>
>>> le = SparkLabelEncoder()
>>> le.fit(y)
>>> le.classes_
array(['amsterdam', 'paris', 'tokyo'],
      dtype='|S9')
>>>
>>> test = ["tokyo", "tokyo", "paris"]
>>> y_test = BlockRDD(sc.parallelize(test))
>>>
>>> le.transform(y_test).toarray()
array([2, 2, 1])
>>>
>>> test = [2, 2, 1]
>>> y_test = BlockRDD(sc.parallelize(test))
>>>
>>> le.inverse_transform(y_test).toarray()
array(['tokyo', 'tokyo', 'paris'],
      dtype='|S9')