TfDif和自定义功能之间的FeatureUnion上的KeyError

RogB 发表于 Dev

罗格

我正在尝试创建一个模型，在该模型中，我将在文本列上使用TfidfVectorizer，同时还要在文本上使用其他数据的其他几个列。下面的代码再现了我正在尝试执行的操作以及出现的错误。

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import FeatureUnion
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction import DictVectorizer
from sklearn.naive_bayes import BernoulliNB

class ParStats(BaseEstimator, TransformerMixin):

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        print(X[0])
        return [{'feat_1': x['feat_1'],
                 'feat_2': x['feat_2']}
                for x in X]

class ItemSelector(BaseEstimator, TransformerMixin):

    def __init__(self, key):
        self.key = key

    def fit(self, x, y=None):
        return self

    def transform(self, data_dict):
        return data_dict[self.key]

def feature_union_test():

    # create test data frame
    test_data = {
        'text': ['And the silken, sad, uncertain rustling of each purple curtain',
                 'Thrilled me filled me with fantastic terrors never felt before',
                 'So that now, to still the beating of my heart, I stood repeating',
                 'Tis some visitor entreating entrance at my chamber door',
                 'Some late visitor entreating entrance at my chamber door',
                 'This it is and nothing more'],
        'feat_1': [4, 7, 10, 7, 4, 6],
        'feat_2': [1, 5, 5, 1, 1, 10],
        'ignore': [1, 1, 1, 0, 0, 0]
    }
    test_df = pd.DataFrame(data=test_data)
    y_train = test_df['ignore'].values.astype('int')

    # Feature Union Pipeline
    pipeline = FeatureUnion([

                ('text', Pipeline([
                    ('selector', ItemSelector(key='text')),
                    ('tfidf', TfidfVectorizer(max_df=0.5)),
                ])),

                ('parstats', Pipeline([
                    ('stats', ParStats()),
                    ('vect', DictVectorizer()),
                ]))

            ])

    tfidf = pipeline.fit_transform(test_df)

    # fits Naive Bayes
    clf = BernoulliNB().fit(tfidf, y_train)

feature_union_test()

运行此命令时，出现以下错误消息：

Traceback (most recent call last):
  File "C:\Users\Rogerio\Python VENV\lib\site-packages\pandas\core\indexes\base.py", line 3064, in get_loc
    return self._engine.get_loc(key)
  File "pandas\_libs\index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 0

我尝试了管道的几次不同迭代，但总是会遇到某种错误，因此显然我缺少了一些东西。我究竟做错了什么？

Vivek Kumar

好。因此，在评论中进行讨论之后，这就是您的问题陈述。

你想通过柱feat_1，feat_2用的TFIDF一起text列到你的模型毫升。

因此，您唯一需要做的就是：

# Feature Union Pipeline
pipeline = FeatureUnion([('text', Pipeline([('selector', ItemSelector(key='text')),
                                            ('tfidf', TfidfVectorizer(max_df=0.5)),
                                           ])),
                         ('non_text', ItemSelector(key=['feat_1', 'feat_2']))
                        ])

tfidf = pipeline.fit_transform(test_df)

默认值ItemSelector可用于一次选择多个功能，这些功能将附加到text部分功能并集的tfidf数据返回的最后。

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。