Python在使用函数时不能接受输入

oo92

我正在研究 Kaggle 上托管的房价问题。在构建我的模型时,我认为在测试集上重用我一直用于训练数据集的一些代码是有意义的,因此我将执行相互操作的代码合并到一个函数定义中。在此函数中,我正在处理缺失值并使用其返回值执行单热编码并将其用于随机森林回归。但是,它抛出以下错误:

Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 56, in <module>
    sel.fit(x_train, y_train)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\feature_selection\from_model.py", line 196, in fit
    self.estimator_.fit(X, y, **fit_params)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\forest.py", line 249, in fit
    X = check_array(X, accept_sparse="csc", dtype=DTYPE)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 542, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
    raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

在使用相同的代码而不将其组织成函数时,我没有遇到这个问题。def feature_selection_and_engineering(df)是有问题的功能。以下是我的全部代码:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")

def feature_selection_and_engineering(df):
    # Creating a series of how many NaN's are in each column
    nan_counts = df.isna().sum()

    # Creating a template list
    nan_columns = []

    # Iterating over the series and if the value is more than 0 (i.e there are some NaN's present)
    for i in range(0, len(nan_counts)):
        if nan_counts[i] > 0:
            nan_columns.append(df.columns[i])

    # Iterating through all the columns which are known to have NaN's
    for i in nan_columns:
        if df[nan_columns][i].dtypes == 'float64':
            df[i] = df[i].fillna(df[i].mean())
        elif df[nan_columns][i].dtypes == 'object':
            df[i] = df[i].fillna('XX')

    # Creating a template list
    categorical_columns = []

    # Iterating across all the columns,
    # checking if they're of the object datatype and if they are, appending them to the categorical list
    for i in range(0, len(df.dtypes)):
        if df.dtypes[i] == 'object':
            categorical_columns.append(df.columns[i])

    return categorical_columns

# take one-hot encoding
OHE_sdf = pd.get_dummies(feature_selection_and_engineering(train))

# drop the old categorical column from original df
train.drop(columns = feature_selection_and_engineering(train), axis = 1, inplace = True)

# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)

# Dividing the training dataset into train/test sets with the test size being 20% of the overall dataset.
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.2, random_state = 42)

randomForestRegressor = RandomForestRegressor(n_estimators=1000)

# Invoking the Random Forest Classifier with a 1.25x the mean threshold to select correlating features
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)

selected = sel.get_support()

# linearRegression.fit(x_train, y_train)
randomForestRegressor.fit(x_train, y_train)

# Assigning the accuracy of the model to the variable "accuracy"
accuracy = randomForestRegressor.score(x_train, y_train)

# Predicting for the data in the test set
predictions = randomForestRegressor.predict(feature_selection_and_engineering(test))

# Writing the predictions to a new CSV file
submission = pd.DataFrame({'Id': test['PassengerId'], 'SalePrice': predictions})
filename = 'Boston-Submission.csv'
submission.to_csv(filename, index=False)

print(accuracy*100, "%")

新错误

    Traceback (most recent call last):
  File "/home/onur/Documents/Boston-Kaggle/Model.py", line 76, in <module>
    x_train, encoder = feature_selection_and_engineering(x_train)
  File "/home/onur/Documents/Boston-Kaggle/Model.py", line 57, in feature_selection_and_engineering
    encoder = train_one_hot_encoder(df, categorical_columns)
  File "/home/onur/Documents/Boston-Kaggle/Model.py", line 30, in train_one_hot_encoder
    return enc.fit(categorical_df)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 493, in fit
    self._fit(X, handle_unknown=self.handle_unknown)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 80, in _fit
    X_list, n_samples, n_features = self._check_X(X)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 67, in _check_X
    force_all_finite=needs_validation)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 542, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 60, in _assert_all_finite
    raise ValueError("Input contains NaN")
ValueError: Input contains NaN
凯尔

重用代码是个好主意,但要注意将代码放入函数时变量的范围如何变化。

你得到的错误是因为NaN你输入到随机森林的数组中值。在您的feature_engineering_and_selection()函数中,您正在删除NaN值,但df永远不会从函数中返回,因此df模型中使用的是原始的、未修改的。

我建议将您的feature_engineering_and_selection()功能拆分为不同的组件。这里我做了一个只删除NaNs的函数

# Iterates through the columns and fixes any NaNs
def remove_nan(df):
    replace_dict = {}

    for col in df.columns:

        # If there are any NaN values in this column
        if pd.isna(df[col]).any():

            # Replace NaN in object columns with 'N/A'
            if df[col].dtypes == 'object':
                replace_dict[col] = 'N/A'

            # Replace NaN in float columns with 0
            elif df[col].dtypes == 'float64':
                replace_dict[col] = 0

    df = df.fillna(replace_dict)

    return df

我建议NaN用 0 而不是平均值填充数值。对于此数据,有 3 个数值列具有 nan 值:(LotFrontage连接到财产的街道英尺)、MasVnrArea(砌体贴面面积)、GarageYrBlt(建造的车库年份)。如果没有车库,那么就没有建造车库年份,因此将年份设为 0 而不是平均年份等是有意义的。

还有一些工作需要使用您设置的一个热编码器来完成。创建单热编码可能很棘手,因为训练数据和测试数据需要具有相同的列。如果你有以下训练和测试数据

火车

| House Type |
| ---------- |
| Mansion    |
| Ranch      |

测试

| House Type |
| ---------- |
| Mansion    |
| Duplex     |

然后,如果使用pd.get_dummies()train 列[house_type_mansion, house_type_ranch]和 test 列将是[house_type_mansion, house_type_duplex],这将不起作用。但是,使用 sklearn,您可以将单热编码器安装到您的训练数据中。转换测试数据集时,它将创建与训练数据集相同的列。handle_unknown参数将告诉编码器是做什么用duplex的测试集,无论是ignoreerror

# Fits an sklearn one hot encoder
def train_one_hot_encoder(df, categorical_columns):
    # take one-hot encoding of categorical columns
    categorical_df = df[categorical_columns]
    enc = OneHotEncoder(sparse=False, handle_unknown='ignore')
    return enc.fit(categorical_df)

为了结合分类和非分类数据,我再次建议创建一个单独的函数

# One hot encodes the given dataframe
def one_hot_encode(df, categorical_columns, encoder):
    # Get dataframe with only categorical columns
    categorical_df = df[categorical_columns]
    # Get one hot encoding
    ohe_df = pd.DataFrame(encoder.transform(categorical_df), columns=encoder.get_feature_names())
    # Get float columns
    float_df = df.drop(categorical_columns, axis=1)
    # Return the combined array
    return pd.concat([float_df, ohe_df], axis=1)

最后,您的feature_engineering_and_selection()函数可以调用所有这些函数。

def feature_selection_and_engineering(df, encoder=None):
    df = remove_nan(df)
    categorical_columns = get_categorical_columns(df)
    # If there is no encoder, train one
    if encoder == None:
        encoder = train_one_hot_encoder(df, categorical_columns)
    # Encode Data
    df = one_hot_encode(df, categorical_columns, encoder)
    # Return the encoded data AND encoder
    return df, encoder

为了使代码运行,我必须解决一些问题,我在此处的要点中包含了整个修改后的脚本https://gist.github.com/kylelrichards11/6be90d92a7dd6a5cc9a5290dae3ff94e

本文收集自互联网,转载请注明来源。

如有侵权,请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章