我正在使用Dask处理数据帧（来自CSV文件），并且正在寻找一种方法来使用map，或apply函数来改进此代码，因为在大型文件中花费的时间太长（我知道嵌套for和使用iterrows()是最糟糕的认为我可以做到）
NAN_VALUES = [-999, "INVALID", -9999]
_all_rows=list()
for index, row in df.iterrows():
_row = list()
for key, value in row.iteritems():
if value in NAN_VALUES or pd.isnull(value):
_row.append(None)
else:
_row.append(apply_transform(key, value))
_all_rows.append(_row)
rows_count += 1
如何使用map_partitions或映射此代码pandas.map？
额外：更多上下文：为了能够应用某些功能，我将NaN值替换为默认值。最后，我需要为每行创建一个列表，将默认值替换为“无”。
1.-原始DF
"name" "age" "money"
---------------------------
"David" NaN 12.345
"Jhon" 22 NaN
"Charles" 30 123.45
NaN NaN NaN
2.-将NaN传递给默认值
"name" "age" "money"
------------------------------
"David" -999 12.345
"Jhon" 22 -9999
"Charles" 30 123.45
"INVALID" -999 -9999
3.-解析到每一行的列表
"name" , "age", "money"
------------------------
["David", None, 12.345]
["Jhon", 22, None]
["Charles", 30, 123.45]
[None, None, None]
我的建议是尝试与熊猫合作，然后尝试将其翻译为dask
pandasimport pandas as pd
import numpy as np
nan = np.nan
df = {'name': {0: 'David', 1: 'John', 2: 'Charles', 3: nan},
'age': {0: nan, 1: 22.0, 2: 30.0, 3: nan},
'money': {0: 12.345, 1: nan, 2: 123.45, 3: nan}}
df = pd.DataFrame(df)
# These are your default values
diz = {"age": -999, "name": "INVALID", "money": -9999}
for k,v in diz.items():
df[k] = df[k].fillna(v)
df.apply(list, axis=1)
0 [David, nan, 12.345]
1 [John, 22.0, nan]
2 [Charles, 30.0, 123.45]
3 [nan, nan, nan]
dtype: object
daskimport pandas as pd
import dask.dataframe as dd
import numpy as np
nan = np.nan
df = {'name': {0: 'David', 1: 'John', 2: 'Charles', 3: nan},
'age': {0: nan, 1: 22.0, 2: 30.0, 3: nan},
'money': {0: 12.345, 1: nan, 2: 123.45, 3: nan}}
df = pd.DataFrame(df)
# These are your default values
diz = {"age": -999, "name": "INVALID", "money": -9999}
# transform to dask dataframe
df = dd.from_pandas(df, npartitions=2)
这和以前完全一样。请注意，dask如果您想查看效果，则应懒惰运行df.compute()
for k,v in diz.items():
df[k] = df[k].fillna(v)
要求您明确声明输出的dtype时，这里的情况有所变化
df.apply(list, axis=1, meta=(None, 'object'))
最终您可以使用map_partitions以下命令
df.map_partitions(lambda x: x.apply(list, axis=1))
备注请考虑，如果你做内存数据拟合并不需要dask和pandas可能会更快。
本文收集自互联网，转载请注明来源。
如有侵权，请联系 [email protected] 删除。
我来说两句