具有可迭代和多个参数的Python多处理

亨利·桑顿

使用多重处理，我想传递一个可迭代的多个参数：

a）运行在n_core cpu上的函数b）一次产生（或返回）n_core结果c）以任何完成顺序

from multiprocessing import Pool 

def func(iterable, args):
    this, that, other = args[0], args[1], args[2]

    for s in iterable:
        return ' '.join([s, this, that, other])        

def main():
    iterable = ['abc', 'bcd', 'cde', 'def', 'efg', 'fgh', 'ghi', 'hij']
    args = ['this', 'that', 'other']
    n_core = 2

    p = Pool(n_core)
    for r in p.imap_unordered(func, iterable, args):
        print(r)

if __name__ == '__main__':
    main()

预期结果是：

"abc this that other"
"bcd this that other"
"cde this that other"
"def this that other" 
"efg this that other" 
"fgh this that other"
"ghi this that other" 
"hij this that other"

使这项工作正确的方法是什么？

其次，并发.futures.ProcessPoolExecutor是否可以解决此问题？

大卫·卡伦

您可以创建一个new_iterable将的值iterable与结合在一起的args：

from multiprocessing import Pool

def func(args):
    iterable, this, that, other = args[0], args[1][0], args[1][1], args[1][2]
    return ' '.join([iterable, this, that, other])

def main():
    iterable = ['abc', 'bcd', 'cde', 'def', 'efg', 'fgh', 'ghi', 'hij']
    args = ['this', 'that', 'other']
    new_iterable = ([x, args] for x in iterable)
    n_core = 2

    p = Pool(n_core)
    for r in p.imap_unordered(func, new_iterable):
        print(r)

if __name__ == '__main__':
    main()

输出量

abc this that other
bcd this that other
cde this that other
def this that other
efg this that other
fgh this that other
ghi this that other
hij this that other

该解决方案使用生成器表达式来创建新的可迭代项iterable，该可迭代项将来自的条目与所需的合并args。您还可以使用生成器函数执行相同的操作。

更新：我进行了修改，func以产生您在评论中提到并添加到您的问题的预期结果。

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。