Python多处理竞争条件

马克西姆·米哈伊洛夫(Maxim Mikhaylov)

concurrent.futures在读取多个文本文件时发现了一个奇怪的错误

这是一个可重现的小示例:

import os
import concurrent.futures

def read_file(file):
    with open(os.path.join(data_dir, file),buffering=1000) as f:
        for row in f:
            try:
                print(row)
            except Exception as e:
                print(str(e))

if __name__ == '__main__':
    data_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data'))
    files = ['file1', 'file2']
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for file,_ in zip(files,executor.map(read_file,files)):
            pass    

file1并且file2在任意的文本文件data目录。

我收到以下错误(基本上是进程尝试data_dir在分配变量之前读取变量):

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\my_username\AppData\Local\Continuum\Anaconda3\lib\concurrent\futures\process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "C:\Users\my_username\AppData\Local\Continuum\Anaconda3\lib\concurrent\futures\process.py", line 153, in _process_chunk
    return [fn(*args) for args in chunk]
  File "C:\Users\my_username\AppData\Local\Continuum\Anaconda3\lib\concurrent\futures\process.py", line 153, in <listcomp>
    return [fn(*args) for args in chunk]
  File "C:\Users\my_username\Downloads\example.py", line 5, in read_file
    with open(os.path.join(data_dir, file),buffering=1000) as f:
NameError: name 'data_dir' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "example.py", line 16, in <module>
    for file,_ in zip(files,executor.map(read_file,files)):
  File "C:\Users\my_username\AppData\Local\Continuum\Anaconda3\lib\concurrent\futures\_base.py", line 556, in result_iterator
    yield future.result()
  File "C:\Users\my_username\AppData\Local\Continuum\Anaconda3\lib\concurrent\futures\_base.py", line 405, in result
    return self.__get_result()
  File "C:\Users\my_username\AppData\Local\Continuum\Anaconda3\lib\concurrent\futures\_base.py", line 357, in __get_result
    raise self._exception
NameError: name 'data_dir' is not defined

如果将data_dir分配放在if __name__ == '__main__':之前,则不会出现此错误,并且代码将按预期执行。

是什么导致此错误?显然,data_dir在两种情况下都应先进行分配,然后再进行任何异步调用。

J·安德森

fork()在Windows上不可用,因此python用于spawn启动新进程,这将启动新的python解释器进程,不会共享任何内存,但是python会尝试在新进程中重新创建辅助函数环境,这就是模块级变量起作用的原因。有关详细信息,请参阅doc

本文收集自互联网,转载请注明来源。

如有侵权,请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章