Python多处理池.map对于多个参数

  • 问题:
  • 在Python多处理库中,是否有的变体池.map哪个支持多个参数?在

    text = "test"
    def harvester(text, case):
    X = case[0]
    text+ str(X)

    if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=6)
    case = RAW_DATASET
    pool.map(harvester(text,case),case, 1)
    pool.close()
    pool.join()

  • 答案:
  • 答案取决于版本和情况。对于Python的最新版本(从3.3开始)最普遍的答案是由J.F. Sebastian.1它使用池.星图方法,它接受一系列参数元组。然后,它会自动从每个元组中解压参数并将它们传递给给定的函数:

    import multiprocessing
    from itertools import product

    def merge_names(a, b):
    return '{} & {}'.format(a, b)

    if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with multiprocessing.Pool(processes=3) as pool:
    results = pool.starmap(merge_names, product(names, repeat=2))
    print(results)

    # Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...

    对于Python的早期版本,您需要编写一个helper函数来显式地解包参数。如果要将一起使用,还需要编写一个包装器将转换为上下文管理器。(感谢muon因为他指出了这一点。)

    import multiprocessing
    from itertools import product
    from contextlib import contextmanager

    def merge_names(a, b):
    return '{} & {}'.format(a, b)

    def merge_names_unpack(args):
    return merge_names(*args)

    @contextmanager
    def poolcontext(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)
    yield pool
    pool.terminate()

    if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with poolcontext(processes=3) as pool:
    results = pool.map(merge_names_unpack, product(names, repeat=2))
    print(results)

    # Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...

    在更简单的情况下,使用固定的第二个参数,也可以使用partial,但只能在python2.7+中使用

    import multiprocessing
    from functools import partial
    from contextlib import contextmanager

    @contextmanager
    def poolcontext(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)
    yield pool
    pool.terminate()

    def merge_names(a, b):
    return '{} & {}'.format(a, b)

    if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with poolcontext(processes=3) as pool:
    results = pool.map(partial(merge_names, b='Sons'), names)
    print(results)

    # Output: ['Brown & Sons', 'Wilson & Sons', 'Bartlett & Sons', ...

    1。这其中很大一部分是受到他的回答的启发,而他的回答可能应该被接受。但是,由于这本书停留在顶端,所以最好为将来的读者改进它。