I am trying to figure out how to parallelize nested loops with prefect, which require an inner fan out/in within the outer fan out/in.
If I use map() on the outer loop, then pass some of the results from outer loop calculations also using map() to inner loop in unmapped(), will this create an inner fan out? Then how do I do a "partial reduce" to use inner loop results in the outer loop?
A pointer or example would be much appreciated.
Linking a conversation with OP about this on the Prefect Github for posterity: https://github.com/PrefectHQ/prefect/issues/2786
My tl;dr of that discussion for convenience: as of time of writing (Prefect version 0.12.0) dynamic pipelines with fan-out/fan-in is not supported by the existing map syntax. Today's map syntax only supports parallel pipelines of a constant cardinality. Instead, you would need to introduce a reduce step that reconfigures your outputs so they can be submitted to a new map.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments