What happened?
In the experimental Python Dask Runner, the DaskBagWindowedIterator (which handles iterators for apache_beam.transforms.sideinputs.SideInputMap) iterates over a Dask Bag by wrapping it in a Python list().
Calling list(self.bag) implicitly triggers a full compute() on the Dask dataset. This blocking operation materializes the entirety of the side input data into the client's local memory. For large side inputs, this completely bypasses Dask's distributed memory management and results in an Out-Of-Memory (OOM) crash, effectively bottlenecking the scalability of pipelines running on Dask.
The code currently includes an explicit FIXME acknowledging this proof-of-concept behavior, but it remains a silent, critical scalability flaw.
Code Pointers / Steps to Reproduce
The issue is located in sdks/python/apache_beam/runners/dask/transform_evaluator.py within the __iter__ method of the DaskBagWindowedIterator class (lines 91-96):
class DaskBagWindowedIterator:
"""Iterator for `apache_beam.transforms.sideinputs.SideInputMap`"""
bag: db.Bag
window_fn: WindowFn
def __iter__(self):
# FIXME(cisaacstern): list() is likely inefficient, since it presumably
# materializes the full result before iterating over it. doing this for
# now as a proof-of-concept. can we can generate results incrementally?
for result in list(self.bag):
yield get_windowed_value(result, self.window_fn)Impact
Any Apache Beam pipeline using DaskRunner that relies on substantial side inputs will crash with OOM errors as soon as the side input data surpasses the available local RAM on the node where the iterator is evaluated. This severely limits the DaskRunner's ability to process real-world distributed datasets and creates a harsh scalability ceiling.
Proposed Solution
The evaluation should generate results incrementally rather than performing a monolithic evaluation. Potential approaches:
- Partition-based iteration: Utilize Dask's
.map_partitions or .to_delayed() to fetch and yield the underlying data partition-by-partition.
- Generators: Instead of eager computation via
list(), retrieve delayed results asynchronously and yield them to allow the Python garbage collector to free memory between partition iterations.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
In the experimental Python Dask Runner, the
DaskBagWindowedIterator(which handles iterators forapache_beam.transforms.sideinputs.SideInputMap) iterates over a DaskBagby wrapping it in a Pythonlist().Calling
list(self.bag)implicitly triggers a fullcompute()on the Dask dataset. This blocking operation materializes the entirety of the side input data into the client's local memory. For large side inputs, this completely bypasses Dask's distributed memory management and results in an Out-Of-Memory (OOM) crash, effectively bottlenecking the scalability of pipelines running on Dask.The code currently includes an explicit
FIXMEacknowledging this proof-of-concept behavior, but it remains a silent, critical scalability flaw.Code Pointers / Steps to Reproduce
The issue is located in
sdks/python/apache_beam/runners/dask/transform_evaluator.pywithin the__iter__method of theDaskBagWindowedIteratorclass (lines 91-96):Impact
Any Apache Beam pipeline using
DaskRunnerthat relies on substantial side inputs will crash with OOM errors as soon as the side input data surpasses the available local RAM on the node where the iterator is evaluated. This severely limits theDaskRunner's ability to process real-world distributed datasets and creates a harsh scalability ceiling.Proposed Solution
The evaluation should generate results incrementally rather than performing a monolithic evaluation. Potential approaches:
.map_partitionsor.to_delayed()to fetch and yield the underlying data partition-by-partition.list(), retrieve delayed results asynchronously and yield them to allow the Python garbage collector to free memory between partition iterations.Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components