Processing a stream. Must layers be violated?
- by Lord Tydus
Theoretical situation:
One trillion foobars are stored in a text file (no fancy databases). Each foobar must have some business logic executed on it. A set of 1 trillion will not fit in memory so the data layer cannot return a big set to the business layer. Instead they are to be streamed in 1 foobar at a time, and have business logic execute on 1 foobar at a time. The stream must be closed when finished.
In order for the stream to be closed the business layer must close the stream (a data operation detail), thus violating the separation of concerns.
Is it possible to incrementally process data without violating layers?
