Processing a stream. Must layers be violated?

Posted by Lord Tydus on Programmers See other posts from Programmers or by Lord Tydus
Published on 2012-10-05T01:01:20Z Indexed on 2012/10/05 3:51 UTC
Read the original article Hit count: 172

Filed under:

layers

Theoretical situation:
One trillion foobars are stored in a text file (no fancy databases). Each foobar must have some business logic executed on it. A set of 1 trillion will not fit in memory so the data layer cannot return a big set to the business layer. Instead they are to be streamed in 1 foobar at a time, and have business logic execute on 1 foobar at a time. The stream must be closed when finished.

In order for the stream to be closed the business layer must close the stream (a data operation detail), thus violating the separation of concerns.

Is it possible to incrementally process data without violating layers?

Related posts about layers

Verbose Listing of All Application Layers/Tiers?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've looked at a few sites now, and I'm still struggling to find a complete listing of all the possible layers/tiers you can have in an application. From back in college (1999) I remember the following: Presentation Layer (Views) Application Layer (Controllers) Business Logic Layer (API/Rules) Persistence… >>> More
Architectural Layers and Tips on how to Achieve Architecture Consistency

as seen on Internet.com - Search for 'Internet.com'
Architectural roles can make a project great or set you back to the drawing board. Read on to discover the many architectural roles in your project life cycle and how to achieve consistency throughout. >>> More
Does it make sense to use ORM in Android development?

as seen on Programmers - Search for 'Programmers'
Does it make sense to use an ORM in Android development or is the framework optimized for a tighter coupling between the UI and the DB layer? Background: I've just started with Android development, and my first instinct (coming from a .net background) was to look for a small object-relational mapper… >>> More
How to document/verify consistent layering?

as seen on Programmers - Search for 'Programmers'
I have recently moved to the dark side: I am now a CUSTOMER of software development -- mainly websites. With this new role comes new concerns. As a programmer i know how solid an application becomes when it is properly layered, and I want to use this knowledge in my new job. I don't want business… >>> More
How accurate is "Business logic should be in a service, not in a model"?

as seen on Programmers - Search for 'Programmers'
Situation Earlier this evening I gave an answer to a question on StackOverflow. The question: Editing of an existing object should be done in repository layer or in service? For example if I have a User that has debt. I want to change his debt. Should I do it in UserRepository or in service… >>> More

Developer IT