I'll admit right up front that the following is a pretty terrible description of what I want to do. Apologies in advance. Please ask questions to help me explain. :-)
I've written ETLs in other languages that consist of individual operations that look something like:
// in class CountOperation
IEnumerable<Row> Execute(IEnumerable<Row> rows) {
var count = 0;
foreach (var row in rows) {
row["record number"] = count++;
yield return row;
}
}
Then you string a number of these operations together, and call The Dispatcher, which is responsible for calling Operations and pushing data between them.
I'm trying to do something similar in Common Lisp, and I want to use the same basic structure, i.e., each operation is defined like a normal function that inputs a list and outputs a list, but lazily.
I can define-condition a condition (have-value) to use for yield-like behavior, and I can run it in a single loop, and it works great. I'm defining the operations the same way, looping through the inputs:
(defun count-records (rows)
(loop for count from 0
for row in rows
do (signal 'have-value :value `(:count ,count @,row))))
The trouble is if I want to string together several operations, and run them. My first attempt at writing a dispatcher for these looks something like:
(let ((next-op ...)) ;; pick an op from the set of all ops
(loop
(handler-bind
((have-value (...))) ;; records output from operation
(setq next-op ...) ;; pick a new next-op
(call next-op)))
But restarts have only dynamic extent: each operation will have the same restart names. The restart isn't a Lisp object I can store, to store the state of a function: it's something you call by name (symbol) inside the handler block, not a continuation you can store for later use.
Is it possible to do something like I want here? Or am I better off just making each operation function explicitly look at its input queue, and explicitly place values on the output queue?
