on Geeks with Blogs
See other posts from Geeks with Blogs
or by Shaun
Published on Tue, 16 Oct 2012 03:43:20 GMT Indexed on 2012/10/16 23:01 UTC
Read the original article Hit count: 488
What’s the Problem: Dense “Callback” Phobia
Let’s firstly back to my second post in this series. As I mentioned in that post, when we wanted to read some records from SQL Server we need to open the database connection, and then execute the query. In Node.js all IO operation are designed as async callback pattern which means when the operation was done, it will invoke a function which was taken from the last parameter.
For example the database connection opening code would be like this.
And then if we need to query the database the code would be like this. It nested in the previous function.
Assuming if we need to copy some data from this database to another then we need to open another connection and execute the command within the function under the query function.
This is just an example. In the real project the logic would be more complicated. This means our application might be messed up and the business process will be fragged by many callback functions. I would like call this “Dense Callback Phobia”.
This might be a challenge how to make code straightforward and easy to read, something like below.
What’s the Problem: Sync-styled Async Programming
Similar as the previous problem, the callback-styled async programming model makes the upcoming operation as a part of the current operation, and mixed with the error handling code. So it’s very hard to understand what on earth this code will do.
And since Node.js utilizes non-blocking IO mode, we cannot invoke those operations one by one, as they will be executed concurrently.
For example, in this post when I tried to copy the records from Windows Azure SQL Database (a.k.a. WASD) to Windows Azure Table Storage, if I just insert the data into table storage one by one and then print the “Finished” message, I will see the message shown before the data had been copied. This is because all operations were executed at the same time.
In order to make the copy operation and print operation executed synchronously I introduced a module named “async” and the code was changed as below.
It ensured that the “Finished” message will be printed when all table entities had been inserted. But it cannot promise that the records will be inserted in sequence. It might be another challenge to make the code looks like in sync-style?
How “Wind” Helps
Now let’s create a very simple Node.js application as the example. This application will take some website URLs from the command arguments and tried to retrieve the body length and print them in console. Then at the end print “Finish”. I’m going to use “request” module to make the HTTP call simple so I also need to install by the command “npm install request”. The code would be like this.
Let’s execute this application. (I made them in multi-lines for better reading.)
Below is the output.
As you can see the finish message was printed at the beginning, and the pages’ length retrieved in a different order than we specified. This is because in this code the request command, console logging command are executed asynchronously and concurrently.
Now let’s introduce “Wind” to make them executed in order, which means it will request the websites one by one, and print the message at the end.
First of all we need to import the “Wind” package and make sure the there’s only one global variant named “Wind”, and ensure it’s “Wind” instead of “wind”.
Next, we need to tell “Wind” which code will be executed asynchronously so that “Wind” can control the execution process. In this case the “request” operation executed asynchronously so we will create a “Task” by using a build-in helps function in “Wind” named Wind.Async.Task.create.
The code above created a “Task” from the original request calling code. In “Wind” a “Task” means an operation will be finished in some time in the future. A “Task” can be started by invoke its start() method, but no one knows when it actually will be finished.
The Wind.Async.Task.create helped us to create a task. The only parameter is a function where we can put the actual operation in, and then notify the task object it’s finished successfully or failed by using the complete() method.
In the code above I invoked the request method. If it retrieved the response successfully I set the status of this task as “success” with the URL and body length. If it failed I set this task as “failure” and pass the error out.
Next, we will change the main() function. In “Wind” if we want a function can be controlled by Wind we need to mark it as “async”. This should be done by using the code below.
When the application is running, Wind will detect “eval(Wind.compile(“async”, function” and generate an anonymous code from the body of this original function. Then the application will run the anonymous code instead of the original one.
In our example the main function will be like this.
As you can see, when I tried to request the URL I use a new command named “$await”. It tells Wind, the operation next to $await will be executed asynchronously, and the main thread should be paused until it finished (or failed). So in this case, my application will be pause when the first response was received, and then print its body length, then try the next one. At the end, print the finish message.
Finally, execute the main function. The full code would be like this.
Run our new application. At the beginning we will see the compiled and generated code by Wind. Then we can see the pages were requested one by one, and at the end the finish message was printed.
Below is the code Wind generated for us. As you can see the original code, the output code were shown.
How Wind Works
Someone may raise a big concern when you find I utilized “eval” in my code. Someone may assume that Wind utilizes “eval” to execute some code dynamically while “eval” is very low performance. But I would say, Wind does NOT use “eval” to run the code. It only use “eval” as a flag to know which code should be compiled at runtime.
Since the code generation was done at the beginning of the application was started, in the future no matter how long our application runs and how many times the async function was invoked, it will use the generated code, no need to generate again. So there’s no significant performance hurt when using Wind.
Wind in My Previous Demo
Let’s adopt Wind into one of my previous demonstration and to see how it helps us to make our code simple, straightforward and easy to read and understand.
In this post when I implemented the functionality that copied the records from my WASD to table storage, the logic would be like this.
1, Open database connection.
2, Execute a query to select all records from the table.
3, Recreate the table in Windows Azure table storage.
4, Create entities from each of the records retrieved previously, and then insert them into table storage.
5, Finally, show message as the HTTP response.
But as the image below, since there are so many callbacks and async operations, it’s very hard to understand my logic from the code.
Now let’s use Wind to rewrite our code. First of all, of course, we need the Wind package.
Then we need to include the package files into project and mark them as “Copy always”.
Add the Wind package into the source code. Pay attention to the variant name, you must use “Wind” instead of “wind”.
Now we need to create some async functions by using Wind. All async functions should be wrapped so that it can be controlled by Wind which are open database, retrieve records, recreate table (delete and create) and insert entity in table.
Below are these new functions. All of them are created by using Wind.Async.Task.create.
Then in order to use these functions we will create a new function which contains all steps for data copying.
Let’s execute steps one by one with the “$await” keyword introduced by Wind so that it will be invoked in sequence. First is to open the database connection.
Then retrieve all records from the database connection.
After recreated the table, we need to create the entities and insert them into table storage.
Finally, send response back to the browser.
If we compared with the previous code we will find now it became more readable and much easy to understand. It’s very easy to know what this function does even though without any comments.
When user go to URL “/was/copyRecords” we will execute the function above. The code would be like this.
And below is the logs printed in local compute emulator console. As we can see the functions executed one by one and then finally the response back to me browser.
Scaffold Functions in Wind
Wind provides not only the async flow control and compile functions, but many scaffold methods as well. We can build our async code more easily by using them. I’m going to introduce some basic scaffold functions here.
In the code above I created some functions which wrapped from the original async function such as open database, create table, etc.. All of them are very similar, created a task by using Wind.Async.Task.create, return error or result object through Task.complete function. In fact, Wind provides some functions for us to create task object from the original async functions.
If the original async function only has a callback parameter, we can use Wind.Async.Binding.fromCallback method to get the task object directly. For example the code below returned the task object which wrapped the file exist check function.
In Node.js a very popular async function pattern is that, the first parameter in the callback function represent the error object, and the other parameters is the return values. In this case we can use another build-in function in Wind named Wind.Async.Binding.fromStandard. For example, the open database function can be created from the code below.
When I was testing the scaffold functions under Wind.Async.Binding I found for some functions, such as the Azure SDK insert entity function, cannot be processed correctly. So I personally suggest writing the wrapped method manually.
Another scaffold method in Wind is the parallel tasks coordination. In this example, the steps of open database, retrieve records and recreated table should be invoked one by one, but it can be executed in parallel when copying data from database to table storage. In Wind there’s a scaffold function named Task.whenAll which can be used here.
Task.whenAll accepts a list of tasks and creates a new task. It will be returned only when all tasks had been completed, or any errors occurred. For example in the code below I used the Task.whenAll to make all copy operation executed at the same time.
Besides the task creation and coordination, Wind supports the cancellation solution so that we can send the cancellation signal to the tasks. It also includes exception solution which means any exceptions will be reported to the caller function.
In this post I introduced a Node.js module named Wind, which created by my friend Jeff Zhao. As you can see, different from other async library and framework, adopted the idea from F# and C#, Wind utilizes runtime code generation technology to make it more easily to write async, callback-based functions in a sync-style way. By using Wind there will be almost no callback, and the code will be very easy to understand.
Currently Wind is still under developed and improved. There might be some problems but the author, Jeff, should be very happy and enthusiastic to learn your problems, feedback, suggestion and comments. You can contact Jeff by
- Email: [email protected]
Source code can be download here.
Hope this helps,
© Geeks with Blogs or respective owner