Node.js Adventure - Storage Services and Service Runtime
- by Shaun
When I described on how to host a Node.js application on Windows Azure, one of questions might be raised about how to consume the vary Windows Azure services, such as the storage, service bus, access control, etc.. Interact with windows azure services is available in Node.js through the Windows Azure Node.js SDK, which is a module available in NPM. In this post I would like to describe on how to use Windows Azure Storage (a.k.a. WAS) as well as the service runtime. Consume Windows Azure Storage Let’s firstly have a look on how to consume WAS through Node.js. As we know in the previous post we can host Node.js application on Windows Azure Web Site (a.k.a. WAWS) as well as Windows Azure Cloud Service (a.k.a. WACS). In theory, WAWS is also built on top of WACS worker roles with some more features. Hence in this post I will only demonstrate for hosting in WACS worker role. The Node.js code can be used when consuming WAS when hosted on WAWS. But since there’s no roles in WAWS, the code for consuming service runtime mentioned in the next section cannot be used for WAWS node application. We can use the solution that I created in my last post. Alternatively we can create a new windows azure project in Visual Studio with a worker role, add the “node.exe” and “index.js” and install “express” and “node-sqlserver” modules, make all files as “Copy always”. In order to use windows azure services we need to have Windows Azure Node.js SDK, as knows as a module named “azure” which can be installed through NPM. Once we downloaded and installed, we need to include them in our worker role project and make them as “Copy always”. You can use my “Copy all always” tool mentioned in my last post to update the currently worker role project file. You can also find the source code of this tool here. The source code of Windows Azure SDK for Node.js can be found in its GitHub page. It contains two parts. One is a CLI tool which provides a cross platform command line package for Mac and Linux to manage WAWS and Windows Azure Virtual Machines (a.k.a. WAVM). The other is a library for managing and consuming vary windows azure services includes tables, blobs, queues, service bus and the service runtime. I will not cover all of them but will only demonstrate on how to use tables and service runtime information in this post. You can find the full document of this SDK here. Back to Visual Studio and open the “index.js”, let’s continue our application from the last post, which was working against Windows Azure SQL Database (a.k.a. WASD). The code should looks like this. 1: var express = require("express");
2: var sql = require("node-sqlserver");
3:
4: var connectionString = "Driver={SQL Server Native Client 10.0};Server=tcp:ac6271ya9e.database.windows.net,1433;Database=synctile;Uid=shaunxu@ac6271ya9e;Pwd={PASSWORD};Encrypt=yes;Connection Timeout=30;";
5: var port = 80;
6:
7: var app = express();
8:
9: app.configure(function () {
10: app.use(express.bodyParser());
11: });
12:
13: app.get("/", function (req, res) {
14: sql.open(connectionString, function (err, conn) {
15: if (err) {
16: console.log(err);
17: res.send(500, "Cannot open connection.");
18: }
19: else {
20: conn.queryRaw("SELECT * FROM [Resource]", function (err, results) {
21: if (err) {
22: console.log(err);
23: res.send(500, "Cannot retrieve records.");
24: }
25: else {
26: res.json(results);
27: }
28: });
29: }
30: });
31: });
32:
33: app.get("/text/:key/:culture", function (req, res) {
34: sql.open(connectionString, function (err, conn) {
35: if (err) {
36: console.log(err);
37: res.send(500, "Cannot open connection.");
38: }
39: else {
40: var key = req.params.key;
41: var culture = req.params.culture;
42: var command = "SELECT * FROM [Resource] WHERE [Key] = '" + key + "' AND [Culture] = '" + culture + "'";
43: conn.queryRaw(command, function (err, results) {
44: if (err) {
45: console.log(err);
46: res.send(500, "Cannot retrieve records.");
47: }
48: else {
49: res.json(results);
50: }
51: });
52: }
53: });
54: });
55:
56: app.get("/sproc/:key/:culture", function (req, res) {
57: sql.open(connectionString, function (err, conn) {
58: if (err) {
59: console.log(err);
60: res.send(500, "Cannot open connection.");
61: }
62: else {
63: var key = req.params.key;
64: var culture = req.params.culture;
65: var command = "EXEC GetItem '" + key + "', '" + culture + "'";
66: conn.queryRaw(command, function (err, results) {
67: if (err) {
68: console.log(err);
69: res.send(500, "Cannot retrieve records.");
70: }
71: else {
72: res.json(results);
73: }
74: });
75: }
76: });
77: });
78:
79: app.post("/new", function (req, res) {
80: var key = req.body.key;
81: var culture = req.body.culture;
82: var val = req.body.val;
83:
84: sql.open(connectionString, function (err, conn) {
85: if (err) {
86: console.log(err);
87: res.send(500, "Cannot open connection.");
88: }
89: else {
90: var command = "INSERT INTO [Resource] VALUES ('" + key + "', '" + culture + "', N'" + val + "')";
91: conn.queryRaw(command, function (err, results) {
92: if (err) {
93: console.log(err);
94: res.send(500, "Cannot retrieve records.");
95: }
96: else {
97: res.send(200, "Inserted Successful");
98: }
99: });
100: }
101: });
102: });
103:
104: app.listen(port);
Now let’s create a new function, copy the records from WASD to table service.
1. Delete the table named “resource”.
2. Create a new table named “resource”. These 2 steps ensures that we have an empty table.
3. Load all records from the “resource” table in WASD.
4. For each records loaded from WASD, insert them into the table one by one.
5. Prompt to user when finished.
In order to use table service we need the storage account and key, which can be found from the developer portal. Just select the storage account and click the Manage Keys button.
Then create two local variants in our Node.js application for the storage account name and key. Since we need to use WAS we need to import the azure module. Also I created another variant stored the table name.
In order to work with table service I need to create the storage client for table service. This is very similar as the Windows Azure SDK for .NET. As the code below I created a new variant named “client” and use “createTableService”, specified my storage account name and key.
1: var azure = require("azure");
2: var storageAccountName = "synctile";
3: var storageAccountKey = "/cOy9L7xysXOgPYU9FjDvjrRAhaMX/5tnOpcjqloPNDJYucbgTy7MOrAW7CbUg6PjaDdmyl+6pkwUnKETsPVNw==";
4: var tableName = "resource";
5: var client = azure.createTableService(storageAccountName, storageAccountKey);
Now create a new function for URL “/was/init” so that we can trigger it through browser. Then in this function we will firstly load all records from WASD.
1: app.get("/was/init", function (req, res) {
2: // load all records from windows azure sql database
3: sql.open(connectionString, function (err, conn) {
4: if (err) {
5: console.log(err);
6: res.send(500, "Cannot open connection.");
7: }
8: else {
9: conn.queryRaw("SELECT * FROM [Resource]", function (err, results) {
10: if (err) {
11: console.log(err);
12: res.send(500, "Cannot retrieve records.");
13: }
14: else {
15: if (results.rows.length > 0) {
16: // begin to transform the records into table service
17: }
18: }
19: });
20: }
21: });
22: });
When we succeed loaded all records we can start to transform them into table service. First I need to recreate the table in table service. This can be done by deleting and creating the table through table client I had just created previously.
1: app.get("/was/init", function (req, res) {
2: // load all records from windows azure sql database
3: sql.open(connectionString, function (err, conn) {
4: if (err) {
5: console.log(err);
6: res.send(500, "Cannot open connection.");
7: }
8: else {
9: conn.queryRaw("SELECT * FROM [Resource]", function (err, results) {
10: if (err) {
11: console.log(err);
12: res.send(500, "Cannot retrieve records.");
13: }
14: else {
15: if (results.rows.length > 0) {
16: // begin to transform the records into table service
17: // recreate the table named 'resource'
18: client.deleteTable(tableName, function (error) {
19: client.createTableIfNotExists(tableName, function (error) {
20: if (error) {
21: error["target"] = "createTableIfNotExists";
22: res.send(500, error);
23: }
24: else {
25: // transform the records
26: }
27: });
28: });
29: }
30: }
31: });
32: }
33: });
34: });
As you can see, the azure SDK provide its methods in callback pattern. In fact, almost all modules in Node.js use the callback pattern.
For example, when I deleted a table I invoked “deleteTable” method, provided the name of the table and a callback function which will be performed when the table had been deleted or failed. Underlying, the azure module will perform the table deletion operation in POSIX async threads pool asynchronously. And once it’s done the callback function will be performed. This is the reason we need to nest the table creation code inside the deletion function. If we perform the table creation code after the deletion code then they will be invoked in parallel.
Next, for each records in WASD I created an entity and then insert into the table service. Finally I send the response to the browser.
Can you find a bug in the code below? I will describe it later in this post.
1: app.get("/was/init", function (req, res) {
2: // load all records from windows azure sql database
3: sql.open(connectionString, function (err, conn) {
4: if (err) {
5: console.log(err);
6: res.send(500, "Cannot open connection.");
7: }
8: else {
9: conn.queryRaw("SELECT * FROM [Resource]", function (err, results) {
10: if (err) {
11: console.log(err);
12: res.send(500, "Cannot retrieve records.");
13: }
14: else {
15: if (results.rows.length > 0) {
16: // begin to transform the records into table service
17: // recreate the table named 'resource'
18: client.deleteTable(tableName, function (error) {
19: client.createTableIfNotExists(tableName, function (error) {
20: if (error) {
21: error["target"] = "createTableIfNotExists";
22: res.send(500, error);
23: }
24: else {
25: // transform the records
26: for (var i = 0; i < results.rows.length; i++) {
27: var entity = {
28: "PartitionKey": results.rows[i][1],
29: "RowKey": results.rows[i][0],
30: "Value": results.rows[i][2]
31: };
32: client.insertEntity(tableName, entity, function (error) {
33: if (error) {
34: error["target"] = "insertEntity";
35: res.send(500, error);
36: }
37: else {
38: console.log("entity inserted");
39: }
40: });
41: }
42: // send the
43: console.log("all done");
44: res.send(200, "All done!");
45: }
46: });
47: });
48: }
49: }
50: });
51: }
52: });
53: });
Now we can publish it to the cloud and have a try. But normally we’d better test it at the local emulator first. In Node.js SDK there are three build-in properties which provides the account name, key and host address for local storage emulator. We can use them to initialize our table service client. We also need to change the SQL connection string to let it use my local database. The code will be changed as below.
1: // windows azure sql database
2: //var connectionString = "Driver={SQL Server Native Client 10.0};Server=tcp:ac6271ya9e.database.windows.net,1433;Database=synctile;Uid=shaunxu@ac6271ya9e;Pwd=eszqu94XZY;Encrypt=yes;Connection Timeout=30;";
3: // sql server
4: var connectionString = "Driver={SQL Server Native Client 11.0};Server={.};Database={Caspar};Trusted_Connection={Yes};";
5:
6: var azure = require("azure");
7: var storageAccountName = "synctile";
8: var storageAccountKey = "/cOy9L7xysXOgPYU9FjDvjrRAhaMX/5tnOpcjqloPNDJYucbgTy7MOrAW7CbUg6PjaDdmyl+6pkwUnKETsPVNw==";
9: var tableName = "resource";
10: // windows azure storage
11: //var client = azure.createTableService(storageAccountName, storageAccountKey);
12: // local storage emulator
13: var client = azure.createTableService(azure.ServiceClient.DEVSTORE_STORAGE_ACCOUNT, azure.ServiceClient.DEVSTORE_STORAGE_ACCESS_KEY, azure.ServiceClient.DEVSTORE_TABLE_HOST);
Now let’s run the application and navigate to “localhost:12345/was/init” as I hosted it on port 12345. We can find it transformed the data from my local database to local table service.
Everything looks fine. But there is a bug in my code. If we have a look on the Node.js command window we will find that it sent response before all records had been inserted, which is not what I expected.
The reason is that, as I mentioned before, Node.js perform all IO operations in non-blocking model. When we inserted the records we executed the table service insert method in parallel, and the operation of sending response was also executed in parallel, even though I wrote it at the end of my logic.
The correct logic should be, when all entities had been copied to table service with no error, then I will send response to the browser, otherwise I should send error message to the browser. To do so I need to import another module named “async”, which helps us to coordinate our asynchronous code.
Install the module and import it at the beginning of the code. Then we can use its “forEach” method for the asynchronous code of inserting table entities. The first argument of “forEach” is the array that will be performed. The second argument is the operation for each items in the array. And the third argument will be invoked then all items had been performed or any errors occurred. Here we can send our response to browser.
1: app.get("/was/init", function (req, res) {
2: // load all records from windows azure sql database
3: sql.open(connectionString, function (err, conn) {
4: if (err) {
5: console.log(err);
6: res.send(500, "Cannot open connection.");
7: }
8: else {
9: conn.queryRaw("SELECT * FROM [Resource]", function (err, results) {
10: if (err) {
11: console.log(err);
12: res.send(500, "Cannot retrieve records.");
13: }
14: else {
15: if (results.rows.length > 0) {
16: // begin to transform the records into table service
17: // recreate the table named 'resource'
18: client.deleteTable(tableName, function (error) {
19: client.createTableIfNotExists(tableName, function (error) {
20: if (error) {
21: error["target"] = "createTableIfNotExists";
22: res.send(500, error);
23: }
24: else {
25: async.forEach(results.rows,
26: // transform the records
27: function (row, callback) {
28: var entity = {
29: "PartitionKey": row[1],
30: "RowKey": row[0],
31: "Value": row[2]
32: };
33: client.insertEntity(tableName, entity, function (error) {
34: if (error) {
35: callback(error);
36: }
37: else {
38: console.log("entity inserted.");
39: callback(null);
40: }
41: });
42: },
43: // send reponse
44: function (error) {
45: if (error) {
46: error["target"] = "insertEntity";
47: res.send(500, error);
48: }
49: else {
50: console.log("all done");
51: res.send(200, "All done!");
52: }
53: }
54: );
55: }
56: });
57: });
58: }
59: }
60: });
61: }
62: });
63: });
Run it locally and now we can find the response was sent after all entities had been inserted.
Query entities against table service is simple as well. Just use the “queryEntity” method from the table service client and providing the partition key and row key. We can also provide a complex query criteria as well, for example the code here.
In the code below I queried an entity by the partition key and row key, and return the proper localization value in response.
1: app.get("/was/:key/:culture", function (req, res) {
2: var key = req.params.key;
3: var culture = req.params.culture;
4: client.queryEntity(tableName, culture, key, function (error, entity) {
5: if (error) {
6: res.send(500, error);
7: }
8: else {
9: res.json(entity);
10: }
11: });
12: });
And then tested it on local emulator.
Finally if we want to publish this application to the cloud we should change the database connection string and storage account.
For more information about how to consume blob and queue service, as well as the service bus please refer to the MSDN page.
Consume Service Runtime
As I mentioned above, before we published our application to the cloud we need to change the connection string and account information in our code. But if you had played with WACS you should have known that the service runtime provides the ability to retrieve configuration settings, endpoints and local resource information at runtime. Which means we can have these values defined in CSCFG and CSDEF files and then the runtime should be able to retrieve the proper values.
For example we can add some role settings though the property window of the role, specify the connection string and storage account for cloud and local.
And the can also use the endpoint which defined in role environment to our Node.js application.
In Node.js SDK we can get an object from “azure.RoleEnvironment”, which provides the functionalities to retrieve the configuration settings and endpoints, etc.. In the code below I defined the connection string variants and then use the SDK to retrieve and initialize the table client.
1: var connectionString = "";
2: var storageAccountName = "";
3: var storageAccountKey = "";
4: var tableName = "";
5: var client;
6:
7: azure.RoleEnvironment.getConfigurationSettings(function (error, settings) {
8: if (error) {
9: console.log("ERROR: getConfigurationSettings");
10: console.log(JSON.stringify(error));
11: }
12: else {
13: console.log(JSON.stringify(settings));
14: connectionString = settings["SqlConnectionString"];
15: storageAccountName = settings["StorageAccountName"];
16: storageAccountKey = settings["StorageAccountKey"];
17: tableName = settings["TableName"];
18:
19: console.log("connectionString = %s", connectionString);
20: console.log("storageAccountName = %s", storageAccountName);
21: console.log("storageAccountKey = %s", storageAccountKey);
22: console.log("tableName = %s", tableName);
23:
24: client = azure.createTableService(storageAccountName, storageAccountKey);
25: }
26: });
In this way we don’t need to amend the code for the configurations between local and cloud environment since the service runtime will take care of it.
At the end of the code we will listen the application on the port retrieved from SDK as well.
1: azure.RoleEnvironment.getCurrentRoleInstance(function (error, instance) {
2: if (error) {
3: console.log("ERROR: getCurrentRoleInstance");
4: console.log(JSON.stringify(error));
5: }
6: else {
7: console.log(JSON.stringify(instance));
8: if (instance["endpoints"] && instance["endpoints"]["nodejs"]) {
9: var endpoint = instance["endpoints"]["nodejs"];
10: app.listen(endpoint["port"]);
11: }
12: else {
13: app.listen(8080);
14: }
15: }
16: });
But if we tested the application right now we will find that it cannot retrieve any values from service runtime. This is because by default, the entry point of this role was defined to the worker role class. In windows azure environment the service runtime will open a named pipeline to the entry point instance, so that it can connect to the runtime and retrieve values. But in this case, since the entry point was worker role and the Node.js was opened inside the role, the named pipeline was established between our worker role class and service runtime, so our Node.js application cannot use it.
To fix this problem we need to open the CSDEF file under the azure project, add a new element named Runtime. Then add an element named EntryPoint which specify the Node.js command line. So that the Node.js application will have the connection to service runtime, then it’s able to read the configurations.
Start the Node.js at local emulator we can find it retrieved the connections, storage account for local.
And if we publish our application to azure then it works with WASD and storage service through the configurations for cloud.
Summary
In this post I demonstrated how to use Windows Azure SDK for Node.js to interact with storage service, especially the table service. I also demonstrated on how to use WACS service runtime, how to retrieve the configuration settings and the endpoint information. And in order to make the service runtime available to my Node.js application I need to create an entry point element in CSDEF file and set “node.exe” as the entry point.
I used five posts to introduce and demonstrate on how to run a Node.js application on Windows platform, how to use Windows Azure Web Site and Windows Azure Cloud Service worker role to host our Node.js application. I also described how to work with other services provided by Windows Azure platform through Windows Azure SDK for Node.js.
Node.js is a very new and young network application platform. But since it’s very simple and easy to learn and deploy, as well as, it utilizes single thread non-blocking IO model, Node.js became more and more popular on web application and web service development especially for those IO sensitive projects. And as Node.js is very good at scaling-out, it’s more useful on cloud computing platform.
Use Node.js on Windows platform is new, too. The modules for SQL database and Windows Azure SDK are still under development and enhancement. It doesn’t support SQL parameter in “node-sqlserver”. It does support using storage connection string to create the storage client in “azure”. But Microsoft is working on make them easier to use, working on add more features and functionalities.
PS, you can download the source code here. You can download the source code of my “Copy all always” tool here.
Hope this helps,
Shaun
All documents and related graphics, codes are provided "AS IS" without warranty of any kind.
Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.
