When designing a service oriented architecture (SOA) system, there will be a lot of services with many service contracts, endpoints and behaviors. Besides the client calling the service, in a large distributed system a service may invoke other services. In this case, one service might need
to know the endpoints it invokes. This might not be a problem in a small system. But when you have more than 10 services this might be a problem. For example in my current product, there are around 10 services, such as the user authentication service, UI integration service, location service, license service, device monitor service, event monitor service, schedule job service, accounting service, player management service, etc.. Benefit of Discovery Service Since almost all my services need
to invoke at least one other service. This would be a difficult task
to make sure all services endpoints are configured correctly in every service. And furthermore, it would be a nightmare when a service changed its endpoint at runtime. Hence, we need a discovery service
to remove the dependency (configuration dependency). A discovery service plays as a service dictionary which stores the relationship between the contracts and the endpoints for every service. By using the discovery service, when service X wants
to invoke service Y, it just need
to ask the discovery service where is service Y, then the discovery service will return all proper endpoints of service Y, then service X can use the endpoint
to send the request
to service Y. And when some services changed their endpoint address, all need
to do is
to update its records in the discovery service then all others will know its new endpoint. In WCF 4.0 Discovery it supports both managed proxy discovery mode and ad-hoc discovery mode. In ad-hoc mode there is no standalone discovery service. When a client wanted
to invoke a service, it will broadcast an message (normally in UDP protocol)
to the entire network with the service match criteria. All services which enabled the discovery behavior will receive this message and only those matched services will send their endpoint back
to the client. The managed proxy discovery service works as I described above. In this post I will only cover the managed proxy mode, where there’s a discovery service. For more information about the ad-hoc mode please refer
to the MSDN. Service Announcement and Probe The main functionality of discovery service should be return the proper endpoint addresses back
to the service who is looking for. In most cases the consume service (as a client) will send the contract which it wanted
to request
to the discovery service. And then the discovery service will find the endpoint and respond. Sometimes the contract and endpoint are not enough. It also contains versioning, extensions attributes. This post I will only cover the case includes contract and endpoint. When a client (or sometimes a service who need
to invoke another service) need
to connect
to a target service, it will firstly request the discovery service through the “Probe” method with the criteria. Basically the criteria contains the contract type name of the target service. Then the discovery service will search its endpoint repository by the criteria. The repository might be a database, a distributed cache or a flat XML file. If it matches, the discovery service will grab the endpoint information (it’s called discovery endpoint metadata in WCF) and send back. And this is called “Probe”. Finally the client received the discovery endpoint metadata and will use the endpoint
to connect
to the target service. Besides the probe, discovery service should take the responsible
to know there is a new service available when it goes online, as well as stopped when it goes offline. This feature is named “Announcement”. When a service started and stopped, it will announce
to the discovery service. So the basic functionality of a discovery service should includes: 1, An endpoint which receive the service online message, and add the service endpoint information in the discovery repository. 2, An endpoint which receive the service offline message, and remove the service endpoint information from the discovery repository. 3, An endpoint which receive the client probe message, and return the matches service endpoints, and return the discovery endpoint metadata. WCF 4.0 discovery service just covers all these features in it's infrastructure classes. Discovery Service in WCF 4.0 WCF 4.0 introduced a new assembly named System.ServiceModel.Discovery which has all necessary classes and interfaces
to build a WS-Discovery compliant discovery service. It supports ad-hoc and managed proxy modes. For the case mentioned in this post, what we need
to build is a standalone discovery service, which is the managed proxy discovery service mode.
To build a managed discovery service in WCF 4.0 just create a new class inherits from the abstract class System.ServiceModel.Discovery.DiscoveryProxy. This class implemented and abstracted the procedures of service announcement and probe. And it exposes 8 abstract methods where we can implement our own endpoint register, unregister and find logic. These 8 methods are asynchronized, which means all invokes
to the discovery service are asynchronously, for better service capability and performance. 1, OnBeginOnlineAnnouncement, OnEndOnlineAnnouncement: Invoked when a service sent the online announcement message. We need
to add the endpoint information
to the repository in this method. 2, OnBeginOfflineAnnouncement, OnEndOfflineAnnouncement: Invoked when a service sent the offline announcement message. We need
to remove the endpoint information from the repository in this method. 3, OnBeginFind, OnEndFind: Invoked when a client sent the probe message that want
to find the service endpoint information. We need
to look for the proper endpoints by matching the client’s criteria through the repository in this method. 4, OnBeginResolve, OnEndResolve: Invoked then a client sent the resolve message. Different from the find method, when using resolve method the discovery service will return the exactly one service endpoint metadata
to the client. In our example we will NOT implement this method. Let’s create our own discovery service, inherit the base System.ServiceModel.Discovery.DiscoveryProxy. We also need
to specify the service behavior in this class. Since the build-in discovery service host class only support the singleton mode, we must set its instance context mode
to single. 1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.ServiceModel.Discovery;
6: using System.ServiceModel;
7:
8: namespace Phare.Service
9: {
10: [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
11: public class ManagedProxyDiscoveryService : DiscoveryProxy
12: {
13: protected override IAsyncResult OnBeginFind(FindRequestContext findRequestContext, AsyncCallback callback, object state)
14: {
15: throw new NotImplementedException();
16: }
17:
18: protected override IAsyncResult OnBeginOfflineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
19: {
20: throw new NotImplementedException();
21: }
22:
23: protected override IAsyncResult OnBeginOnlineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
24: {
25: throw new NotImplementedException();
26: }
27:
28: protected override IAsyncResult OnBeginResolve(ResolveCriteria resolveCriteria, AsyncCallback callback, object state)
29: {
30: throw new NotImplementedException();
31: }
32:
33: protected override void OnEndFind(IAsyncResult result)
34: {
35: throw new NotImplementedException();
36: }
37:
38: protected override void OnEndOfflineAnnouncement(IAsyncResult result)
39: {
40: throw new NotImplementedException();
41: }
42:
43: protected override void OnEndOnlineAnnouncement(IAsyncResult result)
44: {
45: throw new NotImplementedException();
46: }
47:
48: protected override EndpointDiscoveryMetadata OnEndResolve(IAsyncResult result)
49: {
50: throw new NotImplementedException();
51: }
52: }
53: }
Then let’s implement the online, offline and find methods one by one. WCF discovery service gives us full flexibility
to implement the endpoint add, remove and find logic. For the demo purpose we will use an internal dictionary
to store the services’ endpoint metadata.
In the next post we will see how
to serialize and store these information in database.
Define a concurrent dictionary inside the service class since our it will be used in the multiple threads scenario.
1: [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
2: public class ManagedProxyDiscoveryService : DiscoveryProxy
3: {
4: private ConcurrentDictionary<EndpointAddress, EndpointDiscoveryMetadata> _services;
5:
6: public ManagedProxyDiscoveryService()
7: {
8: _services = new ConcurrentDictionary<EndpointAddress, EndpointDiscoveryMetadata>();
9: }
10: }
Then we can simply implement the logic of service online and offline.
1: protected override IAsyncResult OnBeginOnlineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
2: {
3: _services.AddOrUpdate(endpointDiscoveryMetadata.Address, endpointDiscoveryMetadata, (key, value) => endpointDiscoveryMetadata);
4: return new OnOnlineAnnouncementAsyncResult(callback, state);
5: }
6:
7: protected override void OnEndOnlineAnnouncement(IAsyncResult result)
8: {
9: OnOnlineAnnouncementAsyncResult.End(result);
10: }
11:
12: protected override IAsyncResult OnBeginOfflineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
13: {
14: EndpointDiscoveryMetadata endpoint = null;
15: _services.TryRemove(endpointDiscoveryMetadata.Address, out endpoint);
16: return new OnOfflineAnnouncementAsyncResult(callback, state);
17: }
18:
19: protected override void OnEndOfflineAnnouncement(IAsyncResult result)
20: {
21: OnOfflineAnnouncementAsyncResult.End(result);
22: }
Regards the find method, the parameter FindRequestContext.Criteria has a method named IsMatch, which can be use for us
to evaluate which service metadata is satisfied with the criteria. So the implementation of find method would be like this.
1: protected override IAsyncResult OnBeginFind(FindRequestContext findRequestContext, AsyncCallback callback, object state)
2: {
3: _services.Where(s => findRequestContext.Criteria.IsMatch(s.Value))
4: .Select(s => s.Value)
5: .All(meta =>
6: {
7: findRequestContext.AddMatchingEndpoint(meta);
8: return true;
9: });
10: return new OnFindAsyncResult(callback, state);
11: }
12:
13: protected override void OnEndFind(IAsyncResult result)
14: {
15: OnFindAsyncResult.End(result);
16: }
As you can see, we checked all endpoints metadata in repository by invoking the IsMatch method. Then add all proper endpoints metadata into the parameter.
Finally since all these methods are asynchronized we need some AsyncResult classes as well. Below are the base class and the inherited classes used in previous methods.
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.Threading;
6:
7: namespace Phare.Service
8: {
9: abstract internal class AsyncResult : IAsyncResult
10: {
11: AsyncCallback callback;
12: bool completedSynchronously;
13: bool endCalled;
14: Exception exception;
15: bool isCompleted;
16: ManualResetEvent manualResetEvent;
17: object state;
18: object thisLock;
19:
20: protected AsyncResult(AsyncCallback callback, object state)
21: {
22: this.callback = callback;
23: this.state = state;
24: this.thisLock = new object();
25: }
26:
27: public object AsyncState
28: {
29: get
30: {
31: return state;
32: }
33: }
34:
35: public WaitHandle AsyncWaitHandle
36: {
37: get
38: {
39: if (manualResetEvent != null)
40: {
41: return manualResetEvent;
42: }
43: lock (ThisLock)
44: {
45: if (manualResetEvent == null)
46: {
47: manualResetEvent = new ManualResetEvent(isCompleted);
48: }
49: }
50: return manualResetEvent;
51: }
52: }
53:
54: public bool CompletedSynchronously
55: {
56: get
57: {
58: return completedSynchronously;
59: }
60: }
61:
62: public bool IsCompleted
63: {
64: get
65: {
66: return isCompleted;
67: }
68: }
69:
70: object ThisLock
71: {
72: get
73: {
74: return this.thisLock;
75: }
76: }
77:
78: protected static TAsyncResult End<TAsyncResult>(IAsyncResult result)
79: where TAsyncResult : AsyncResult
80: {
81: if (result == null)
82: {
83: throw new ArgumentNullException("result");
84: }
85:
86: TAsyncResult asyncResult = result as TAsyncResult;
87:
88: if (asyncResult == null)
89: {
90: throw new ArgumentException("Invalid async result.", "result");
91: }
92:
93: if (asyncResult.endCalled)
94: {
95: throw new InvalidOperationException("Async object already ended.");
96: }
97:
98: asyncResult.endCalled = true;
99:
100: if (!asyncResult.isCompleted)
101: {
102: asyncResult.AsyncWaitHandle.WaitOne();
103: }
104:
105: if (asyncResult.manualResetEvent != null)
106: {
107: asyncResult.manualResetEvent.Close();
108: }
109:
110: if (asyncResult.exception != null)
111: {
112: throw asyncResult.exception;
113: }
114:
115: return asyncResult;
116: }
117:
118: protected void Complete(bool completedSynchronously)
119: {
120: if (isCompleted)
121: {
122: throw new InvalidOperationException("This async result is already completed.");
123: }
124:
125: this.completedSynchronously = completedSynchronously;
126:
127: if (completedSynchronously)
128: {
129: this.isCompleted = true;
130: }
131: else
132: {
133: lock (ThisLock)
134: {
135: this.isCompleted = true;
136: if (this.manualResetEvent != null)
137: {
138: this.manualResetEvent.Set();
139: }
140: }
141: }
142:
143: if (callback != null)
144: {
145: callback(this);
146: }
147: }
148:
149: protected void Complete(bool completedSynchronously, Exception exception)
150: {
151: this.exception = exception;
152: Complete(completedSynchronously);
153: }
154: }
155: }
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.ServiceModel.Discovery;
6: using Phare.Service;
7:
8: namespace Phare.Service
9: {
10: internal sealed class OnOnlineAnnouncementAsyncResult : AsyncResult
11: {
12: public OnOnlineAnnouncementAsyncResult(AsyncCallback callback, object state)
13: : base(callback, state)
14: {
15: this.Complete(true);
16: }
17:
18: public static void End(IAsyncResult result)
19: {
20: AsyncResult.End<OnOnlineAnnouncementAsyncResult>(result);
21: }
22:
23: }
24:
25: sealed class OnOfflineAnnouncementAsyncResult : AsyncResult
26: {
27: public OnOfflineAnnouncementAsyncResult(AsyncCallback callback, object state)
28: : base(callback, state)
29: {
30: this.Complete(true);
31: }
32:
33: public static void End(IAsyncResult result)
34: {
35: AsyncResult.End<OnOfflineAnnouncementAsyncResult>(result);
36: }
37: }
38:
39: sealed class OnFindAsyncResult : AsyncResult
40: {
41: public OnFindAsyncResult(AsyncCallback callback, object state)
42: : base(callback, state)
43: {
44: this.Complete(true);
45: }
46:
47: public static void End(IAsyncResult result)
48: {
49: AsyncResult.End<OnFindAsyncResult>(result);
50: }
51: }
52:
53: sealed class OnResolveAsyncResult : AsyncResult
54: {
55: EndpointDiscoveryMetadata matchingEndpoint;
56:
57: public OnResolveAsyncResult(EndpointDiscoveryMetadata matchingEndpoint, AsyncCallback callback, object state)
58: : base(callback, state)
59: {
60: this.matchingEndpoint = matchingEndpoint;
61: this.Complete(true);
62: }
63:
64: public static EndpointDiscoveryMetadata End(IAsyncResult result)
65: {
66: OnResolveAsyncResult thisPtr = AsyncResult.End<OnResolveAsyncResult>(result);
67: return thisPtr.matchingEndpoint;
68: }
69: }
70: }
Now we have finished the discovery service. The next step is
to host it. The discovery service is a standard WCF service. So we can use ServiceHost on a console application, windows service, or in IIS as usual. The following code is how
to host the discovery service we had just created in a console application.
1: static void Main(string[] args)
2: {
3: using (var host = new ServiceHost(new ManagedProxyDiscoveryService()))
4: {
5: host.Opened += (sender, e) =>
6: {
7: host.Description.Endpoints.All((ep) =>
8: {
9: Console.WriteLine(ep.ListenUri);
10: return true;
11: });
12: };
13:
14: try
15: {
16: // retrieve the announcement, probe endpoint and binding from configuration
17: var announcementEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["announcementEndpointAddress"]);
18: var probeEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["probeEndpointAddress"]);
19: var binding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
20: var announcementEndpoint = new AnnouncementEndpoint(binding, announcementEndpointAddress);
21: var probeEndpoint = new DiscoveryEndpoint(binding, probeEndpointAddress);
22: probeEndpoint.IsSystemEndpoint = false;
23: // append the service endpoint for announcement and probe
24: host.AddServiceEndpoint(announcementEndpoint);
25: host.AddServiceEndpoint(probeEndpoint);
26:
27: host.Open();
28:
29: Console.WriteLine("Press any key
to exit.");
30: Console.ReadKey();
31: }
32: catch (Exception ex)
33: {
34: Console.WriteLine(ex.ToString());
35: }
36: }
37:
38: Console.WriteLine("Done.");
39: Console.ReadKey();
40: }
What we need
to notice is that, the discovery service needs two endpoints for announcement and probe. In this example I just retrieve them from the configuration file. I also specified the binding of these two endpoints in configuration file as well.
1: <?xml version="1.0"?>
2: <configuration>
3: <startup>
4: <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
5: </startup>
6: <appSettings>
7: <add key="announcementEndpointAddress" value="net.tcp://localhost:10010/announcement"/>
8: <add key="probeEndpointAddress" value="net.tcp://localhost:10011/probe"/>
9: <add key="bindingType" value="System.ServiceModel.NetTcpBinding, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
10: </appSettings>
11: </configuration>
And this is the console screen when I ran my discovery service. As you can see there are two endpoints listening for announcement message and probe message.
Discoverable Service and Client
Next, let’s create a WCF service that is discoverable, which means it can be found by the discovery service.
To do so, we need
to let the service send the online announcement message
to the discovery service, as well as offline message before it shutdown.
Just create a simple service which can make the incoming string
to upper. The service contract and implementation would be like this.
1: [ServiceContract]
2: public interface IStringService
3: {
4: [OperationContract]
5: string ToUpper(string content);
6: }
1: public class StringService : IStringService
2: {
3: public string ToUpper(string content)
4: {
5: return content.ToUpper();
6: }
7: }
Then host this service in the console application. In order
to make the discovery service easy
to be tested the service address will be changed each time it’s started.
1: static void Main(string[] args)
2: {
3: var baseAddress = new Uri(string.Format("net.tcp://localhost:11001/stringservice/{0}/", Guid.NewGuid().ToString()));
4:
5: using (var host = new ServiceHost(typeof(StringService), baseAddress))
6: {
7: host.Opened += (sender, e) =>
8: {
9: Console.WriteLine("Service opened at {0}", host.Description.Endpoints.First().ListenUri);
10: };
11:
12: host.AddServiceEndpoint(typeof(IStringService), new NetTcpBinding(), string.Empty);
13:
14: host.Open();
15:
16: Console.WriteLine("Press any key
to exit.");
17: Console.ReadKey();
18: }
19: }
Currently this service is NOT discoverable. We need
to add a special service behavior so that it could send the online and offline message
to the discovery service announcement endpoint when the host is opened and closed. WCF 4.0 introduced a service behavior named ServiceDiscoveryBehavior. When we specified the announcement endpoint address and appended it
to the service behaviors this service will be discoverable.
1: var announcementAddress = new EndpointAddress(ConfigurationManager.AppSettings["announcementEndpointAddress"]);
2: var announcementBinding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
3: var announcementEndpoint = new AnnouncementEndpoint(announcementBinding, announcementAddress);
4: var discoveryBehavior = new ServiceDiscoveryBehavior();
5: discoveryBehavior.AnnouncementEndpoints.Add(announcementEndpoint);
6: host.Description.Behaviors.Add(discoveryBehavior);
The ServiceDiscoveryBehavior utilizes the service extension and channel dispatcher
to implement the online and offline announcement logic. In short, it injected the channel open and close procedure and send the online and offline message
to the announcement endpoint.
On client side, when we have the discovery service, a client can invoke a service without knowing its endpoint. WCF discovery assembly provides a class named DiscoveryClient, which can be used
to find the proper service endpoint by passing the criteria.
In the code below I initialized the DiscoveryClient, specified the discovery service probe endpoint address. Then I created the find criteria by specifying the service contract I wanted
to use and invoke the Find method. This will send the probe message
to the discovery service and it will find the endpoints back
to me.
The discovery service will return all endpoints that matches the find criteria, which means in the result of the find method there might be more than one endpoints. In this example I just returned the first matched one back. In the next post I will show how
to extend our discovery service
to make it work like a service load balancer.
1: static EndpointAddress FindServiceEndpoint()
2: {
3: var probeEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["probeEndpointAddress"]);
4: var probeBinding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
5: var discoveryEndpoint = new DiscoveryEndpoint(probeBinding, probeEndpointAddress);
6:
7: EndpointAddress address = null;
8: FindResponse result = null;
9: using (var discoveryClient = new DiscoveryClient(discoveryEndpoint))
10: {
11: result = discoveryClient.Find(new FindCriteria(typeof(IStringService)));
12: }
13:
14: if (result != null && result.Endpoints.Any())
15: {
16: var endpointMetadata = result.Endpoints.First();
17: address = endpointMetadata.Address;
18: }
19: return address;
20: }
Once we probed the discovery service we will receive the endpoint. So in the client code we can created the channel factory from the endpoint and binding, and invoke
to the service.
When creating the client side channel factory we need
to make sure that the client side binding should be the same as the service side. WCF discovery service can be used
to find the endpoint for a service contract, but the binding is NOT included. This is because the binding was not in the WS-Discovery specification.
In the next post I will demonstrate how
to add the binding information into the discovery service. At that moment the client don’t need
to create the binding by itself. Instead it will use the binding received from the discovery service.
1: static void Main(string[] args)
2: {
3: Console.WriteLine("Say something...");
4: var content = Console.ReadLine();
5: while (!string.IsNullOrWhiteSpace(content))
6: {
7: Console.WriteLine("Finding the service endpoint...");
8: var address = FindServiceEndpoint();
9: if (address == null)
10: {
11: Console.WriteLine("There is no endpoint matches the criteria.");
12: }
13: else
14: {
15: Console.WriteLine("Found the endpoint {0}", address.Uri);
16:
17: var factory = new ChannelFactory<IStringService>(new NetTcpBinding(), address);
18: factory.Opened += (sender, e) =>
19: {
20: Console.WriteLine("Connecting
to {0}.", factory.Endpoint.ListenUri);
21: };
22: var proxy = factory.CreateChannel();
23: using (proxy as IDisposable)
24: {
25: Console.WriteLine("ToUpper: {0} => {1}", content, proxy.ToUpper(content));
26: }
27: }
28:
29: Console.WriteLine("Say something...");
30: content = Console.ReadLine();
31: }
32: }
Similarly, the discovery service probe endpoint and binding were defined in the configuration file.
1: <?xml version="1.0"?>
2: <configuration>
3: <startup>
4: <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
5: </startup>
6: <appSettings>
7: <add key="announcementEndpointAddress" value="net.tcp://localhost:10010/announcement"/>
8: <add key="probeEndpointAddress" value="net.tcp://localhost:10011/probe"/>
9: <add key="bindingType" value="System.ServiceModel.NetTcpBinding, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
10: </appSettings>
11: </configuration>
OK, now let’s have a test. Firstly start the discovery service, and then start our discoverable service. When it started it will announced
to the discovery service and registered its endpoint into the repository, which is the local dictionary. And then start the client and type something. As you can see the client asked the discovery service for the endpoint and then establish the connection
to the discoverable service.
And more interesting, do NOT close the client console but terminate the discoverable service but press the enter key. This will make the service send the offline message
to the discovery service. Then start the discoverable service again. Since we made it use a different address each time it started, currently it should be hosted on another address. If we enter something in the client we could see that it asked the discovery service and retrieve the new endpoint, and connect the the service.
Summary
In this post I discussed the benefit of using the discovery service and the procedures of service announcement and probe. I also demonstrated how
to leverage the WCF Discovery feature in WCF 4.0
to build a simple managed discovery service.
For test purpose, in this example I used the in memory dictionary as the discovery endpoint metadata repository. And when finding I also just return the first matched endpoint back. I also hard coded the bindings between the discoverable service and the client.
In next post I will show you how
to solve the problem mentioned above, as well as some additional feature for production usage.
You can download the code here.
Hope this helps,
Shaun
All documents and related graphics, codes are provided "AS IS" without warranty of any kind.
Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.
