Form, function and complexity in rule processing
- by Charles Young
Tim Bass posted on ‘Orwellian Event Processing’. I was involved in a heated exchange in the comments, and he has more recently published a post entitled ‘Disadvantages of Rule-Based Systems (Part 1)’. Whatever the rights and wrongs of our exchange, it clearly failed to generate any agreement or understanding of our different positions. I don't particularly want to promote further argument of that kind, but I do want to take the opportunity of offering a different perspective on rule-processing and an explanation of my comments.
For me, the ‘red rag’ lay in Tim’s claim that “...rules alone are highly inefficient for most classes of (not simple) problems” and a later paragraph that appears to equate the simplicity of form (‘IF-THEN-ELSE’) with simplicity of function. It is not the first time Tim has expressed these views and not the first time I have responded to his assertions. Indeed, Tim has a long history of commenting on the subject of complex event processing (CEP) and, less often, rule processing in ‘robust’ terms, often asserting that very many other people’s opinions on this subject are mistaken. In turn, I am of the opinion that, certainly in terms of rule processing, which is an area in which I have a specific interest and knowledge, he is often mistaken.
There is no simple answer to the fundamental question ‘what is a rule?’ We use the word in a very fluid fashion in English. Likewise, the term ‘rule processing’, as used widely in IT, is equally difficult to define simplistically. The best way to envisage the term is as a ‘centre of gravity’ within a wider domain. That domain contains many other ‘centres of gravity’, including CEP, statistical analytics, neural networks, natural language processing and so much more. Whole communities tend to gravitate towards and build themselves around some of these centres.
The term 'rule processing' is associated with many different technology types, various software products, different architectural patterns, the functional capability of many applications and services, etc. There is considerable variation amongst these different technologies, techniques and products. Very broadly, a common theme is their ability to manage certain types of processing and problem solving through declarative, or semi-declarative, statements of propositional logic bound to action-based consequences. It is generally important to be able to decouple these statements from other parts of an overall system or architecture so that they can be managed and deployed independently.
As a centre of gravity, ‘rule processing’ is no island. It exists in the context of a domain of discourse that is, itself, highly interconnected and continuous. Rule processing does not, for example, exist in splendid isolation to natural language processing. On the contrary, an on-going theme of rule processing is to find better ways to express rules in natural language and map these to executable forms. Rule processing does not exist in splendid isolation to CEP. On the contrary, an event processing agent can reasonably be considered as a rule engine (a theme in ‘Power of Events’ by David Luckham). Rule processing does not live in splendid isolation to statistical approaches such as Bayesian analytics. On the contrary, rule processing and statistical analytics are highly synergistic. Rule processing does not even live in splendid isolation to neural networks. For example, significant research has centred on finding ways to translate trained nets into explicit rule sets in order to support forms of validation and facilitate insight into the knowledge stored in those nets.
What about simplicity of form? Many rule processing technologies do indeed use a very simple form (‘If...Then’, ‘When...Do’, etc.) However, it is a fundamental mistake to equate simplicity of form with simplicity of function. It is absolutely mistaken to suggest that simplicity of form is a barrier to the efficient handling of complexity. There are countless real-world examples which serve to disprove that notion. Indeed, simplicity of form is often the key to handling complexity.
Does rule processing offer a ‘one size fits all’. No, of course not. No serious commentator suggests it does. Does the design and management of large knowledge bases, expressed as rules, become difficult? Yes, it can do, but that is true of any large knowledge base, regardless of the form in which knowledge is expressed.
The measure of complexity is not a function of rule set size or rule form. It tends to be correlated more strongly with the size of the ‘problem space’ (‘search space’) which is something quite different. Analysis of the problem space and the algorithms we use to search through that space are, of course, the very things we use to derive objective measures of the complexity of a given problem. This is basic computer science and common practice.
Sailing a Dreadnaught through the sea of information technology and lobbing shells at some of the islands we encounter along the way does no one any good. Building bridges and causeways between islands so that the inhabitants can collaborate in open discourse offers hope of real progress.
