Oracle BI Server Modeling, Part 1- Designing a Query Factory
- by bob.ertl(at)oracle.com
Welcome to Oracle BI Development's BI Foundation blog, focused on helping you get the most value from your Oracle Business Intelligence Enterprise Edition (BI EE) platform deployments.
In my first series of posts, I plan to show developers the concepts and best practices for modeling in the Common Enterprise Information Model (CEIM), the semantic layer of Oracle BI EE. In this segment, I will lay the groundwork for the modeling concepts. First, I will cover the big picture of how the BI Server fits into the system, and how the CEIM controls the query processing.
Oracle BI EE Query Cycle
The purpose of the Oracle BI Server is to bridge the gap between the presentation services and the data sources. There are typically a variety of data sources in a variety of technologies: relational, normalized transaction systems; relational star-schema data warehouses and marts; multidimensional analytic cubes and financial applications; flat files, Excel files, XML files, and so on. Business datasets can reside in a single type of source, or, most of the time, are spread across various types of sources.
Presentation services users are generally business people who need to be able to query that set of sources without any knowledge of technologies, schemas, or how sources are organized in their company. They think of business analysis in terms of measures with specific calculations, hierarchical dimensions for breaking those measures down, and detailed reports of the business transactions themselves. Most of them create queries without knowing it, by picking a dashboard page and some filters. Others create their own analysis by selecting metrics and dimensional attributes, and possibly creating additional calculations.
The BI Server bridges that gap from simple business terms to technical physical queries by exposing just the business focused measures and dimensional attributes that business people can use in their analyses and dashboards. After they make their selections and start the analysis, the BI Server plans the best way to query the data sources, writes the optimized sequence of physical queries to those sources, post-processes the results, and presents them to the client as a single result set suitable for tables, pivots and charts.
The CEIM is a model that controls the processing of the BI Server. It provides the subject areas that presentation services exposes for business users to select simplified metrics and dimensional attributes for their analysis. It models the mappings to the physical data access, the calculations and logical transformations, and the data access security rules. The CEIM consists of metadata stored in the repository, authored by developers using the Administration Tool client.
Presentation services and other query clients create their queries in BI EE's SQL-92 language, called Logical SQL or LSQL. The API simply uses ODBC or JDBC to pass the query to the BI Server. Presentation services writes the LSQL query in terms of the simplified objects presented to the users. The BI Server creates a query plan, and rewrites the LSQL into fully-detailed SQL or other languages suitable for querying the physical sources. For example, the LSQL on the left below was rewritten into the physical SQL for an Oracle 11g database on the right.
Logical SQL
Physical SQL
SELECT
"D0 Time"."T02 Per Name Month" saw_0,
"D4 Product"."P01 Product" saw_1,
"F2 Units"."2-01 Billed Qty (Sum All)" saw_2
FROM "Sample Sales"
ORDER BY saw_0, saw_1
WITH SAWITH0 AS (
select T986.Per_Name_Month as c1, T879.Prod_Dsc as c2,
sum(T835.Units) as c3, T879.Prod_Key as c4
from
Product T879 /* A05 Product */ ,
Time_Mth T986 /* A08 Time Mth */ ,
FactsRev T835 /* A11 Revenue (Billed Time Join) */
where ( T835.Prod_Key = T879.Prod_Key and T835.Bill_Mth = T986.Row_Wid)
group by T879.Prod_Dsc, T879.Prod_Key, T986.Per_Name_Month
)
select SAWITH0.c1 as c1, SAWITH0.c2 as c2, SAWITH0.c3 as c3
from SAWITH0
order by c1, c2
Probably everybody reading this blog can write SQL or MDX. However, the trick in designing the CEIM is that you are modeling a query-generation factory. Rather than hand-crafting individual queries, you model behavior and relationships, thus configuring the BI Server machinery to manufacture millions of different queries in response to random user requests. This mass production requires a different mindset and approach than when you are designing individual SQL statements in tools such as Oracle SQL Developer, Oracle Hyperion Interactive Reporting (formerly Brio), or Oracle BI Publisher.
The Structure of the Common Enterprise Information Model (CEIM)
The CEIM has a unique structure specifically for modeling the relationships and behaviors that fill the gap from logical user requests to physical data source queries and back to the result. The model divides the functionality into three specialized layers, called Presentation, Business Model and Mapping, and Physical, as shown below.
Presentation services clients can generally only see the presentation layer, and the objects in the presentation layer are normally the only ones used in the LSQL request. When a request comes into the BI Server from presentation services or another client, the relationships and objects in the model allow the BI Server to select the appropriate data sources, create a query plan, and generate the physical queries. That's the left to right flow in the diagram below. When the results come back from the data source queries, the right to left relationships in the model show how to transform the results and perform any final calculations and functions that could not be pushed down to the databases.
Business Model
Think of the business model as the heart of the CEIM you are designing. This is where you define the analytic behavior seen by the users, and the superset library of metric and dimension objects available to the user community as a whole. It also provides the baseline business-friendly names and user-readable dictionary. For these reasons, it is often called the "logical" model--it is a virtual database schema that persists no data, but can be queried as if it is a database. The business model always has a dimensional shape (more on this in future posts), and its simple shape and terminology hides the complexity of the source data models.
Besides hiding complexity and normalizing terminology, this layer adds most of the analytic value, as well. This is where you define the rich, dimensional behavior of the metrics and complex business calculations, as well as the conformed dimensions and hierarchies. It contributes to the ease of use for business users, since the dimensional metric definitions apply in any context of filters and drill-downs, and the conformed dimensions enable dashboard-wide filters and guided analysis links that bring context along from one page to the next. The conformed dimensions also provide a key to hiding the complexity of many sources, including federation of different databases, behind the simple business model.
Note that the expression language in this layer is LSQL, so that any expression can be rewritten into any data source's query language at run time. This is important for federation, where a given logical object can map to several different physical objects in different databases. It is also important to portability of the CEIM to different database brands, which is a key requirement for Oracle's BI Applications products.
Your requirements process with your user community will mostly affect the business model. This is where you will define most of the things they specifically ask for, such as metric definitions. For this reason, many of the best-practice methodologies of our consulting partners start with the high-level definition of this layer.
Physical Model
The physical model connects the business model that meets your users' requirements to the reality of the data sources you have available. In the query factory analogy, think of the physical layer as the bill of materials for generating physical queries. Every schema, table, column, join, cube, hierarchy, etc., that will appear in any physical query manufactured at run time must be modeled here at design time.
Each physical data source will have its own physical model, or "database" object in the CEIM. The shape of each physical model matches the shape of its physical source. In other words, if the source is normalized relational, the physical model will mimic that normalized shape. If it is a hypercube, the physical model will have a hypercube shape. If it is a flat file, it will have a denormalized tabular shape.
To aid in query optimization, the physical layer also tracks the specifics of the database brand and release. This allows the BI Server to make the most of each physical source's distinct capabilities, writing queries in its syntax, and using its specific functions. This allows the BI Server to push processing work as deep as possible into the physical source, which minimizes data movement and takes full advantage of the database's own optimizer. For most data sources, native APIs are used to further optimize performance and functionality.
The value of having a distinct separation between the logical (business) and physical models is encapsulation of the physical characteristics. This encapsulation is another enabler of packaged BI applications and federation. It is also key to hiding the complex shapes and relationships in the physical sources from the end users. Consider a routine drill-down in the business model: physically, it can require a drill-through where the first query is MDX to a multidimensional cube, followed by the drill-down query in SQL to a normalized relational database. The only difference from the user's point of view is that the 2nd query added a more detailed dimension level column - everything else was the same.
Mappings
Within the Business Model and Mapping Layer, the mappings provide the binding from each logical column and join in the dimensional business model, to each of the objects that can provide its data in the physical layer. When there is more than one option for a physical source, rules in the mappings are applied to the query context to determine which of the data sources should be hit, and how to combine their results if more than one is used. These rules specify aggregate navigation, vertical partitioning (fragmentation), and horizontal partitioning, any of which can be federated across multiple, heterogeneous sources. These mappings are usually the most sophisticated part of the CEIM.
Presentation
You might think of the presentation layer as a set of very simple relational-like views into the business model. Over ODBC/JDBC, they present a relational catalog consisting of databases, tables and columns. For business users, presentation services interprets these as subject areas, folders and columns, respectively. (Note that in 10g, subject areas were called presentation catalogs in the CEIM. In this blog, I will stick to 11g terminology.) Generally speaking, presentation services and other clients can query only these objects (there are exceptions for certain clients such as BI Publisher and Essbase Studio).
The purpose of the presentation layer is to specialize the business model for different categories of users. Based on a user's role, they will be restricted to specific subject areas, tables and columns for security. The breakdown of the model into multiple subject areas organizes the content for users, and subjects superfluous to a particular business role can be hidden from that set of users. Customized names and descriptions can be used to override the business model names for a specific audience. Variables in the object names can be used for localization.
For these reasons, you are better off thinking of the tables in the presentation layer as folders than as strict relational tables. The real semantics of tables and how they function is in the business model, and any grouping of columns can be included in any table in the presentation layer. In 11g, an LSQL query can also span multiple presentation subject areas, as long as they map to the same business model.
Other Model Objects
There are some objects that apply to multiple layers. These include security-related objects, such as application roles, users, data filters, and query limits (governors). There are also variables you can use in parameters and expressions, and initialization blocks for loading their initial values on a static or user session basis. Finally, there are Multi-User Development (MUD) projects for developers to check out units of work, and objects for the marketing feature used by our packaged customer relationship management (CRM) software.
The Query Factory
At this point, you should have a grasp on the query factory concept. When developing the CEIM model, you are configuring the BI Server to automatically manufacture millions of queries in response to random user requests. You do this by defining the analytic behavior in the business model, mapping that to the physical data sources, and exposing it through the presentation layer's role-based subject areas. While configuring mass production requires a different mindset than when you hand-craft individual SQL or MDX statements, it builds on the modeling and query concepts you already understand.
The following posts in this series will walk through the CEIM modeling concepts and best practices in detail. We will initially review dimensional concepts so you can understand the business model, and then present a pattern-based approach to learning the mappings from a variety of physical schema shapes and deployments to the dimensional model. Along the way, we will also present the dimensional calculation template, and learn how to configure the many additivity patterns.
