In my reading I recently came across the concept of the Data, Information, Knowledge, Wisdom (DIKW) hierarchy. While it is perhaps loosely defined, I see the concept as a helpful way to illustrate the progression from raw data to actionable information (requiring wisdom to handle it correctly). There are valid criticisms of this model, but it does reflect the goal of collecting and managing data - turning it into something that allows business users to make wise decisions.

So how do we get our business customers from data (which is available in quantities so big they are difficult to grasp) to knowledge and wisdom? Let's consider the illustration of buying a house.

 There is a lot of buzz lately around Agile BI and Agile data warehousing.  Typical questions asked include - Does it produce better results?  Is it faster?  How much does it cost?  And of course, most importantly, is the Agile methodology a fit for Data Warehousing and Business Intelligence projects?

What is Agile?

Margaret, who was an average sales person, moved from Washington, DC to Richmond, VA, whose market is one fifth the size, during the month of June.  When the annual evaluations of sales performance were done in the month of December, she was listed as the top performer in the Richmond market resulting in the company promoting her to Sales Director.  The next two highest ranked Richmond salespeople had been the consistent leaders for the last several years and outperformed Margaret since she arrived in Richmond.  Her very high sales numbers during the first six months of the year skewed her average, placing her above the rest of the Richmond area.  In this example, if the decision makers had correct information handy, and used it appropriately, would they have promoted Margaret over her new Richmond peers?

Here is another example.

This write up contains a high level investigation of the Business Intelligence solution offering from Microsoft (SQL Server Reporting Services or SSRS) and the offering from SAP, the Business Objects base reporting package (BOBJ).  While BOBJ does have more options for reporting and presentation, from a basic report feature standpoint both of the tools offer similar functionality and offer the user a great deal of flexibility in the presentation of their data.  The other difference between the two solutions that needs to be considered is the expense associated with the Total Cost of Ownership.  While you will have similar costs in the requirements gathering, design, development, testing, and ongoing administration, there is a significant difference in the licensing cost of these products.  While BOBJ charges by either named user or CPU, SSRS comes with SQL Server so there are no additional costs with adding a BI tool set.

SQL Server and Sharepoint offer a quality BI solution, which meets basic architectural principles and business requirements.  Because of Microsoft’s desire to establish itself in the BI space, it offers the BI components with a license to SQL Server.  The lack of flashy, AJAX style reporting features (which are often shown in demos of BOBJ) may limit the business’s interest in SQL Server.  Additionally, BOBJ’s reporting, ad hoc queries, dashboard, data visualization capabilities are key strengths of the SAP BOBJ product suite and are among top rated BI tools.

When considering the total cost of ownership, a company must consider the individual components that make up this expense.  Total Cost of Ownership (TCO) comes from the High Level Business Requirements, Software Selection Process, Software Installation, Detailed Requirements, Design, Development, System and User Acceptance Testing, Production Software Licenses, the ongoing Maintenance of the solution.   While many of these costs would be similar across the two platforms, a company needs to assess the differences in development time, and ongoing maintenance and understand which tool their personnel and IT infrastructure can support.  Specific costs and return on investments are highly dependent on company’s specific situations and deployment choices.  From our specific client exposure, mid-market companies do not opt for BOBJ, and we find that SQL Server is more prevalent.  An SSRS solution will often be lower cost from a licensing perspective as all components are included with a SQL Server license.  However, SSRS requires a developer to build their reports, where BOBJ supports an end business user self-service model.  So long term technical support and development costs could actually be lower with BOBJ.

Because many companies already own SQL Server licenses within their infrastructure, the ease and low cost benefits of implementation may be too good to pass up.  However, companies either without SQL Server in house or requiring heavily visual reports accessible to business users or self-service access to information with minimal IT support may want to implement BOBJ as their BI stack.

The terms data and information are often used interchangeably.  However, in the data warehousing world they are quite different from each other.  Remembering the difference is as simple as the difference between Charlie and Raymond Babbitt.  You may remember these two characters from the movie Rainman.  Charlie is the younger brother of austitic savant Raymond. 

Recently there’s been a lot of discussion about alternatives to the standard relational DBMS (note this widely read article).  Some of the feedback comes from OO developers.  There’s a fundamental dissonance between OO and relational approaches, requiring an intermediate object/relational mapping (ORM) layer for OO systems to operate effectively with relational DBMSs.  Others decry the overhead imposed by the relational DBMS, seeking a way to store and use massive datasets for applications needing fast response.  In response a number of vendors are now providing SaaS-like database services in the “cloud”: open source, lightly structured data services provided via the internet, capable of storing and delivering large data stores for high availability, fast response applications.

Vendors have only recently arrived on the scene with these offerings, but according to Kevin Hazzard, CapTech architect, this was inevitable: “In my own writings and talks, I sometimes call databases an unfortunate consequence of history. This sentiment isn’t borne of any sort of animosity toward databases. On the contrary, I can’t imagine doing what I do without the services that databases provide. However, I mustered just enough intellectual honesty to admit that if I were able to give a handful of 4Gb DIMMs to the creators of the ENIAC computer at University of Pittsburgh in 1948 and if I were smart enough to show them how to use that memory effectively, the database as we know it today would simply never have evolved. Instead, databases would have become an integral part of the software development model. The database would be the file system with some automatic hashing and locking capabilities.

“Well, that sounds an awful lot like Amazon’s SimpleDB, Google’s BigTable (and Hypertable), Microsoft SQL Data Services (in the cloud), and other emerging DB platforms.  The ORM trend is also pushing developers toward these models by providing language extensions for query comprehension, query optimization and parallelism in the application domain. Is there a chance that pre-relational or post-relational database engines will eclipse relational databases in the marketplace in the near-term? No way. “

Still, Data Management professionals should be on the lookout for bringing competitive edge to their BI environment with these new approaches – but carefully.  According to CapTech DM/BI specialist Ashwini Kumar, these new database approaches are “a good fit for simple data structures, structured data, and applications with queries using procedural codes. Also, they may be good fit for customized applications. However, Business Intelligence/Enterprise Information Management type applications are more dynamic in nature, especially in exploring or mining data.”  

So where to from here?  Over the coming months we’ll watch as these new database technologies emerge into wider commercial application and lessons-learned emerge.