Here’s a little-recognized fact about data integration: if you run a business or any sizable chunk of one, someone is integrating your data.

In my professional life I have, on occasion, suggested data integration efforts.  Sometimes my suggestions have been accepted and sometimes not.  As an IT professional I understand that different managers have different priorities, and in a given business situation sometimes other things may be more important than, for example, having a single, consistent source for all customer records, or making sure production data matches financial data.

But as a customer?  That’s different.

As Business Intelligence consultants we always aim to help clients overcome the operational inefficiencies, find gaps in their current processes, enable cost curtailment and better decision making. My last business intelligence implementation was for a leading pharmaceutical company in the US. The biggest challenge pharmaceutical companies face today is to reduce the cycle time involved to bring a new drug into the market. Before a drug reaches the consumer it must go through the following phases:

Discovery > Clinical Trials > Approval > Manufacturing > Marketing

How often do we find the currency and gold lying unprotected in the office cubes or corridors?  How often do we find piles of garbage in and around the office buildings? Even if we do find them occasionally, how often do we find the gist of it getting summarized, packaged and sent to most of the senior managers, along with many other goodies undetected?

Information Management recently sent around their pick of best IM blog articles of 2009.  Among them was Forrester’s James Kobelius’s reaction to Bill Inmon’s “incineration of a straw man concept that he refers to as ‘virtual data warehousing (DW).’”

Recently there’s been a lot of discussion about alternatives to the standard relational DBMS (note this widely read article).  Some of the feedback comes from OO developers.  There’s a fundamental dissonance between OO and relational approaches, requiring an intermediate object/relational mapping (ORM) layer for OO systems to operate effectively with relational DBMSs.  Others decry the overhead imposed by the relational DBMS, seeking a way to store and use massive datasets for applications needing fast response.  In response a number of vendors are now providing SaaS-like database services in the “cloud”: open source, lightly structured data services provided via the internet, capable of storing and delivering large data stores for high availability, fast response applications.

Vendors have only recently arrived on the scene with these offerings, but according to Kevin Hazzard, CapTech architect, this was inevitable: “In my own writings and talks, I sometimes call databases an unfortunate consequence of history. This sentiment isn’t borne of any sort of animosity toward databases. On the contrary, I can’t imagine doing what I do without the services that databases provide. However, I mustered just enough intellectual honesty to admit that if I were able to give a handful of 4Gb DIMMs to the creators of the ENIAC computer at University of Pittsburgh in 1948 and if I were smart enough to show them how to use that memory effectively, the database as we know it today would simply never have evolved. Instead, databases would have become an integral part of the software development model. The database would be the file system with some automatic hashing and locking capabilities.

“Well, that sounds an awful lot like Amazon’s SimpleDB, Google’s BigTable (and Hypertable), Microsoft SQL Data Services (in the cloud), and other emerging DB platforms.  The ORM trend is also pushing developers toward these models by providing language extensions for query comprehension, query optimization and parallelism in the application domain. Is there a chance that pre-relational or post-relational database engines will eclipse relational databases in the marketplace in the near-term? No way. “

Still, Data Management professionals should be on the lookout for bringing competitive edge to their BI environment with these new approaches – but carefully.  According to CapTech DM/BI specialist Ashwini Kumar, these new database approaches are “a good fit for simple data structures, structured data, and applications with queries using procedural codes. Also, they may be good fit for customized applications. However, Business Intelligence/Enterprise Information Management type applications are more dynamic in nature, especially in exploring or mining data.”  

So where to from here?  Over the coming months we’ll watch as these new database technologies emerge into wider commercial application and lessons-learned emerge.   

Here are three things to remember when putting together a BI business case:

It is a commonplace to say we should manage data like a resource. But when you think about it, data is an asset but not a resource.  Data isn’t a thing like real estate, employees, or customers, but rather it represents all of those things.  In data-geek-speak, data is a meta-resource that holds information about resources.  That makes data a lot like money.

In a recent very thoughtful post on data quality, Paul Erb plays out an analogy comparing data users with Don Quixote and data quality professionals with Sancho Panza, then reverses the analogy to cleverly coin the “Sancho Panza” test of data quality professionals.  He encourages data quality professionals promoting the critical role of data quality to apply a what would Sancho say test to ensure tha