The traditional EDW, however, has been sidelined even at a time when data is exploding at a tremendous rate and new data types, from sensor data to smartphone and social media data to video data are becoming common. IBM recently projected a 44-fold increase in data and content, reach 35 zettabytes by 2020. In short, the world of data has changed dramatically since organizations began building conventional data warehouses. Now the EDW should accommodate these new types of data and be flexible enough to handle rapidly changing forms of data.
Data warehousing as it is mainly practiced today is too complex, difficult to deploy, requires too much tuning, and is too inefficient when it comes to bringing in analytics, which delays delivering the answers from the EDW that business managers need, observed Phil Francisco, VP at Netezza, an IBM acquisition that makes data warehouse appliances. And without fast analytics to deliver business insights, well, what’s the point?
In addition, the typical EDW requires too many people to maintain and administer, which makes it too costly, Francisco continued. Restructuring the conventional EDW to accommodate new data types and new data formats—in short, a new enterprise data model—is a mammoth undertaking that companies wisely shy away from. But IBM is moving beyond basic EDW to something Francisco describes as an enterprise data hub, which entails an enterprise data store surrounded by myriad special purpose data marts and special purpose processors for various analytics and such.
IBM’s recommendation: evolve the traditional enterprise data warehouse into what it calls the enterprise data hub, a more flexible systems architecture. This will entail consolidating the infrastructure and reducing the data mart sprawl. It also will simplify analytics, mainly by deploying analytic appliances like IBM’s Netezza. Finally, organizations will need data governance and lifecycle management, probably through automated policy-based controls. The result should be better information faster and delivered in a more flexible and cost-effective way.
Ultimately, IBM wants to see organizations build out this enterprise data hub with a variety of BI and analytic engines connected to it for analyzing streamed data and vast amounts of unstructured data of the type Hadoop has shown itself particularly good at handling. BottomlineIT wrote about Hadoop in the enterprise back in February here.
The payback from all of this, according to IBM, will be increased enterprise agility and faster deployment of analytics, which should result in increased business performance. The consolidated enterprise data warehouse also should lower the TCO for the EDW and speed time to business value. All desirable things, no doubt, but for many organizations this will have require a gradual process and a significant investment in new tools and technologies, from specialized appliances to analytics.
Case in point is Florida Hospital, Orlando, which deployed a z10 mainframe with DB2 10, which provides enhanced temporal data capabilities, with the primary goal of converting its 15 years of clinical patient data into an analytical data warehouse for use in leading edge medical and genetics research. The hospital calls for getting the data up and running on DB2 10 this year and attaching the Smart Analytics Optimizer as an appliance in Q1 2012. Then it can begin cranking up the research analytics. Top management has bought into this plan for now, but a lot can change in the next year, the earliest the first fruits of the hospital’s analytical medical data exploration are likely to hit.
Oracle has its own EDW success stories here. Hotwire, a leading discount travel site, for example, works with major travel providers to help them fill seats, hotel rooms, and rental cars that would otherwise go unsold. It deployed Oracle’s Exadata Database Machine to improve data warehouse performance and to scale for growing business needs.
IBM does not envision the enterprise data hub as a platform-specific effort. Although EDW runs on IBM’s mainframe much of the activity is steered to the company’s midsize UNIX/Linux Power Systems server platform. Oracle and HP offer x86-based EDW platforms, and HP is actively partnering with Microsoft on its EDW offering.
In an IBM study, 50% business managers complained they don’t have the information they need to do their jobs and 60% of CEOs admitted they need to do a better job of capturing and understanding information rapidly in order to make swift business decisions. That should be a signal to revamp to your EDW now.