Big Data Platform Strategies: Considerations While Choosing a Platform
More and more organizations are starting the journey to become a data-driven enterprise. The ability to analyze and optimize large volumes of data to make better decisions and products is a formidable competitive advantage. In my last article, I outlined reasons why the big data analytics can fail. In this article, we’ll cover points to consider for developing a successful data analytics program.
A successful data analytics program, like any other organizational initiative, must work on two counts: – personnel and technology. A data-driven culture is something top management must introduce into the culture of the organization if technology is to have a profound impact on the success of the big data analytics program.
Let us first look at the personnel or cultural aspects of a data analytics program. Much has been said about the demand and supply mismatch in the analytics domain and how certain types of analytical professionals are the real panacea for all of the analytical woes of an organization. However, the real mantra for success lies in adoption.
A big data analytics platform (or any other technology product) cannot bring results unless it adequately empowers users to perform their own analyses in a simplistic manner, without being too dependent on anyone else. Citizen business intelligence professional ideology is merely this concept. In a successful data analytics platform which all stakeholders can use, the first attribute your analytics platform must have is simplicity. A simple and powerful analytics platform that business users can work with, without the need to learn arcane terminologies and highly sophisticated statistical algorithms, is will go a long way towards making your analytics platform a success. Unless each and every stakeholder actively engages with the platform, the platform cannot deliver the resulted expected.
While the cultural aspect of analytics can be driven by leadership through careful implementation of “data-driven strategies,” the technology aspect is more complex. Organizations can choose from two approaches to the architecture of a big data platform.
A “Contraption of Components” Architecture Approach
The current avatar of most big data platforms is a mix and match combination of specialized tools that only perform a specific function. Typical “contraption of components” architecture may include:
- Connectors or a data integration tool to transmit data into the operational data store (ODS) for additional operations
- NoSQL or Hadoop database storage
- Reporting and visualization tools or a custom visualization layer
Today’s rapidly evolving technologies means that too often, as soon as a platform becomes widely adopted, an individual component becomes obsolete and must be upgraded or replaced. The consequences of this are rework and frequent changes in the platform. With a multi-tool big data platform, a minor change in one of the tools necessitates all of the other components in the architecture must also be changed. Imagine, you have just added a new data source for analysis and you must change everything, right from integration layer to the visualization layer, even though all you have added is a few more columns.
Thus, the platform must not only address analytics, but how integration affects the many different components as well.
Unified Analytics Approach
In this approach, a single end-to-end analytics platform caters to the entire gamut of analytics functions, namely data ingestion, data processing, data analysis and data presentation (i.e., reporting and visualization). The advantage of this approach is that it helps you focus on the analytics, rather than integration and compatibility of individual components. This approach also simplifies the data analyst’s tasks by providing a unified way of handling all of the data operations through a single interface, rather than using highly sophisticated coding languages for each of the functions in the analytical pipeline (ingestion, processing, analysis and presentation) like one has to do in the “contraption of components” approach.
An important point to consider with this approach is extensibility of the platform. While the “contraption of components” approach has its disadvantages, it empowers you to extend your platform by replacing a component easily. In the unified analytics approach, to include a feature that has not been built in as yet, you must wait for a newer version, which typically can takes months. An extensible platform, which allows you to extend its functionality by simply writing your own custom code, really provides the golden combination of agility and speed — which is extremely important in analytics.
When evaluating big data platforms, an API-based architectural approach, where each component of the data platform architecture can integrate with disparate data sources and formats, as well as share raw data and processed aggregates with other components, is the best approach as you look to the future.