As the world of big data continues to expand, a growing number of storage options have emerged. When compared, data lakes vs. data hubs vs. data warehouses all have similar core functions, and are often mistaken as interchangeable terms. While all three structures are used to house data for business reporting and analysis, they often store different types of data and work with different data generation systems and access standards.

The data lake vs data warehouse argument is not always well-defined, with the term ‘data lake’ often used when something doesn’t fit the traditional data warehouse architecture. While the use of multiple terms has been criticised in some quarters, it’s important to understand the differences between data structures in order to find a solution that works for your organisation,

What is a data lake?

A data lake is a system or data repository used to store data in its natural format. Coined in 2010 by James Dixon, then chief technology officer at Pentaho, this term typically refers to a single store of all enterprise data in the form of files or object blobs. Whether it’s source system data or other unprocessed data, a data lake refers to a vast pool of raw data for which the purpose has yet to be defined.

The information stored in a data lake is both highly accessible and quick to update, which makes it ideal for tasks such as reporting, visualisation, analytics and machine learning. While the raw nature of the data requires a large storage capacity compared to other repository systems, organisations may prefer to use a data lake when they need a malleable system that is quicker to analyse and better suited to machine learning. According to an Aberdeen survey, organisations who’ve implemented a data lake outperformed similar companies by nine per cent in terms of organic revenue growth.

A data lake can be implemented in numerous ways, including structured data from relational databases, semi-structured data, unstructured data, and raw binary data. A number of cloud services already use this fast-growing method of data storage, including Azure Data Lake and Amazon S3. Distributed file systems such as Apache Hadoop are another example of data lake technology that continues to grow in popularity.

What is a data warehouse?

A data warehouse is a data storage system used for reporting and data analysis. Also known as an enterprise data warehouse, this type of repository system deals with data that has been uploaded directly from the operational systems of a business. Unlike a data lake, a data warehouse only deals with processed data, which offers advantages in terms of storage space and accessibility to a larger audience.

A data warehouse is used to create ongoing analytical reports, and is therefore considered a core component of business intelligence. Most warehouses are based on a standard ETL (extract, transform, and load) system that uses staging, data integration, and access layers to house critical business functions. Enterprises will normally choose a data warehouse over a data lake when they need data from operational systems to be ready and waiting for analysis. A data warehouse is more accessible to a wider range of people, and is generally used by business analysts rather than data scientists.

A number of different methods can be used to organise a data warehouse, with overall functionality defined by the hardware, software, and data resources that define the specific warehouse architecture. Organisations can utilise a dimensional approach or a normalised approach, with the dimensional approach partitioning data into “facts” and “dimensions” and the normalised approach creating data segments according to conventional database normalisation rules. Popular data warehousing solutions include Amazon Redshift, Panoply, and BigQuery among others.

What is a data hub?

A data hub is a simple collection of organised data objects from multiple sources. Distributed in the form of a hub and spoke architecture, a data hub is useful when businesses want to share and distribute data efficiently in one or more desired formats. Although a data hub shares many similarities with a data warehouse, a hub is not limited to operational data and is typically unintegrated and available at different scalable grains.

Whilst not limited to operational data like a data warehouse or operational store, it also differs from a data lake by providing access to homogenised data in multiple desired formats. Instead of storing data in one place to ensure easy access, a data hub utilises multiple locations and formats to enable easy de-duplication, quality, security, and standardisation across an enterprise. Data hub architecture enables intermediate nodes to store and execute a variety of information and access key business systems. While not as cost-effective in terms of storage, this approach offers real operational efficiencies and drives agility.

There are a range of storage products which function as data hubs, some of which also function as data lakes and other repositories. Leading products include Apache Hadoop, Google MapReduce, Cloudera CDH, Cassandra, CKAN, and Quandl among others. Enterprises choose data hubs when they want to benefit from a hub and spoke architecture, data normalisation, security, and flexibility.

Data hubs don’t store transaction information and are often small in comparison to data lakes and data warehouses. A data hub can be a great option for enterprises who want to benefit from more efficient data quality, sharing, scalability, and greater access to query services.

Data hub vs. data lake vs. data warehouse comparison table

Large enterprises continue to search for new and efficient ways to manage their big data. There are more options out there than ever, with businesses needing to make tough decisions based on costs, storage capacity, and operational needs. Compromises often need to be made along the way, with different storage formats and database types balanced against the need to access and transform data in a useful format.

Related Articles

• Post & Infographic: Data vs. Information: What’s the Difference? 
• Post: What is Information Management? Organisational Barriers & Consequences of Poor IM
• Post & Infographic: Digital Asset Management, Federated Records Management, Master Data Management Explained