Data Lake
A data lake is a central store that holds large volumes of raw data from many sources – structured and unstructured alike – in its original format and at low cost, applying structure only when the data is analysed (schema-on-read).
A data lake is a central data store that ingests arbitrarily large volumes of raw data from many different sources in their original format – structured tables as well as semi-structured files and unstructured content such as images, log files or free text. Unlike a classic data warehouse, which forces data into a fixed model before loading, the data lake first stores everything unchanged and only defines the structure when the data is read for a specific analysis. This principle is called schema-on-read and turns the data lake into a flexible, low-cost collection point for all of a company's data.
The core idea: data need not be laboriously prepared before it is even allowed to be stored. Instead, everything available flows into a shared store – often a low-cost object store in the cloud – where it is ready for later analyses, data-science projects or machine-learning models. The data lake is therefore less a finished analytics system than a reservoir from which different tools and teams draw depending on the question at hand. Only the analysis decides what structure the data takes on.
At a glance
- Central store for raw data in its original format
- Ingests structured, semi-structured and unstructured data
- Structure emerges only at analysis time (schema-on-read)
- Low-cost and flexible, usually as a cloud object store
- Foundation for data science, machine learning and exploratory analysis
How does a data lake work?
A data lake ingests data from operational systems such as ERP, online shop, CRM, machine and sensor data or external sources and initially stores it unchanged. Instead of a fixed database schema, a low-cost object store usually serves as the foundation, holding files of any format. Only when an analysis is due are the relevant data read, interpreted and shaped into the required form. This is precisely where it differs from the warehouse: the effort for structure and cleansing arises not when storing, but when analysing.
Schema-on-read instead of schema-on-write
The classic data warehouse follows schema-on-write: the data model is fixed, and every record must fit that schema when loaded. The data lake reverses this. Following the schema-on-read principle, raw data is stored without constraints, and only the individual query determines how it is to be read and structured. This makes it possible to interpret the same data differently for different purposes, and to ingest data sources whose structure is not yet known or defined at storage time.
Zones in the data lake
In practice, a data lake is usually organised into zones to keep order despite the freedom of format. A raw zone holds the unchanged input data, a cleansed zone the checked and harmonised data, and a curated zone the datasets prepared for business use. Without such structure and without a data catalogue, the lake risks becoming a "data swamp" – an unmanageable dump of data in which no one knows anymore which data came from where or whether it is reliable.
Why a data lake matters
The value of a data lake lies in flexibility, cost and the ability to handle unstructured and very large volumes of data. Because data need not be modelled before storage, new sources can be connected quickly and analyses tried out without having to design an elaborate data model. This is especially valuable for exploratory questions where, at the outset, it is not yet clear which analysis will ultimately be needed.
For data science and machine learning in particular, the data lake is attractive because such methods often require large, raw and varied data holdings. The low-cost object store makes it economical to retain even data whose value is still unclear today but which might enable an analysis tomorrow. The flip side: without governing data quality, metadata and access rights, the effort grows with every analysis, and the results become unreliable. The data lake shifts complexity from storing to using – good data governance is therefore a prerequisite, not a nice-to-have.
Data lake and the ERP system
The ERP system is one of the most important sources for a data lake, because orders, invoices, stock, purchases and master data converge there in structured form. This ERP data is transferred into the data lake via an API or a connector and combined there with data of other origin that is not represented in the ERP itself – for example clickstreams from the online shop, sensor data from production, images or service tickets in free text.
The data lake thus complements the ERP but does not replace it. The ERP remains the leading system for day-to-day operations and for robust, structured metrics. The data lake, by contrast, serves as an overarching reservoir for varied raw data and for analyses that go beyond the ERP's built-in reporting functions. For up-to-date standard reporting in mid-sized companies, the ERP's own reports or a lean data warehouse often suffice; a data lake pays off once large, heterogeneous or unstructured volumes of data come into play.
Distinctions: data lake vs. data warehouse and lakehouse
Data lake and data warehouse are often confused but pursue opposing approaches. A data warehouse stores structured, already-prepared data according to a fixed schema and is suited to defined, recurring analyses with high reliability. A data lake stores raw data of every kind in its original format and is designed for flexibility and exploratory analysis. Put simply: the warehouse first provides order and then answers, the lake first provides room and creates order only when needed.
Data lakehouse as a hybrid
Because both concepts have strengths and weaknesses, the hybrid form of the "data lakehouse" has become established. It combines the low-cost, format-free storage of a data lake with the management and structure features of a warehouse – such as transactional safety, schema management and performant queries directly on the raw data. The aim is to unite raw-stored data and reliable reporting on a single platform instead of operating lake and warehouse separately. A data mart remains unaffected by this: it is a business-tailored subset and is typically associated with the warehouse.
Data lakes in mid-sized DACH businesses
For a long time the data lake was seen as a topic for large corporations with their own big-data teams. Cloud object stores and managed analytics platforms have lowered the barrier to entry, so that increasingly mid-sized retail and manufacturing businesses in the DACH region operate such stores too – for instance to bring shop, marketplace, ERP and machine data together in one place. What matters is a clear goal: a data lake without a concrete use case and without governance incurs costs without creating value.
Particular care is required around data protection. As soon as personal data such as customer or employee data flows into the lake, the requirements of the GDPR apply – especially purpose limitation, data minimisation and deletion concepts. It is precisely the "just in case" collecting typical of a data lake that is in tension with these principles: retaining raw personal data without a defined purpose is problematic under data protection law. With cloud services outside the EU, the rules on transfers to third countries and the storage location of the data must additionally be checked.
Example
Data lake at a multichannel retailer
A mid-sized retailer runs an online shop, sells across several marketplaces and has its own warehouse with picking hardware. Alongside the structured order and stock data from the ERP, large volumes of other data arise: click and search histories in the shop, review texts, images of returns, log files from the warehouse devices. This data does not fit into a fixed warehouse schema but could contain valuable clues about buying behaviour and reasons for returns.
The retailer sets up a data lake in a cloud object store. A connector feeds ERP data, shop events and return images in their raw form into separate zones. A data-science team reads out precisely the relevant datasets, links clickstreams with sales and trains a model that predicts returns. Monthly standard reporting remains the responsibility of the structured data warehouse – the data lake provides the exploratory basis for analyses that would not be possible there.
Frequently asked questions
Related services
Sources
Questions about Data Lake in your ERP project?
We advise vendor-neutrally – and implement it ourselves on request.