When a data lake becomes an expensive dumping ground

The original appeal of a data lake is sensible: retain diverse raw data cheaply, preserve future analytical options and separate storage from the constraints of a traditional warehouse. The trouble begins when “store now, decide later” becomes the permanent operating model.

Data arrives without an owner. Schemas drift. Files are duplicated. Sensitive fields are copied into broad zones. Nobody knows which dataset is current, useful or safe. Compute scans grow. Retention obligations become harder to execute. The lake remains inexpensive per unit of storage while the cost of understanding every unit rises.

That is not a storage failure. It is deferred product and governance work accumulating interest.

Raw does not mean unmanaged

A raw layer should preserve source fidelity and support replay. It still needs control.

Every ingestion should carry source, owner, capture time, classification, retention, schema or format, lineage and a stable identifier. The platform should know whether a load was complete, whether it replaced or appended data and how consumers can reproduce a derived result.

“Schema on read” means structure can be interpreted at consumption. It does not mean there is no schema, quality expectation or responsibility. If every consumer has to rediscover file conventions and business meaning, the lake has shifted integration cost onto every future use case.

The warning signs

A lake is becoming a dumping ground when:

• ingestion success is measured by bytes landed rather than usable products created;
• nobody can identify an accountable owner for most datasets;
• multiple copies exist with names such as final, final_v2 or dates nobody can interpret;
• analysts build private curated layers because shared layers are not trusted;
• raw data is queried directly for recurring business reporting;
• access is granted at broad bucket or account level;
• deletion and retention requests require manual archaeology;
• compute cost rises without a clear link to consumers or outcomes;
• pipelines cannot be replayed safely; and
• the catalogue records physical objects but not endorsed use.

These signals are related. Weak ownership produces weak lifecycle decisions; weak lifecycle produces clutter; clutter increases access, cost and trust problems.

Layering helps only when promotion has meaning

Bronze, silver and gold—or raw, cleansed and curated—are useful concepts. They become decorative when teams copy data between zones without explicit acceptance criteria.

Define what promotion means.

The raw layer preserves source evidence and ingestion metadata. The cleansed layer applies validated structure, standardises representation, handles duplicates and records quality. The curated layer implements governed business meaning for an identified consumer or decision.

A dataset should not become “gold” because a scheduled job wrote it into a directory. It should have a product owner, definition, service expectations, quality evidence, security policy and active consumers.

Storage cost is the smallest visible number

Object storage prices encourage teams to retain everything. Total cost includes more:

• repeated ingestion and egress;
• cataloguing and classification;
• compute scanning poorly partitioned or duplicated data;
• backup, replication and cross-region requirements;
• security monitoring and access reviews;
• incident response for unknown sensitive data;
• engineer time discovering and reconciling datasets;
• legal and operational work to satisfy retention; and
• the opportunity cost of decisions delayed by low trust.

Tag storage and compute to domains and products. Show cost per active consumer, query and decision where feasible. Data with no owner, no use and recurring cost needs an explicit retain, archive or delete decision.

Security risk grows with uncertainty

A lake centralises attractive data while making context harder to see. Broad access, copied credentials, unmanaged notebooks and unknown classifications expand the attack surface.

Use least privilege at data-product boundaries, not only infrastructure accounts. Separate duties for ingestion, curation and administration. Tokenise or minimise sensitive fields. Propagate classification and purpose restrictions. Audit actual access, not just granted access. Treat exports and temporary workspaces as part of the estate.

Most importantly, do not ingest data that the platform cannot protect or delete appropriately. “We will govern it later” is not a control.

A recovery plan that does not require a new platform

Technology replacement is rarely the first move. Begin by making the current lake legible.

Observe usage. Identify datasets read in the last 30, 90 and 365 days; their consumers; cost; sensitivity; and downstream dependencies.

Quarantine uncertainty. Restrict new consumption of unowned or unclassified areas while preserving evidence needed for investigation.

Define product zones. Select high-value datasets and give them owners, contracts, quality indicators, access policies and lifecycle rules.

Stop uncontrolled ingestion. Require minimum metadata and ownership at the boundary. Reject or isolate feeds that do not meet it.

Remove duplication safely. Use lineage and consumer evidence, announce deprecation, measure remaining access and retain recovery windows.

Automate lifecycle. Apply retention, tiering, archive and deletion policy with exceptions that expire.

Simplify processing. Fix partitioning, file size, format and repeated transformation where measurement shows cost or reliability problems.

This creates a smaller governed surface before any architectural migration. If a new lakehouse or warehouse is still justified, the organisation now knows what to move.

Data products are not labels

Calling every table a product will not improve the lake. A data product serves an identified need and has an owner, interface, documentation, quality expectations, policy and lifecycle.

The product boundary should reduce cognitive load for consumers. It should hide unnecessary internal complexity while making limitations and evidence visible. A product with no users is a candidate, not a product.

What to measure

• storage and compute cost by domain and product;
• percentage of spend attributable to active consumers;
• critical datasets with owners, classification and retention;
• direct raw-layer use in recurring production decisions;
• duplicate and abandoned assets retired;
• time to find and gain access to an endorsed product;
• failed quality expectations and recovery time;
• sensitive-data policy violations; and
• ability to reproduce a curated output from retained inputs and code.

Do not use total petabytes as a maturity measure. Volume proves collection, not value.

When a broad raw lake is justified

Research, fraud, security and machine-learning use cases may need long-lived, detailed source data whose future value is uncertain. That is a valid reason for broad retention when legal basis, security, lineage, cost and lifecycle are managed.

The distinction is deliberate optionality versus indefinite neglect. A governed archive preserves possibility. A dumping ground merely postpones responsibility.

Adovo’s view

Cheap storage is useful because it lets enterprises preserve evidence and explore. It is dangerous because it makes every avoided decision look inexpensive.

Do not begin by draining the lake. Make ownership, use, sensitivity and cost visible. Stop adding ungoverned data. Curate the products that matter and retire what does not. A data lake becomes valuable when the organisation can explain why data is there, who may use it and what decision it improves.

Primary documentation and further reading

• Microsoft data lake architecture guide
• AWS Glue data quality