The 2026 State of Data Engineering survey of 1,101 data professionals identified that 44% still rely on cloud data warehouses as their primary paradigm, while 27% have moved to lakehouse architectures. The remaining teams use a mix of both, and 25% name legacy systems and technical debt as their biggest bottleneck. For organizations stuck in that last group, the root cause is almost always the same: the data lake was built as a storage project instead of an architecture project.
The storage itself is rarely the issue. S3 is cheap, ADLS scales well, GCS is reliable. Where data lake architecture breaks down is in the decisions made (or not made) before the first byte lands:
- how zones are structured
- which open table format governs transactions
- whether a catalog exists to make data discoverable.
Skip any of those three, and the lake drifts toward a swamp, regardless of how much you spent on compute.
This article focuses on the architectural decisions: open table format selection, catalog and metastore strategy, AI-specific zone design, and the concrete triggers for evolving a lake into a lakehouse. If you already know what a data lake is, this is the article about how to build one that holds up in production.
Summary
- Data lake architecture fails when teams treat it as a storage problem. Three decisions made before ingestion determine success: zone structure, open table format, and metadata catalog.
- Open table formats (Iceberg, Delta Lake, Hudi) are now essential. The 2026 State of Data Engineering survey found that 27% of data professionals already use lakehouse architectures built on these formats.
- AI workloads need specific architectural patterns. Feature store integration, unstructured data pipelines, and model training data lineage require purpose-built zones that traditional lake designs don’t include.
- Governance cannot be an afterthought. 25% of data professionals cite legacy systems and technical debt as their biggest bottleneck. Most of that debt accumulates from deferred governance decisions.
What is data lake architecture?
Unlike data warehouses that enforce schema-on-write, data lakes accept data in its original format, making them well-suited for exploratory analytics, log processing, and training machine learning models. The architecture encompasses ingestion pipelines, storage layers, processing engines, metadata catalogs, and governance frameworks that work together to keep data accessible, trustworthy, and queryable.
Core data lake design patterns
Medallion architecture (bronze, silver, gold)
The medallion pattern, popularized by Databricks, organizes data into three quality tiers.
- The bronze layer holds raw, unprocessed data exactly as ingested.
- Silver applies cleaning, deduplication, and schema enforcement.
- Gold serves curated, business-ready datasets optimized for analytics and reporting.
This works well when different teams need data at different stages of refinement. Data scientists might query bronze for raw signals, while finance teams rely on gold for reconciled numbers. The medallion architecture also simplifies debugging, because every transformation step is preserved and replayable.
Data lake zones (landing, raw, curated, sandbox)
Zone-based architecture organizes the lake by access patterns and data maturity rather than quality tiers.
A typical layout includes:
- a landing zone (temporary staging for incoming data)
- a raw zone (immutable, append-only storage)
- a curated zone (governed, validated datasets)
- a sandbox zone (experimental space for data science teams).
Zones enforce different security and governance rules: the raw zone might restrict access to data engineering teams only, while the sandbox zone allows broader access with reduced governance overhead. The key decision is how many zones to create. Xenoss engineers recommend starting with three or four and expanding only when a clear business need arises. Over-engineering zones adds complexity without adding value.
Lambda and kappa architectures
Lambda architecture runs batch and real-time processing in parallel, merging results in a serving layer. It handles historical reprocessing well, but creates maintenance overhead because teams maintain two codebases.
Kappa architecture simplifies this by treating all data as a stream, replaying historical data through the same streaming pipeline when reprocessing is needed.
For enterprise use cases in 2026, kappa-influenced designs (stream-first, with batch as a fallback) are gaining traction. Apache Kafka and Confluent Cloud support this pattern natively, and platforms like Databricks unify batch and streaming under a single API.
Three decisions to make before your first ingestion pipeline runs
Across Xenoss client engagements, data lakes that succeed share one trait: the team made three explicit architectural decisions before ingesting data. Each decision, if deferred or skipped, creates compounding problems as the lake grows.
The sequence matters: zones define the physical structure, the open table format defines transactional behavior within those zones, and the catalog makes everything discoverable. Skipping any of the three means the next one cannot function properly.
Open table formats: Choosing between Iceberg, Delta Lake, and Hudi
Open table formats bring warehouse-grade capabilities (ACID transactions, time travel, schema evolution) to data lake storage.
27% of data professionals now use lakehouse architectures, up significantly from prior years. Three formats dominate the space.
| Format | Best for | Strengths | Considerations |
|---|---|---|---|
| Apache Iceberg | Multi-engine environments (Spark, Trino, Flink, Presto) and teams avoiding vendor lock-in | Engine-agnostic design, hidden partitioning, strong community momentum across AWS, Snowflake, Databricks | Newer ecosystem, fewer mature tooling integrations than Delta Lake |
| Delta Lake | Databricks-centric environments and teams already on Spark | Tight Spark integration, mature tooling, strong documentation, built-in optimization (Z-ordering, liquid clustering) | Historically tighter coupling to Databricks, though open-source compatibility is improving |
| Apache Hudi | Streaming-heavy workloads with frequent upserts and CDC | Record-level upserts, incremental processing, designed for streaming-first architectures | Smaller community than Iceberg or Delta. Best suited for specific ingestion patterns |
In practice, the market is converging toward Apache Iceberg as the default for new deployments. AWS, Snowflake, and Databricks all now support Iceberg REST catalogs, and the format’s engine-agnostic design aligns with the multi-cloud direction most enterprises are moving toward. For teams already invested in Databricks, Delta Lake remains a strong choice. Hudi is best suited for teams with heavy CDC and streaming upsert requirements.
Why this matters: Choosing a table format after data is already in the lake means migrating terabytes of files and rewriting transformation logic. The format decision should be locked before the first ingestion pipeline runs.
Data lake vs lakehouse: When to evolve your architecture
The lakehouse concept merges the flexibility of data lakes with the transactional guarantees of data warehouses. In the 2026 State of Data Engineering survey, 44% of respondents still use cloud data warehouses as their primary paradigm, while 27% have adopted lakehouse architectures. The remaining teams use a mix of both.
A pure data lake makes sense when the primary consumers are data scientists and ML engineers who need raw, flexible access to diverse data types. A lakehouse becomes necessary when business analysts, BI tools, and governance requirements enter the picture. The lakehouse adds structure without losing flexibility.
The practical trigger for migration is usually the moment when a team needs to run both SQL analytics and ML training on the same data. In a pure lake, maintaining separate ETL pipelines for each use case is required. In a lakehouse, both workloads read from the same governed, transactionally consistent tables.
Why this matters: Premature lakehouse adoption adds complexity without business value. But delaying it too long means accumulating technical debt in the form of duplicated datasets, inconsistent metrics, and ungoverned ML training data. Xenoss engineers recommend evaluating the transition when the data pipeline count exceeds 50 or when more than three teams consume the same datasets for different purposes.
Architecting data lakes for AI and ML workloads
85% of Lakehouse users are either developing AI models or plan to. At the same time, 36% cite governance as a major challenge for AI-driven analytics. Teams are pushing AI workloads onto data lakes that were designed for dashboards and batch reporting. The architecture gaps only become visible when the first ML pipeline goes to production.
AI workloads place four specific demands on data lake architecture that traditional designs don’t address.
- Feature store integration. ML models consume features, not raw tables. A feature store (such as Feast, Tecton, or Databricks Feature Store) sits between the curated zone and the training pipeline, providing versioned, point-in-time correct feature sets. The data lake must support the feature store’s read patterns, which typically involve large sequential scans for training and low-latency lookups for inference.
- Unstructured data pipelines. Text documents, images, audio, sensor readings, and log files are increasingly valuable for AI use cases. The data lake needs a dedicated zone for unstructured data with its own ingestion and cataloging pipeline. Parquet and Iceberg work well for structured features, but unstructured data often requires object-level metadata tagging and separate indexing.
- Training data lineage. Regulatory and compliance requirements increasingly demand traceability from model predictions back to training data. The catalog must track which datasets were used to train which model version, including the specific time-travel snapshot. Without this lineage, models in regulated industries (banking, healthcare, insurance) cannot pass an audit.
- Data versioning and reproducibility. ML experiments require reproducing exact training conditions. Open table formats with time-travel support (Iceberg, Delta Lake) enable this by letting teams query the lake as it existed at any point in time. The architecture must preserve historical snapshots long enough to support experiment reproducibility, which means retention policies need to account for ML workflows, not just analytics use cases.
Why this matters: The data lake is increasingly the foundation for AI, not just analytics. Architectures that don’t account for ML-specific requirements will need expensive retrofitting as AI adoption scales.
Data lake governance: Three failure patterns and how to avoid them
One in two Chief Data and Analytics Officers now considers optimizing the technology landscape a primary responsibility. That urgency exists because governance failures compound faster than most teams expect. Data lakes degrade through three specific patterns.
Missing metadata. Without a catalog that describes what each dataset contains, who owns it, and when it was last updated, the lake becomes unsearchable. Teams create duplicate copies of the same data rather than finding the authoritative source. Storage costs grow while data utility shrinks.
Absent ownership. When no team is accountable for a dataset’s quality, accuracy degrades silently. Stale records, schema drift, and broken pipelines go unnoticed until a downstream report produces wrong numbers. Data mesh principles (domain ownership, data-as-a-product) solve this by assigning clear accountability to the team closest to the data source.
Deferred governance decisions. The most common mistake is treating governance as a future initiative. Teams plan to add access controls, quality monitoring, and retention policies “later,” after the lake is operational.
By the time “later” arrives, the lake holds terabytes of ungoverned data, and retroactive governance becomes a multi-month remediation project. 25% of data professionals cite legacy systems and technical debt as their single biggest bottleneck. Much of that debt originates from governance decisions that were deferred during the initial build.
Bottom line
Data lake architecture is a solved problem in the sense that the design patterns are well understood. Medallion zones, open table formats, and metadata catalogs have been validated across thousands of enterprise deployments. The architecture fails when teams skip the foundational decisions.
The practical checklist is short: define your zone structure before ingesting data, select an open table format before building pipelines, and deploy a metadata catalog before granting access. These three decisions, made upfront, prevent the governance drift that turns data lakes into swamps.
For teams preparing to serve AI workloads, the architecture needs to go further: feature store integration, unstructured data zones, training data lineage, and experiment-grade versioning. These are not future requirements. With 82% of data professionals already using AI tools daily, they are current ones.
