Build a data lakehouse ingestion: schema-on-write, quality checks, and lineage.
- User
Build a data lakehouse ingestion: schema-on-write, quality checks, and lineage.
Assistant
The user wants to build a data lakehouse ingestion system. The key requirements are:
- Schema-on-write: Defining the structure before loading data.
- Quality checks: Ensuring data integrity and correctness.
- Lineage: Tracking where data comes from and where it goes.
I need to provide a technical guide or a reference architecture for this. Python (with libraries like Pandas, PyArrow, or frameworks like Spark/DBT) or using a cloud service (like Azure Data Factory/Databricks or AWS Glue) are common approaches. Since no specific technology stack was requested, I will use a modern, open-source Python-based stack using Pandas/PySpark, Great Expectations (for quality), Delta Lake (for the lakehouse format), and OpenLineage (for lineage) concepts, or perhaps a simpler framework like Dagster or Airflow to tie it together.
Let's go with a Python-based approach using Pandas for logic, Great Expectations (or simple assertions) for quality, and Delta Lake for storage. For lineage, manual tracking or a framework like Marquez/OpenLineage is ideal, but for a "build" request, code examples are best.
Plan:
- Architecture Overview: Explain the components (Source -> Ingestion -> Quality -> Sink -> Lineage).
- Schema-on-Write: Show how to define a schema (e.g., using PyArrow or SQL DDL) and enforce it during write.
- Quality Checks: Implement validation (e.g., null checks, range checks).
- Lineage: Explain how to emit events or track data flow programmatically.
- Code Example: A complete Python script demonstrating these concepts using
pandas,pyarrow(for schema), anddelta-lake.
Detailed thought process on the code:
- Source: Generate dummy data.
- Schema: Define a
pyarrowschema. - Validation: Use a simple custom validator class or Great Expectations (GE is heavy for a simple snippet, maybe just a dictionary of rules).
- Write: Save as Parquet or Delta.
- Lineage: Print metadata or send a JSON payload (simulating an OpenLineage event).
Refining the "Lineage" part: Real lineage usually requires a backend (like DataHub or Marquez). Since I can't spin up a backend in a text response, I will simulate the generation of an OpenLineage event.
Refining the "Schema-on-Write" part: Ideally, in a Lakehouse (like Delta Lake), schema enforcement is a feature. I should highlight that.
Technology Choice: Python + Delta Lake (using
deltalakelibrary) is the most modern standard for "Lakehouse". Or I could use Spark. Spark is more enterprise-standard for this. Let's stick to Python withdeltalakepackage for simplicity, or a Spark snippet if the user implies scale. I will provide a Python solution usingpandasanddeltalakelibrary for accessibility, but mention Spark.Let's structure the response:
- Conceptual Architecture: High-level design.
- The Stack: Tools selected (Pandas, Delta Lake, Great Expectations/Great Expectations alternative, OpenLineage).
- Step-by-Step Implementation (Python Code):
- Define Schema.
- Ingest & Validate.
- Write with Enforcement.
- Emit Lineage.
Self-Correction during drafting: "Schema-on-write" specifically contrasts with "schema-on-read". It means we reject data that doesn't fit. The
deltalakewriter has amode="overwrite"with schema merging options, but strict enforcement means rejecting mismatched columns.Code Structure:
import pandas as pd from deltalake import write_deltalake import pyarrow as pa # 1. Schema Definition schema = pa.schema([...]) # 2. Ingestion + Quality def process_data(raw_df): # Clean # Check return clean_df # 3. Write (Schema-on-write) write_deltalake(table, df, schema_mode="overwrite") # Check params # 4. Lineage # Construct a JSON object describing the run.Let's polish the Lineage section. OpenLineage is the standard. I will show what a lineage event looks like.
Final Output Structure:
- Architecture Diagram (Textual).
- Key Concepts (Defining Schema, DQ, Lineage).
- Implementation (The Code).
- Production Considerations (Orchestration, Catalogs).
This looks solid. Building a robust Data Lakehouse ingestion pipeline requires shifting from traditional "schema-on-read" (dumping raw files) to "Schema-on-Write" (strict contracts) to ensure high-quality analytics.
Below is a comprehensive guide and architectural implementation using Python, Delta Lake (the de facto Lakehouse format), and Great Expectations (for quality), following OpenLineage standards.
1. Architecture Overview
We will build a "Medallion Architecture" (Bronze -> Silver) ingestion step.
- Source: Raw Data (e.g., CSV, JSON, API).
- Schema Enforcement: Converting raw loose data to strictly typed Apache Arrow/Parquet schema.
- Quality Checks: Validating against expectations before writing.
- Storage: Writing to a Delta Table (ACID transactions).
- Lineage: Emitting metadata to an OpenLineage backend (like Marquez or DataHub).
2. Prerequisites
You would typically need the following libraries:
pip install pandas deltalake great_expectations openlineage-python
3. The Implementation
Here is a complete, modular Python class that handles ingestion, schema enforcement, quality checks, and lineage emission.
import pandas as pd import pyarrow as pa from deltalake import write_deltalake, DeltaTable from datetime import datetime import uuid import json # --- Mock OpenLineage Client --- # In production, you would use: from openlineage.client import OpenLineageClient class LineageTracker: def __init__(self, namespace="my_lakehouse"): self.namespace = namespace self.run_id = str(uuid.uuid4()) def _create_facets(self, schema, stats): return { "schema": schema, "dataSource": {"name": self.namespace}, "stats": stats } def log_event(self, job_name, inputs, outputs, schema_fields, stats): event = { "eventType": "COMPLETE", "eventTime": datetime.utcnow().isoformat(), "run": {"runId": self.run_id}, "job": {"namespace": self.namespace, "name": job_name}, "inputs": inputs, "outputs": outputs, "facets": self._create_facets(schema_fields, stats) } # In production, send to Marquez/DataHub via HTTP print(f"📡 LINEAGE EVENT: {json.dumps(event, indent=2)}") # --- Ingestion Engine --- class LakehouseIngester: def __init__(self, table_name, layer="bronze"): self.table_name = table_name self.layer = layer self.path = f"./lakehouse_db/{layer}/{table_name}" self.lineage = LineageTracker() def enforce_schema(self, df: pd.DataFrame, schema: pa.Schema) -> pa.Table: """ Schema-on-Write: Convert Pandas DF to strict PyArrow schema. This will fail if types cannot be cast. """ try: # Cast columns to match schema table = pa.Table.from_pandas(df) # Use PyArrow's cast to enforce specific types (e.g., string to timestamp) return table.cast(schema) except Exception as e: raise ValueError(f"Schema Enforcement Failed: {e}") def run_quality_checks(self, table: pa.Table, expectations: list) -> bool: """ Run validation rules. Returns True if passed, raises Error if failed. """ df = table.to_pandas() print("⚙️ Running Quality Checks...") for rule in expectations: col = rule['column'] check = rule['check'] value = rule.get('value') if check == "no_nulls": assert df[col].notna().all(), f"Quality Fail: Nulls found in {col}" elif check == "unique": assert df[col].nunique() == len(df), f"Quality Fail: Duplicates in {col}" elif check == "min_value": assert df[col].min() >= value, f"Quality Fail: Min value violation in {col}" print(f" ✅ Passed: {col} - {check}") return True def ingest(self, source_df: pd.DataFrame, schema: pa.Schema, expectations: list): start_time = datetime.now() # 1. Define Lineage Inputs (Source System) input_facet = {"namespace": "source_system", "name": f"raw_{self.table_name}"} # 2. Schema-on-Write Enforcement print("🔨 Enforcing Schema...") arrow_table = self.enforce_schema(source_df, schema) # 3. Quality Checks self.run_quality_checks(arrow_table, expectations) # 4. Write to Delta Lake (Transactional) print(f"💾 Writing to Delta Table: {self.path}") write_deltalake( self.path, arrow_table, mode="overwrite", engine="rust" ) # 5. Generate Statistics for Lineage stats = { "rowCount": len(arrow_table), "sizeBytes": arrow_table.nbytes, "processingTimeMs": (datetime.now() - start_time).milliseconds } # 6. Log Lineage (Output) output_facet = { "namespace": self.lineage.namespace, "name": f"{self.layer}.{self.table_name}", "outputFacets": { "schema": [field.name for field in schema] } } self.lineage.log_event( job_name=f"ingest_{self.table_name}", inputs=[input_facet], outputs=[output_facet], schema_fields=[f"{f.name}:{f.type}" for f in schema], stats=stats ) # --- Execution Example --- # 1. Define strict Schema (The Contract) user_schema = pa.schema([ ("user_id", pa.int64()), ("email", pa.string()), ("signup_ts", pa.timestamp('s')), ("age", pa.int64()) ]) # 2. Define Quality Rules quality_rules = [ {"column": "user_id", "check": "unique"}, {"column": "email", "check": "no_nulls"}, {"column": "age", "check": "min_value", "value": 0} ] # 3. Mock Raw Data raw_data = pd.DataFrame({ 'user_id': [1, 2, 3], 'email': ['a@test.com', 'b@test.com', 'c@test.com'], 'signup_ts': ['2023-01-01', '2023-01-02', '2023-01-03'], 'age': [25, 30, 35] }) # 4. Run Ingestion ingester = LakehouseIngester(table_name="users", layer="silver") ingester.ingest(raw_data, user_schema, quality_rules)
4. Breakdown of Key Concepts
A. Schema-on-Write
In traditional data lakes, you dump a CSV and figure out the columns later ("Schema-on-Read"). This creates analytics debt.
- The Implementation: We define a
pa.schema(PyArrow Schema). - The Enforcement:
table.cast(schema)forces the data types. If the raw data contains a string "foo" in an integer column defined in the schema, the ingestion fails immediately. This prevents "garbage in, garbage out."
B. Quality Checks
Quality should not be an afterthought. It should be a gatekeeper before the write operation commits.
- The Logic: We iterate through a list of
expectations. - Fail-Fast: If
assertfails, the Python script crashes, and thewrite_deltalakecommand is never reached. This ensures that the Lakehouse table never contains invalid data.
C. Lineage
Lineage answers "Where did this data come from?" and "What did we do to it?".
- OpenLineage Standard: The
LineageTrackerclass constructs a JSON payload following the OpenLineage spec. - Metadata: It captures:
- Inputs: The logical name of the source system.
- Outputs: The specific table in the Lakehouse.
- Facets: Schema details (column names/types) and Statistics (row count/size).
- Integration: In a real production environment, this JSON is sent to a lineage backend (like DataHub or Marquez) which visualizes the graph.
D. The Lakehouse Format (Delta Lake)
Using
deltalake(or Apache Iceberg) provides:- ACID Transactions: Ensures readers don't see partial data while the ingest is writing.
- Time Travel: Because we wrote successfully, you can query
VERSION AS OF 1if a future ingestion breaks the data. - Schema Evolution: If you need to add a column later, Delta handles the metadata update gracefully.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.