Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a data lakehouse ingestion: schema-on-write, quality checks, and lineage.

  1. User

    Build a data lakehouse ingestion: schema-on-write, quality checks, and lineage.

  2. GLM Assistant

    The user wants to build a data lakehouse ingestion system. The key requirements are:

    1. Schema-on-write: Defining the structure before loading data.
    2. Quality checks: Ensuring data integrity and correctness.
    3. Lineage: Tracking where data comes from and where it goes.

    I need to provide a technical guide or a reference architecture for this. Python (with libraries like Pandas, PyArrow, or frameworks like Spark/DBT) or using a cloud service (like Azure Data Factory/Databricks or AWS Glue) are common approaches. Since no specific technology stack was requested, I will use a modern, open-source Python-based stack using Pandas/PySpark, Great Expectations (for quality), Delta Lake (for the lakehouse format), and OpenLineage (for lineage) concepts, or perhaps a simpler framework like Dagster or Airflow to tie it together.

    Let's go with a Python-based approach using Pandas for logic, Great Expectations (or simple assertions) for quality, and Delta Lake for storage. For lineage, manual tracking or a framework like Marquez/OpenLineage is ideal, but for a "build" request, code examples are best.

    Plan:

    1. Architecture Overview: Explain the components (Source -> Ingestion -> Quality -> Sink -> Lineage).
    2. Schema-on-Write: Show how to define a schema (e.g., using PyArrow or SQL DDL) and enforce it during write.
    3. Quality Checks: Implement validation (e.g., null checks, range checks).
    4. Lineage: Explain how to emit events or track data flow programmatically.
    5. Code Example: A complete Python script demonstrating these concepts using pandas, pyarrow (for schema), and delta-lake.

    Detailed thought process on the code:

    • Source: Generate dummy data.
    • Schema: Define a pyarrow schema.
    • Validation: Use a simple custom validator class or Great Expectations (GE is heavy for a simple snippet, maybe just a dictionary of rules).
    • Write: Save as Parquet or Delta.
    • Lineage: Print metadata or send a JSON payload (simulating an OpenLineage event).

    Refining the "Lineage" part: Real lineage usually requires a backend (like DataHub or Marquez). Since I can't spin up a backend in a text response, I will simulate the generation of an OpenLineage event.

    Refining the "Schema-on-Write" part: Ideally, in a Lakehouse (like Delta Lake), schema enforcement is a feature. I should highlight that.

    Technology Choice: Python + Delta Lake (using deltalake library) is the most modern standard for "Lakehouse". Or I could use Spark. Spark is more enterprise-standard for this. Let's stick to Python with deltalake package for simplicity, or a Spark snippet if the user implies scale. I will provide a Python solution using pandas and deltalake library for accessibility, but mention Spark.

    Let's structure the response:

    1. Conceptual Architecture: High-level design.
    2. The Stack: Tools selected (Pandas, Delta Lake, Great Expectations/Great Expectations alternative, OpenLineage).
    3. Step-by-Step Implementation (Python Code):
      • Define Schema.
      • Ingest & Validate.
      • Write with Enforcement.
      • Emit Lineage.

    Self-Correction during drafting: "Schema-on-write" specifically contrasts with "schema-on-read". It means we reject data that doesn't fit. The deltalake writer has a mode="overwrite" with schema merging options, but strict enforcement means rejecting mismatched columns.

    Code Structure:

    import pandas as pd
    from deltalake import write_deltalake
    import pyarrow as pa
    
    # 1. Schema Definition
    schema = pa.schema([...])
    
    # 2. Ingestion + Quality
    def process_data(raw_df):
        # Clean
        # Check
        return clean_df
    
    # 3. Write (Schema-on-write)
    write_deltalake(table, df, schema_mode="overwrite") # Check params
    
    # 4. Lineage
    # Construct a JSON object describing the run.
    

    Let's polish the Lineage section. OpenLineage is the standard. I will show what a lineage event looks like.

    Final Output Structure:

    1. Architecture Diagram (Textual).
    2. Key Concepts (Defining Schema, DQ, Lineage).
    3. Implementation (The Code).
    4. Production Considerations (Orchestration, Catalogs).

    This looks solid. Building a robust Data Lakehouse ingestion pipeline requires shifting from traditional "schema-on-read" (dumping raw files) to "Schema-on-Write" (strict contracts) to ensure high-quality analytics.

    Below is a comprehensive guide and architectural implementation using Python, Delta Lake (the de facto Lakehouse format), and Great Expectations (for quality), following OpenLineage standards.


    1. Architecture Overview

    We will build a "Medallion Architecture" (Bronze -> Silver) ingestion step.

    • Source: Raw Data (e.g., CSV, JSON, API).
    • Schema Enforcement: Converting raw loose data to strictly typed Apache Arrow/Parquet schema.
    • Quality Checks: Validating against expectations before writing.
    • Storage: Writing to a Delta Table (ACID transactions).
    • Lineage: Emitting metadata to an OpenLineage backend (like Marquez or DataHub).

    2. Prerequisites

    You would typically need the following libraries:

    pip install pandas deltalake great_expectations openlineage-python
    

    3. The Implementation

    Here is a complete, modular Python class that handles ingestion, schema enforcement, quality checks, and lineage emission.

    import pandas as pd
    import pyarrow as pa
    from deltalake import write_deltalake, DeltaTable
    from datetime import datetime
    import uuid
    import json
    
    # --- Mock OpenLineage Client ---
    # In production, you would use: from openlineage.client import OpenLineageClient
    class LineageTracker:
        def __init__(self, namespace="my_lakehouse"):
            self.namespace = namespace
            self.run_id = str(uuid.uuid4())
            
        def _create_facets(self, schema, stats):
            return {
                "schema": schema,
                "dataSource": {"name": self.namespace},
                "stats": stats
            }
    
        def log_event(self, job_name, inputs, outputs, schema_fields, stats):
            event = {
                "eventType": "COMPLETE",
                "eventTime": datetime.utcnow().isoformat(),
                "run": {"runId": self.run_id},
                "job": {"namespace": self.namespace, "name": job_name},
                "inputs": inputs,
                "outputs": outputs,
                "facets": self._create_facets(schema_fields, stats)
            }
            # In production, send to Marquez/DataHub via HTTP
            print(f"📡 LINEAGE EVENT: {json.dumps(event, indent=2)}")
    
    # --- Ingestion Engine ---
    class LakehouseIngester:
        def __init__(self, table_name, layer="bronze"):
            self.table_name = table_name
            self.layer = layer
            self.path = f"./lakehouse_db/{layer}/{table_name}"
            self.lineage = LineageTracker()
            
        def enforce_schema(self, df: pd.DataFrame, schema: pa.Schema) -> pa.Table:
            """
            Schema-on-Write: Convert Pandas DF to strict PyArrow schema.
            This will fail if types cannot be cast.
            """
            try:
                # Cast columns to match schema
                table = pa.Table.from_pandas(df)
                # Use PyArrow's cast to enforce specific types (e.g., string to timestamp)
                return table.cast(schema)
            except Exception as e:
                raise ValueError(f"Schema Enforcement Failed: {e}")
    
        def run_quality_checks(self, table: pa.Table, expectations: list) -> bool:
            """
            Run validation rules. Returns True if passed, raises Error if failed.
            """
            df = table.to_pandas()
            
            print("⚙️  Running Quality Checks...")
            for rule in expectations:
                col = rule['column']
                check = rule['check']
                value = rule.get('value')
                
                if check == "no_nulls":
                    assert df[col].notna().all(), f"Quality Fail: Nulls found in {col}"
                elif check == "unique":
                    assert df[col].nunique() == len(df), f"Quality Fail: Duplicates in {col}"
                elif check == "min_value":
                    assert df[col].min() >= value, f"Quality Fail: Min value violation in {col}"
                
                print(f"   ✅ Passed: {col} - {check}")
            
            return True
    
        def ingest(self, source_df: pd.DataFrame, schema: pa.Schema, expectations: list):
            start_time = datetime.now()
            
            # 1. Define Lineage Inputs (Source System)
            input_facet = {"namespace": "source_system", "name": f"raw_{self.table_name}"}
            
            # 2. Schema-on-Write Enforcement
            print("🔨 Enforcing Schema...")
            arrow_table = self.enforce_schema(source_df, schema)
            
            # 3. Quality Checks
            self.run_quality_checks(arrow_table, expectations)
            
            # 4. Write to Delta Lake (Transactional)
            print(f"💾 Writing to Delta Table: {self.path}")
            write_deltalake(
                self.path, 
                arrow_table, 
                mode="overwrite", 
                engine="rust"
            )
            
            # 5. Generate Statistics for Lineage
            stats = {
                "rowCount": len(arrow_table),
                "sizeBytes": arrow_table.nbytes,
                "processingTimeMs": (datetime.now() - start_time).milliseconds
            }
            
            # 6. Log Lineage (Output)
            output_facet = {
                "namespace": self.lineage.namespace, 
                "name": f"{self.layer}.{self.table_name}",
                "outputFacets": {
                    "schema": [field.name for field in schema]
                }
            }
            
            self.lineage.log_event(
                job_name=f"ingest_{self.table_name}",
                inputs=[input_facet],
                outputs=[output_facet],
                schema_fields=[f"{f.name}:{f.type}" for f in schema],
                stats=stats
            )
    
    # --- Execution Example ---
    
    # 1. Define strict Schema (The Contract)
    user_schema = pa.schema([
        ("user_id", pa.int64()),
        ("email", pa.string()),
        ("signup_ts", pa.timestamp('s')),
        ("age", pa.int64())
    ])
    
    # 2. Define Quality Rules
    quality_rules = [
        {"column": "user_id", "check": "unique"},
        {"column": "email", "check": "no_nulls"},
        {"column": "age", "check": "min_value", "value": 0}
    ]
    
    # 3. Mock Raw Data
    raw_data = pd.DataFrame({
        'user_id': [1, 2, 3],
        'email': ['a@test.com', 'b@test.com', 'c@test.com'],
        'signup_ts': ['2023-01-01', '2023-01-02', '2023-01-03'],
        'age': [25, 30, 35]
    })
    
    # 4. Run Ingestion
    ingester = LakehouseIngester(table_name="users", layer="silver")
    ingester.ingest(raw_data, user_schema, quality_rules)
    

    4. Breakdown of Key Concepts

    A. Schema-on-Write

    In traditional data lakes, you dump a CSV and figure out the columns later ("Schema-on-Read"). This creates analytics debt.

    • The Implementation: We define a pa.schema (PyArrow Schema).
    • The Enforcement: table.cast(schema) forces the data types. If the raw data contains a string "foo" in an integer column defined in the schema, the ingestion fails immediately. This prevents "garbage in, garbage out."

    B. Quality Checks

    Quality should not be an afterthought. It should be a gatekeeper before the write operation commits.

    • The Logic: We iterate through a list of expectations.
    • Fail-Fast: If assert fails, the Python script crashes, and the write_deltalake command is never reached. This ensures that the Lakehouse table never contains invalid data.

    C. Lineage

    Lineage answers "Where did this data come from?" and "What did we do to it?".

    • OpenLineage Standard: The LineageTracker class constructs a JSON payload following the OpenLineage spec.
    • Metadata: It captures:
      • Inputs: The logical name of the source system.
      • Outputs: The specific table in the Lakehouse.
      • Facets: Schema details (column names/types) and Statistics (row count/size).
    • Integration: In a real production environment, this JSON is sent to a lineage backend (like DataHub or Marquez) which visualizes the graph.

    D. The Lakehouse Format (Delta Lake)

    Using deltalake (or Apache Iceberg) provides:

    • ACID Transactions: Ensures readers don't see partial data while the ingest is writing.
    • Time Travel: Because we wrote successfully, you can query VERSION AS OF 1 if a future ingestion breaks the data.
    • Schema Evolution: If you need to add a column later, Delta handles the metadata update gracefully.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.