Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1 Codex Max· openaiHugging Face Frontier Reasoning Traces

Provide a Python data science boilerplate integrating Kedro for pipeline management.…

  1. User

    Provide a Python data science boilerplate integrating Kedro for pipeline management. Define modular pipelines with inputs and outputs in catalog.yml and implement a hook that validates data schema with Great Expectations before nodes run.

  2. Assistant

    Providing Kedro boilerplate with Great Expectations hook

    Here's a minimal Kedro setup illustrating modular pipelines with catalog entries and a hook that validates pandas DataFrames using Great Expectations before each node runs. It defines dataset inputs/outputs in catalog.yml, a simple data-processing pipeline, and a hook that loads expectation suites per dataset from JSON files and validates inputs before node execution. The hook is registered in settings for automatic use. Adjust paths and suite details to fit the actual project. Below is a minimal Kedro + Great Expectations boilerplate you can use as a template. It shows:

    • A simple, modular Kedro pipeline with inputs/outputs defined in conf/base/catalog.yml.
    • A hook that runs before each node to validate any pandas DataFrame inputs with Great Expectations.
    • Registration of the hook in settings.py.

    You can extend the same pattern to multiple pipelines/nodes.


    Project layout (key files)

    .
    ├─ src/
    │  ├─ my_project/
    │  │  ├─ pipeline/
    │  │  │  └─ data_processing/
    │  │  │     ├─ nodes.py
    │  │  │     └─ pipeline.py
    │  │  ├─ hooks.py
    │  │  ├─ settings.py
    │  │  └─ __init__.py
    │  └─ run.py
    └─ conf/
       └─ base/
          ├─ catalog.yml
          ├─ parameters.yml   # optional
          └─ expectations/    # expectation suites live here
    

    conf/base/catalog.yml

    Define your modular pipeline inputs/outputs here. For example:

    # conf/base/catalog.yml
    
    raw_data:
      type: pandas.CSVDataSet
      filepath: data/01_raw/raw_data.csv
    
    processed_data:
      type: pandas.CSVDataSet
      filepath: data/02_intermediate/processed_data.csv
    
    model_input:
      type: pandas.CSVDataSet
      filepath: data/03_primary/model_input.csv
    
    model_output:
      type: pickle.PickleDataSet
      filepath: data/06_models/model.pkl
    

    src/my_project/pipeline/data_processing/nodes.py

    Minimal node functions:

    # src/my_project/pipeline/data_processing/nodes.py
    import pandas as pd
    
    def preprocess(raw_df: pd.DataFrame) -> pd.DataFrame:
        # Example transform
        df = raw_df.dropna().copy()
        df["total"] = df["quantity"] * df["price"]
        return df
    
    def prepare_model_input(processed_df: pd.DataFrame) -> pd.DataFrame:
        # Example feature selection
        return processed_df[["feature_a", "feature_b", "total"]]
    

    src/my_project/pipeline/data_processing/pipeline.py

    Wire nodes into a Kedro pipeline:

    # src/my_project/pipeline/data_processing/pipeline.py
    from kedro.pipeline import Pipeline, node
    from .nodes import preprocess, prepare_model_input
    
    def create_pipeline(**kwargs) -> Pipeline:
        return Pipeline(
            [
                node(
                    func=preprocess,
                    inputs="raw_data",
                    outputs="processed_data",
                    name="preprocess_node",
                    tags=["validate_raw_data"],
                ),
                node(
                    func=prepare_model_input,
                    inputs="processed_data",
                    outputs="model_input",
                    name="prepare_model_input_node",
                    tags=["validate_processed_data"],
                ),
            ]
        )
    

    Note: The tags field is used here to indicate which expectation suite to apply; you can also key off of node.name, inputs, or custom metadata if you prefer.

    Great Expectations expectation suites

    Place expectation suite JSON files under conf/base/expectations/. Name them to match your node tags or input dataset names. For example:

    • conf/base/expectations/validate_raw_data.json
    • conf/base/expectations/validate_processed_data.json

    Each file should be a standard Great Expectations expectation suite definition, e.g.:

    {
      "data_asset_type": "Dataset",
      "expectation_suite_name": "validate_raw_data",
      "expectations": [
        {
          "expectation_type": "expect_column_values_to_not_be_null",
          "kwargs": { "column": "price" }
        },
        {
          "expectation_type": "expect_column_values_to_be_between",
          "kwargs": { "column": "quantity", "min_value": 0 }
        }
      ]
    }
    

    src/my_project/hooks.py

    A Kedro hook that validates DataFrame inputs using Great Expectations before each node runs:

    # src/my_project/hooks.py
    import json
    from pathlib import Path
    from typing import Any, Dict
    
    import pandas as pd
    import great_expectations as ge
    from kedro.framework.hooks import hook_impl
    from kedro.io import DataCatalog
    from kedro.pipeline.node import Node
    
    class DataValidationHook:
        """
        A hook that validates pandas.DataFrame inputs with Great Expectations
        before each node runs. It looks for an expectation suite named after
        the node tag (or input dataset name) in conf/base/expectations/.
        """
    
        def __init__(self, expectations_dir: str = "conf/base/expectations"):
            self.expectations_dir = Path(expectations_dir)
    
        def _load_suite(self, suite_name: str):
            suite_path = self.expectations_dir / f"{suite_name}.json"
            if not suite_path.exists():
                return None
            with suite_path.open("r") as f:
                return json.load(f)
    
        @hook_impl
        def before_node_run(
            self,
            node: Node,
            catalog: DataCatalog,
            inputs: Dict[str, Any],
            is_async: bool,
            session_id: str,
        ) -> None:
            # Derive which suite to use: try tags, otherwise input dataset names
            suite_names = list(node.tags) or list(inputs.keys())
    
            for suite_name in suite_names:
                suite_json = self._load_suite(suite_name)
                if suite_json is None:
                    # No suite found for this tag/input; skip
                    continue
    
                for input_name, data in inputs.items():
                    if not isinstance(data, pd.DataFrame):
                        continue
                    gx_df = ge.from_pandas(data)
                    validation_result = gx_df.validate(expectation_suite=suite_json)
                    if not validation_result.success:
                        raise ValueError(
                            f"Great Expectations validation failed for node={node.name}, "
                            f"input={input_name}, suite={suite_name}, "
                            f"result={validation_result}"
                        )
                    # Optionally log or persist validation results
                    print(
                        f"Validation passed for node={node.name}, "
                        f"input={input_name}, suite={suite_name}"
                    )
    

    You can customize the logic that chooses which expectation suite to load (by tag, by input name, or by node name).

    src/my_project/settings.py

    Register the hook so Kedro picks it up:

    # src/my_project/settings.py
    from my_project.hooks import DataValidationHook
    
    HOOKS = (DataValidationHook(),)
    
    # Optionally register pipelines here if you want a single place:
    # PIPELINE_REGISTRY = { "data_processing": data_processing.create_pipeline() }
    

    src/run.py (standard Kedro runner)

    Use the default Kedro entrypoint, or add a custom runner. Minimal example:

    # src/run.py
    from kedro.framework.context import KedroContext
    from kedro.framework.project import configure_project
    
    from my_project import __version__ as project_version
    
    def main():
        configure_project("my_project")
        context = KedroContext(package_name="my_project", project_path=".")
        context.run()  # runs all registered pipelines with hooks active
    
    if __name__ == "__main__":
        main()
    

    How it works

    1. You define your datasets in catalog.yml (CSV, Pickle, etc.).
    2. You create modular pipelines under src/my_project/pipeline/... with clear inputs/outputs.
    3. You create Great Expectations suites under conf/base/expectations/ named after tags or input names.
    4. The DataValidationHook.before_node_run hook loads the relevant suite and validates any DataFrame inputs before the node executes. If validation fails, the run is halted with an error.

    Extend this boilerplate by adding more pipelines/nodes, more expectation suites, and richer validation or logging as needed.