Provide a Python data science boilerplate integrating Kedro for pipeline management.…
- User
Provide a Python data science boilerplate integrating Kedro for pipeline management. Define modular pipelines with inputs and outputs in catalog.yml and implement a hook that validates data schema with Great Expectations before nodes run.
- Assistant
Providing Kedro boilerplate with Great Expectations hook
Here's a minimal Kedro setup illustrating modular pipelines with catalog entries and a hook that validates pandas DataFrames using Great Expectations before each node runs. It defines dataset inputs/outputs in
catalog.yml, a simple data-processing pipeline, and a hook that loads expectation suites per dataset from JSON files and validates inputs before node execution. The hook is registered in settings for automatic use. Adjust paths and suite details to fit the actual project. Below is a minimal Kedro + Great Expectations boilerplate you can use as a template. It shows:- A simple, modular Kedro pipeline with inputs/outputs defined in
conf/base/catalog.yml. - A hook that runs before each node to validate any pandas DataFrame inputs with Great Expectations.
- Registration of the hook in
settings.py.
You can extend the same pattern to multiple pipelines/nodes.
Project layout (key files)
. ├─ src/ │ ├─ my_project/ │ │ ├─ pipeline/ │ │ │ └─ data_processing/ │ │ │ ├─ nodes.py │ │ │ └─ pipeline.py │ │ ├─ hooks.py │ │ ├─ settings.py │ │ └─ __init__.py │ └─ run.py └─ conf/ └─ base/ ├─ catalog.yml ├─ parameters.yml # optional └─ expectations/ # expectation suites live hereconf/base/catalog.ymlDefine your modular pipeline inputs/outputs here. For example:
# conf/base/catalog.yml raw_data: type: pandas.CSVDataSet filepath: data/01_raw/raw_data.csv processed_data: type: pandas.CSVDataSet filepath: data/02_intermediate/processed_data.csv model_input: type: pandas.CSVDataSet filepath: data/03_primary/model_input.csv model_output: type: pickle.PickleDataSet filepath: data/06_models/model.pklsrc/my_project/pipeline/data_processing/nodes.pyMinimal node functions:
# src/my_project/pipeline/data_processing/nodes.py import pandas as pd def preprocess(raw_df: pd.DataFrame) -> pd.DataFrame: # Example transform df = raw_df.dropna().copy() df["total"] = df["quantity"] * df["price"] return df def prepare_model_input(processed_df: pd.DataFrame) -> pd.DataFrame: # Example feature selection return processed_df[["feature_a", "feature_b", "total"]]src/my_project/pipeline/data_processing/pipeline.pyWire nodes into a Kedro pipeline:
# src/my_project/pipeline/data_processing/pipeline.py from kedro.pipeline import Pipeline, node from .nodes import preprocess, prepare_model_input def create_pipeline(**kwargs) -> Pipeline: return Pipeline( [ node( func=preprocess, inputs="raw_data", outputs="processed_data", name="preprocess_node", tags=["validate_raw_data"], ), node( func=prepare_model_input, inputs="processed_data", outputs="model_input", name="prepare_model_input_node", tags=["validate_processed_data"], ), ] )Note: The
tagsfield is used here to indicate which expectation suite to apply; you can also key off ofnode.name,inputs, or custom metadata if you prefer.Great Expectations expectation suites
Place expectation suite JSON files under
conf/base/expectations/. Name them to match your node tags or input dataset names. For example:conf/base/expectations/validate_raw_data.jsonconf/base/expectations/validate_processed_data.json
Each file should be a standard Great Expectations expectation suite definition, e.g.:
{ "data_asset_type": "Dataset", "expectation_suite_name": "validate_raw_data", "expectations": [ { "expectation_type": "expect_column_values_to_not_be_null", "kwargs": { "column": "price" } }, { "expectation_type": "expect_column_values_to_be_between", "kwargs": { "column": "quantity", "min_value": 0 } } ] }src/my_project/hooks.pyA Kedro hook that validates DataFrame inputs using Great Expectations before each node runs:
# src/my_project/hooks.py import json from pathlib import Path from typing import Any, Dict import pandas as pd import great_expectations as ge from kedro.framework.hooks import hook_impl from kedro.io import DataCatalog from kedro.pipeline.node import Node class DataValidationHook: """ A hook that validates pandas.DataFrame inputs with Great Expectations before each node runs. It looks for an expectation suite named after the node tag (or input dataset name) in conf/base/expectations/. """ def __init__(self, expectations_dir: str = "conf/base/expectations"): self.expectations_dir = Path(expectations_dir) def _load_suite(self, suite_name: str): suite_path = self.expectations_dir / f"{suite_name}.json" if not suite_path.exists(): return None with suite_path.open("r") as f: return json.load(f) @hook_impl def before_node_run( self, node: Node, catalog: DataCatalog, inputs: Dict[str, Any], is_async: bool, session_id: str, ) -> None: # Derive which suite to use: try tags, otherwise input dataset names suite_names = list(node.tags) or list(inputs.keys()) for suite_name in suite_names: suite_json = self._load_suite(suite_name) if suite_json is None: # No suite found for this tag/input; skip continue for input_name, data in inputs.items(): if not isinstance(data, pd.DataFrame): continue gx_df = ge.from_pandas(data) validation_result = gx_df.validate(expectation_suite=suite_json) if not validation_result.success: raise ValueError( f"Great Expectations validation failed for node={node.name}, " f"input={input_name}, suite={suite_name}, " f"result={validation_result}" ) # Optionally log or persist validation results print( f"Validation passed for node={node.name}, " f"input={input_name}, suite={suite_name}" )You can customize the logic that chooses which expectation suite to load (by tag, by input name, or by node name).
src/my_project/settings.pyRegister the hook so Kedro picks it up:
# src/my_project/settings.py from my_project.hooks import DataValidationHook HOOKS = (DataValidationHook(),) # Optionally register pipelines here if you want a single place: # PIPELINE_REGISTRY = { "data_processing": data_processing.create_pipeline() }src/run.py(standard Kedro runner)Use the default Kedro entrypoint, or add a custom runner. Minimal example:
# src/run.py from kedro.framework.context import KedroContext from kedro.framework.project import configure_project from my_project import __version__ as project_version def main(): configure_project("my_project") context = KedroContext(package_name="my_project", project_path=".") context.run() # runs all registered pipelines with hooks active if __name__ == "__main__": main()
How it works
- You define your datasets in
catalog.yml(CSV, Pickle, etc.). - You create modular pipelines under
src/my_project/pipeline/...with clear inputs/outputs. - You create Great Expectations suites under
conf/base/expectations/named after tags or input names. - The
DataValidationHook.before_node_runhook loads the relevant suite and validates any DataFrame inputs before the node executes. If validation fails, the run is halted with an error.
Extend this boilerplate by adding more pipelines/nodes, more expectation suites, and richer validation or logging as needed.
- A simple, modular Kedro pipeline with inputs/outputs defined in