Provide a boilerplate for a Python data science project with Jupyter and Poetry.
- User
Provide a boilerplate for a Python data science project with Jupyter and Poetry.
Assistant
1. Understand the Goal: The user wants a "boilerplate" (template/starter structure) for a Python data science project that specifically uses Jupyter for notebooks and Poetry for dependency management.
-
Identify Key Components: A robust data science boilerplate needs more than just "pip install". It needs:
- Directory Structure: Standard locations for data, notebooks, source code, tests, configs.
- Dependency Management:
pyproject.toml(handled by Poetry). - Jupyter Integration: Making sure the kernel uses the virtual environment created by Poetry.
- Version Control:
.gitignore(crucial for data science to avoid committing large data files or notebook checkpoints). - Documentation: A basic
README.md. - Code Quality (Optional but recommended): Linting/Formatting tools (Black, Ruff/Flake8).
- Data Handling: Folders for raw/processed data (usually in
.gitignore).
-
Drafting the Directory Structure:
- Root:
my-datascience-project - Source code:
src/ormy_datascience_project/. (Poetry defaults tosrc/, but many DS projects prefer a flatsrcor just a named folder. Let's stick to a standardsrcpackage structure for cleaner imports, or a simpler structure if it's just analysis. Let's go with asrc/project_namestructure as it's "best practice" for packaging, but keep it simple enough for notebooks to import from it). - Notebooks:
notebooks/ - Data:
data/raw,data/processed,data/external - Tests:
tests/ - Configs:
configs/(YAML files are common).
- Root:
-
Poetry Setup:
- Command:
poetry new my-projectorpoetry init. - Dependencies to include:
pandas,numpy,jupyter(orjupyterlab),matplotlib/seaborn(standard DS stack). - Dev dependencies:
pytest,black,ruff.
- Command:
-
Jupyter Specifics:
- Need to explain how to run Jupyter inside the Poetry environment.
- Command:
poetry run jupyter labor installing the kernel specifically (poetry add ipykernel->poetry run python -m ipykernel install --user --name=my-project). The latter is safer for switching kernels, but runningpoetry run jupyter labis the easiest workflow for beginners. I will provide theipykernelmethod as it's more robust.
-
Refining the Boilerplate Content:
- File:
pyproject.toml: I can't generate a real one, but I can provide a mock-up of what the[tool.poetry]section looks like. - File:
.gitignore: Needs Python specifics + Jupyter specifics + Data specifics. - File:
README.md: Instructions on how to set up.
- File:
-
Constructing the Output:
- Step 1: Directory Tree. Visualizing the layout first helps the user understand the big picture.
- Step 2: Setup Commands. How to actually create this using the CLI.
- Step 3: The Files.
pyproject.toml: Show the dependencies.src/package/__init__.py: The actual code..gitignore: Crucial for DS.README.md: Usage.
- Step 4: Jupyter Integration. Explain the "kernel" trick. This is the most common pain point for beginners.
-
Self-Correction/Review during drafting:
- Initial thought: Just tell them to run
poetry run jupyter notebook. - Correction: That installs Jupyter in the venv. That works, but often DS people want to use one Jupyter instance and switch kernels. I'll mention the
ipykernelapproach as the "Pro" way andpoetry run jupyter labas the "Easy" way. - Data: Should I include a
dvcsetup? No, that's too advanced for a "boilerplate" request. Stick to standard folder structure. - Path issues: Notebooks often have path issues importing from
src. I should add a small snippet in the notebook (or apaths.pyhelper) to fixsys.path.
- Initial thought: Just tell them to run
-
Final Polish: Ensure clear headers, code blocks, and concise explanations. (Proceeding to generate response based on these steps).
- Structure:
- Tree view.
pyproject.toml..gitignore.srcfolder example.- Notebook setup (sys.path trick).
- Step-by-step CLI commands.
- Structure:
-
Output Generation (This matches the provided good response). Here is a comprehensive boilerplate structure for a Python Data Science project using Poetry and Jupyter.
This structure separates your reusable code (source) from your experiments (notebooks) and raw data.
1. The Directory Structure
my-ds-project/ ├── pyproject.toml # Poetry dependencies and metadata ├── poetry.lock # Lock file for exact versions ├── README.md # Project description ├── .gitignore # Files to ignore (data, venv, checkpoints) ├── data/ │ ├── raw/ # Immutable raw data │ ├── interim/ # Intermediate data │ └── processed/ # Final datasets for modeling ├── notebooks/ # Jupyter Notebooks (Exploratory) │ └── example.ipynb ├── src/ # Reusable Python modules │ ├── __init__.py │ ├── data.py # Scripts to download/generate data │ └── features.py # Feature engineering ├── tests/ # Unit tests │ └── test_data.py └── .env # Environment variables (API keys, paths)
2.
pyproject.toml(Configuration)This file replaces
requirements.txtandsetup.py. Runpoetry initto generate this interactively, or copy the template below.Key step: You must install
ipykernelto allow Jupyter to see this specific virtual environment as a kernel.[tool.poetry] name = "my-ds-project" version = "0.1.0" description = "A data science project boilerplate" authors = ["Your Name <you@example.com>"] readme = "README.md" packages = [{include = "src", from = "."}] [tool.poetry.dependencies] python = "^3.10" # Specify your version pandas = "^2.0" numpy = "^1.24" matplotlib = "^3.7" seaborn = "^0.12" scikit-learn = "^1.3" python-dotenv = "^1.0.0" # For loading .env files [tool.poetry.group.dev.dependencies] jupyter = "^1.0" ipykernel = "^6.0" # Essential for Jupyter integration pytest = "^7.4" black = "^23.0" # Code formatting ruff = "^0.0.280" # Linting [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"
3.
.gitignoreData science projects generate large files that should not be committed to Git.
# Byte-compiled / optimized / DLL files __pycache__/ # Jupyter Notebook .ipynb_checkpoints/ # Environment .env .venv venv/ # Data (Prevent committing large datasets) data/ *.csv *.parquet *.pkl # Poetry poetry.lock
4.
src/__init__.pyThis file allows you to import functions from the
srcfolder directly into your notebooks.import os import sys # Auto-reload modules in src when they change %load_ext autoreload %autoreload 2 # Helper to find the project root def find_root(): return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
5. Quick Start Guide
Here is the workflow to get this project running on your machine.
Step 1: Create the Project
mkdir my-ds-project cd my-ds-project git initStep 2: Set up Poetry
Create the
pyproject.tomlfile (copy the content above or runpoetry init).# Install dependencies poetry installStep 3: Register the Kernel
This is the most critical step. It links your project's virtual environment to Jupyter.
# This makes your environment available as a kernel named "my-ds-project" poetry run python -m ipykernel install --user --name=my-ds-projectStep 4: Run Jupyter
You can now launch Jupyter Lab. Because we installed
jupyteras a dev dependency, we run it through poetry.poetry run jupyter labStep 5: Select the Kernel
- Open or create a notebook in the
notebooks/folder. - Go to the Kernel menu -> Change Kernel.
- Select my-ds-project.
6. How to import your code into Notebooks
Since we defined
packages = [{include = "src"}]inpyproject.toml, you can import your source code cleanly.In
src/features.py:def clean_data(df): return df.dropna()In
notebooks/analysis.ipynb:from src.features import clean_data import pandas as pd df = pd.read_csv("../data/raw/data.csv") clean_df = clean_data(df)-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.