Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Provide a boilerplate for a Python data science project with Jupyter and Poetry.

  1. User

    Provide a boilerplate for a Python data science project with Jupyter and Poetry.

  2. GLM Assistant

    1. Understand the Goal: The user wants a "boilerplate" (template/starter structure) for a Python data science project that specifically uses Jupyter for notebooks and Poetry for dependency management.

    1. Identify Key Components: A robust data science boilerplate needs more than just "pip install". It needs:

      • Directory Structure: Standard locations for data, notebooks, source code, tests, configs.
      • Dependency Management: pyproject.toml (handled by Poetry).
      • Jupyter Integration: Making sure the kernel uses the virtual environment created by Poetry.
      • Version Control: .gitignore (crucial for data science to avoid committing large data files or notebook checkpoints).
      • Documentation: A basic README.md.
      • Code Quality (Optional but recommended): Linting/Formatting tools (Black, Ruff/Flake8).
      • Data Handling: Folders for raw/processed data (usually in .gitignore).
    2. Drafting the Directory Structure:

      • Root: my-datascience-project
      • Source code: src/ or my_datascience_project/. (Poetry defaults to src/, but many DS projects prefer a flat src or just a named folder. Let's stick to a standard src package structure for cleaner imports, or a simpler structure if it's just analysis. Let's go with a src/project_name structure as it's "best practice" for packaging, but keep it simple enough for notebooks to import from it).
      • Notebooks: notebooks/
      • Data: data/raw, data/processed, data/external
      • Tests: tests/
      • Configs: configs/ (YAML files are common).
    3. Poetry Setup:

      • Command: poetry new my-project or poetry init.
      • Dependencies to include: pandas, numpy, jupyter (or jupyterlab), matplotlib/seaborn (standard DS stack).
      • Dev dependencies: pytest, black, ruff.
    4. Jupyter Specifics:

      • Need to explain how to run Jupyter inside the Poetry environment.
      • Command: poetry run jupyter lab or installing the kernel specifically (poetry add ipykernel -> poetry run python -m ipykernel install --user --name=my-project). The latter is safer for switching kernels, but running poetry run jupyter lab is the easiest workflow for beginners. I will provide the ipykernel method as it's more robust.
    5. Refining the Boilerplate Content:

      • File: pyproject.toml: I can't generate a real one, but I can provide a mock-up of what the [tool.poetry] section looks like.
      • File: .gitignore: Needs Python specifics + Jupyter specifics + Data specifics.
      • File: README.md: Instructions on how to set up.
    6. Constructing the Output:

      • Step 1: Directory Tree. Visualizing the layout first helps the user understand the big picture.
      • Step 2: Setup Commands. How to actually create this using the CLI.
      • Step 3: The Files.
        • pyproject.toml: Show the dependencies.
        • src/package/__init__.py: The actual code.
        • .gitignore: Crucial for DS.
        • README.md: Usage.
      • Step 4: Jupyter Integration. Explain the "kernel" trick. This is the most common pain point for beginners.
    7. Self-Correction/Review during drafting:

      • Initial thought: Just tell them to run poetry run jupyter notebook.
      • Correction: That installs Jupyter in the venv. That works, but often DS people want to use one Jupyter instance and switch kernels. I'll mention the ipykernel approach as the "Pro" way and poetry run jupyter lab as the "Easy" way.
      • Data: Should I include a dvc setup? No, that's too advanced for a "boilerplate" request. Stick to standard folder structure.
      • Path issues: Notebooks often have path issues importing from src. I should add a small snippet in the notebook (or a paths.py helper) to fix sys.path.
    8. Final Polish: Ensure clear headers, code blocks, and concise explanations. (Proceeding to generate response based on these steps).

      • Structure:
        1. Tree view.
        2. pyproject.toml.
        3. .gitignore.
        4. src folder example.
        5. Notebook setup (sys.path trick).
        6. Step-by-step CLI commands.
    9. Output Generation (This matches the provided good response). Here is a comprehensive boilerplate structure for a Python Data Science project using Poetry and Jupyter.

    This structure separates your reusable code (source) from your experiments (notebooks) and raw data.

    1. The Directory Structure

    my-ds-project/
    ├── pyproject.toml       # Poetry dependencies and metadata
    ├── poetry.lock          # Lock file for exact versions
    ├── README.md            # Project description
    ├── .gitignore           # Files to ignore (data, venv, checkpoints)
    ├── data/
    │   ├── raw/             # Immutable raw data
    │   ├── interim/         # Intermediate data
    │   └── processed/       # Final datasets for modeling
    ├── notebooks/           # Jupyter Notebooks (Exploratory)
    │   └── example.ipynb
    ├── src/                 # Reusable Python modules
    │   ├── __init__.py
    │   ├── data.py          # Scripts to download/generate data
    │   └── features.py      # Feature engineering
    ├── tests/               # Unit tests
    │   └── test_data.py
    └── .env                 # Environment variables (API keys, paths)
    

    2. pyproject.toml (Configuration)

    This file replaces requirements.txt and setup.py. Run poetry init to generate this interactively, or copy the template below.

    Key step: You must install ipykernel to allow Jupyter to see this specific virtual environment as a kernel.

    [tool.poetry]
    name = "my-ds-project"
    version = "0.1.0"
    description = "A data science project boilerplate"
    authors = ["Your Name <you@example.com>"]
    readme = "README.md"
    packages = [{include = "src", from = "."}]
    
    [tool.poetry.dependencies]
    python = "^3.10"  # Specify your version
    pandas = "^2.0"
    numpy = "^1.24"
    matplotlib = "^3.7"
    seaborn = "^0.12"
    scikit-learn = "^1.3"
    python-dotenv = "^1.0.0" # For loading .env files
    
    [tool.poetry.group.dev.dependencies]
    jupyter = "^1.0"
    ipykernel = "^6.0"       # Essential for Jupyter integration
    pytest = "^7.4"
    black = "^23.0"          # Code formatting
    ruff = "^0.0.280"        # Linting
    
    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
    

    3. .gitignore

    Data science projects generate large files that should not be committed to Git.

    # Byte-compiled / optimized / DLL files
    __pycache__/
    
    # Jupyter Notebook
    .ipynb_checkpoints/
    
    # Environment
    .env
    .venv
    venv/
    
    # Data (Prevent committing large datasets)
    data/
    *.csv
    *.parquet
    *.pkl
    
    # Poetry
    poetry.lock
    

    4. src/__init__.py

    This file allows you to import functions from the src folder directly into your notebooks.

    import os
    import sys
    
    # Auto-reload modules in src when they change
    %load_ext autoreload
    %autoreload 2
    
    # Helper to find the project root
    def find_root():
        return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    

    5. Quick Start Guide

    Here is the workflow to get this project running on your machine.

    Step 1: Create the Project

    mkdir my-ds-project
    cd my-ds-project
    git init
    

    Step 2: Set up Poetry

    Create the pyproject.toml file (copy the content above or run poetry init).

    # Install dependencies
    poetry install
    

    Step 3: Register the Kernel

    This is the most critical step. It links your project's virtual environment to Jupyter.

    # This makes your environment available as a kernel named "my-ds-project"
    poetry run python -m ipykernel install --user --name=my-ds-project
    

    Step 4: Run Jupyter

    You can now launch Jupyter Lab. Because we installed jupyter as a dev dependency, we run it through poetry.

    poetry run jupyter lab
    

    Step 5: Select the Kernel

    1. Open or create a notebook in the notebooks/ folder.
    2. Go to the Kernel menu -> Change Kernel.
    3. Select my-ds-project.

    6. How to import your code into Notebooks

    Since we defined packages = [{include = "src"}] in pyproject.toml, you can import your source code cleanly.

    In src/features.py:

    def clean_data(df):
        return df.dropna()
    

    In notebooks/analysis.ipynb:

    from src.features import clean_data
    import pandas as pd
    
    df = pd.read_csv("../data/raw/data.csv")
    clean_df = clean_data(df)
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.