Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini 3 Pro Preview· googleHugging Face Frontier Reasoning Traces

Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.

  1. User

    Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.

  2. Gemini Assistant

    Conceptualizing Project Setup

    I'm focused on the fundamental building blocks now. Currently, I'm identifying the core components: FastAPI for the web framework, Celery for task queuing, and Redis acting as both a message broker and the result backend. My next step involves defining the dependencies needed to make these components work seamlessly together within the Python environment. I'm also thinking about structuring the project directory for clarity.

    Outlining the Project Components

    I've been drilling down into the technical specifics. I'm focusing on the app structure now. My plan is for FastAPI to live in main.py, with Celery tasks defined in worker.py, and project-wide configurations set in config.py. I'm leaning toward including a routers directory for organization. Docker and Docker Compose are non-negotiable for running Redis.

    Formulating Code Implementations

    I'm now diving into the code specifics. I'm focusing on defining requirements.txt with essential packages. Next, I plan to flesh out config.py with environment variable handling. Then, I'll build the Celery structure and the FastAPI endpoints and how they interact. Finally, I must define the Dockerfile and Docker Compose setup for deployment.

    Finalizing Boilerplate Details

    I'm now consolidating the project structure and its components. I've focused on the optimal placement of files, and I'm ready to proceed with writing the explanations, including how the different pieces fit together. I'm also ensuring the code follows best practices for a scalable boilerplate. I need to make sure the user fully understands the project.

    Constructing Project Explanations

    I'm synthesizing the explanations now. I'm focusing on providing users with clear, step-by-step instructions. I'm also preparing to structure the response by presenting the files, code snippets, and deployment commands. I'm carefully reviewing all aspects to guarantee accuracy and usability.

    Here is a clean, production-ready boilerplate for a project using FastAPI, Celery, and Redis.

    I have structured this using Docker Compose, as it is the standard way to spin up the Web Server, the Background Worker, and the Redis broker simultaneously.

    Project Structure

    project_root/
    ├── app/
    │   ├── __init__.py
    │   ├── config.py
    │   ├── main.py
    │   ├── models.py
    │   └── worker.py
    ├── Dockerfile
    ├── docker-compose.yml
    └── requirements.txt
    

    1. Dependencies (requirements.txt)

    fastapi
    uvicorn[standard]
    celery
    redis
    python-dotenv
    pydantic
    

    2. Configuration (app/config.py)

    Centralize your settings to make it easy to switch between local and docker environments.

    import os
    
    class Settings:
        CELERY_BROKER_URL: str = os.environ.get("CELERY_BROKER_URL", "redis://127.0.0.1:6379/0")
        CELERY_RESULT_BACKEND: str = os.environ.get("CELERY_RESULT_BACKEND", "redis://127.0.0.1:6379/0")
    
    settings = Settings()
    

    3. Celery Application (app/worker.py)

    This file initializes the Celery app and defines the tasks.

    import time
    from celery import Celery
    from app.config import settings
    
    celery = Celery(
        __name__,
        broker=settings.CELERY_BROKER_URL,
        backend=settings.CELERY_RESULT_BACKEND
    )
    
    @celery.task(name="create_task")
    def create_task(task_type: str):
        """Simulates a slow computation."""
        time.sleep(10)  # Simulate a 10-second operation
        return {"status": True, "message": f"Task {task_type} completed successfully!"}
    

    4. Models (app/models.py)

    Pydantic models for request validation.

    from pydantic import BaseModel
    
    class TaskIn(BaseModel):
        type: str
    
    class TaskOut(BaseModel):
        task_id: str
        status: str
    

    5. FastAPI Application (app/main.py)

    The API endpoints to trigger tasks and check their status.

    from fastapi import FastAPI
    from celery.result import AsyncResult
    from app.worker import create_task
    from app.models import TaskIn, TaskOut
    
    app = FastAPI()
    
    @app.get("/")
    def root():
        return {"message": "Hello World"}
    
    @app.post("/tasks", response_model=TaskOut, status_code=201)
    def run_task(task_in: TaskIn):
        """Create a new celery task."""
        task = create_task.delay(task_in.type)
        return {"task_id": task.id, "status": "Processing"}
    
    @app.get("/tasks/{task_id}")
    def get_status(task_id: str):
        """Check the status of a task."""
        task_result = AsyncResult(task_id)
        
        result = {
            "task_id": task_id,
            "task_status": task_result.status,
            "task_result": task_result.result
        }
        return result
    

    6. Containerization

    Dockerfile

    We use a single Dockerfile for both the Web API and the Celery Worker.

    FROM python:3.11-slim
    
    WORKDIR /usr/src/app
    
    # Prevent Python from writing pyc files to disc
    ENV PYTHONDONTWRITEBYTECODE 1
    # Prevent Python from buffering stdout and stderr
    ENV PYTHONUNBUFFERED 1
    
    COPY ./requirements.txt .
    RUN pip install --no-cache-dir -r requirements.txt
    
    COPY . .
    

    docker-compose.yml

    This orchestrates three services:

    1. Redis: The message broker.
    2. Web: The FastAPI application.
    3. Worker: The Celery process that executes tasks.
    services:
      redis:
        image: redis:7-alpine
        container_name: redis_broker
        ports:
          - "6379:6379"
    
      web:
        build: .
        container_name: fastapi_web
        command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
        volumes:
          - .:/usr/src/app
        ports:
          - "8000:8000"
        environment:
          - CELERY_BROKER_URL=redis://redis:6379/0
          - CELERY_RESULT_BACKEND=redis://redis:6379/0
        depends_on:
          - redis
    
      worker:
        build: .
        container_name: celery_worker
        # Command to start the Celery worker
        command: celery -A app.worker.celery worker --loglevel=info
        volumes:
          - .:/usr/src/app
        environment:
          - CELERY_BROKER_URL=redis://redis:6379/0
          - CELERY_RESULT_BACKEND=redis://redis:6379/0
        depends_on:
          - redis
    

    How to run it

    1. Start the stack:

      docker-compose up --build
      
    2. Trigger a background task: Open your browser to http://127.0.0.1:8000/docs (Swagger UI).

      Use the POST /tasks endpoint:

      {
        "type": "email_report"
      }
      

      Response: You will get a task_id (e.g., c0613...).

    3. Check the logs: Check your terminal. You will see the Worker container pick up the task, sleep for 10 seconds, and verify completion.

    4. Check status: Use the GET /tasks/{task_id} endpoint with the ID received in step 2.

      • Within 10 seconds: Status PENDING.
      • After 10 seconds: Status SUCCESS with the result.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.