Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini 3 Pro Preview· googleHugging Face Frontier Reasoning Traces

Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.

  1. User

    Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.

  2. Gemini Assistant

    Outlining the Framework

    I'm currently focused on defining the core components needed for the boilerplate. I've begun by identifying the user's requirements: a FastAPI web app, Celery for task management, and Redis for both the message broker and result storage. My next step will be to detail how these pieces fit together architecturally.

    Detailing the Requirements

    I'm now diving deeper into the technical specifics. I'm considering FastAPI's app structure, including route definitions and Uvicorn integration. For Celery, I'm focusing on worker configuration and task definition best practices. I also need to ensure effective communication between FastAPI and Celery. Finally, I'm researching environment variables management, and the use of Docker to containerize the services.

    Defining the Structure

    Now, I'm fleshing out the structure within the app directory. I'm leaning towards the use of routers and will structure the main.py entry point. I'm also considering the use of config.py with Pydantic Settings for handling environment variables. I'm also planning the Dockerfile and docker-compose.yml files, to ensure quick deployment. I'll make sure to add requirements.txt soon.

    Formulating the Implementation

    I'm presently focusing on the concrete steps for implementation, starting with a streamlined requirements.txt. I'm also carefully mapping out the config.py structure using Pydantic, making sure to include necessary Celery and Redis connection strings. Then I'm planning out the worker.py, including initializing the Celery app and defining a sample task. I'm checking the dependencies between these modules.

    Implementing the Components

    I'm now detailing each code section. I'm writing requirements.txt first, and then constructing the config.py using BaseSettings for environment variable handling. Then, I'll move on to worker.py for Celery configuration and sample task definitions, ensuring they correctly import the configurations. Next, the FastAPI app's structure and the two endpoints will be made.

    Here is a clean, production-ready boilerplate structure for a FastAPI application integrated with Celery and Redis.

    I have containerized it using Docker and Docker Compose, as this is the standard way to run these three services working together.

    Project Structure

    proto-project/
    ├── app/
    │   ├── __init__.py
    │   ├── config.py       # Environment configuration
    │   ├── main.py         # FastAPI endpoints
    │   └── worker.py       # Celery app and task definitions
    ├── .env                # Environment variables (optional)
    ├── docker-compose.yml  # Orchestrates Redis, API, and Worker
    ├── Dockerfile          # Environment for Python
    └── requirements.txt    # Python dependencies
    

    1. The Code

    requirements.txt

    fastapi
    uvicorn
    celery
    redis
    pydantic-settings
    

    app/config.py

    We use Pydantic for strict environment validation.

    from pydantic_settings import BaseSettings
    
    class Settings(BaseSettings):
        CELERY_BROKER_URL: str = "redis://redis:6379/0"
        CELERY_RESULT_BACKEND: str = "redis://redis:6379/0"
    
        class Config:
            env_file = ".env"
    
    settings = Settings()
    

    app/worker.py

    There is where the Celery instance lives and where we define the background tasks.

    import time
    from celery import Celery
    from app.config import settings
    
    # Initialize Celery
    celery = Celery(
        __name__,
        broker=settings.CELERY_BROKER_URL,
        backend=settings.CELERY_RESULT_BACKEND
    )
    
    @celery.task(name="create_task")
    def create_task(task_type: int):
        """
        A dummy mock task that simulates a long running process.
        """
        time.sleep(int(task_type) * 3)  # Simulate heavy work
        return True
    

    app/main.py

    The FastAPI application. It pushes tasks to the queue and checks their status.

    from fastapi import FastAPI
    from celery.result import AsyncResult
    from app.worker import create_task, celery
    from pydantic import BaseModel
    
    app = FastAPI()
    
    class TaskIn(BaseModel):
        complexity: int
    
    @app.post("/tasks", status_code=201)
    def run_task(payload: TaskIn):
        # Send task to Celery Worker
        task = create_task.delay(int(payload.complexity))
        return {"task_id": task.id}
    
    @app.get("/tasks/{task_id}")
    def get_status(task_id: str):
        task_result = AsyncResult(task_id, app=celery)
        result = {
            "task_id": task_id,
            "task_status": task_result.status,
            "task_result": task_result.result
        }
        return result
    

    2. Docker Configuration

    Dockerfile

    This image is shared by both the FastAPI web server and the Celery worker.

    # Use an official Python runtime as a parent image
    FROM python:3.11-slim
    
    # Set work directory
    WORKDIR /code
    
    # Set env variables
    # PYTHONDONTWRITEBYTECODE: Prevents Python from writing pyc files to disc
    # PYTHONUNBUFFERED: Prevents Python from buffering stdout and stderr
    ENV PYTHONDONTWRITEBYTECODE 1
    ENV PYTHONUNBUFFERED 1
    
    # Install dependencies
    COPY requirements.txt .
    RUN pip install --no-cache-dir -r requirements.txt
    
    # Copy project
    COPY . .
    

    docker-compose.yml

    This spins up three containers:

    1. Redis: The message broker.
    2. Web: The FastAPI server.
    3. Worker: The Celery process that executes the tasks.
    services:
      redis:
        image: redis:7-alpine
    
      web:
        build: .
        command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
        volumes:
          - .:/code
        ports:
          - 8000:8000
        environment:
          - CELERY_BROKER_URL=redis://redis:6379/0
          - CELERY_RESULT_BACKEND=redis://redis:6379/0
        depends_on:
          - redis
    
      worker:
        build: .
        # Run the worker. The -A flag points to the celery instance in app.worker
        command: celery -A app.worker.celery worker --loglevel=info
        volumes:
          - .:/code
        environment:
          - CELERY_BROKER_URL=redis://redis:6379/0
          - CELERY_RESULT_BACKEND=redis://redis:6379/0
        depends_on:
          - redis
    

    3. How to Run It

    1. Start the services:

      docker-compose up --build
      
    2. Trigger a Task: Send a POST request to start a background job (complexity sets the simulated sleep time).

      curl -X POST "http://localhost:8000/tasks" \
           -H "Content-Type: application/json" \
           -d '{"complexity": 5}'
      

      Response:

      {"task_id":"b2ba9462-63dc-4329-abe6-9da85814a34f"}
      
    3. Check Task Status: Take the ID from the previous step and check the status immediately.

      curl http://localhost:8000/tasks/b2ba9462-63dc-4329-abe6-9da85814a34f
      

      Output (Pending):

      {"task_id": "...", "task_status": "PENDING", "task_result": null}
      

      Wait 15 seconds (since we sent complexity 5 * 3 seconds delay) and try again: Output (Success):

      {"task_id": "...", "task_status": "SUCCESS", "task_result": true}
      

    Key Concepts in this Boilerplate:

    1. Shared Image: The Web and Worker services use the same Docker image. This ensures they have the exact same code and task definitions, which is required for Celery to work properly (the worker needs to know the function definition to execute it).
    2. AsyncResult: Used in main.py to query Redis for the status of a specific Task ID.
    3. Volumes: The volumes mapping in Docker Compose allows you to edit your Python files locally and have the server/worker reload automatically without rebuilding the container.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.