Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.
- User
Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.
Assistant
Conceptualizing Project Setup
I'm focused on the fundamental building blocks now. Currently, I'm identifying the core components: FastAPI for the web framework, Celery for task queuing, and Redis acting as both a message broker and the result backend. My next step involves defining the dependencies needed to make these components work seamlessly together within the Python environment. I'm also thinking about structuring the project directory for clarity.
Outlining the Project Components
I've been drilling down into the technical specifics. I'm focusing on the
appstructure now. My plan is for FastAPI to live inmain.py, with Celery tasks defined inworker.py, and project-wide configurations set inconfig.py. I'm leaning toward including aroutersdirectory for organization. Docker and Docker Compose are non-negotiable for running Redis.Formulating Code Implementations
I'm now diving into the code specifics. I'm focusing on defining
requirements.txtwith essential packages. Next, I plan to flesh outconfig.pywith environment variable handling. Then, I'll build the Celery structure and the FastAPI endpoints and how they interact. Finally, I must define the Dockerfile and Docker Compose setup for deployment.Finalizing Boilerplate Details
I'm now consolidating the project structure and its components. I've focused on the optimal placement of files, and I'm ready to proceed with writing the explanations, including how the different pieces fit together. I'm also ensuring the code follows best practices for a scalable boilerplate. I need to make sure the user fully understands the project.
Constructing Project Explanations
I'm synthesizing the explanations now. I'm focusing on providing users with clear, step-by-step instructions. I'm also preparing to structure the response by presenting the files, code snippets, and deployment commands. I'm carefully reviewing all aspects to guarantee accuracy and usability.
Here is a clean, production-ready boilerplate for a project using FastAPI, Celery, and Redis.
I have structured this using Docker Compose, as it is the standard way to spin up the Web Server, the Background Worker, and the Redis broker simultaneously.
Project Structure
project_root/ ├── app/ │ ├── __init__.py │ ├── config.py │ ├── main.py │ ├── models.py │ └── worker.py ├── Dockerfile ├── docker-compose.yml └── requirements.txt
1. Dependencies (
requirements.txt)fastapi uvicorn[standard] celery redis python-dotenv pydantic2. Configuration (
app/config.py)Centralize your settings to make it easy to switch between local and docker environments.
import os class Settings: CELERY_BROKER_URL: str = os.environ.get("CELERY_BROKER_URL", "redis://127.0.0.1:6379/0") CELERY_RESULT_BACKEND: str = os.environ.get("CELERY_RESULT_BACKEND", "redis://127.0.0.1:6379/0") settings = Settings()3. Celery Application (
app/worker.py)This file initializes the Celery app and defines the tasks.
import time from celery import Celery from app.config import settings celery = Celery( __name__, broker=settings.CELERY_BROKER_URL, backend=settings.CELERY_RESULT_BACKEND ) @celery.task(name="create_task") def create_task(task_type: str): """Simulates a slow computation.""" time.sleep(10) # Simulate a 10-second operation return {"status": True, "message": f"Task {task_type} completed successfully!"}4. Models (
app/models.py)Pydantic models for request validation.
from pydantic import BaseModel class TaskIn(BaseModel): type: str class TaskOut(BaseModel): task_id: str status: str5. FastAPI Application (
app/main.py)The API endpoints to trigger tasks and check their status.
from fastapi import FastAPI from celery.result import AsyncResult from app.worker import create_task from app.models import TaskIn, TaskOut app = FastAPI() @app.get("/") def root(): return {"message": "Hello World"} @app.post("/tasks", response_model=TaskOut, status_code=201) def run_task(task_in: TaskIn): """Create a new celery task.""" task = create_task.delay(task_in.type) return {"task_id": task.id, "status": "Processing"} @app.get("/tasks/{task_id}") def get_status(task_id: str): """Check the status of a task.""" task_result = AsyncResult(task_id) result = { "task_id": task_id, "task_status": task_result.status, "task_result": task_result.result } return result
6. Containerization
DockerfileWe use a single Dockerfile for both the Web API and the Celery Worker.
FROM python:3.11-slim WORKDIR /usr/src/app # Prevent Python from writing pyc files to disc ENV PYTHONDONTWRITEBYTECODE 1 # Prevent Python from buffering stdout and stderr ENV PYTHONUNBUFFERED 1 COPY ./requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . .docker-compose.ymlThis orchestrates three services:
- Redis: The message broker.
- Web: The FastAPI application.
- Worker: The Celery process that executes tasks.
services: redis: image: redis:7-alpine container_name: redis_broker ports: - "6379:6379" web: build: . container_name: fastapi_web command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload volumes: - .:/usr/src/app ports: - "8000:8000" environment: - CELERY_BROKER_URL=redis://redis:6379/0 - CELERY_RESULT_BACKEND=redis://redis:6379/0 depends_on: - redis worker: build: . container_name: celery_worker # Command to start the Celery worker command: celery -A app.worker.celery worker --loglevel=info volumes: - .:/usr/src/app environment: - CELERY_BROKER_URL=redis://redis:6379/0 - CELERY_RESULT_BACKEND=redis://redis:6379/0 depends_on: - redis
How to run it
-
Start the stack:
docker-compose up --build -
Trigger a background task: Open your browser to
http://127.0.0.1:8000/docs(Swagger UI).Use the POST /tasks endpoint:
{ "type": "email_report" }Response: You will get a
task_id(e.g.,c0613...). -
Check the logs: Check your terminal. You will see the Worker container pick up the task, sleep for 10 seconds, and verify completion.
-
Check status: Use the GET /tasks/{task_id} endpoint with the ID received in step 2.
- Within 10 seconds: Status
PENDING. - After 10 seconds: Status
SUCCESSwith the result.
- Within 10 seconds: Status
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.