Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.
- User
Provide a boilerplate for a Python FastAPI project with Celery and Redis integration.
Assistant
Outlining the Framework
I'm currently focused on defining the core components needed for the boilerplate. I've begun by identifying the user's requirements: a FastAPI web app, Celery for task management, and Redis for both the message broker and result storage. My next step will be to detail how these pieces fit together architecturally.
Detailing the Requirements
I'm now diving deeper into the technical specifics. I'm considering FastAPI's app structure, including route definitions and Uvicorn integration. For Celery, I'm focusing on worker configuration and task definition best practices. I also need to ensure effective communication between FastAPI and Celery. Finally, I'm researching environment variables management, and the use of Docker to containerize the services.
Defining the Structure
Now, I'm fleshing out the structure within the
appdirectory. I'm leaning towards the use ofroutersand will structure themain.pyentry point. I'm also considering the use ofconfig.pywithPydantic Settingsfor handling environment variables. I'm also planning theDockerfileanddocker-compose.ymlfiles, to ensure quick deployment. I'll make sure to addrequirements.txtsoon.Formulating the Implementation
I'm presently focusing on the concrete steps for implementation, starting with a streamlined
requirements.txt. I'm also carefully mapping out theconfig.pystructure using Pydantic, making sure to include necessary Celery and Redis connection strings. Then I'm planning out theworker.py, including initializing the Celery app and defining a sample task. I'm checking the dependencies between these modules.Implementing the Components
I'm now detailing each code section. I'm writing
requirements.txtfirst, and then constructing theconfig.pyusingBaseSettingsfor environment variable handling. Then, I'll move on toworker.pyfor Celery configuration and sample task definitions, ensuring they correctly import the configurations. Next, the FastAPI app's structure and the two endpoints will be made.Here is a clean, production-ready boilerplate structure for a FastAPI application integrated with Celery and Redis.
I have containerized it using Docker and Docker Compose, as this is the standard way to run these three services working together.
Project Structure
proto-project/ ├── app/ │ ├── __init__.py │ ├── config.py # Environment configuration │ ├── main.py # FastAPI endpoints │ └── worker.py # Celery app and task definitions ├── .env # Environment variables (optional) ├── docker-compose.yml # Orchestrates Redis, API, and Worker ├── Dockerfile # Environment for Python └── requirements.txt # Python dependencies
1. The Code
requirements.txtfastapi uvicorn celery redis pydantic-settingsapp/config.pyWe use Pydantic for strict environment validation.
from pydantic_settings import BaseSettings class Settings(BaseSettings): CELERY_BROKER_URL: str = "redis://redis:6379/0" CELERY_RESULT_BACKEND: str = "redis://redis:6379/0" class Config: env_file = ".env" settings = Settings()app/worker.pyThere is where the Celery instance lives and where we define the background tasks.
import time from celery import Celery from app.config import settings # Initialize Celery celery = Celery( __name__, broker=settings.CELERY_BROKER_URL, backend=settings.CELERY_RESULT_BACKEND ) @celery.task(name="create_task") def create_task(task_type: int): """ A dummy mock task that simulates a long running process. """ time.sleep(int(task_type) * 3) # Simulate heavy work return Trueapp/main.pyThe FastAPI application. It pushes tasks to the queue and checks their status.
from fastapi import FastAPI from celery.result import AsyncResult from app.worker import create_task, celery from pydantic import BaseModel app = FastAPI() class TaskIn(BaseModel): complexity: int @app.post("/tasks", status_code=201) def run_task(payload: TaskIn): # Send task to Celery Worker task = create_task.delay(int(payload.complexity)) return {"task_id": task.id} @app.get("/tasks/{task_id}") def get_status(task_id: str): task_result = AsyncResult(task_id, app=celery) result = { "task_id": task_id, "task_status": task_result.status, "task_result": task_result.result } return result
2. Docker Configuration
DockerfileThis image is shared by both the FastAPI web server and the Celery worker.
# Use an official Python runtime as a parent image FROM python:3.11-slim # Set work directory WORKDIR /code # Set env variables # PYTHONDONTWRITEBYTECODE: Prevents Python from writing pyc files to disc # PYTHONUNBUFFERED: Prevents Python from buffering stdout and stderr ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # Install dependencies COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy project COPY . .docker-compose.ymlThis spins up three containers:
- Redis: The message broker.
- Web: The FastAPI server.
- Worker: The Celery process that executes the tasks.
services: redis: image: redis:7-alpine web: build: . command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload volumes: - .:/code ports: - 8000:8000 environment: - CELERY_BROKER_URL=redis://redis:6379/0 - CELERY_RESULT_BACKEND=redis://redis:6379/0 depends_on: - redis worker: build: . # Run the worker. The -A flag points to the celery instance in app.worker command: celery -A app.worker.celery worker --loglevel=info volumes: - .:/code environment: - CELERY_BROKER_URL=redis://redis:6379/0 - CELERY_RESULT_BACKEND=redis://redis:6379/0 depends_on: - redis
3. How to Run It
-
Start the services:
docker-compose up --build -
Trigger a Task: Send a POST request to start a background job (complexity sets the simulated sleep time).
curl -X POST "http://localhost:8000/tasks" \ -H "Content-Type: application/json" \ -d '{"complexity": 5}'Response:
{"task_id":"b2ba9462-63dc-4329-abe6-9da85814a34f"} -
Check Task Status: Take the ID from the previous step and check the status immediately.
curl http://localhost:8000/tasks/b2ba9462-63dc-4329-abe6-9da85814a34fOutput (Pending):
{"task_id": "...", "task_status": "PENDING", "task_result": null}Wait 15 seconds (since we sent complexity 5 * 3 seconds delay) and try again: Output (Success):
{"task_id": "...", "task_status": "SUCCESS", "task_result": true}
Key Concepts in this Boilerplate:
- Shared Image: The Web and Worker services use the same Docker image. This ensures they have the exact same code and task definitions, which is required for Celery to work properly (the worker needs to know the function definition to execute it).
- AsyncResult: Used in
main.pyto query Redis for the status of a specific Task ID. - Volumes: The
volumesmapping in Docker Compose allows you to edit your Python files locally and have the server/worker reload automatically without rebuilding the container.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.