I have a Python FastAPI app. The basic mechanism of the main handler of this app is:…
- User
I have a Python FastAPI app. The basic mechanism of the main handler of this app is: User click claim points, request goes to FastAPI it makes some validation etc and the produce message with data needed to recalculate points, this message goes to Apache Pulsar and worker subscribed to the topic handles recalculations and storing to DB, after that this worker send another message with updated values to another Pulsar topic to which the main FastAPI app is subscribed and it consumes this message and push those updates back to user via websocket connection. The question is: Currently this worker that handles the recalculation and db update is just a part of this FastAPI app, but because it is pretty independant and to increase thrououtput of the app I think about the ways to separate this worker from the main app. SO the app and the worker would be run on the separate servers. But at the same time this worker in terms of code reuse a lot from the app. So I thought of making this a monorep with two kinda separate parts: 1 FastAPI app and 2 async recalculation worker. But I don't know how to organize it and how to run it propperly? Currently I run this app in DIgitalOcean App Platform. Could you please help me? I need it to be production ready and super efficient, done super professional as it was made by Netlix Senior Staff Python Engineer. This should be a monorepo, so Worker could reuse part of code from the app(like Repository and DB parts, or Redis and Pulsar client which are pretty generic and don't contain business logic at all + both app and worker share all of this: DB, Redis and Pulsar) I also want to do this so I would have only one place to use Alembic to make my DB migrations and won't have problems because they share it. I will restart my project from the begining to make it as clean as possible so help me please.
- Assistant
Certainly! Structuring your application as a monorepo with separate services for the FastAPI app and the worker is a sound approach for scalability, maintainability, and code reuse. Below is a comprehensive guide to help you design and implement this architecture effectively, ensuring it is production-ready and efficient.
1. Monorepo Structure
A monorepo allows you to maintain both services (FastAPI app and worker) within a single repository, facilitating code sharing and unified dependency management. Here's a recommended directory structure:
my_project/ ├── app/ │ ├── fastapi_app/ │ │ ├── main.py │ │ ├── api/ │ │ ├── websocket/ │ │ └── ... │ ├── worker/ │ │ ├── worker.py │ │ └── ... ├── common/ │ ├── db/ │ │ ├── models.py │ │ ├── repositories.py │ │ └── ... │ ├── config/ │ │ ├── settings.py │ │ └── ... │ ├── utils/ │ │ └── ... │ └── ... ├── migrations/ │ ├── versions/ │ └── env.py ├── tests/ │ ├── fastapi_app/ │ ├── worker/ │ └── ... ├── Dockerfile.fastapi ├── Dockerfile.worker ├── docker-compose.yml ├── requirements.txt or pyproject.toml ├── alembic.ini └── README.mdExplanation:
- app/fastapi_app/: Contains the FastAPI application code.
- app/worker/: Contains the standalone worker service code.
- common/: Shared modules like database models, repositories, configuration, and utilities.
- migrations/: Alembic migration scripts.
- tests/: Unit and integration tests for both services.
- Dockerfile.fastapi & Dockerfile.worker: Docker configurations for containerizing each service.
- docker-compose.yml: For local development, orchestrating multiple services.
- requirements.txt or pyproject.toml: Dependency management.
- alembic.ini: Configuration for Alembic migrations.
2. Shared Code and Configuration
common/config/settings.py
Use a configuration management library like Pydantic's BaseSettings to manage configurations centrally.
from pydantic import BaseSettings class Settings(BaseSettings): DATABASE_URL: str REDIS_URL: str PULSAR_URL: str # Add other settings as needed class Config: env_file = ".env"common/db/models.py
Define your SQLAlchemy models here. Ensure both services can import these models.
from sqlalchemy import Column, Integer, String from sqlalchemy.ext.declarative import declarative_base Base = declarative_base() class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True, index=True) name = Column(String, index=True) points = Column(Integer, default=0) # Add other fields as necessarycommon/db/repositories.py
Implement your data access layer. This promotes code reuse and abstraction.
from sqlalchemy.orm import Session from .models import User def get_user(db: Session, user_id: int): return db.query(User).filter(User.id == user_id).first() def update_user_points(db: Session, user_id: int, points: int): user = get_user(db, user_id) if user: user.points = points db.commit() db.refresh(user) return user3. Dependency Management
Use a tool like Poetry for managing dependencies, which allows you to handle dependencies for both services elegantly.
pyproject.toml
[tool.poetry] name = "my_project" version = "0.1.0" description = "FastAPI and Worker monorepo" authors = ["Your Name <you@example.com>"] [tool.poetry.dependencies] python = "^3.11" fastapi = "^0.95.0" pulsar-client = "^2.9.1" sqlalchemy = "^1.4.0" alembic = "^1.7.0" redis = "^4.5.4" uvicorn = "^0.20.0" # Add other shared dependencies [tool.poetry.dev-dependencies] pytest = "^7.2.0" black = "^23.3.0" mypy = "^0.991.2" # Add other development dependencies [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"Benefits:
- Single Source of Truth: Both FastAPI app and worker share the same
pyproject.toml. - Version Consistency: Ensures that all shared packages are using the same versions, preventing conflicts.
4. Alembic for Database Migrations
Centralize your migrations to manage them from a single place.
alembic.ini
Configure Alembic to point to your
common.db.modelsfortarget_metadata.[alembic] # path to migration scripts script_location = migrations # sqlalchemy.url = driver://user:pass@localhost/dbname sqlalchemy.url = env:DATABASE_URL [loggers] keys = root,sqlalchemy,alembic # ... rest of the Alembic configmigrations/env.py
Modify the Alembic environment to import
target_metadatafrom your models.from logging.config import fileConfig from sqlalchemy import engine_from_config from sqlalchemy import pool from alembic import context import os import sys # Add the project directory to sys.path sys.path.append(os.path.abspath(os.path.join(os.getcwd()))) from common.db.models import Base # Your declarative base # this is the Alembic Config object, which provides # access to the values within the .ini file config = context.config # Interpret the config file for Python logging. fileConfig(config.config_file_name) target_metadata = Base.metadata def run_migrations_offline(): url = config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, literal_binds=True ) with context.begin_transaction(): context.run_migrations() def run_migrations_online(): connectable = engine_from_config( config.get_section(config.config_ini_section), prefix='sqlalchemy.', poolclass=pool.NullPool ) with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata ) with context.begin_transaction(): context.run_migrations() if context.is_offline_mode(): run_migrations_offline() else: run_migrations_online()Managing Migrations
- Create Migration: Run
alembic revision --autogenerate -m "Initial migration" - Apply Migration: Run
alembic upgrade head - Rollback Migration: Run
alembic downgrade -1
5. Service Implementations
app/fastapi_app/main.py
from fastapi import FastAPI, WebSocket from common.config.settings import Settings from common.db.repositories import get_user from sqlalchemy.orm import sessionmaker from sqlalchemy import create_engine import pulsar import asyncio settings = Settings() engine = create_engine(settings.DATABASE_URL) SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) app = FastAPI() # Initialize Pulsar client pulsar_client = pulsar.Client(settings.PULSAR_URL) producer = pulsar_client.create_producer('recalculate-points-topic') consumer = pulsar_client.subscribe('updated-values-topic', 'fastapi-subscription') @app.post("/claim-points") async def claim_points(user_id: int, points: int): # Validation logic message = {'user_id': user_id, 'points': points} producer.send(bytes(str(message), 'utf-8')) return {"status": "Points claimed, processing!"} @app.websocket("/ws/{user_id}") async def websocket_endpoint(websocket: WebSocket, user_id: int): await websocket.accept() while True: msg = consumer.receive() try: data = msg.data().decode('utf-8') # Process and send to WebSocket await websocket.send_text(data) consumer.acknowledge(msg) except: consumer.negative_acknowledge(msg)app/worker/worker.py
import pulsar import json from common.config.settings import Settings from common.db.repositories import update_user_points from sqlalchemy.orm import sessionmaker from sqlalchemy import create_engine settings = Settings() engine = create_engine(settings.DATABASE_URL) SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) pulsar_client = pulsar.Client(settings.PULSAR_URL) consumer = pulsar_client.subscribe('recalculate-points-topic', 'worker-subscription') producer = pulsar_client.create_producer('updated-values-topic') def process_message(message): data = json.loads(message.data().decode('utf-8')) user_id = data['user_id'] points = data['points'] # Recalculation logic new_points = points * 2 # Example recalculation # Update DB db = SessionLocal() user = update_user_points(db, user_id, new_points) db.close() # Send updated values updated_data = json.dumps({'user_id': user_id, 'new_points': new_points}) producer.send(bytes(updated_data, 'utf-8')) def run(): while True: msg = consumer.receive() try: process_message(msg) consumer.acknowledge(msg) except Exception as e: print(f"Failed to process message: {e}") consumer.negative_acknowledge(msg) if __name__ == "__main__": run()Key Points:
- Reusable Components: Both services use the same
Settings, database connections, and repositories from thecommonmodule. - Isolation: Each service runs independently, allowing for horizontal scaling.
- Error Handling: Ensure robust error handling and retry mechanisms, especially in the worker to handle message processing failures.
6. Containerization with Docker
Containerizing each service ensures consistency across environments and simplifies deployment.
Dockerfile.fastapi
FROM python:3.11-slim WORKDIR /app # Install dependencies COPY pyproject.toml poetry.lock ./ RUN pip install --upgrade pip RUN pip install poetry RUN poetry config virtualenvs.create false RUN poetry install --no-dev --no-root # Copy application code COPY app/fastapi_app /app/app/fastapi_app COPY common /app/common COPY alembic.ini migrations /app/ # Expose port EXPOSE 8000 # Command to run FastAPI CMD ["uvicorn", "app.fastapi_app.main:app", "--host", "0.0.0.0", "--port", "8000"]Dockerfile.worker
FROM python:3.11-slim WORKDIR /app # Install dependencies COPY pyproject.toml poetry.lock ./ RUN pip install --upgrade pip RUN pip install poetry RUN poetry config virtualenvs.create false RUN poetry install --no-dev --no-root # Copy application code COPY app/worker /app/app/worker COPY common /app/common COPY alembic.ini migrations /app/ # Command to run Worker CMD ["python", "app/worker/worker.py"]docker-compose.yml
For local development, you can use Docker Compose to orchestrate services along with dependencies like PostgreSQL, Redis, and Pulsar.
version: '3.8' services: fastapi: build: context: . dockerfile: Dockerfile.fastapi ports: - "8000:8000" environment: - DATABASE_URL=postgresql://user:password@db:5432/mydb - REDIS_URL=redis://redis:6379/0 - PULSAR_URL=pulsar://pulsar:6650 depends_on: - db - redis - pulsar worker: build: context: . dockerfile: Dockerfile.worker environment: - DATABASE_URL=postgresql://user:password@db:5432/mydb - REDIS_URL=redis://redis:6379/0 - PULSAR_URL=pulsar://pulsar:6650 depends_on: - db - redis - pulsar db: image: postgres:14 environment: POSTGRES_USER: user POSTGRES_PASSWORD: password POSTGRES_DB: mydb ports: - "5432:5432" redis: image: redis:7 ports: - "6379:6379" pulsar: image: apachepulsar/pulsar:2.9.1 command: bin/pulsar standalone ports: - "6650:6650" - "8080:8080"7. Deployment on DigitalOcean
Using Docker Containers
Given that you’re already using DigitalOcean App Platform, you can leverage Docker for deployment.
-
Build Docker Images: Ensure both
Dockerfile.fastapiandDockerfile.workerare correctly set up. -
Push to Container Registry: Push your images to DigitalOcean Container Registry or another registry like Docker Hub.
-
Deploy Services:
- FastAPI Service: Deploy the FastAPI service as a web service.
- Worker Service: Deploy the worker as a background/worker service.
-
Environment Variables: Configure environment variables in the App Platform settings to match your
Settings.
Using Kubernetes (Optional)
For more advanced deployments, consider using DigitalOcean Kubernetes (DOKS). This allows for better scalability, service discovery, and management.
8. Continuous Integration and Continuous Deployment (CI/CD)
Implement CI/CD pipelines to automate testing, building, and deployment.
Using GitHub Actions
Create a
.github/workflows/ci.ymlfile:name: CI/CD Pipeline on: push: branches: - main jobs: build: runs-on: ubuntu-latest services: postgres: image: postgres:14 env: POSTGRES_USER: user POSTGRES_PASSWORD: password POSTGRES_DB: mydb ports: - 5432:5432 options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 redis: image: redis:7 ports: - 6379:6379 pulsar: image: apachepulsar/pulsar:2.9.1 ports: - 6650:6650 - 8080:8080 options: --entrypoint /bin/bash -c "bin/pulsar standalone & sleep 10 && tail -f /dev/null" steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.11' - name: Install dependencies run: | python -m pip install --upgrade pip pip install poetry poetry install - name: Run Migrations run: | alembic upgrade head - name: Run Tests run: | poetry run pytestBenefits:
- Automated Testing: Ensures code integrity before deployment.
- Automated Deployment: Integrate with DigitalOcean’s deployment triggers to automate the release of Docker images upon passing tests.
9. Logging and Monitoring
Ensure you have robust logging and monitoring to maintain application health.
Logging
- Structured Logging: Use libraries like loguru or Python’s built-in
loggingmodule with structured log formats (e.g., JSON) for easier parsing. - Centralized Logging: Aggregate logs using services like DigitalOcean’s Managed Logging, ELK Stack, or third-party services like LogDNA.
Monitoring
- Application Monitoring: Integrate tools like Prometheus and Grafana for metrics collection and visualization.
- Health Checks: Implement health check endpoints in your FastAPI application.
- Alerts: Set up alerts for critical metrics (e.g., high error rates, downtime).
10. Best Practices and Considerations
1. Decouple Services
Ensure that the FastAPI app and worker are as decoupled as possible. They should communicate strictly via Pulsar, reducing inter-service dependencies.
2. Environment Management
- Configuration Separation: Use environment variables to manage different configurations for development, staging, and production.
- Secrets Management: Utilize DigitalOcean’s Secrets or other secret management solutions to handle sensitive information securely.
3. Scalability
- Horizontal Scaling: Both FastAPI and worker services can be scaled horizontally based on load.
- Pulsar Configuration: Ensure your Pulsar setup can handle the increased message throughput as services scale.
4. Testing
- Unit Tests: For individual modules and components.
- Integration Tests: To test interactions between services, database, and Pulsar.
- End-to-End Tests: Simulate real-world usage scenarios.
5. Documentation
Maintain clear documentation for:
- Project Structure: To help onboard new developers.
- API Endpoints: Using tools like FastAPI’s built-in documentation.
- Setup Instructions: For local development and deployment.
6. Security
- Input Validation: Always validate and sanitize user inputs.
- Authentication and Authorization: Implement robust authentication mechanisms for your API endpoints.
- Dependency Management: Regularly update dependencies to patch security vulnerabilities.
11. Example
README.mdProvide a comprehensive README to guide contributors.
# My Project ## Overview This project consists of a FastAPI application and an asynchronous worker service, structured as a monorepo for efficient code reuse and maintainability. ## Directory Structure - `app/fastapi_app/`: FastAPI application. - `app/worker/`: Worker service for recalculations. - `common/`: Shared modules including database models and utilities. - `migrations/`: Database migration scripts with Alembic. - `tests/`: Unit and integration tests. - `Dockerfile.fastapi`: Dockerfile for FastAPI service. - `Dockerfile.worker`: Dockerfile for Worker service. ## Getting Started ### Prerequisites - [Poetry](https://python-poetry.org/) - [Docker](https://www.docker.com/) - [Docker Compose](https://docs.docker.com/compose/) ### Local Development 1. **Clone the Repository** ```bash git clone https://github.com/yourusername/my_project.git cd my_project ``` 2. **Set Up Environment Variables** Create a `.env` file based on the provided `.env.example`. 3. **Build and Run Services** ```bash docker-compose up --build ``` 4. **Apply Migrations** ```bash docker-compose exec fastapi alembic upgrade head ``` 5. **Access the Application** - FastAPI: [http://localhost:8000](http://localhost:8000) - Documentation: [http://localhost:8000/docs](http://localhost:8000/docs) ### Running Tests ```bash poetry run pytestDeployment
Deploy using Docker containers on DigitalOcean App Platform. Ensure environment variables are correctly set in the deployment settings.
Contributing
- Fork the Repository
- Create a Feature Branch
- Commit Your Changes
- Push to the Branch
- Open a Pull Request
License
MIT License
## **Conclusion** By following this structured approach, you can create a clean, scalable, and maintainable monorepo setup that separates your FastAPI application and worker service while leveraging shared code and resources. This setup not only promotes code reuse but also ensures that each service can be independently scaled and managed, adhering to professional software engineering practices. Feel free to adapt the structure and configurations to better fit your specific requirements and preferences. Good luck with your project!