Develop a refactoring playbook for legacy ORMs introducing carthography, anti-corruption…
- User
Develop a refactoring playbook for legacy ORMs introducing carthography, anti-corruption layers, and tests.
- Assistant
We are developing a refactoring playbook for legacy ORMs (Object-Relational Mappings) with the goal of introducing:
- Carthography (likely meaning mapping documentation or visualization)
- Anti-corruption layers (ACL) to isolate the legacy system from new code
- Tests to ensure correctness during and after refactoring
Steps:
-
Assessment and Documentation (Carthography)
- Inventory existing ORM mappings, models, and database schema.
- Document current relationships, constraints, and business logic embedded in the ORM.
- Create a map (carthography) of the current state:
- Entity Relationship Diagrams (ERD) from the database and ORM models.
- Identify any custom queries, stored procedures, or raw SQL.
- Note any known issues or technical debt.
-
Define the Target State
- Decide on the desired structure for the new code. This might involve:
- Splitting large models (if following Domain-Driven Design, into bounded contexts).
- Deciding on the new ORM or data access pattern (if changing ORM) or keeping the same but restructuring.
- Plan the anti-corruption layer (ACL) to translate between the legacy ORM and the new models.
- Decide on the desired structure for the new code. This might involve:
-
Establish Testing Strategy
- Start by writing characterization tests for the current behavior (if not already present). These tests capture the existing behavior of the system, so that we know if we break something during refactoring.
- Set up a test environment that uses the real database (or a realistic test database) to run these tests.
- Plan for unit tests for the new ACL and refactored code.
-
Incremental Refactoring with ACL
- Implement the anti-corruption layer (ACL) that sits between the legacy ORM and the new code. The ACL should:
- Present a clean, simplified interface to the new code.
- Translate requests to the legacy ORM and database.
- Handle any necessary data transformation to isolate the new code from the legacy model.
- Start by building the ACL for a small, non-critical part of the system. Write tests (using the characterization tests and new unit tests) to ensure the ACL behaves as expected.
- Gradually move functionality from the legacy ORM to the new code via the ACL. This might involve:
- Writing new code that uses the ACL instead of the legacy ORM directly.
- Moving business logic out of the ORM models and into service or domain layers that use the ACL.
- For each piece of functionality, ensure tests pass (both old characterization tests and new tests).
- Implement the anti-corruption layer (ACL) that sits between the legacy ORM and the new code. The ACL should:
-
Introduce New ORM (if desired)
- If the goal is to replace the legacy ORM with a new one, do it incrementally behind the ACL.
- The ACL can initially use the legacy ORM. Then, for a specific bounded context or set of entities, change the ACL to use the new ORM for that part.
- The ACL should maintain the same interface to the new code, so the new code doesn't need to change.
- Test thoroughly to ensure the new ORM part behaves the same as the legacy part.
- If the goal is to replace the legacy ORM with a new one, do it incrementally behind the ACL.
-
Remove Legacy Dependencies
- Once all functionality is behind the ACL and the new code is using the ACL, and the ACL has been transitioned to the new ORM (if that's the goal), then you can remove the legacy ORM from the project.
- Clean up any remaining direct dependencies on the legacy ORM in the codebase.
-
Continuous Integration and Deployment
- Ensure that the tests (characterization and new) are run in a CI pipeline.
- Use feature toggles if necessary to switch between the legacy and new code during deployment.
-
Documentation and Knowledge Sharing
- Update the carthography (mapping documentation) to reflect the new state.
- Document the ACL and its purpose for future developers.
-
Retire the Legacy System
- Once all functionality is moved and the legacy ORM is removed, monitor the system for any issues.
- Consider decommissioning the old database schema if a new one was introduced (but this is a separate migration).
Important: The anti-corruption layer is key because it allows us to change the underlying data access without affecting the rest of the application. It also allows us to gradually migrate.
Example steps for a specific entity:
Step 1: Write characterization tests for the current behavior of
Userentity (using the legacy ORM). Step 2: Create an ACL interfaceUserRepository(or similar) that the new code will use. Step 3: ImplementLegacyUserRepositorythat uses the legacy ORM and passes the characterization tests. Step 4: Change the new code to useUserRepository(via dependency injection) instead of the legacy ORM directly. Step 5: Now, if we want to change the ORM forUser, we can implement aNewUserRepositorythat uses the new ORM and ensure it passes the same tests.Repeat for each entity or bounded context.
If the legacy ORM is deeply embedded in the models (active record pattern), then we might need to extract the data access from the models. In that case, we might:
- Create a repository for each model that uses the legacy ORM to fetch data and then builds a new domain object (or DTO) that the new code uses. - The old model might still be used in some places, but we gradually replace those with the new objects.Alternatively, if we are moving to a different pattern (like data mapper), we might:
- Create a new set of domain objects and use the ACL to map between the legacy ORM's records and the new domain objects.The key is to keep the system working at every step and have tests to verify.
Let's structure the playbook as a step-by-step guide with checkpoints.
Playbook Structure:
-
Phase 0: Preparation
- Set up version control (if not already) and a staging environment.
- Ensure you have a good backup and rollback plan.
- Assemble the team and ensure everyone understands the goals and the steps.
-
Phase 1: Document the Legacy (Carthography)
- Use tools to generate ERDs from the database.
- Use static analysis or manual inspection to map ORM models to tables.
- Document any custom queries and their purposes.
- Output: A set of diagrams and documents that describe the current data model and access patterns.
-
Phase 2: Characterization Tests
- Write tests that cover the current behavior of the system from the perspective of the application (black-box tests) and also for the ORM models (if possible, white-box).
- These tests should be runnable against the current production database (or a copy) and should pass.
- Output: A test suite that captures the current behavior.
-
Phase 3: Design the ACL and Target Architecture
- Define the interfaces (repositories, services) that the new code will use.
- Decide on the granularity of the ACL (per entity, per aggregate, per bounded context).
- Design the mapping between the legacy ORM and the new interfaces.
- Output: Design documents and interface definitions.
-
Phase 4: Implement ACL for One Bounded Context
- Choose a bounded context that is less critical and has fewer dependencies.
- Implement the ACL for that context, starting with read operations (which are usually safer).
- Write unit tests for the ACL implementation (mocking the legacy ORM if necessary) and integration tests that use the real database.
- Change the code in that bounded context to use the ACL instead of the legacy ORM directly.
- Run the characterization tests and the new tests to ensure nothing breaks.
- Output: A bounded context that uses the ACL.
-
Phase 5: Expand ACL to More Contexts
- Repeat phase 4 for other bounded contexts, gradually expanding the coverage.
- If you are changing the ORM, you can now start implementing the new ORM in the ACL for some contexts, one by one.
-
Phase 6: Remove Legacy ORM
- Once all contexts are using the ACL and the ACL no longer depends on the legacy ORM (or if you are keeping the same ORM but the ACL is the only point of contact), then remove the legacy ORM from the project.
- Remove any remaining direct usages of the legacy ORM models in the codebase (they should have been replaced by the objects provided by the ACL).
- Run all tests to ensure the system still works.
-
Phase 7: Cleanup and Optimization
- Now that the legacy ORM is gone, you can optimize the new code (e.g., adjust the new ORM mappings, add indexes, etc.) without worrying about breaking the old structure.
- Consider cleaning up the database schema if it was designed for the old ORM and is not optimal for the new one (but this is a separate database migration and should be done carefully).
-
Phase 8: Update Documentation
- Update the carthography to reflect the new architecture.
- Document the ACL and the new data access patterns.
Risks and Mitigations:
-
Risk: The characterization tests might not capture all edge cases. Mitigation: Complement with manual testing and monitor logs after deployment.
-
Risk: The ACL might become a performance bottleneck. Mitigation: Profile and optimize as needed. The ACL should be thin, but if complex transformations are needed, consider caching or other optimizations.
-
Risk: The legacy ORM might have subtle bugs that are now considered features. Mitigation: The characterization tests should capture these. If a bug is found that is considered a feature, adjust the test accordingly.
-
Risk: The migration might take too long and cause the team to lose focus. Mitigation: Break down the work by bounded contexts and celebrate small wins.
This playbook is a high-level guide. Each step will require detailed planning and adaptation to the specific project.
Let me know if you have any questions about a specific step.
Legacy ORM Refactoring Playbook: Carthography, ACLs & Tests
1. Guiding Principles
- Strangler Fig Pattern: Gradually strangulate legacy ORM usage with new layers
- Behavior Preservation: Never break existing functionality without explicit intent
- Incremental Migration: Small, reversible steps with constant verification
- Testing as Safety Net: Characterization tests first, then contract tests
2. Phase 0: Foundation Setup (Week 1-2)
2.1 Tooling & Infrastructure
Essential Tools: - Database schema diff tool (sqldiff, Liquibase, Flyway) - ORM mapping analyzer (if available) - Test coverage tool (JaCoCo, Istanbul, Coverage.py) - Contract testing framework (Pact, Spring Cloud Contract) - Visualization tools (dbdiagram.io, Mermaid, Graphviz)2.2 Establish Baselines
-
Current State Document
- Generate full ERD from database
- Map ORM entities to tables (many-to-many relationships)
- Document raw SQL queries/hints in code
- Identify N+1 query hotspots via DB query logs
-
Baseline Metrics
- Test coverage % (aim for 0% → 20% in first 2 weeks)
- Query performance benchmarks
- Current bug count related to ORM issues
3. Phase 1: Carthography (Week 2-4)
Create the "map" of your data landscape
3.1 Automated Discovery
# Generate entity relationship map python scripts/map_orm_entities.py \ --orm-type django/sqlalchemy/activerecord \ --output docs/entity-map.json # Extract raw SQL usage grep -r "raw\|execute\|query" app/models/ > docs/raw-sql-inventory.md3.2 Documentation Deliverables
-
Entity Map Matrix
| Entity | Table | Relationships | Raw SQL? | Critical Path | |---------|----------|---------------|----------|---------------| | User | users | 1:N orders | No | Auth flow | | Order | orders | N:1 user, N:M | Yes[1] | Checkout | -
Data Flow Diagrams
- Show how entities traverse layers (Controller → Service → ORM → DB)
- Highlight circular dependencies
-
Change Impact Matrix
- Which entities affect which business capabilities?
- Prediction: Changing
Order× breaks 3 payment flows
4. Phase 2: Characterization Tests (Week 3-5)
Capture current behavior before changing anything
4.1 Test Strategy by Layer
┌─────────────────┬─────────────────────────────┬──────────────┐ │ Layer │ Test Type │ Tool │ ├─────────────────┼─────────────────────────────┼──────────────┤ │ Database Schema │ Schema comparison │ sqldiff │ │ ORM Layer │ Record retrieval │ ORM fixtures │ │ Business Logic │ Given-When-Then scenarios │ pytest │ │ API Contract │ Request/response snapshots │ Postman/Rest │ └─────────────────┴─────────────────────────────┴──────────────┘4.2 Characterization Test Template
# tests/characterization/test_user_orm_behavior.py def test_user_retrieves_orders_with_correct_status(): """Legacy behavior: User.orders returns ACTIVE orders only""" # Given user = User.objects.get(id=42) # When orders = user.orders.all() # Legacy ORM call # Then assert all(o.status == 'ACTIVE' for o in orders) # NOTE: This behavior exists because of a filter in User model4.3 Contract Testing
# contracts/order_service.yaml name: Order Service Contract given: - user exists with id: 42 when: - GET /api/users/42/orders then: - status: 200 - body[0].status in [ACTIVE, PENDING] # Current actual behavior
5. Phase 3: Anti-Corruption Layer Implementation (Week 4-8)
5.1 ACL Design Principles
# ACL Pattern: Repository with Adapter class OrderRepository: """ACL Interface - Business-facing""" def get_active_orders_for_user(user_id: int) -> List[OrderDTO]: raise NotImplementedError class LegacyOrderAdapter(OrderRepository): """Adapter for legacy ORM""" def __init__(self, legacy_orm_session): self.session = legacy_orm_session def get_active_orders_for_user(self, user_id): # Translate ACL request → Legacy ORM call # Map ORM models → DTOs legacy_orders = (self.session.query(OrderModel) .filter_by(user_id=user_id, status='ACTIVE') .all()) return [OrderDTO.from_orm(o) for o in legacy_orders]5.2 Incremental ACL Rollout
Step 1: Read-Only ACL
# Start with READ operations (safer) class UserReadACL: def get_user_profile(user_id) -> UserProfileDTO: # Uses legacy ORM underneath pass # Replace in service layer: # OLD: user = LegacyORM.User.get(id) # NEW: user = user_read_acl.get_user_profile(id)Step 2: Add Write Methods Gradually
# After read coverage is stable class UserWriteACL: def update_user_email(user_id, email): # Still uses legacy ORM but isolates write logic passStep 3: Begin Data Transformation
# ACL now transforms between legacy and new models def _to_new_user(legacy_user: LegacyUserORM) -> NewUserDomain: # Handle legacy quirks: # - Legacy stores full_name in 'name' column # - New model wants first_name/last_name split return NewUserDomain( first_name=legacy_user.name.split()[0], last_name=' '.join(legacy_user.name.split()[1:]) )5.3 ACL Testing Strategy
1. Unit tests for ACL adapters (mock legacy ORM) 2. Integration tests: ACL → real DB (small dataset) 3. Contract tests: ACL meets business expectations 4. Performance tests: ACL doesn't add >10% latency
6. Phase 4: Progressive Decoupling (Week 8-12)
6.1 Strangler Fig Migration Pattern
Current State: [Business Code] → [Legacy ORM] → [DB] Step 1: [Business Code] → [ACL Read] → [Legacy ORM] → [DB] ↘ [ACL Write] ↗ Step 2 (Partial): [Business Code] → [ACL] → [New Repository] → [New ORM] → [DB] ↘ [ACL] → [Legacy ORM] ↗ (for uncoved entities) Step 3 (Complete): [Business Code] → [ACL] → [New Repository] → [New ORM] → [DB]6.2 Migration Priority Matrix
High Value, Low Risk: - Read-heavy entities (User profiles) - Stable schemas (no recent changes) - Well-tested paths Low Value, High Risk: - Write-heavy financial transactions - Complex relationships (many-to-many with payload) - Schema in flux6.3 Feature Flag Integration
# config/feature_flags.py USE_LEGACY_ACL = { 'orders': False, # Using new ACL 'payments': True, # Still legacy 'users': False } # In code: def get_orders(user_id): if USE_LEGACY_ACL['orders']: return legacy_order_orm.get_orders(user_id) else: return order_acl.get_orders(user_id) # New ACL
7. Phase 5: New ORM Integration (Optional, Week 12-16)
7.1 When to Replace ORM
- ✅ ACL abstracts all data access already
- ✅ New ORM offers significant benefits (performance, type safety)
- ❌ Don't replace just for "modernization"
7.2 Safe ORM Replacement
class SQLAlchemyOrderAdapter(OrderRepository): """New ORM implementation behind same ACL""" def __init__(self, sqlalchemy_session): self.session = sqlalchemy_session def get_active_orders_for_user(self, user_id): # Different query style, same result return (self.session.query(NewOrder) .filter(NewOrder.user_id == user_id, NewOrder.status == 'ACTIVE') .all())7.3 Dual-Write Strategy (If Schema Changes)
# During transition period def create_order(order_data): # Write to both legacy_order = LegacyOrder(**order_data) new_order = NewOrder(**transform_order_data(order_data)) session.add(legacy_order) session.add(new_order) session.commit() # Validation: Compare both sides periodically assert legacy_order.id == new_order.legacy_id
8. Phase 6: Legacy Retirement (Week 16+)
8.1 Decommissioning Checklist
- All business code uses ACL (0 direct ORM imports)
- Characterization tests pass with new stack
- Performance benchmarks meet SLAs
- No queries hitting legacy tables (DB monitoring)
- Documentation updated (carthography reflects new state)
- Team trained on new patterns
8.2 Safe Removal
# 1. Remove ORM from dependency file # 2. Search for remaining imports git grep "from legacy_orm" || echo "Clean" # 3. Drop legacy tables gradually (backup first!)
9. Testing Pyramid for Refactoring
┌─────────────────────┐ │ Contract Tests │ ← Business expectations │ (ACL interfaces) │ (Pact, OpenAPI) └──────────┬──────────┘ │ ┌────────────────────┼────────────────────┐ │ │ │ ┌─────▼─────┐ ┌────▼─────┐ ┌────▼─────┐ │ Unit │ │ Unit │ │ E2E │ │ ACL │ │ Business│ │ (Smoke) │ │ Tests │ │ Logic │ │ │ └───────────┘ └──────────┘ └──────────┘ ▲ ▲ ▲ │ │ │ ┌─────┴─────┐ ┌────┴─────┐ ┌────┴─────┐ │ Mocked │ │ Real DB │ │ Staging │ │ Legacy │ │ (Test) │ │ Env │ └───────────┘ └──────────┘ └──────────┘
10. Common Pitfalls & Mitigation
Pitfall Symptom Solution ACL becomes God Object 2000+ line ACL class Split by bounded context; use multiple ACLs Performance Regression Queries 2x slower Profile ACL; add caching; batch operations Circular Dependencies ACL imports models that import ACL Extract interfaces; use dependency injection Test Brittleness Characterization tests fail on benign changes Group tests by business capability, not implementation Partial Migration Some endpoints use old, some new Feature flags everywhere; monitor split traffic
11. Success Metrics
Quantitative:
- ≥70% of data access through ACL
- 0 N+1 queries in critical paths
- ≤5% performance degradation (if any)
- Test coverage ≥60% for data access layer
Qualitative:
- New developers can understand data flow in <1 day
- No "magic" hidden in ORM models
- Business logic testable without DB
- Schema changes localized to one place
12. Rollback Plan
- Immediate (0-2 hours): Feature flag off new ACL, revert to legacy
- Short-term (1 day): Keep both ACL and legacy paths running, compare outputs
- Long-term (1 week): Gradual traffic shifting with monitoring
Command to rollback:
# Emergency rollback script kubectl set env deployment/app USE_LEGACY_ACL=true # OR update feature_flags set orders_legacy=true where id=1;
13. Communication Plan
Stakeholder Updates:
- Product owners: "We're making changes safer, not adding features"
- QA team: "Your regression suite just became 10x more valuable"
- New hires: "Here's the carthography map—our single source of truth"
Signaling Progress:
# Dashboard command ./scripts/migration_status.sh # Output: # ✅ ACL Coverage: 73% (↑5% this week) # ⚠️ Raw SQL remaining: 12 queries # 🔴 High-risk unmigrated: Payments module
14. References & Templates
- Sample Carthography Template
- ACL Boilerplate Code
- Characterization Test Generator
- Migration Checklist
Final Note: This isn't a one-time project—it's a permanent shift in how your team thinks about data access. The ACL becomes your data access contract, and carthography becomes your living map. Invest in keeping them updated! 🗺️
"The best time to refactor a legacy ORM was before you wrote it. The second best time is now—with tests, maps, and protective layers."
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.