Mission: Design and implement a scalable, extensible “D‑and‑D Cutscene Audio/Video…
- User
Mission: Design and implement a scalable, extensible “D‑and‑D Cutscene Audio/Video Manager”—a browser‑based timeline editor that lets Dungeon Masters assemble atmospheric cut‑scenes from any mix of audio tracks and video clips.
Core user goals Multi‑file import – drag‑and‑drop any number of audio and video files; each file appears as a draggable “clip.”
Dual‑lane timeline – one lane for audio, one for video, sharing a common time‑ruler.
Clip editing – split, trim, resize, slip‑edit, duplicate, group, multi‑select, undo/redo; edges snap to grid or neighbour clips.
Clip roles – any clip or clip‑group can be tagged Intro, Loop, Outro.
Playback engine – • space‑bar preview that honours the current timeline layout • seamless intro → loop → outro flow; Loop may repeat until the DM triggers Outro • cross‑fade at all segment boundaries
Export – • project JSON for later re‑load • WAV mix‑down of the audio lane • MP4 mux (mixed audio + arranged video lane) via WebAssembly/ffmpeg
Quality of life – zoom, snap toggle, keyboard shortcuts, role‑coloured clips, theming, accessibility, i18n hooks, performance on >30 tracks.
Engineering expectations Framework freedom – you may pick the front‑end stack (React/Redux, Vue, Svelte, vanilla, etc.) but justify the choice for scalability and maintainability.
Modular architecture – design with clear domain layers (UI / state / audio‑engine / export), dependency injection where sensible, plugin points for future effects or MIDI triggers.
Type safety – favour TypeScript or equivalent.
Worker off‑loading – long media operations (decode, mix‑down, ffmpeg) should not block the main thread.
Testing strategy – unit tests for utilities, integration tests for timeline ops, and e2e playback sanity checks (e.g. Playwright).
Tooling – lint, prettier, commit hooks, CI pipeline.
Documentation – high‑level architecture diagram, public API docs for plugin authors, and an onboarding README.
Deliverables Solution outline – chosen tech stack, core libraries (waveform‑playlist, Tone.js, ffmpeg.wasm, etc.), high‑level component diagram, data models, and build pipeline description.
Incremental roadmap – sprint plan from MVP (load/preview a single audio clip) to full feature‑set.
Source code – organised in a mono‑repo (e.g. PNPM workspaces) with lint/test scripts, ready to run via pnpm install && pnpm dev.
Sample project files – demo JSON scene, example cutscene assets, and a short ‘getting started’ guide.
(Optional stretch) – plug‑in SDK example: add a real‑time reverb effect or MIDI foot‑switch trigger.
Important: This prompt is spec‑only—do not output production code until asked; focus first on architecture and planning to maximise extensibility and developer velocity.
Assistant
Solution Outline
Chosen Tech Stack
-
Front-End Framework: React + TypeScript
- Justification: React's component-based architecture is ideal for building modular, reusable UI components. TypeScript ensures type safety, which is critical for maintaining a scalable codebase and reducing runtime errors.
- React's ecosystem has strong support for state management (Redux Toolkit), testing (Jest, React Testing Library), and performance optimization (React.memo, concurrent mode).
-
State Management: Redux Toolkit
- Redux Toolkit simplifies state management with built-in immutability and middleware support. It also integrates seamlessly with TypeScript.
-
Audio Engine: Tone.js
- Tone.js is a robust library for handling audio playback, scheduling, and effects. It supports seamless looping, cross-fading, and real-time audio manipulation.
-
Video Handling: HTML5
<video>API- For video rendering and playback, the native
<video>API is sufficient. It can be augmented with libraries likevideo.jsfor advanced features if needed.
- For video rendering and playback, the native
-
Waveform Visualization: wavesurfer.js
- Wavesurfer.js provides waveform visualization for audio clips, enabling precise editing and trimming.
-
Export Pipeline: ffmpeg.wasm
- ffmpeg.wasm allows for WebAssembly-based encoding of audio/video files directly in the browser, enabling MP4 muxing and WAV export without server-side dependencies.
-
Build Tooling: Vite
- Vite offers fast development builds and hot module replacement (HMR), which is crucial for productivity during development.
-
Testing: Jest + Playwright
- Jest for unit and integration tests, Playwright for end-to-end testing to ensure playback and export functionality.
-
Linting & Formatting: ESLint + Prettier
- Enforces consistent coding standards and prevents common mistakes.
Core Libraries
- Waveform Rendering:
wavesurfer.js - Audio Playback/Effects:
Tone.js - Video Playback: Native
<video>API - Export Encoding:
ffmpeg.wasm - Drag-and-Drop:
react-beautiful-dnd - UI Components:
MUI(Material-UI) orRadix UIfor pre-built, accessible components.
High-Level Component Diagram
+---------------------------------------------+ | App Shell | | | | +---------------------------------------+ | | | Header | | | +---------------------------------------+ | | | | +-------------------+ +---------------+ | | | Timeline | | Sidebar | | | | | | | | | | +-------------+ | | +---------+ | | | | | Audio Lane | | | | File | | | | | +-------------+ | | | Browser | | | | | | | +---------+ | | | | +-------------+ | | +---------+ | | | | | Video Lane | | | | Clip | | | | | +-------------+ | | | Editor | | | | | | | +---------+ | | | +-------------------+ +---------------+ | | | | +---------------------------------------+ | | | Playback Controls | | | +---------------------------------------+ | +---------------------------------------------+
Data Models
-
Clip Model
interface Clip { id: string; type: 'audio' | 'video'; src: string; // URL or Blob reference start: number; // Start time in seconds duration: number; // Duration in seconds role: 'intro' | 'loop' | 'outro'; groupId?: string; // Optional group ID for grouped clips } -
Timeline Model
interface Timeline { audioClips: Clip[]; videoClips: Clip[]; currentTime: number; // Current playhead position zoomLevel: number; // Zoom level of the timeline snapToGrid: boolean; } -
Project Model
interface Project { name: string; timeline: Timeline; metadata: Record<string, any>; // Custom metadata }
Modular Architecture
-
UI Layer
- Handles user interactions and renders the timeline, sidebar, and playback controls.
- Components:
Timeline,Sidebar,PlaybackControls.
-
State Layer
- Manages application state using Redux Toolkit.
- Slices:
timelineSlice,projectSlice.
-
Audio Engine
- Handles audio playback, cross-fading, and looping using Tone.js.
- Services:
AudioEngineService.
-
Export Module
- Handles project export (JSON, WAV, MP4).
- Utilizes Web Workers for offloading heavy tasks like encoding.
-
Plugin Points
- Exposes hooks for adding custom effects or triggers (e.g., MIDI foot-switch).
Incremental Roadmap
Sprint 1: MVP
- Basic drag-and-drop file import.
- Single audio clip playback with a timeline.
- Save project as JSON.
Sprint 2: Core Features
- Add dual-lane timeline (audio + video).
- Implement basic clip editing (trim, split, resize).
- Playback engine with intro → loop → outro flow.
Sprint 3: Advanced Editing
- Multi-select, undo/redo, and grouping.
- Snap-to-grid and zoom functionality.
- Cross-fade at segment boundaries.
Sprint 4: Export
- WAV mix-down for audio lane.
- MP4 muxing using ffmpeg.wasm.
Sprint 5: Quality of Life
- Keyboard shortcuts, theming, and accessibility improvements.
- Internationalization (i18n) hooks.
Stretch Goals
- Real-time reverb effect plugin.
- MIDI foot-switch trigger for playback control.
Build Pipeline Description
-
Development
- Use Vite for fast HMR and development builds.
- Run
pnpm devto start the development server.
-
Testing
- Run unit tests with
pnpm test. - Run e2e tests with
pnpm e2e.
- Run unit tests with
-
Production Build
- Run
pnpm buildto generate optimized production assets.
- Run
-
CI Pipeline
- Linting, formatting, and testing are enforced via GitHub Actions or similar CI tools.
Documentation
-
Architecture Diagram
- A high-level diagram showing the interaction between UI, state, audio engine, and export modules.
-
Public API Docs
- Document plugin points and APIs for extending the editor.
-
Onboarding README
- Step-by-step guide to set up the project locally and create a sample cutscene.
Sample Project Files
-
Demo JSON Scene
{ "name": "Sample Cutscene", "timeline": { "audioClips": [ { "id": "1", "type": "audio", "src": "intro.mp3", "start": 0, "duration": 10, "role": "intro" }, { "id": "2", "type": "audio", "src": "loop.mp3", "start": 10, "duration": 20, "role": "loop" } ], "videoClips": [ { "id": "3", "type": "video", "src": "scene.mp4", "start": 0, "duration": 30, "role": "intro" } ] } } -
Example Assets
- Include sample audio (
intro.mp3,loop.mp3) and video (scene.mp4) files.
- Include sample audio (
-
Getting Started Guide
- Instructions to install dependencies, run the app, and create a simple project.
This architecture ensures scalability, maintainability, and extensibility while providing a clear path to implementation.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
gemini-2.0-flash-001Compare side by side →
gemma-3-27b-itCompare side by side →
- o3-2025-04-16Compare side by side →
- gpt-4o-mini-2024-07-18Compare side by side →
gemini-2.5-pro-preview-03-25Compare side by side →
claude-3-5-haiku-20241022Compare side by side →
gemini-2.0-flash-001Compare side by side →
- gpt-4o-mini-2024-07-18Compare side by side →
- o4-mini-2025-04-16Compare side by side →
- o3-2025-04-16Compare side by side →
claude-3-7-sonnet-20250219Compare side by side →