I want to build a CLI tool that automates the process of downloading videos from a list…
- User
I want to build a CLI tool that automates the process of downloading videos from a list of youtube channels and playlists and then sorts them into the right folder on my computer. Essentially its job is to look for new videos from the provided source list, figure out which folder in my local media folder each respective video belongs in, and finally downloads + places those videos into those respective folders.
I already have a folder structure where I've organized hundreds of videos manually. It's a nested folder structure of different custom categories I've defined, where each folder is a category. Most of the videos I've already downloaded (several hundred) use the same title as the YouTube source. The idea is we can use an LLM to perform the categorization by passing in a file tree structure of the existing folders and file names within them. Ideally we would use the LLM service's caching function for cost optimization.
It should use a config file which lets me specify the sources (channels and playlists) we should download from. Each source should have a 'last synced date' parameter to track when that source was last checked and downloaded from. This is to make it easy for me to add new sources over time and prevent the script from checking videos from before the last synced date. The script should use the yt-dlp package to manage downloads/metadata from youtube (that's what I use right now for manually downloading stuff). Because many of the videos I'm downloading are in foreign languages I should be able to specify the subtitle languages I want downloaded and attached to every video that gets downloaded. The config file should also let me enable debug logging.
For reference, here's the command I currently use for downloading these videos (note that this command embeds subtitles too):
yt-dlp -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4" --embed-subs --write-auto-subs --sub-langs "en,en-US,en*,en-*"
Functionally I should be able to run the CLI command and the script will begin checking the sources from the config file, finding videos that were added or uploaded past the last sync date and getting the metadata for them. It will then use the metadata (the video's title) to figure out which folder each new video should go into. Importantly it shouldn't start downloading the videos yet because it should allow me to verify that I'm happy with the videos to download list as well as their designated folders. This verification step should let me make modifications to videos that I'm not happy with, such as removing videos from being downloaded or changing the designated folder (either by re-running the LLM categorization process or by manually selecting a folder myself). Finally, when I'm satisfied, the script should begin downloading videos. A progress UI should be present that lets me see the status of the downloads (including errors, such as if some downloads fail or if subtitles can't be found for example)
Lastly, the CLI should have an override that lets me directly provide a bunch of youtube video, channel, or playlist links which should be used as sources instead of the sources provided in the config file. This is to make it easy for me to download ad-hoc videos I find that aren't in the usual channels/playlists present in the config file.
Give me an overview of how this script would work, from a UX and technological standpoint. Feel free to ask clarifying questions if necessary. When we're both satisfied, we can begin the technical implementation.
Assistant
Okay, this is a well-defined project! Here's an overview of the CLI tool, covering UX and technological aspects. Let's break it down into stages:
I. Configuration & CLI Structure
- Configuration File (config.yaml/config.json):
- Structure: YAML or JSON format for easy readability and modification.
- Top-Level:
debug: Boolean (true/false) to enable/disable debug logging.media_root: String. The absolute path to your root media folder (e.g.,/home/user/media). Crucial for file organization.subtitle_languages: List of strings. Subtitle language codes (e.g.,["en", "en-US", "en*", "en-*"]).yt_dlp_options: List of strings. Command-line options to pass to yt-dlp.
sources: A list of dictionaries, each representing a YouTube source:type: String. Either"channel"or"playlist".url: String. The YouTube channel or playlist URL.last_synced: Date string (ISO 8601 format:YYYY-MM-DDorYYYY-MM-DDTHH:MM:SSZ) indicating the last time this source was checked. If the source has never been synced, use a date long in the past (e.g.,"1970-01-01").category: String. The folder path to use by default when LLM fails to categorize (e.g. "Uncategorized").
- Example:
debug: true media_root: /home/user/MyVideos subtitle_languages: ["en", "en-US", "en*", "en-*"] yt_dlp_options: ["-f", "bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4", "--embed-subs", "--write-auto-subs"] sources: - type: channel url: https://www.youtube.com/c/LinusTechTips last_synced: 2023-10-26 category: Tech/LinusTechTips - type: playlist url: https://www.youtube.com/playlist?list=PL7D2RMSi08H4_t8-zR54AjeE00wzR-R-2 last_synced: 2023-10-27 category: Documentaries/Nature-
CLI Commands:
-
sync: The main command to start the synchronization process.--config <path>: (Optional) Specifies the path to the configuration file. Defaults toconfig.yamlorconfig.jsonin the current directory.--override-sources <url1> <url2> ...: (Optional) Overrides thesourcesin the config file. Accepts a space-separated list of YouTube URLs (channels, playlists, individual videos).--no-verify: (Optional) Skips the verification step and immediately starts downloading. Use with caution!
-
categorize <video_title>: Manually categorizes a video title. This is useful for correcting LLM errors or for adding videos that are difficult to categorize.<video_title>: The title of the video to categorize.--category <category_path>: The desired category path (e.g.,Documentaries/History).
-
II. Synchronization Workflow
-
CLI Argument Parsing & Configuration Loading:
- The script parses command-line arguments using a library like
argparseorclick. - It loads the configuration file using a library like
PyYAMLorjson. - It handles the
--override-sourcesoption. If provided, it ignores thesourcesfrom the configuration file and uses the provided URLs instead.
- The script parses command-line arguments using a library like
-
Source Enumeration & Metadata Retrieval:
- Iterate through each source in the configuration (or overridden sources).
- Use
yt-dlp's Python API (or subprocess calls if necessary) to:- Fetch a list of videos for each channel or playlist.
- Filter videos to include only those uploaded after the
last_synceddate for that source. - Extract video metadata: title, URL, upload date, description (optional, for LLM context).
-
Categorization (LLM-Assisted):
- For each new video:
- Format the file tree structure into an input prompt
- Construct a prompt for the LLM. This prompt should:
- Include the video title.
- Optionally include the video description or channel name for context.
- Include the file tree structure
- Clearly instruct the LLM to choose the best matching category from the file tree. The prompt should emphasize that the output must exactly match one of the folders and should explain what to do in case none of the folders match the video
- Example:
Video Title: "The Future of Quantum Computing" File Tree: - Tech - Quantum Computing - Artificial Intelligence - Documentaries - Science - History Based on the video title above, select the best matching folder from the file tree:
- Send the prompt to the LLM API (e.g., OpenAI's API).
- Store the LLM's predicted category.
- If the LLM fails to categorize (e.g., returns an error or a category that doesn't exist), default to the
categoryspecified in the config file for that source.
- For each new video:
-
Verification Stage (Interactive):
- Display a list of videos to be downloaded, including:
- Video Title
- Predicted Category
- URL
- Provide options to:
- Approve a video for download.
- Reject a video (remove it from the download list).
- Re-categorize a video using the LLM (re-run the LLM categorization process for that specific video).
- Manually specify a category for a video.
- Use a library like
inquirerorrichto create an interactive command-line interface.
- Display a list of videos to be downloaded, including:
-
Downloading & File Management:
- Once the verification is complete, iterate through the approved video list.
- For each video:
- Construct the full file path:
media_root/predicted_category/video_title.mp4(or other appropriate extension). Sanitize thevideo_titleto remove invalid characters for filenames. - Construct the
yt-dlpcommand. Include thesubtitle_languagesfrom the config file. - Execute the
yt-dlpcommand usingsubprocess.run(). Capture the output (stdout and stderr) for logging. - Implement robust error handling:
- If
yt-dlpfails (e.g., network error, video unavailable), log the error and continue to the next video. - If subtitles are not found, log a warning but continue downloading the video.
- If
- Construct the full file path:
-
Updating
last_syncedDates:- After downloading all videos from a source (or encountering an unrecoverable error), update the
last_synceddate in the configuration file for that source. Use the current date and time. - Write the updated configuration back to the file. Make sure to handle potential file writing errors.
- After downloading all videos from a source (or encountering an unrecoverable error), update the
III. UX Considerations
- Clear Output: Provide informative messages to the user throughout the process. Use logging levels (DEBUG, INFO, WARNING, ERROR) to control the verbosity of the output.
- Progress Bar: Use a library like
tqdmto display a progress bar during the download phase. - Error Handling: Gracefully handle errors and provide informative error messages to the user. Don't crash the script!
- Interactive Verification: The verification step is crucial. Make it easy for the user to review and modify the video list. Use a clear and concise interface.
- Configuration File: Make the configuration file easy to understand and modify. Provide clear documentation.
IV. Technology Stack
- Python: The core language.
- yt-dlp: For YouTube video downloading and metadata extraction. Use the Python API if possible, otherwise use
subprocess. - PyYAML or json: For reading and writing the configuration file.
- argparse or click: For command-line argument parsing.
- tqdm: For progress bars.
- inquirer or rich: For creating an interactive command-line interface.
- openai: For using the LLM. You'll need an API key and a suitable model (e.g., GPT-3.5 or GPT-4). You can use the official openai python package to interact with the API.
- logging: For logging events and errors.
- os, pathlib: For file system operations (creating directories, sanitizing filenames).
V. Further Considerations
- LLM Caching: Implement a caching mechanism to avoid repeatedly querying the LLM for the same video titles. You can use a simple in-memory cache or a more persistent cache using a database like SQLite. This will save on API costs.
- Concurrency: Consider using threading or asynchronous programming to download multiple videos concurrently. This can significantly speed up the download process. Be careful not to overwhelm your network connection or the YouTube servers.
- Testing: Write unit tests to ensure that the script is working correctly.
- Packaging: Package the script as a standalone executable using a tool like PyInstaller.
Clarifying Questions:
- LLM Service: What specific LLM service do you plan to use (e.g., OpenAI, Google Vertex AI, etc.)? This will influence the API integration.
- Error Handling Preferences: How aggressively should the script handle errors during the download phase? Should it retry failed downloads? Should it skip videos with missing subtitles?
- File Naming: Besides sanitizing the video title, do you have any specific file naming conventions (e.g., adding upload dates to filenames)?
- Verification UX: Do you have a specific vision for the interactive verification stage? (e.g. should all the options be presented in a TUI, or should it let me re-categorize the videos through a web interface, etc)
- Dependencies: Are you willing to install extra dependencies besides the usual? (Like installing a TUI lib like 'inquirer')
Let me know if you have any other questions. Once we're aligned on the overall design, we can start with the technical implementation. We can start with the core CLI structure, configuration loading, and source enumeration.
- Configuration File (config.yaml/config.json):
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
gemini-2.5-pro-preview-03-25Compare side by side →
- chatgpt-4o-latest-20250326Compare side by side →
- grok-3-mini-betaCompare side by side →
- chatgpt-4o-latest-20250326Compare side by side →
claude-3-5-sonnet-20241022Compare side by side →
- grok-3-mini-betaCompare side by side →
llama-3.3-70b-instructCompare side by side →
- o3-miniCompare side by side →
- o4-mini-2025-04-16Compare side by side →
gemini-2.5-pro-preview-03-25Compare side by side →
claude-3-5-sonnet-20241022Compare side by side →