based off the info below, what do I do is flash attention enabled by default or how do I…
- User
based off the info below, what do I do is flash attention enabled by default or how do I enable it?
Welcome to YuE-Exllamav2-UI, the ultimate optimized interface for music generation using YuE models with ExLlamaV2 acceleration. This project delivers the best possible performance for YuE models, achieving exceptional speed and efficiency on modern NVIDIA GPUs like the RTX 4090 and RTX 3060.
Note: This project is a fork of the fork made by user sgsdxzy which has been optimized for performance using exllamav2 from the YuE-exllamav2 repository. For more official information about the model, follow the official repository.
Key Features ExLlamaV2 Optimization: Inference acceleration with Flash Attention 2 and BF16 support. Exceptional Performance: Up to 500% speedup compared to the original implementation. NVIDIA GPU Support: Compatible with CUDA 12.2 and optimized for modern GPUs. User-Friendly Interface: Web-based Gradio interface for easy configuration and music generation. Audio Playback and Download: Listen to generated audio and download it directly from the interface. Pre-configured Docker: Easy deployment with Docker and support for persistent volumes. News and Updates Note: Keep an eye on updates to know when you need to do a docker pull alissonpereiraanjos/yue-exllamav2-interface
2025.02.04 🔥: Added field to save the result file with a custom name 2025.02.03 🔥: Added ExLlamaV2 integration with up to 500% speedup by sgsdxzy (https://github.com/sgsdxzy/YuE-exllamav2) 2025.02.01 (Oficial Repository) 🧑🏻💻 Inference Update: 1. Support dual-track ICL mode. 2. Fix "instrumental" naming bug in output files. 3. Support seeding. 2025.01.31 🔥: Added possibility to select existing models using a dropdown, added Refresh File Explorer button and Warnings are ignored in the log. 2025.01.30 🔥: Initial release with BF16 model support. Performance Benchmarks RTX 4090 24GB Stage Original ExLlamaV2 Speedup Stage 1 282s 125s 2.25x Stage 2 666s 49s 13.6x Total 948s 174s 5.45x Configuration: BF16 models, Flash Attention 2 enabled, CFG enabled, batch size 4, 64k cache.
RTX 3060 Mobile 6GB Stage ExLlamaV2 Stage 1 317s Stage 2 350s Total 667s Configuration: Quantized models with Q4/Q8 cache, Flash Attention 2 enabled.
How to Use Prerequisites Docker Ensure you have Docker installed on your system. Follow the official Docker installation guides for your platform:
Install Docker on Windows Install Docker on Ubuntu NVIDIA GPU Support This interface requires NVIDIA GPUs for acceleration. Ensure you have the necessary hardware and drivers set up.
Linux:
Install the NVIDIA Container Toolkit. Ensure your NVIDIA drivers are properly installed and up to date. Windows/macOS:
Refer to the respective Docker and NVIDIA documentation for GPU passthrough (e.g., WSL2 on Windows). Note: GPU support is mandatory. Without compatible NVIDIA GPUs, the container will not function correctly. Docker Compose To simplify the setup and management of the YuE-Exllamav2-UI, you can use Docker Compose. Docker Compose allows you to define and run multi-container Docker applications with a single configuration file (docker-compose.yml). Below are the steps to get started.
Note: This docker-compose.yml file already exists in the root of the repository, you just need to download or copy the file and replace the directory mapping and run the command in the same directory as the file, see the explanation below.
Create a docker-compose.yml file:
version: '3.8'
services: yue-exllamav2: image: alissonpereiraanjos/yue-exllamav2-interface:latest container_name: yue-exllamav2 restart: unless-stopped ports: - "7860:7860" environment: - DOWNLOAD_MODELS=all_bf16 volumes: - /path/to/models:/workspace/models - /path/to/outputs:/workspace/outputs deploy: resources: reservations: devices: - capabilities: [gpu] Run the container:
docker-compose up -d Access the Interface: Once the container is running, access the Gradio UI at http://localhost:7860.
Explanation of the Configuration image: Specifies the Docker image to use (alissonpereiraanjos/yue-exllamav2-interface:latest). container_name: Sets a name for the container (yue-interface). restart: unless-stopped: Ensures the container restarts automatically unless manually stopped. ports: Maps container ports to the host: 7860:7860: Port for accessing the Gradio UI. 8888:8888: Optional additional port (JupyterLab). environment: Defines environment variables: DOWNLOAD_MODELS=all: Downloads all available models. Replace all with specific model keys (e.g., YuE-s2-1B-general,YuE-s1-7B-anneal-en-cot) to download only selected models. volumes: Maps host directories to the container: /path/to/models:/workspace/models: Directory where models will be stored. /path/to/outputs:/workspace/outputs: Directory where generated outputs will be saved. Replace /path/to/models and /path/to/outputs with the actual paths on your system. deploy.resources.reservations.devices: Enables GPU support in the container (requires NVIDIA GPU and drivers). Customization Specific models: To download only specific models, modify the DOWNLOAD_MODELS environment variable in the docker-compose.yml file. For example:
environment:
- DOWNLOAD_MODELS=YuE-s2-1B-general,YuE-s1-7B-anneal-en-cot Different ports: If you need to use different ports, adjust the port mappings under the ports section.
Direct Docker Run To run without Docker Compose:
docker run --gpus all -d
-p 7860:7860
-e DOWNLOAD_MODELS=all_bf16
-v /path/to/models:/workspace/models
-v /path/to/outputs:/workspace/outputs
alissonpereiraanjos/yue-exllamav2-interface:latest --gpus all: Enables NVIDIA GPU support. -d: Runs the container in detached mode (background). -p 7860:7860: Exposes port 7860 for accessing the Gradio UI at http://localhost:7860. -p 8888:8888: Exposes port 8888 for additional services if applicable. -e DOWNLOAD_MODELS=all_bf16: Downloads all bf16 available models upon initialization. Environment Variables DOWNLOAD_MODELS: Determines which models to download. Set to all_bf16 to download all available models (BF16). Alternatively, specify a comma-separated list of model keys to download specific models (e.g., DOWNLOAD_MODELS=YuE-s2-1B-general,YuE-s1-7B-anneal-en-cot). Supported Models Model Key Model HF Repository Container Path Quantization xcodec_mini_infer m-a-p/xcodec_mini_infer /workspace/YuE-Interface/inference/xcodec_mini_infer N/A YuE-s1-7B-anneal-en-cot m-a-p/YuE-s1-7B-anneal-en-cot /workspace/models/YuE-s1-7B-anneal-en-cot BF16 YuE-s1-7B-anneal-en-icl m-a-p/YuE-s1-7B-anneal-en-icl /workspace/models/YuE-s1-7B-anneal-en-icl BF16 YuE-s1-7B-anneal-jp-kr-cot m-a-p/YuE-s1-7B-anneal-jp-kr-cot /workspace/models/YuE-s1-7B-anneal-jp-kr-cot BF16 YuE-s1-7B-anneal-jp-kr-icl m-a-p/YuE-s1-7B-anneal-jp-kr-icl /workspace/models/YuE-s1-7B-anneal-jp-kr-icl BF16 YuE-s1-7B-anneal-zh-cot m-a-p/YuE-s1-7B-anneal-zh-cot /workspace/models/YuE-s1-7B-anneal-zh-cot BF16 YuE-s1-7B-anneal-zh-icl m-a-p/YuE-s1-7B-anneal-zh-icl /workspace/models/YuE-s1-7B-anneal-zh-icl BF16 YuE-s2-1B-general m-a-p/YuE-s2-1B-general /workspace/models/YuE-s2-1B-general BF16 YuE-upsampler m-a-p/YuE-upsampler /workspace/models/YuE-upsampler BF16 Mapping Directories for Models and Output You can mount host directories to store models and outputs outside the container:docker run --gpus all -it
-v /path/to/models:/workspace/models
-v /path/to/outputs:/workspace/outputs
-p 7860:7860
-p 8888:8888
-e DOWNLOAD_MODELS=false
alissonpereiraanjos/yue-exllamav2-interface:latest -v /path/to/models:/workspace/models: Mounts the host's /path/to/models directory to /workspace/models inside the container. -v /path/to/outputs:/workspace/outputs: Mounts the host's /path/to/outputs directory to /workspace/outputs inside the container. -e DOWNLOAD_MODELS=false: Skips automatic model downloads (useful if models are already present in the mounted directories). RunPod Deployment If you prefer to use RunPod, you can quickly deploy an instance based on this image by using the following template link:Deploy on RunPod
This link directs you to the RunPod console, allowing you to set up a machine directly with the YuE Interface image. Configure your GPU, volume mounts, and environment variables as needed.
Tip: If you generate music frequently, consider creating a Network Volume in RunPod. This allows you to store models and data persistently, avoiding repeated downloads and saving time.
Recommended Settings:
GPU: RTX 4090 or A100 Network Volume: Minimum 100GB for model storage Environment Variables: DOWNLOAD_MODELS=all_bf16 Update Docker Image (Important) To update the Docker image with the latest changes, run:
docker pull alissonpereiraanjos/yue-exllamav2-interface:latest Note: Always update the image before running the container to ensure you have the latest features and fixes. This is especially important when deploying on RunPod, as it pulls the latest image upon creating a new pod.
Model Suffixes Explained The suffixes in the model keys indicate specific training or optimization techniques applied to the models:
Suffix Meaning Description COT Chain-of-Thought Models trained with Chain-of-Thought to enhance reasoning and logical generation capabilities. ICL In-Context Learning Models optimized for In-Context Learning, allowing dynamic adaptation based on the provided context. Examples:
YuE-s1-7B-anneal-en-cot: A model trained with Chain-of-Thought techniques. YuE-s1-7B-anneal-en-icl: A model optimized for In-Context Learning. Acknowledgements A special thanks to the YuE-exllamav2 repository for their incredible optimization work, which made this project possible.
Support For technical support or questions:
GitHub Issues CivitAI Profile Citation If you use this project in your research, please consider citing:
@misc{yuan2025yue, title={YuE: Open Music Foundation Models for Full-Song Generation}, author={Ruibin Yuan et al.}, year={2025}, howpublished={\url{https://github.com/multimodal-art-projection/YuE}} } Experience music generation with accelerated performance! 🎵🚀
- Assistant
Based on the information provided, Flash Attention (specifically, Flash Attention 2) is enabled by default in the YuE-Exllamav2-UI setup. This is indicated in the performance benchmarks, which mention that Flash Attention 2 is included in the configuration for achieving improved speed up in the model's performance.
If you’re looking to ensure that Flash Attention is enabled or if you want to confirm your configuration:
-
Default Behavior: If you use the provided Docker image (
alissonpereiraanjos/yue-exllamav2-interface:latest) as-is, Flash Attention should be enabled by default as part of the ExLlamaV2 optimization noted in the key features. -
Custom Configuration: If you're customizing your setup or looking deeper into configuration files, you might want to check any environment variables or settings that are associated with model inference, though the documentation does not provide specific toggles for Flash Attention.
-
Running Docker: When you run the Docker container using the provided commands (either the
docker runordocker-composecommands), ensure that you are using the parameters specified in the examples:- Use the environment variable
-e DOWNLOAD_MODELS=all_bf16to load models that support BF16 and Flash Attention.
- Use the environment variable
-
Confirming in Code: If you're comfortable with source code, checking the specific implementation files within the Docker image or repo related to the model's inference might provide insights or options explicitly enabling or toggling Flash Attention.
-
Testing Performance: You could conduct performance benchmarks yourself to see if you're achieving speedups that align with the stated benefits of Flash Attention 2.
If you follow the standard setup and run the provided commands, Flash Attention should be enabled by default without needing additional configuration.
-
- User
where do I edit batch size
- Assistant
To edit the batch size for the YuE-Exllamav2-UI, you'll need to adjust the relevant settings in the configuration file or environment variables typically used by the application. Here’s a general approach based on the Docker setup:
Step 1: Check for a Configuration File
-
Look for a config file: In many applications, settings like batch size might be specified in a configuration file, potentially located in the root of the repository or within a designated config directory. Search for files like
config.json,settings.yaml, or similar. -
Environment Variables: If there is no configuration file, check if there's a section in the
docker-compose.ymlor the relevant Docker run command to set environment variables.
Step 2: Modify the Docker Compose File
If you’re using
docker-compose.yml, you might specify the batch size as an environment variable. Here’s a hypothetical example:version: '3.8' services: yue-exllamav2: image: alissonpereiraanjos/yue-exllamav2-interface:latest container_name: yue-exllamav2 restart: unless-stopped ports: - "7860:7860" environment: - DOWNLOAD_MODELS=all_bf16 - BATCH_SIZE=4 # Add this line to set the batch size volumes: - /path/to/models:/workspace/models - /path/to/outputs:/workspace/outputsStep 3: Direct Docker Run
If you are running the interface directly using the Docker command line, add the batch size as an environment variable:
docker run --gpus all -d \ -p 7860:7860 \ -e DOWNLOAD_MODELS=all_bf16 \ -e BATCH_SIZE=4 \ # Adjust the batch size here -v /path/to/models:/workspace/models \ -v /path/to/outputs:/workspace/outputs \ alissonpereiraanjos/yue-exllamav2-interface:latestStep 4: Check Documentation
Since the documentation you provided doesn’t specify batch size settings explicitly, you may still want to check:
- The official documentation or README files.
- Issues or discussions in the project’s GitHub repository to see if other users have found where to configure batch size.
Step 5: Test Changes
After modifying the configuration or environment variables:
- Run your Docker container again (using
docker-compose upor the Docker run command). - Monitor the application to ensure it operates correctly with the new batch size.
By following these steps, you should be able to adjust the batch size for the YuE-Exllamav2-UI to suit your performance needs.
-