<div align="center"> ⚡️Pyramid Flow⚡️ [[Paper]](https://arxiv.org/abs/2410.05954)…
- User
⚡️Pyramid Flow⚡️
[Paper] [Project Page ✨] [miniFLUX Model 🚀] [SD3 Model ⚡️] [demo 🤗]
This is the official repository for Pyramid Flow, a training-efficient Autoregressive Video Generation method based on Flow Matching. By training only on open-source datasets, it can generate high-quality 10-second videos at 768p resolution and 24 FPS, and naturally supports image-to-video generation.
News
-
2024.11.13🚀🚀🚀 We release the 768p miniFLUX checkpoint (up to 10s).We have switched the model structure from SD3 to a mini FLUX to fix human structure issues, please try our 1024p image checkpoint, 384p video checkpoint (up to 5s) and 768p video checkpoint (up to 10s). The new miniflux model shows great improvement on human structure and motion stability
-
2024.10.29⚡️⚡️⚡️ We release training code for VAE, finetuning code for DiT and new model checkpoints with FLUX structure trained from scratch. -
2024.10.13✨✨✨ Multi-GPU inference and CPU offloading are supported. Use it with less than 8GB of GPU memory, with great speedup on multiple GPUs. -
2024.10.11🤗🤗🤗 Hugging Face demo is available. Thanks @multimodalart for the commit! -
2024.10.10🚀🚀🚀 We release the technical report, project page and model checkpoint of Pyramid Flow.
Table of Contents
Introduction
[image: motivation]
Existing video diffusion models operate at full resolution, spending a lot of computation on very noisy latents. By contrast, our method harnesses the flexibility of flow matching (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden-Eijnden, 2023) to interpolate between latents of different resolutions and noise levels, allowing for simultaneous generation and decompression of visual content with better computational efficiency. The entire framework is end-to-end optimized with a single DiT (Peebles & Xie, 2023), generating high-quality 10-second videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours.
Installation
We recommend setting up the environment with conda. The codebase currently uses Python 3.8.10 and PyTorch 2.1.2 (guide), and we are actively working to support a wider range of versions.
git clone https://github.com/jy0205/Pyramid-Flow cd Pyramid-Flow # create env using conda conda create -n pyramid python==3.8.10 conda activate pyramid pip install -r requirements.txtThen, download the model from Huggingface (there are two variants: miniFLUX or SD3). The miniFLUX models support 1024p image, 384p and 768p video generation, and the SD3-based models support 768p and 384p video generation. The 384p checkpoint generates 5-second video at 24FPS, while the 768p checkpoint generates up to 10-second video at 24FPS.
from huggingface_hub import snapshot_download model_path = 'PATH' # The local directory to save downloaded checkpoint snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')Inference
1. Quick start with Gradio
To get started, first install Gradio, set your model path at #L36, and then run on your local machine:
python app.pyThe Gradio demo will be opened in a browser. Thanks to @tpc2233 the commit, see #48 for details.
Or, try it out effortlessly on Hugging Face Space 🤗 created by @multimodalart. Due to GPU limits, this online demo can only generate 25 frames (export at 8FPS or 24FPS). Duplicate the space to generate longer videos.
Quick Start on Google Colab
To quickly try out Pyramid Flow on Google Colab, run the code below:
# Setup !git clone https://github.com/jy0205/Pyramid-Flow %cd Pyramid-Flow !pip install -r requirements.txt !pip install gradio # This code downloads miniFLUX from huggingface_hub import snapshot_download model_path = '/content/Pyramid-Flow' snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model') # Start !python app.py2. Inference Code
To use our model, please follow the inference code in
video_generation_demo.ipynbat this link. We strongly recommend you to try the latest published pyramid-miniflux, which shows great improvement on human structure and motion stability. Set the parammodel_nametopyramid_fluxto use. We further simplify it into the following two-step procedure. First, load the downloaded model:import torch from PIL import Image from pyramid_dit import PyramidDiTForVideoGeneration from diffusers.utils import load_image, export_to_video torch.cuda.set_device(0) model_dtype, torch_dtype = 'bf16', torch.bfloat16 # Use bf16 (not support fp16 yet) model = PyramidDiTForVideoGeneration( 'PATH', # The downloaded checkpoint dir model_name="pyramid_flux", model_dtype=model_dtype, model_variant='diffusion_transformer_768p', ) model.vae.enable_tiling() # model.vae.to("cuda") # model.dit.to("cuda") # model.text_encoder.to("cuda") # if you're not using sequential offloading bellow uncomment the lines above ^ model.enable_sequential_cpu_offload()Then, you can try text-to-video generation on your own prompts. Noting that the 384p version only support 5s now (set temp up to 16)!
prompt = "A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors" # used for 384p model variant # width = 640 # height = 384 # used for 768p model variant width = 1280 height = 768 with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype): frames = model.generate( prompt=prompt, num_inference_steps=[20, 20, 20], video_num_inference_steps=[10, 10, 10], height=height, width=width, temp=16, # temp=16: 5s, temp=31: 10s guidance_scale=7.0, # The guidance for the first frame, set it to 7 for 384p variant video_guidance_scale=5.0, # The guidance for the other video latent output_type="pil", save_memory=True, # If you have enough GPU memory, set it to `False` to improve vae decoding speed ) export_to_video(frames, "./text_to_video_sample.mp4", fps=24)As an autoregressive model, our model also supports (text conditioned) image-to-video generation:
# used for 384p model variant # width = 640 # height = 384 # used for 768p model variant width = 1280 height = 768 image = Image.open('assets/the_great_wall.jpg').convert("RGB").resize((width, height)) prompt = "FPV flying over the Great Wall" with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype): frames = model.generate_i2v( prompt=prompt, input_image=image, num_inference_steps=[10, 10, 10], temp=16, video_guidance_scale=4.0, output_type="pil", save_memory=True, # If you have enough GPU memory, set it to `False` to improve vae decoding speed ) export_to_video(frames, "./image_to_video_sample.mp4", fps=24)CPU offloading
We also support two types of CPU offloading to reduce GPU memory requirements. Note that they may sacrifice efficiency.
- Adding a
cpu_offloading=Trueparameter to the generate function allows inference with less than 12GB of GPU memory. This feature was contributed by @Ednaordinary, see #23 for details. - Calling
model.enable_sequential_cpu_offload()before the above procedure allows inference with less than 8GB of GPU memory. This feature was contributed by @rodjjo, see #75 for details.
MPS backend
Thanks to @niw, Apple Silicon users (e.g. MacBook Pro with M2 24GB) can also try our model using the MPS backend! Please see #113 for the details.
3. Multi-GPU Inference
For users with multiple GPUs, we provide an inference script that uses sequence parallelism to save memory on each GPU. This also brings a big speedup, taking only 2.5 minutes to generate a 5s, 768p, 24fps video on 4 A100 GPUs (vs. 5.5 minutes on a single A100 GPU). Run it on 2 GPUs with the following command:
CUDA_VISIBLE_DEVICES=0,1 sh scripts/inference_multigpu.shIt currently supports 2 or 4 GPUs (For SD3 Version), with more configurations available in the original script. You can also launch a multi-GPU Gradio demo created by @tpc2233, see #59 for details.
Spoiler: We didn't even use sequence parallelism in training, thanks to our efficient pyramid flow designs.
4. Usage tips
- The
guidance_scaleparameter controls the visual quality. We suggest using a guidance within [7, 9] for the 768p checkpoint during text-to-video generation, and 7 for the 384p checkpoint. - The
video_guidance_scaleparameter controls the motion. A larger value increases the dynamic degree and mitigates the autoregressive generation degradation, while a smaller value stabilizes the video. - For 10-second video generation, we recommend using a guidance scale of 7 and a video guidance scale of 5.
Training
1. Training VAE
The hardware requirements for training VAE are at least 8 A100 GPUs. Please refer to this document. This is a MAGVIT-v2 like continuous 3D VAE, which should be quite flexible. Feel free to build your own video generative model on this part of VAE training code.
2. Finetuning DiT
The hardware requirements for finetuning DiT are at least 8 A100 GPUs. Please refer to this document. We provide instructions for both autoregressive and non-autoregressive versions of Pyramid Flow. The former is more research oriented and the latter is more stable (but less efficient without temporal pyramid).
Gallery
The following video examples are generated at 5s, 768p, 24fps. For more results, please visit our project page.
Comparison
On VBench (Huang et al., 2024), our method surpasses all the compared open-source baselines. Even with only public video data, it achieves comparable performance to commercial models like Kling (Kuaishou, 2024) and Gen-3 Alpha (Runway, 2024), especially in the quality score (84.74 vs. 84.11 of Gen-3) and motion smoothness.
[image: vbench]
We conduct an additional user study with 20+ participants. As can be seen, our method is preferred over open-source models such as Open-Sora and CogVideoX-2B especially in terms of motion smoothness.
[image: user_study]
Acknowledgement
We are grateful for the following awesome projects when implementing Pyramid Flow:
- SD3 Medium and Flux 1.0: State-of-the-art image generation models based on flow matching.
- Diffusion Forcing and GameNGen: Next-token prediction meets full-sequence diffusion.
- WebVid-10M, OpenVid-1M and Open-Sora Plan: Large-scale datasets for text-to-video generation.
- CogVideoX: An open-source text-to-video generation model that shares many training details.
- Video-LLaMA2: An open-source video LLM for our video recaptioning.
Citation
Consider giving this repository a star and cite Pyramid Flow in your publications if it helps your research.
@article{jin2024pyramidal, title={Pyramidal Flow Matching for Efficient Video Generative Modeling}, author={Jin, Yang and Sun, Zhicheng and Li, Ningyuan and Xu, Kun and Xu, Kun and Jiang, Hao and Zhuang, Nan and Huang, Quzhe and Song, Yang and Mu, Yadong and Lin, Zhouchen}, jounal={arXiv preprint arXiv:2410.05954}, year={2024} } -
- Assistant
Pyramid Flow
Pyramid Flow is a revolutionary autoregressive video generation method that utilizes flow matching techniques for enhanced computational efficiency. By exclusively training on open-source datasets, Pyramid Flow is capable of producing high-quality 10-second videos at a resolution of 768p, at a frame rate of 24 FPS. One of its standout features is the support for image-to-video transformation.
Highlights
- High-Quality Video Generation: Capable of generating 10-second videos at a resolution of 768p and 24 FPS.
- Image-to-Video Support: Allows users to transform images into videos seamlessly.
- Training-Efficient: The entire model can achieve impressive results without the need for extensive proprietary datasets.
Showcase
Video Samples
10s, 768p, 24fps 5s, 768p, 24fps Image-to-video [image: 10s video] open [image: 5s video] open [image: Image to video] open Important Updates
- Latest Release (2024.11.13): Launched the 768p miniFLUX checkpoint for enhanced human structure and motion stability.
- Previous updates include new model checkpoints, multi-GPU inference capability, and a Hugging Face demo.
Getting Started
Installation Steps
To install Pyramid Flow, follow these steps:
-
Clone the repository:
git clone https://github.com/jy0205/Pyramid-Flow cd Pyramid-Flow -
Create the conda environment:
conda create -n pyramid python==3.8.10 conda activate pyramid pip install -r requirements.txt -
Download the model from Hugging Face:
from huggingface_hub import snapshot_download model_path = 'YOUR_MODEL_PATH' snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
Running Inference
Quick Start with Gradio
To run the Gradio demo, ensure your model path is set, then execute:
python app.pyYou can also test directly via the Hugging Face Space for immediate results.
Using the Model
To generate videos, load your desired model and use the following example code:
import torch from pyramid_dit import PyramidDiTForVideoGeneration model = PyramidDiTForVideoGeneration('YOUR_MODEL_PATH', model_name="pyramid_flux") prompt = "Your video prompt here" frames = model.generate(prompt=prompt, num_inference_steps=[20, 20, 20], temp=16)Comparison and Performance
Pyramid Flow excels in benchmarks like VBench and user studies, showcasing superior quality scores and motion smoothness compared to existing models.
Acknowledgements
Pyramid Flow is built upon exceptional contributions from various foundational video generation projects such as SD3 Medium, FLUX, and CogVideoX.
Citation
If you find Pyramid Flow useful in your work, please consider citing:
@article{jin2024pyramidal, title={Pyramidal Flow Matching for Efficient Video Generative Modeling}, author={Jin, Yang et al.}, journal={arXiv preprint arXiv:2410.05954}, year={2024} }For more details and updates, refer to the Pyramid Flow project page. Enjoy creating with Pyramid Flow!
- User
How can I install this in colab?
- Assistant
To install and run Pyramid Flow in Google Colab, you can follow these steps. This includes setting up the environment, installing required libraries, and running the inference demo. Here's a step-by-step guide:
Step 1: Set Up the Environment
- Open a new Google Colab notebook.
- Run the following code to clone the Pyramid Flow repository, install the required libraries, and download the pre-trained model from Hugging Face:
# Clone the repository !git clone https://github.com/jy0205/Pyramid-Flow %cd Pyramid-Flow # Install required libraries !pip install -r requirements.txt !pip install gradio !pip install huggingface-hub --upgradeStep 2: Download the Pre-trained Model
Decide which model you want to use, either the
miniFLUXor theSD3model. Here’s how to download theminiFLUXmodel. Adjust the model name in the shared code for theSD3model if needed:from huggingface_hub import snapshot_download # Download the miniFLUX model model_path = './pyramid-flow-miniflux' # Directory to save the downloaded checkpoint snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')Step 3: Run the Inference Demo
Now that the environment is set up and the model is downloaded, run the Gradio app:
# Now, run the app !python app.pyStep 4: Access the Gradio Interface
Once the app is running, you will see a URL generated in the output, which typically looks like this:
Running on local URL: http://127.0.0.1:7860/To access the Gradio interface, you will need to click the link and open it in a new tab, or directly visit it in your browser:
- Paste the URL into a new browser tab (if Colab does not directly provide a clickable link).
- From there, you can enter prompts for video generation or upload images for the image-to-video feature.
Step 5: Additional Options
If you want to perform inference programmatically instead of using the Gradio demo, you can use the provided code snippets to generate videos based on text prompts or images using the loaded model. Here's an example for using text prompts:
import torch from PIL import Image from pyramid_dit import PyramidDiTForVideoGeneration from diffusers.utils import load_image, export_to_video torch.cuda.set_device(0) model_dtype, torch_dtype = 'bf16', torch.bfloat16 # Load the model model = PyramidDiTForVideoGeneration( model_path, model_name="pyramid_flux", model_dtype=model_dtype, model_variant='diffusion_transformer_768p', ) # Example prompt for video generation prompt = "A serene landscape during autumn, vibrant colors." width = 1280 # For 768p model height = 768 with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype): frames = model.generate( prompt=prompt, num_inference_steps=[20, 20, 20], video_num_inference_steps=[10, 10, 10], height=height, width=width, temp=16, guidance_scale=7.0, video_guidance_scale=5.0, output_type="pil", save_memory=True, ) # Save the generated video export_to_video(frames, "./generated_video.mp4", fps=24)Make sure to customize the prompt and settings (like resolution, inference steps, etc.) according to your needs.
Note
Due to Colab's GPU resource limits, generating longer videos might take a considerable amount of time, so plan accordingly. Enjoy experimenting with Pyramid Flow on Google Colab!
- User
You set
add_prefix_space. The tokenizer needs to be converted from the slow tokenizers Loading checkpoint shards: 50% 1/2 [00:27<00:27, 27.25s/it] - Assistant
The warning message you are encountering indicates a couple of potential issues while loading the model, specifically regarding the tokenizer. Here's what you can do to address this problem:
1. Update Your Installations
First, ensure that your environment is set up correctly with the latest versions of the required libraries. You can add the following commands to your Colab notebook to upgrade the libraries before running your main code:
# Upgrade transformers and diffusers to the latest version !pip install --upgrade transformers diffusers2. Set Up the Tokenizer Properly
Depending on the specific tokenizer setup, you might need to ensure that you are initializing the tokenizer correctly. Here is an example snippet to load the tokenizer which you should include in your code:
from transformers import AutoTokenizer # Load the tokenizer relevant to your model tokenizer = AutoTokenizer.from_pretrained( "rain1011/pyramid-flow-miniflux", use_fast=True # Ensure you're using the fast tokenizer if available )3. Handling the Checkpoint Loading
Make sure you're correctly handling the checkpoint shards. The message indicates that the checkpoint is being loaded; ensure that it is properly set up. If you're working with the
PyramidDiTForVideoGenerationmodel, make sure it corresponds to the tokenizer:from pyramid_dit import PyramidDiTForVideoGeneration # Load the model model = PyramidDiTForVideoGeneration( model_path, model_name="pyramid_flux", model_dtype='bf16', model_variant='diffusion_transformer_768p', )4. Verifying Model Directory Structure
Ensure that the model directory you downloaded contains all necessary files for loading correctly. Check if it contains files like:
config.jsonpytorch_model.bintokenizer.jsonortokenizer_config.json
If any essential files are missing, consider re-downloading the model.
5. Possible Issues with Fast Tokenizer
The warning about
add_prefix_spacelikely suggests that the tokenizer isn't being properly configured to manage certain types of token inputs. Ensure that your input handling is following the expected format for the chosen tokenizer.Full Colab Code Snippet Example
Here’s a complete example that encompasses the upgrades and proper loading of the model and tokenizer:
# Install necessary libraries !pip install --upgrade transformers diffusers gradio huggingface-hub # Clone the repository and navigate to it !git clone https://github.com/jy0205/Pyramid-Flow %cd Pyramid-Flow # Install other requirements if necessary !pip install -r requirements.txt from huggingface_hub import snapshot_download # Download the model model_path = './pyramid-flow-miniflux' snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model') # Import necessary libraries import torch from pyramid_dit import PyramidDiTForVideoGeneration from transformers import AutoTokenizer # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True) # Load the model model = PyramidDiTForVideoGeneration( model_path, model_name="pyramid_flux", model_dtype='bf16', model_variant='diffusion_transformer_768p', ) # Now perform inference or any operations you require # Example prompt prompt = "A serene landscape during autumn, vibrant colors." # Proceed with model generationConclusion
By ensuring that your environment is updated and that the tokenizer is set up properly, you should be able to avoid those warnings and load the model smoothly. If issues persist, verify the specific version compatibilities for the libraries used in your code.