Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-2024-07-18· OpenAIAi2 WildChat-4.8M (AI2)

<div align="center"> ⚡️Pyramid Flow⚡️ [[Paper]](https://arxiv.org/abs/2410.05954)…

  1. User

    ⚡️Pyramid Flow⚡️

    [Paper] [Project Page ✨] [miniFLUX Model 🚀] [SD3 Model ⚡️] [demo 🤗]

    This is the official repository for Pyramid Flow, a training-efficient Autoregressive Video Generation method based on Flow Matching. By training only on open-source datasets, it can generate high-quality 10-second videos at 768p resolution and 24 FPS, and naturally supports image-to-video generation.

    News

    Table of Contents

    Introduction

    [image: motivation]

    Existing video diffusion models operate at full resolution, spending a lot of computation on very noisy latents. By contrast, our method harnesses the flexibility of flow matching (Lipman et al., 2023; Liu et al., 2023; Albergo & Vanden-Eijnden, 2023) to interpolate between latents of different resolutions and noise levels, allowing for simultaneous generation and decompression of visual content with better computational efficiency. The entire framework is end-to-end optimized with a single DiT (Peebles & Xie, 2023), generating high-quality 10-second videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours.

    Installation

    We recommend setting up the environment with conda. The codebase currently uses Python 3.8.10 and PyTorch 2.1.2 (guide), and we are actively working to support a wider range of versions.

    git clone https://github.com/jy0205/Pyramid-Flow
    cd Pyramid-Flow
    
    # create env using conda
    conda create -n pyramid python==3.8.10
    conda activate pyramid
    pip install -r requirements.txt
    

    Then, download the model from Huggingface (there are two variants: miniFLUX or SD3). The miniFLUX models support 1024p image, 384p and 768p video generation, and the SD3-based models support 768p and 384p video generation. The 384p checkpoint generates 5-second video at 24FPS, while the 768p checkpoint generates up to 10-second video at 24FPS.

    from huggingface_hub import snapshot_download
    
    model_path = 'PATH'   # The local directory to save downloaded checkpoint
    snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
    

    Inference

    1. Quick start with Gradio

    To get started, first install Gradio, set your model path at #L36, and then run on your local machine:

    python app.py
    

    The Gradio demo will be opened in a browser. Thanks to @tpc2233 the commit, see #48 for details.

    Or, try it out effortlessly on Hugging Face Space 🤗 created by @multimodalart. Due to GPU limits, this online demo can only generate 25 frames (export at 8FPS or 24FPS). Duplicate the space to generate longer videos.

    Quick Start on Google Colab

    To quickly try out Pyramid Flow on Google Colab, run the code below:

    # Setup
    !git clone https://github.com/jy0205/Pyramid-Flow
    %cd Pyramid-Flow
    !pip install -r requirements.txt
    !pip install gradio
    
    # This code downloads miniFLUX
    from huggingface_hub import snapshot_download
    
    model_path = '/content/Pyramid-Flow'
    snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
    
    # Start
    !python app.py
    

    2. Inference Code

    To use our model, please follow the inference code in video_generation_demo.ipynb at this link. We strongly recommend you to try the latest published pyramid-miniflux, which shows great improvement on human structure and motion stability. Set the param model_name to pyramid_flux to use. We further simplify it into the following two-step procedure. First, load the downloaded model:

    import torch
    from PIL import Image
    from pyramid_dit import PyramidDiTForVideoGeneration
    from diffusers.utils import load_image, export_to_video
    
    torch.cuda.set_device(0)
    model_dtype, torch_dtype = 'bf16', torch.bfloat16   # Use bf16 (not support fp16 yet)
    
    model = PyramidDiTForVideoGeneration(
        'PATH',                                         # The downloaded checkpoint dir
        model_name="pyramid_flux",
        model_dtype=model_dtype,
        model_variant='diffusion_transformer_768p',
    )
    
    model.vae.enable_tiling()
    # model.vae.to("cuda")
    # model.dit.to("cuda")
    # model.text_encoder.to("cuda")
    
    # if you're not using sequential offloading bellow uncomment the lines above ^
    model.enable_sequential_cpu_offload()
    

    Then, you can try text-to-video generation on your own prompts. Noting that the 384p version only support 5s now (set temp up to 16)!

    prompt = "A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors"
    
    # used for 384p model variant
    # width = 640
    # height = 384
    
    # used for 768p model variant
    width = 1280
    height = 768
    
    with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
        frames = model.generate(
            prompt=prompt,
            num_inference_steps=[20, 20, 20],
            video_num_inference_steps=[10, 10, 10],
            height=height,     
            width=width,
            temp=16,                    # temp=16: 5s, temp=31: 10s
            guidance_scale=7.0,         # The guidance for the first frame, set it to 7 for 384p variant
            video_guidance_scale=5.0,   # The guidance for the other video latent
            output_type="pil",
            save_memory=True,           # If you have enough GPU memory, set it to `False` to improve vae decoding speed
        )
    
    export_to_video(frames, "./text_to_video_sample.mp4", fps=24)
    

    As an autoregressive model, our model also supports (text conditioned) image-to-video generation:

    # used for 384p model variant
    # width = 640
    # height = 384
    
    # used for 768p model variant
    width = 1280
    height = 768
    
    image = Image.open('assets/the_great_wall.jpg').convert("RGB").resize((width, height))
    prompt = "FPV flying over the Great Wall"
    
    with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
        frames = model.generate_i2v(
            prompt=prompt,
            input_image=image,
            num_inference_steps=[10, 10, 10],
            temp=16,
            video_guidance_scale=4.0,
            output_type="pil",
            save_memory=True,           # If you have enough GPU memory, set it to `False` to improve vae decoding speed
        )
    
    export_to_video(frames, "./image_to_video_sample.mp4", fps=24)
    

    CPU offloading

    We also support two types of CPU offloading to reduce GPU memory requirements. Note that they may sacrifice efficiency.

    • Adding a cpu_offloading=True parameter to the generate function allows inference with less than 12GB of GPU memory. This feature was contributed by @Ednaordinary, see #23 for details.
    • Calling model.enable_sequential_cpu_offload() before the above procedure allows inference with less than 8GB of GPU memory. This feature was contributed by @rodjjo, see #75 for details.

    MPS backend

    Thanks to @niw, Apple Silicon users (e.g. MacBook Pro with M2 24GB) can also try our model using the MPS backend! Please see #113 for the details.

    3. Multi-GPU Inference

    For users with multiple GPUs, we provide an inference script that uses sequence parallelism to save memory on each GPU. This also brings a big speedup, taking only 2.5 minutes to generate a 5s, 768p, 24fps video on 4 A100 GPUs (vs. 5.5 minutes on a single A100 GPU). Run it on 2 GPUs with the following command:

    CUDA_VISIBLE_DEVICES=0,1 sh scripts/inference_multigpu.sh
    

    It currently supports 2 or 4 GPUs (For SD3 Version), with more configurations available in the original script. You can also launch a multi-GPU Gradio demo created by @tpc2233, see #59 for details.

    Spoiler: We didn't even use sequence parallelism in training, thanks to our efficient pyramid flow designs.

    4. Usage tips

    • The guidance_scale parameter controls the visual quality. We suggest using a guidance within [7, 9] for the 768p checkpoint during text-to-video generation, and 7 for the 384p checkpoint.
    • The video_guidance_scale parameter controls the motion. A larger value increases the dynamic degree and mitigates the autoregressive generation degradation, while a smaller value stabilizes the video.
    • For 10-second video generation, we recommend using a guidance scale of 7 and a video guidance scale of 5.

    Training

    1. Training VAE

    The hardware requirements for training VAE are at least 8 A100 GPUs. Please refer to this document. This is a MAGVIT-v2 like continuous 3D VAE, which should be quite flexible. Feel free to build your own video generative model on this part of VAE training code.

    2. Finetuning DiT

    The hardware requirements for finetuning DiT are at least 8 A100 GPUs. Please refer to this document. We provide instructions for both autoregressive and non-autoregressive versions of Pyramid Flow. The former is more research oriented and the latter is more stable (but less efficient without temporal pyramid).

    Gallery

    The following video examples are generated at 5s, 768p, 24fps. For more results, please visit our project page.

    Comparison

    On VBench (Huang et al., 2024), our method surpasses all the compared open-source baselines. Even with only public video data, it achieves comparable performance to commercial models like Kling (Kuaishou, 2024) and Gen-3 Alpha (Runway, 2024), especially in the quality score (84.74 vs. 84.11 of Gen-3) and motion smoothness.

    [image: vbench]

    We conduct an additional user study with 20+ participants. As can be seen, our method is preferred over open-source models such as Open-Sora and CogVideoX-2B especially in terms of motion smoothness.

    [image: user_study]

    Acknowledgement

    We are grateful for the following awesome projects when implementing Pyramid Flow:

    Citation

    Consider giving this repository a star and cite Pyramid Flow in your publications if it helps your research.

    @article{jin2024pyramidal,
      title={Pyramidal Flow Matching for Efficient Video Generative Modeling},
      author={Jin, Yang and Sun, Zhicheng and Li, Ningyuan and Xu, Kun and Xu, Kun and Jiang, Hao and Zhuang, Nan and Huang, Quzhe and Song, Yang and Mu, Yadong and Lin, Zhouchen},
      jounal={arXiv preprint arXiv:2410.05954},
      year={2024}
    }
    
  2. Assistant

    Pyramid Flow

    Pyramid Flow is a revolutionary autoregressive video generation method that utilizes flow matching techniques for enhanced computational efficiency. By exclusively training on open-source datasets, Pyramid Flow is capable of producing high-quality 10-second videos at a resolution of 768p, at a frame rate of 24 FPS. One of its standout features is the support for image-to-video transformation.

    Highlights

    • High-Quality Video Generation: Capable of generating 10-second videos at a resolution of 768p and 24 FPS.
    • Image-to-Video Support: Allows users to transform images into videos seamlessly.
    • Training-Efficient: The entire model can achieve impressive results without the need for extensive proprietary datasets.

    Showcase

    Video Samples

    10s, 768p, 24fps5s, 768p, 24fpsImage-to-video
    [image: 10s video] open[image: 5s video] open[image: Image to video] open

    Important Updates

    • Latest Release (2024.11.13): Launched the 768p miniFLUX checkpoint for enhanced human structure and motion stability.
    • Previous updates include new model checkpoints, multi-GPU inference capability, and a Hugging Face demo.

    Getting Started

    Installation Steps

    To install Pyramid Flow, follow these steps:

    1. Clone the repository:

      git clone https://github.com/jy0205/Pyramid-Flow
      cd Pyramid-Flow
      
    2. Create the conda environment:

      conda create -n pyramid python==3.8.10
      conda activate pyramid
      pip install -r requirements.txt
      
    3. Download the model from Hugging Face:

      from huggingface_hub import snapshot_download
      
      model_path = 'YOUR_MODEL_PATH'
      snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
      

    Running Inference

    Quick Start with Gradio

    To run the Gradio demo, ensure your model path is set, then execute:

    python app.py
    

    You can also test directly via the Hugging Face Space for immediate results.

    Using the Model

    To generate videos, load your desired model and use the following example code:

    import torch
    from pyramid_dit import PyramidDiTForVideoGeneration
    
    model = PyramidDiTForVideoGeneration('YOUR_MODEL_PATH', model_name="pyramid_flux")
    
    prompt = "Your video prompt here"
    frames = model.generate(prompt=prompt, num_inference_steps=[20, 20, 20], temp=16)
    

    Comparison and Performance

    Pyramid Flow excels in benchmarks like VBench and user studies, showcasing superior quality scores and motion smoothness compared to existing models.

    Acknowledgements

    Pyramid Flow is built upon exceptional contributions from various foundational video generation projects such as SD3 Medium, FLUX, and CogVideoX.

    Citation

    If you find Pyramid Flow useful in your work, please consider citing:

    @article{jin2024pyramidal,
      title={Pyramidal Flow Matching for Efficient Video Generative Modeling},
      author={Jin, Yang et al.},
      journal={arXiv preprint arXiv:2410.05954},
      year={2024}
    }
    

    For more details and updates, refer to the Pyramid Flow project page. Enjoy creating with Pyramid Flow!

  3. User

    How can I install this in colab?

  4. Assistant

    To install and run Pyramid Flow in Google Colab, you can follow these steps. This includes setting up the environment, installing required libraries, and running the inference demo. Here's a step-by-step guide:

    Step 1: Set Up the Environment

    1. Open a new Google Colab notebook.
    2. Run the following code to clone the Pyramid Flow repository, install the required libraries, and download the pre-trained model from Hugging Face:
    # Clone the repository
    !git clone https://github.com/jy0205/Pyramid-Flow
    %cd Pyramid-Flow
    
    # Install required libraries
    !pip install -r requirements.txt
    !pip install gradio
    !pip install huggingface-hub --upgrade
    

    Step 2: Download the Pre-trained Model

    Decide which model you want to use, either the miniFLUX or the SD3 model. Here’s how to download the miniFLUX model. Adjust the model name in the shared code for the SD3 model if needed:

    from huggingface_hub import snapshot_download
    
    # Download the miniFLUX model
    model_path = './pyramid-flow-miniflux'  # Directory to save the downloaded checkpoint
    snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
    

    Step 3: Run the Inference Demo

    Now that the environment is set up and the model is downloaded, run the Gradio app:

    # Now, run the app
    !python app.py
    

    Step 4: Access the Gradio Interface

    Once the app is running, you will see a URL generated in the output, which typically looks like this:

    Running on local URL:  http://127.0.0.1:7860/
    

    To access the Gradio interface, you will need to click the link and open it in a new tab, or directly visit it in your browser:

    1. Paste the URL into a new browser tab (if Colab does not directly provide a clickable link).
    2. From there, you can enter prompts for video generation or upload images for the image-to-video feature.

    Step 5: Additional Options

    If you want to perform inference programmatically instead of using the Gradio demo, you can use the provided code snippets to generate videos based on text prompts or images using the loaded model. Here's an example for using text prompts:

    import torch
    from PIL import Image
    from pyramid_dit import PyramidDiTForVideoGeneration
    from diffusers.utils import load_image, export_to_video
    
    torch.cuda.set_device(0)
    model_dtype, torch_dtype = 'bf16', torch.bfloat16 
    
    # Load the model
    model = PyramidDiTForVideoGeneration(
        model_path,
        model_name="pyramid_flux",
        model_dtype=model_dtype,
        model_variant='diffusion_transformer_768p',
    )
    
    # Example prompt for video generation
    prompt = "A serene landscape during autumn, vibrant colors."
    width = 1280  # For 768p model
    height = 768
    
    with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
        frames = model.generate(
            prompt=prompt,
            num_inference_steps=[20, 20, 20],
            video_num_inference_steps=[10, 10, 10],
            height=height,
            width=width,
            temp=16,
            guidance_scale=7.0,
            video_guidance_scale=5.0,
            output_type="pil",
            save_memory=True,
        )
    
    # Save the generated video
    export_to_video(frames, "./generated_video.mp4", fps=24)
    

    Make sure to customize the prompt and settings (like resolution, inference steps, etc.) according to your needs.

    Note

    Due to Colab's GPU resource limits, generating longer videos might take a considerable amount of time, so plan accordingly. Enjoy experimenting with Pyramid Flow on Google Colab!

  5. User

    You set add_prefix_space. The tokenizer needs to be converted from the slow tokenizers Loading checkpoint shards: 50% 1/2 [00:27<00:27, 27.25s/it]

  6. Assistant

    The warning message you are encountering indicates a couple of potential issues while loading the model, specifically regarding the tokenizer. Here's what you can do to address this problem:

    1. Update Your Installations

    First, ensure that your environment is set up correctly with the latest versions of the required libraries. You can add the following commands to your Colab notebook to upgrade the libraries before running your main code:

    # Upgrade transformers and diffusers to the latest version
    !pip install --upgrade transformers diffusers
    

    2. Set Up the Tokenizer Properly

    Depending on the specific tokenizer setup, you might need to ensure that you are initializing the tokenizer correctly. Here is an example snippet to load the tokenizer which you should include in your code:

    from transformers import AutoTokenizer
    
    # Load the tokenizer relevant to your model
    tokenizer = AutoTokenizer.from_pretrained(
        "rain1011/pyramid-flow-miniflux",
        use_fast=True  # Ensure you're using the fast tokenizer if available
    )
    

    3. Handling the Checkpoint Loading

    Make sure you're correctly handling the checkpoint shards. The message indicates that the checkpoint is being loaded; ensure that it is properly set up. If you're working with the PyramidDiTForVideoGeneration model, make sure it corresponds to the tokenizer:

    from pyramid_dit import PyramidDiTForVideoGeneration
    
    # Load the model
    model = PyramidDiTForVideoGeneration(
        model_path,
        model_name="pyramid_flux",
        model_dtype='bf16',
        model_variant='diffusion_transformer_768p',
    )
    

    4. Verifying Model Directory Structure

    Ensure that the model directory you downloaded contains all necessary files for loading correctly. Check if it contains files like:

    • config.json
    • pytorch_model.bin
    • tokenizer.json or tokenizer_config.json

    If any essential files are missing, consider re-downloading the model.

    5. Possible Issues with Fast Tokenizer

    The warning about add_prefix_space likely suggests that the tokenizer isn't being properly configured to manage certain types of token inputs. Ensure that your input handling is following the expected format for the chosen tokenizer.

    Full Colab Code Snippet Example

    Here’s a complete example that encompasses the upgrades and proper loading of the model and tokenizer:

    # Install necessary libraries
    !pip install --upgrade transformers diffusers gradio huggingface-hub
    
    # Clone the repository and navigate to it
    !git clone https://github.com/jy0205/Pyramid-Flow
    %cd Pyramid-Flow
    
    # Install other requirements if necessary
    !pip install -r requirements.txt
    
    from huggingface_hub import snapshot_download
    
    # Download the model
    model_path = './pyramid-flow-miniflux'
    snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
    
    # Import necessary libraries
    import torch
    from pyramid_dit import PyramidDiTForVideoGeneration
    from transformers import AutoTokenizer
    
    # Load the tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
    
    # Load the model
    model = PyramidDiTForVideoGeneration(
        model_path,
        model_name="pyramid_flux",
        model_dtype='bf16',
        model_variant='diffusion_transformer_768p',
    )
    
    # Now perform inference or any operations you require
    # Example prompt
    prompt = "A serene landscape during autumn, vibrant colors."
    
    # Proceed with model generation
    

    Conclusion

    By ensuring that your environment is updated and that the tokenizer is set up properly, you should be able to avoid those warnings and load the model smoothly. If issues persist, verify the specific version compatibilities for the libraries used in your code.