what can i install it in colab? <div align="center"> LTX-Video This is the official…
- User
what can i install it in colab?
LTX-Video
This is the official repository for LTX-Video.
Website | Model | Demo | Paper
Table of Contents
- Introduction
- Quick Start Guide
- Model User Guide
- Community Contribution
- Training
- Join Us!
- Acknowledgement
Introduction
LTX-Video is the first DiT-based video generation model that can generate high-quality videos in real-time. It can generate 24 FPS videos at 768x512 resolution, faster than it takes to watch them. The model is trained on a large-scale dataset of diverse videos and can generate high-resolution videos with realistic and diverse content.
[image: example1] A woman with long brown hair and light skin smiles at another woman...A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage. [image: example2] A woman walks away from a white Jeep parked on a city street at night...A woman walks away from a white Jeep parked on a city street at night, then ascends a staircase and knocks on a door. The woman, wearing a dark jacket and jeans, walks away from the Jeep parked on the left side of the street, her back to the camera; she walks at a steady pace, her arms swinging slightly by her sides; the street is dimly lit, with streetlights casting pools of light on the wet pavement; a man in a dark jacket and jeans walks past the Jeep in the opposite direction; the camera follows the woman from behind as she walks up a set of stairs towards a building with a green door; she reaches the top of the stairs and turns left, continuing to walk towards the building; she reaches the door and knocks on it with her right hand; the camera remains stationary, focused on the doorway; the scene is captured in real-life footage. [image: example3] A woman with blonde hair styled up, wearing a black dress...A woman with blonde hair styled up, wearing a black dress with sequins and pearl earrings, looks down with a sad expression on her face. The camera remains stationary, focused on the woman's face. The lighting is dim, casting soft shadows on her face. The scene appears to be from a movie or TV show. [image: example4] The camera pans over a snow-covered mountain range...The camera pans over a snow-covered mountain range, revealing a vast expanse of snow-capped peaks and valleys.The mountains are covered in a thick layer of snow, with some areas appearing almost white while others have a slightly darker, almost grayish hue. The peaks are jagged and irregular, with some rising sharply into the sky while others are more rounded. The valleys are deep and narrow, with steep slopes that are also covered in snow. The trees in the foreground are mostly bare, with only a few leaves remaining on their branches. The sky is overcast, with thick clouds obscuring the sun. The overall impression is one of peace and tranquility, with the snow-covered mountains standing as a testament to the power and beauty of nature. [image: example5] A woman with light skin, wearing a blue jacket and a black hat...A woman with light skin, wearing a blue jacket and a black hat with a veil, looks down and to her right, then back up as she speaks; she has brown hair styled in an updo, light brown eyebrows, and is wearing a white collared shirt under her jacket; the camera remains stationary on her face as she speaks; the background is out of focus, but shows trees and people in period clothing; the scene is captured in real-life footage. [image: example6] A man in a dimly lit room talks on a vintage telephone...A man in a dimly lit room talks on a vintage telephone, hangs up, and looks down with a sad expression. He holds the black rotary phone to his right ear with his right hand, his left hand holding a rocks glass with amber liquid. He wears a brown suit jacket over a white shirt, and a gold ring on his left ring finger. His short hair is neatly combed, and he has light skin with visible wrinkles around his eyes. The camera remains stationary, focused on his face and upper body. The room is dark, lit only by a warm light source off-screen to the left, casting shadows on the wall behind him. The scene appears to be from a movie. [image: example7] A prison guard unlocks and opens a cell door...A prison guard unlocks and opens a cell door to reveal a young man sitting at a table with a woman. The guard, wearing a dark blue uniform with a badge on his left chest, unlocks the cell door with a key held in his right hand and pulls it open; he has short brown hair, light skin, and a neutral expression. The young man, wearing a black and white striped shirt, sits at a table covered with a white tablecloth, facing the woman; he has short brown hair, light skin, and a neutral expression. The woman, wearing a dark blue shirt, sits opposite the young man, her face turned towards him; she has short blonde hair and light skin. The camera remains stationary, capturing the scene from a medium distance, positioned slightly to the right of the guard. The room is dimly lit, with a single light fixture illuminating the table and the two figures. The walls are made of large, grey concrete blocks, and a metal door is visible in the background. The scene is captured in real-life footage. [image: example8] A woman with blood on her face and a white tank top...A woman with blood on her face and a white tank top looks down and to her right, then back up as she speaks. She has dark hair pulled back, light skin, and her face and chest are covered in blood. The camera angle is a close-up, focused on the woman's face and upper torso. The lighting is dim and blue-toned, creating a somber and intense atmosphere. The scene appears to be from a movie or TV show. [image: example9] A man with graying hair, a beard, and a gray shirt...A man with graying hair, a beard, and a gray shirt looks down and to his right, then turns his head to the left. The camera angle is a close-up, focused on the man's face. The lighting is dim, with a greenish tint. The scene appears to be real-life footage. Step [image: example10] A clear, turquoise river flows through a rocky canyon...A clear, turquoise river flows through a rocky canyon, cascading over a small waterfall and forming a pool of water at the bottom.The river is the main focus of the scene, with its clear water reflecting the surrounding trees and rocks. The canyon walls are steep and rocky, with some vegetation growing on them. The trees are mostly pine trees, with their green needles contrasting with the brown and gray rocks. The overall tone of the scene is one of peace and tranquility. [image: example11] A man in a suit enters a room and speaks to two women...A man in a suit enters a room and speaks to two women sitting on a couch. The man, wearing a dark suit with a gold tie, enters the room from the left and walks towards the center of the frame. He has short gray hair, light skin, and a serious expression. He places his right hand on the back of a chair as he approaches the couch. Two women are seated on a light-colored couch in the background. The woman on the left wears a light blue sweater and has short blonde hair. The woman on the right wears a white sweater and has short blonde hair. The camera remains stationary, focusing on the man as he enters the room. The room is brightly lit, with warm tones reflecting off the walls and furniture. The scene appears to be from a film or television show. [image: example12] The waves crash against the jagged rocks of the shoreline...The waves crash against the jagged rocks of the shoreline, sending spray high into the air.The rocks are a dark gray color, with sharp edges and deep crevices. The water is a clear blue-green, with white foam where the waves break against the rocks. The sky is a light gray, with a few white clouds dotting the horizon. [image: example13] The camera pans across a cityscape of tall buildings...The camera pans across a cityscape of tall buildings with a circular building in the center. The camera moves from left to right, showing the tops of the buildings and the circular building in the center. The buildings are various shades of gray and white, and the circular building has a green roof. The camera angle is high, looking down at the city. The lighting is bright, with the sun shining from the upper left, casting shadows from the buildings. The scene is computer-generated imagery. [image: example14] A man walks towards a window, looks out, and then turns around...A man walks towards a window, looks out, and then turns around. He has short, dark hair, dark skin, and is wearing a brown coat over a red and gray scarf. He walks from left to right towards a window, his gaze fixed on something outside. The camera follows him from behind at a medium distance. The room is brightly lit, with white walls and a large window covered by a white curtain. As he approaches the window, he turns his head slightly to the left, then back to the right. He then turns his entire body to the right, facing the window. The camera remains stationary as he stands in front of the window. The scene is captured in real-life footage. [image: example15] Two police officers in dark blue uniforms and matching hats...Two police officers in dark blue uniforms and matching hats enter a dimly lit room through a doorway on the left side of the frame. The first officer, with short brown hair and a mustache, steps inside first, followed by his partner, who has a shaved head and a goatee. Both officers have serious expressions and maintain a steady pace as they move deeper into the room. The camera remains stationary, capturing them from a slightly low angle as they enter. The room has exposed brick walls and a corrugated metal ceiling, with a barred window visible in the background. The lighting is low-key, casting shadows on the officers' faces and emphasizing the grim atmosphere. The scene appears to be from a film or television show. [image: example16] A woman with short brown hair, wearing a maroon sleeveless top...A woman with short brown hair, wearing a maroon sleeveless top and a silver necklace, walks through a room while talking, then a woman with pink hair and a white shirt appears in the doorway and yells. The first woman walks from left to right, her expression serious; she has light skin and her eyebrows are slightly furrowed. The second woman stands in the doorway, her mouth open in a yell; she has light skin and her eyes are wide. The room is dimly lit, with a bookshelf visible in the background. The camera follows the first woman as she walks, then cuts to a close-up of the second woman's face. The scene is captured in real-life footage. Quick Start Guide
Online demo
The model is accessible right away via following links:
Run locally
Installation
The codebase was tested with Python 3.10.5, CUDA version 12.2, and supports PyTorch >= 2.1.2.
git clone https://github.com/Lightricks/LTX-Video.git cd LTX-Video # create env python -m venv env source env/bin/activate python -m pip install -e .\[inference-script\]Then, download the model from Hugging Face
from huggingface_hub import hf_hub_download model_path = 'PATH' # The local directory to save downloaded checkpoint hf_hub_download(repo_id="Lightricks/LTX-Video", filename="ltx-video-2b-v0.9.safetensors", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')Inference
To use our model, please follow the inference code in inference.py:
For text-to-video generation:
python inference.py --ckpt_path 'PATH' --prompt "PROMPT" --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEEDFor image-to-video generation:
python inference.py --ckpt_path 'PATH' --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEEDComfyUI Integration
To use our model with ComfyUI, please follow the instructions at https://github.com/Lightricks/ComfyUI-LTXVideo/.
Diffusers Integration
To use our model with the Diffusers Python library, check out the official documentation.
Model User Guide
📝 Prompt Engineering
When writing prompts, focus on detailed, chronological descriptions of actions and scenes. Include specific movements, appearances, camera angles, and environmental details - all in a single flowing paragraph. Start directly with the action, and keep descriptions literal and precise. Think like a cinematographer describing a shot list. Keep within 200 words. For best results, build your prompts using this structure:
- Start with main action in a single sentence
- Add specific details about movements and gestures
- Describe character/object appearances precisely
- Include background and environment details
- Specify camera angles and movements
- Describe lighting and colors
- Note any changes or sudden events
- See examples for more inspiration.
🎮 Parameter Guide
- Resolution Preset: Higher resolutions for detailed scenes, lower for faster generation and simpler scenes. The model works on resolutions that are divisible by 32 and number of frames that are divisible by 8 + 1 (e.g. 257). In case the resolution or number of frames are not divisible by 32 or 8 + 1, the input will be padded with -1 and then cropped to the desired resolution and number of frames. The model works best on resolutions under 720 x 1280 and number of frames below 257
- Seed: Save seed values to recreate specific styles or compositions you like
- Guidance Scale: 3-3.5 are the recommended values
- Inference Steps: More steps (40+) for quality, fewer steps (20-30) for speed
Community Contribution
ComfyUI-LTXTricks 🛠️
A community project providing additional nodes for enhanced control over the LTX Video model. It includes implementations of advanced techniques like RF-Inversion, RF-Edit, FlowEdit, and more. These nodes enable workflows such as Image and Video to Video (I+V2V), enhanced sampling via Spatiotemporal Skip Guidance (STG), and interpolation with precise frame settings.
- Repository: ComfyUI-LTXTricks
- Features:
- 🔄 RF-Inversion: Implements RF-Inversion with an example workflow here.
- ✂️ RF-Edit: Implements RF-Solver-Edit with an example workflow here.
- 🌊 FlowEdit: Implements FlowEdit with an example workflow here.
- 🎥 I+V2V: Enables Video to Video with a reference image. Example workflow.
- ✨ Enhance: Partial implementation of STGuidance. Example workflow.
- 🖼️ Interpolation and Frame Setting: Nodes for precise control of latents per frame. Example workflow.
LTX-VideoQ8 🎱
LTX-VideoQ8 is an 8-bit optimized version of LTX-Video, designed for faster performance on NVIDIA ADA GPUs.
- Repository: LTX-VideoQ8
- Features:
- 🚀 Up to 3X speed-up with no accuracy loss
- 🎥 Generate 720x480x121 videos in under a minute on RTX 4060 (8GB VRAM)
- 🛠️ Fine-tune 2B transformer models with precalculated latents
- Community Discussion: Reddit Thread
Your Contribution
...is welcome! If you have a project or tool that integrates with LTX-Video, please let us know by opening an issue or pull request.
Training
Diffusers
Diffusers implemented LoRA support, with a training script for fine-tuning. More information and training script in finetrainers.
Diffusion-Pipe
An experimental training framework with pipeline parallelism, enabling fine-tuning of large models like LTX-Video across multiple GPUs.
- Repository: Diffusion-Pipe
- Features:
- 🛠️ Full fine-tune support for LTX-Video using LoRA
- 📊 Useful metrics logged to Tensorboard
- 🔄 Training state checkpointing and resumption
- ⚡ Efficient pre-caching of latents and text embeddings for multi-GPU setups
Join Us 🚀
Want to work on cutting-edge AI research and make a real impact on millions of users worldwide?
At Lightricks, an AI-first company, we’re revolutionizing how visual content is created.
If you are passionate about AI, computer vision, and video generation, we would love to hear from you!
Please visit our careers page for more information.
Acknowledgement
We are grateful for the following awesome projects when implementing LTX-Video:
- DiT and PixArt-alpha: vision transformers for image generation.
Citation
📄 Our tech report is out! If you find our work helpful, please ⭐️ star the repository and cite our paper.
@article{HaCohen2024LTXVideo, title={LTX-Video: Realtime Video Latent Diffusion}, author={HaCohen, Yoav and Chiprut, Nisan and Brazowski, Benny and Shalem, Daniel and Moshe, Dudu and Richardson, Eitan and Levin, Eran and Shiran, Guy and Zabari, Nir and Gordon, Ori and Panet, Poriya and Weissbuch, Sapir and Kulikov, Victor and Bitterman, Yaki and Melumian, Zeev and Bibi, Ofir}, journal={arXiv preprint arXiv:2501.00103}, year={2024} } - Assistant
To use the LTX-Video model in Google Colab, you'll need to follow certain steps to set up the environment and run the necessary code for video generation. Below are the generic instructions on how to do this, adapted specifically for Google Colab.
Setting Up LTX-Video in Google Colab
-
Open a New Colab Notebook: Go to Google Colab and create a new notebook.
-
Clone the Repository: You'll need to clone the LTX-Video repository and install its dependencies.
!git clone https://github.com/Lightricks/LTX-Video.git %cd LTX-Video -
Install Dependencies: The repository requires a specific environment and dependencies. Use the following code to install them:
!pip install -e .\[inference-script\] -
Download the Model from Hugging Face: You will need to download the model weights. Substitute
PATH_TO_SAVE_MODELwith the directory where you want to save the model.from huggingface_hub import hf_hub_download model_path = 'PATH_TO_SAVE_MODEL' # Set to a suitable path in Colab hf_hub_download( repo_id="Lightricks/LTX-Video", filename="ltx-video-2b-v0.9.safetensors", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model' ) -
Run Inference: You can now run the inference code. Replace the placeholder values with your desired inputs.
For text-to-video generation:
!python inference.py --ckpt_path 'PATH_TO_MODEL/ltx-video-2b-v0.9.safetensors' --prompt "Your prompt here" --height 512 --width 768 --num_frames 24 --seed 42For image-to-video generation:
!python inference.py --ckpt_path 'PATH_TO_MODEL/ltx-video-2b-v0.9.safetensors' --prompt "Your prompt here" --input_image_path "path/to/your/input/image.jpg" --height 512 --width 768 --num_frames 24 --seed 42 -
Display Results: The generated video might be saved to a directory. You can display or download the video using the following examples depending on how the output is handled in the repository.
Note
- Make sure to ensure that your Colab instance has the necessary resources; if you're working with large models or generating long videos, a GPU instance is recommended. You can select a GPU runtime by going to "Runtime" > "Change runtime type" and choosing "GPU".
- Be aware of Colab's usage limits, and make sure not to exceed the computation limits with extensive or frequent processing.
Additional Considerations
- You might need to adapt paths and specific commands according to the structure of the LTX-Video repository and any updates to it.
- If you face any issues, consult the repository's documentation, especially the sections on installation and inference.
By following these steps, you will be able to install and use LTX-Video in Google Colab successfully.
-
- User
Traceback (most recent call last): File "/content/LTX-Video/inference.py", line 488, in main() File "/content/LTX-Video/inference.py", line 327, in main vae = CausalVideoAutoencoder.from_pretrained(ckpt_path) File "/content/LTX-Video/ltx_video/models/autoencoders/causal_video_autoencoder.py", line 115, in from_pretrained video_vae = cls.from_config(config) UnboundLocalError: local variable 'config' referenced before assignment
- Assistant
The error you're encountering indicates that there's an issue with the
from_pretrainedmethod in theCausalVideoAutoencoderclass. Specifically, it seems like theconfigvariable is not being defined properly before it is being used.Possible Solutions
1. Check the Model Files
Ensure that you have downloaded all necessary model files correctly, especially the configuration files that are typically needed for model loading. The error might be due to a missing or malformed configuration file.
If you've cloned the LTX-Video repository and downloaded the weights, ensure that the following files are present in the model directory:
- The weights file (e.g.,
ltx-video-2b-v0.9.safetensors) - Configuration files (e.g.,
.jsonor similar for the model architecture)
2. Modify the Inference Code
If you have access to the
inference.pyfile, you might want to add error handling or print statements to understand where the issue arises. You might find this at the beginning of thefrom_pretrainedmethod:def from_pretrained(cls, ckpt_path): config = some_function_to_load_config(ckpt_path) # Make sure there is proper loading logic here. # rest of the code...Make sure that
some_function_to_load_config(ckpt_path)is indeed called and that it's properly reporting the loading or failing with a clear message.3. Check for Dependencies
Make sure all dependencies required for running LTX-Video are installed. Sometimes missing libraries might lead to unexpected errors.
If you haven’t done so, use the following to install any missing packages:
!pip install torch torchvision torchaudio !pip install -U huggingface_hub # Include any other dependencies specified in LTX-Video requirements4. Verify the Colab Environment
If you're running in Google Colab, ensure that your runtime is set to GPU and you're correctly initializing any GPU-related settings. This can influence the behavior of how models are loaded.
5. Use Debugging Prints
Add debugging prints in the
from_pretrainedmethod to track the flow. For example, you could print theckpt_path, whether the config is loading successfully, etc.Example Modifications
Here’s an example of how you might modify the code:
def from_pretrained(cls, ckpt_path): try: config = some_function_to_load_config(ckpt_path) # Ensure this function is available and working. if config is None: raise ValueError("Could not load configuration.") # proceed with model creation except Exception as e: print(f"Error loading model from {ckpt_path}: {e}") raiseConclusion
If the above suggestions do not solve your problem, consider checking the project's GitHub issues page for similar problems or reaching out in their community discussions for specific guidance related to the repository.
- The weights file (e.g.,
- User
import json import os from functools import partial from types import SimpleNamespace from typing import Any, Mapping, Optional, Tuple, Union, List from pathlib import Path
import torch import numpy as np from einops import rearrange from torch import nn from diffusers.utils import logging import torch.nn.functional as F from diffusers.models.embeddings import PixArtAlphaCombinedTimestepSizeEmbeddings from safetensors import safe_open
from ltx_video.models.autoencoders.conv_nd_factory import make_conv_nd, make_linear_nd from ltx_video.models.autoencoders.pixel_norm import PixelNorm from ltx_video.models.autoencoders.vae import AutoencoderKLWrapper from ltx_video.models.transformers.attention import Attention from ltx_video.utils.diffusers_config_mapping import ( diffusers_and_ours_config_mapping, make_hashable_key, VAE_KEYS_RENAME_DICT, )
PER_CHANNEL_STATISTICS_PREFIX = "per_channel_statistics." logger = logging.get_logger(name) # pylint: disable=invalid-name
class CausalVideoAutoencoder(AutoencoderKLWrapper): @classmethod def from_pretrained( cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], *args, **kwargs, ): pretrained_model_name_or_path = Path(pretrained_model_name_or_path) if ( pretrained_model_name_or_path.is_dir() and (pretrained_model_name_or_path / "autoencoder.pth").exists() ): config_local_path = pretrained_model_name_or_path / "config.json" config = cls.load_config(config_local_path, **kwargs)
model_local_path = pretrained_model_name_or_path / "autoencoder.pth" state_dict = torch.load(model_local_path, map_location=torch.device("cpu")) statistics_local_path = ( pretrained_model_name_or_path / "per_channel_statistics.json" ) if statistics_local_path.exists(): with open(statistics_local_path, "r") as file: data = json.load(file) transposed_data = list(zip(*data["data"])) data_dict = { col: torch.tensor(vals) for col, vals in zip(data["columns"], transposed_data) } std_of_means = data_dict["std-of-means"] mean_of_means = data_dict.get( "mean-of-means", torch.zeros_like(data_dict["std-of-means"]) ) state_dict[f"{PER_CHANNEL_STATISTICS_PREFIX}std-of-means"] = ( std_of_means ) state_dict[f"{PER_CHANNEL_STATISTICS_PREFIX}mean-of-means"] = ( mean_of_means ) elif pretrained_model_name_or_path.is_dir(): config_path = pretrained_model_name_or_path / "vae" / "config.json" with open(config_path, "r") as f: config = make_hashable_key(json.load(f)) assert config in diffusers_and_ours_config_mapping, ( "Provided diffusers checkpoint config for VAE is not suppported. " "We only support diffusers configs found in Lightricks/LTX-Video." ) config = diffusers_and_ours_config_mapping[config] state_dict_path = ( pretrained_model_name_or_path / "vae" / "diffusion_pytorch_model.safetensors" ) state_dict = {} with safe_open(state_dict_path, framework="pt", device="cpu") as f: for k in f.keys(): state_dict[k] = f.get_tensor(k) for key in list(state_dict.keys()): new_key = key for replace_key, rename_key in VAE_KEYS_RENAME_DICT.items(): new_key = new_key.replace(replace_key, rename_key) state_dict[new_key] = state_dict.pop(key) elif pretrained_model_name_or_path.is_file() and str( pretrained_model_name_or_path ).endswith(".safetensors"): state_dict = {} with safe_open( pretrained_model_name_or_path, framework="pt", device="cpu" ) as f: metadata = f.metadata() for k in f.keys(): state_dict[k] = f.get_tensor(k) configs = json.loads(metadata["config"]) config = configs["vae"] video_vae = cls.from_config(config) if "torch_dtype" in kwargs: video_vae.to(kwargs["torch_dtype"]) video_vae.load_state_dict(state_dict) return video_vae @staticmethod def from_config(config): assert ( config["_class_name"] == "CausalVideoAutoencoder" ), "config must have _class_name=CausalVideoAutoencoder" if isinstance(config["dims"], list): config["dims"] = tuple(config["dims"]) assert config["dims"] in [2, 3, (2, 1)], "dims must be 2, 3 or (2, 1)" double_z = config.get("double_z", True) latent_log_var = config.get( "latent_log_var", "per_channel" if double_z else "none" ) use_quant_conv = config.get("use_quant_conv", True) if use_quant_conv and latent_log_var == "uniform": raise ValueError("uniform latent_log_var requires use_quant_conv=False") encoder = Encoder( dims=config["dims"], in_channels=config.get("in_channels", 3), out_channels=config["latent_channels"], blocks=config.get("encoder_blocks", config.get("blocks")), patch_size=config.get("patch_size", 1), latent_log_var=latent_log_var, norm_layer=config.get("norm_layer", "group_norm"), ) decoder = Decoder( dims=config["dims"], in_channels=config["latent_channels"], out_channels=config.get("out_channels", 3), blocks=config.get("decoder_blocks", config.get("blocks")), patch_size=config.get("patch_size", 1), norm_layer=config.get("norm_layer", "group_norm"), causal=config.get("causal_decoder", False), timestep_conditioning=config.get("timestep_conditioning", False), ) dims = config["dims"] return CausalVideoAutoencoder( encoder=encoder, decoder=decoder, latent_channels=config["latent_channels"], dims=dims, use_quant_conv=use_quant_conv, ) @property def config(self): return SimpleNamespace( _class_name="CausalVideoAutoencoder", dims=self.dims, in_channels=self.encoder.conv_in.in_channels // self.encoder.patch_size**2, out_channels=self.decoder.conv_out.out_channels // self.decoder.patch_size**2, latent_channels=self.decoder.conv_in.in_channels, encoder_blocks=self.encoder.blocks_desc, decoder_blocks=self.decoder.blocks_desc, scaling_factor=1.0, norm_layer=self.encoder.norm_layer, patch_size=self.encoder.patch_size, latent_log_var=self.encoder.latent_log_var, use_quant_conv=self.use_quant_conv, causal_decoder=self.decoder.causal, timestep_conditioning=self.decoder.timestep_conditioning, ) @property def is_video_supported(self): """ Check if the model supports video inputs of shape (B, C, F, H, W). Otherwise, the model only supports 2D images. """ return self.dims != 2 @property def spatial_downscale_factor(self): return ( 2 ** len( [ block for block in self.encoder.blocks_desc if block[0] in ["compress_space", "compress_all"] ] ) * self.encoder.patch_size ) @property def temporal_downscale_factor(self): return 2 ** len( [ block for block in self.encoder.blocks_desc if block[0] in ["compress_time", "compress_all"] ] ) def to_json_string(self) -> str: import json return json.dumps(self.config.__dict__) def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool = True): if any([key.startswith("vae.") for key in state_dict.keys()]): state_dict = { key.replace("vae.", ""): value for key, value in state_dict.items() if key.startswith("vae.") } ckpt_state_dict = { key: value for key, value in state_dict.items() if not key.startswith(PER_CHANNEL_STATISTICS_PREFIX) } model_keys = set(name for name, _ in self.named_parameters()) key_mapping = { ".resnets.": ".res_blocks.", "downsamplers.0": "downsample", "upsamplers.0": "upsample", } converted_state_dict = {} for key, value in ckpt_state_dict.items(): for k, v in key_mapping.items(): key = key.replace(k, v) if "norm" in key and key not in model_keys: logger.info( f"Removing key {key} from state_dict as it is not present in the model" ) continue converted_state_dict[key] = value super().load_state_dict(converted_state_dict, strict=strict) data_dict = { key.removeprefix(PER_CHANNEL_STATISTICS_PREFIX): value for key, value in state_dict.items() if key.startswith(PER_CHANNEL_STATISTICS_PREFIX) } if len(data_dict) > 0: self.register_buffer("std_of_means", data_dict["std-of-means"]) self.register_buffer( "mean_of_means", data_dict.get( "mean-of-means", torch.zeros_like(data_dict["std-of-means"]) ), ) def last_layer(self): if hasattr(self.decoder, "conv_out"): if isinstance(self.decoder.conv_out, nn.Sequential): last_layer = self.decoder.conv_out[-1] else: last_layer = self.decoder.conv_out else: last_layer = self.decoder.layers[-1] return last_layer def set_use_tpu_flash_attention(self): for block in self.decoder.up_blocks: if isinstance(block, UNetMidBlock3D) and block.attention_blocks: for attention_block in block.attention_blocks: attention_block.set_use_tpu_flash_attention()class Encoder(nn.Module): r""" The
Encoderlayer of a variational autoencoder that encodes its input into a latent representation.Args: dims (`int` or `Tuple[int, int]`, *optional*, defaults to 3): The number of dimensions to use in convolutions. in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. blocks (`List[Tuple[str, int]]`, *optional*, defaults to `[("res_x", 1)]`): The blocks to use. Each block is a tuple of the block name and the number of layers. base_channels (`int`, *optional*, defaults to 128): The number of output channels for the first convolutional layer. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. patch_size (`int`, *optional*, defaults to 1): The patch size to use. Should be a power of 2. norm_layer (`str`, *optional*, defaults to `group_norm`): The normalization layer to use. Can be either `group_norm` or `pixel_norm`. latent_log_var (`str`, *optional*, defaults to `per_channel`): The number of channels for the log variance. Can be either `per_channel`, `uniform`, or `none`. """ def __init__( self, dims: Union[int, Tuple[int, int]] = 3, in_channels: int = 3, out_channels: int = 3, blocks: List[Tuple[str, int | dict]] = [("res_x", 1)], base_channels: int = 128, norm_num_groups: int = 32, patch_size: Union[int, Tuple[int]] = 1, norm_layer: str = "group_norm", # group_norm, pixel_norm latent_log_var: str = "per_channel", ): super().__init__() self.patch_size = patch_size self.norm_layer = norm_layer self.latent_channels = out_channels self.latent_log_var = latent_log_var self.blocks_desc = blocks in_channels = in_channels * patch_size**2 output_channel = base_channels self.conv_in = make_conv_nd( dims=dims, in_channels=in_channels, out_channels=output_channel, kernel_size=3, stride=1, padding=1, causal=True, ) self.down_blocks = nn.ModuleList([]) for block_name, block_params in blocks: input_channel = output_channel if isinstance(block_params, int): block_params = {"num_layers": block_params} if block_name == "res_x": block = UNetMidBlock3D( dims=dims, in_channels=input_channel, num_layers=block_params["num_layers"], resnet_eps=1e-6, resnet_groups=norm_num_groups, norm_layer=norm_layer, ) elif block_name == "res_x_y": output_channel = block_params.get("multiplier", 2) * output_channel block = ResnetBlock3D( dims=dims, in_channels=input_channel, out_channels=output_channel, eps=1e-6, groups=norm_num_groups, norm_layer=norm_layer, ) elif block_name == "compress_time": block = make_conv_nd( dims=dims, in_channels=input_channel, out_channels=output_channel, kernel_size=3, stride=(2, 1, 1), causal=True, ) elif block_name == "compress_space": block = make_conv_nd( dims=dims, in_channels=input_channel, out_channels=output_channel, kernel_size=3, stride=(1, 2, 2), causal=True, ) elif block_name == "compress_all": block = make_conv_nd( dims=dims, in_channels=input_channel, out_channels=output_channel, kernel_size=3, stride=(2, 2, 2), causal=True, ) elif block_name == "compress_all_x_y": output_channel = block_params.get("multiplier", 2) * output_channel block = make_conv_nd( dims=dims, in_channels=input_channel, out_channels=output_channel, kernel_size=3, stride=(2, 2, 2), causal=True, ) else: raise ValueError(f"unknown block: {block_name}") self.down_blocks.append(block) # out if norm_layer == "group_norm": self.conv_norm_out = nn.GroupNorm( num_channels=output_channel, num_groups=norm_num_groups, eps=1e-6 ) elif norm_layer == "pixel_norm": self.conv_norm_out = PixelNorm() elif norm_layer == "layer_norm": self.conv_norm_out = LayerNorm(output_channel, eps=1e-6) self.conv_act = nn.SiLU() conv_out_channels = out_channels if latent_log_var == "per_channel": conv_out_channels *= 2 elif latent_log_var == "uniform": conv_out_channels += 1 elif latent_log_var != "none": raise ValueError(f"Invalid latent_log_var: {latent_log_var}") self.conv_out = make_conv_nd( dims, output_channel, conv_out_channels, 3, padding=1, causal=True ) self.gradient_checkpointing = False def forward(self, sample: torch.FloatTensor) -> torch.FloatTensor: r"""The forward method of the `Encoder` class.""" sample = patchify(sample, patch_size_hw=self.patch_size, patch_size_t=1) sample = self.conv_in(sample) checkpoint_fn = ( partial(torch.utils.checkpoint.checkpoint, use_reentrant=False) if self.gradient_checkpointing and self.training else lambda x: x ) for down_block in self.down_blocks: sample = checkpoint_fn(down_block)(sample) sample = self.conv_norm_out(sample) sample = self.conv_act(sample) sample = self.conv_out(sample) if self.latent_log_var == "uniform": last_channel = sample[:, -1:, ...] num_dims = sample.dim() if num_dims == 4: # For shape (B, C, H, W) repeated_last_channel = last_channel.repeat( 1, sample.shape[1] - 2, 1, 1 ) sample = torch.cat([sample, repeated_last_channel], dim=1) elif num_dims == 5: # For shape (B, C, F, H, W) repeated_last_channel = last_channel.repeat( 1, sample.shape[1] - 2, 1, 1, 1 ) sample = torch.cat([sample, repeated_last_channel], dim=1) else: raise ValueError(f"Invalid input shape: {sample.shape}") return sampleclass Decoder(nn.Module): r""" The
Decoderlayer of a variational autoencoder that decodes its latent representation into an output sample.Args: dims (`int` or `Tuple[int, int]`, *optional*, defaults to 3): The number of dimensions to use in convolutions. in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. blocks (`List[Tuple[str, int]]`, *optional*, defaults to `[("res_x", 1)]`): The blocks to use. Each block is a tuple of the block name and the number of layers. base_channels (`int`, *optional*, defaults to 128): The number of output channels for the first convolutional layer. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. patch_size (`int`, *optional*, defaults to 1): The patch size to use. Should be a power of 2. norm_layer (`str`, *optional*, defaults to `group_norm`): The normalization layer to use. Can be either `group_norm` or `pixel_norm`. causal (`bool`, *optional*, defaults to `True`): Whether to use causal convolutions or not. """ def __init__( self, dims, in_channels: int = 3, out_channels: int = 3, blocks: List[Tuple[str, int | dict]] = [("res_x", 1)], base_channels: int = 128, layers_per_block: int = 2, norm_num_groups: int = 32, patch_size: int = 1, norm_layer: str = "group_norm", causal: bool = True, timestep_conditioning: bool = False, ): super().__init__() self.patch_size = patch_size self.layers_per_block = layers_per_block out_channels = out_channels * patch_size**2 self.causal = causal self.blocks_desc = blocks # Compute output channel to be product of all channel-multiplier blocks output_channel = base_channels for block_name, block_params in list(reversed(blocks)): block_params = block_params if isinstance(block_params, dict) else {} if block_name == "res_x_y": output_channel = output_channel * block_params.get("multiplier", 2) if block_name == "compress_all": output_channel = output_channel * block_params.get("multiplier", 1) self.conv_in = make_conv_nd( dims, in_channels, output_channel, kernel_size=3, stride=1, padding=1, causal=True, ) self.up_blocks = nn.ModuleList([]) for block_name, block_params in list(reversed(blocks)): input_channel = output_channel if isinstance(block_params, int): block_params = {"num_layers": block_params} if block_name == "res_x": block = UNetMidBlock3D( dims=dims, in_channels=input_channel, num_layers=block_params["num_layers"], resnet_eps=1e-6, resnet_groups=norm_num_groups, norm_layer=norm_layer, inject_noise=block_params.get("inject_noise", False), timestep_conditioning=timestep_conditioning, ) elif block_name == "attn_res_x": block = UNetMidBlock3D( dims=dims, in_channels=input_channel, num_layers=block_params["num_layers"], resnet_groups=norm_num_groups, norm_layer=norm_layer, inject_noise=block_params.get("inject_noise", False), timestep_conditioning=timestep_conditioning, attention_head_dim=block_params["attention_head_dim"], ) elif block_name == "res_x_y": output_channel = output_channel // block_params.get("multiplier", 2) block = ResnetBlock3D( dims=dims, in_channels=input_channel, out_channels=output_channel, eps=1e-6, groups=norm_num_groups, norm_layer=norm_layer, inject_noise=block_params.get("inject_noise", False), timestep_conditioning=False, ) elif block_name == "compress_time": block = DepthToSpaceUpsample( dims=dims, in_channels=input_channel, stride=(2, 1, 1) ) elif block_name == "compress_space": block = DepthToSpaceUpsample( dims=dims, in_channels=input_channel, stride=(1, 2, 2) ) elif block_name == "compress_all": output_channel = output_channel // block_params.get("multiplier", 1) block = DepthToSpaceUpsample( dims=dims, in_channels=input_channel, stride=(2, 2, 2), residual=block_params.get("residual", False), out_channels_reduction_factor=block_params.get("multiplier", 1), ) else: raise ValueError(f"unknown layer: {block_name}") self.up_blocks.append(block) if norm_layer == "group_norm": self.conv_norm_out = nn.GroupNorm( num_channels=output_channel, num_groups=norm_num_groups, eps=1e-6 ) elif norm_layer == "pixel_norm": self.conv_norm_out = PixelNorm() elif norm_layer == "layer_norm": self.conv_norm_out = LayerNorm(output_channel, eps=1e-6) self.conv_act = nn.SiLU() self.conv_out = make_conv_nd( dims, output_channel, out_channels, 3, padding=1, causal=True ) self.gradient_checkpointing = False self.timestep_conditioning = timestep_conditioning if timestep_conditioning: self.timestep_scale_multiplier = nn.Parameter( torch.tensor(1000.0, dtype=torch.float32) ) self.last_time_embedder = PixArtAlphaCombinedTimestepSizeEmbeddings( output_channel * 2, 0 ) self.last_scale_shift_table = nn.Parameter( torch.randn(2, output_channel) / output_channel**0.5 ) def forward( self, sample: torch.FloatTensor, target_shape, timestep: Optional[torch.Tensor] = None, ) -> torch.FloatTensor: r"""The forward method of the `Decoder` class.""" assert target_shape is not None, "target_shape must be provided" batch_size = sample.shape[0] sample = self.conv_in(sample, causal=self.causal) upscale_dtype = next(iter(self.up_blocks.parameters())).dtype checkpoint_fn = ( partial(torch.utils.checkpoint.checkpoint, use_reentrant=False) if self.gradient_checkpointing and self.training else lambda x: x ) sample = sample.to(upscale_dtype) if self.timestep_conditioning: assert ( timestep is not None ), "should pass timestep with timestep_conditioning=True" scaled_timestep = timestep * self.timestep_scale_multiplier for up_block in self.up_blocks: if self.timestep_conditioning and isinstance(up_block, UNetMidBlock3D): sample = checkpoint_fn(up_block)( sample, causal=self.causal, timestep=scaled_timestep ) else: sample = checkpoint_fn(up_block)(sample, causal=self.causal) sample = self.conv_norm_out(sample) if self.timestep_conditioning: embedded_timestep = self.last_time_embedder( timestep=scaled_timestep.flatten(), resolution=None, aspect_ratio=None, batch_size=sample.shape[0], hidden_dtype=sample.dtype, ) embedded_timestep = embedded_timestep.view( batch_size, embedded_timestep.shape[-1], 1, 1, 1 ) ada_values = self.last_scale_shift_table[ None, ..., None, None, None ] + embedded_timestep.reshape( batch_size, 2, -1, embedded_timestep.shape[-3], embedded_timestep.shape[-2], embedded_timestep.shape[-1], ) shift, scale = ada_values.unbind(dim=1) sample = sample * (1 + scale) + shift sample = self.conv_act(sample) sample = self.conv_out(sample, causal=self.causal) sample = unpatchify(sample, patch_size_hw=self.patch_size, patch_size_t=1) return sampleclass UNetMidBlock3D(nn.Module): """ A 3D UNet mid-block [
UNetMidBlock3D] with multiple residual blocks.Args: in_channels (`int`): The number of input channels. dropout (`float`, *optional*, defaults to 0.0): The dropout rate. num_layers (`int`, *optional*, defaults to 1): The number of residual blocks. resnet_eps (`float`, *optional*, 1e-6 ): The epsilon value for the resnet blocks. resnet_groups (`int`, *optional*, defaults to 32): The number of groups to use in the group normalization layers of the resnet blocks. norm_layer (`str`, *optional*, defaults to `group_norm`): The normalization layer to use. Can be either `group_norm` or `pixel_norm`. inject_noise (`bool`, *optional*, defaults to `False`): Whether to inject noise into the hidden states. timestep_conditioning (`bool`, *optional*, defaults to `False`): Whether to condition the hidden states on the timestep. attention_head_dim (`int`, *optional*, defaults to -1): The dimension of the attention head. If -1, no attention is used. Returns: `torch.FloatTensor`: The output of the last residual block, which is a tensor of shape `(batch_size, in_channels, height, width)`. """ def __init__( self, dims: Union[int, Tuple[int, int]], in_channels: int, dropout: float = 0.0, num_layers: int = 1, resnet_eps: float = 1e-6, resnet_groups: int = 32, norm_layer: str = "group_norm", inject_noise: bool = False, timestep_conditioning: bool = False, attention_head_dim: int = -1, ): super().__init__() resnet_groups = ( resnet_groups if resnet_groups is not None else min(in_channels // 4, 32) ) self.timestep_conditioning = timestep_conditioning if timestep_conditioning: self.time_embedder = PixArtAlphaCombinedTimestepSizeEmbeddings( in_channels * 4, 0 ) self.res_blocks = nn.ModuleList( [ ResnetBlock3D( dims=dims, in_channels=in_channels, out_channels=in_channels, eps=resnet_eps, groups=resnet_groups, dropout=dropout, norm_layer=norm_layer, inject_noise=inject_noise, timestep_conditioning=timestep_conditioning, ) for _ in range(num_layers) ] ) self.attention_blocks = None if attention_head_dim > 0: if attention_head_dim > in_channels: raise ValueError( "attention_head_dim must be less than or equal to in_channels" ) self.attention_blocks = nn.ModuleList( [ Attention( query_dim=in_channels, heads=in_channels // attention_head_dim, dim_head=attention_head_dim, bias=True, out_bias=True, qk_norm="rms_norm", residual_connection=True, ) for _ in range(num_layers) ] ) def forward( self, hidden_states: torch.FloatTensor, causal: bool = True, timestep: Optional[torch.Tensor] = None, ) -> torch.FloatTensor: timestep_embed = None if self.timestep_conditioning: assert ( timestep is not None ), "should pass timestep with timestep_conditioning=True" batch_size = hidden_states.shape[0] timestep_embed = self.time_embedder( timestep=timestep.flatten(), resolution=None, aspect_ratio=None, batch_size=batch_size, hidden_dtype=hidden_states.dtype, ) timestep_embed = timestep_embed.view( batch_size, timestep_embed.shape[-1], 1, 1, 1 ) if self.attention_blocks: for resnet, attention in zip(self.res_blocks, self.attention_blocks): hidden_states = resnet( hidden_states, causal=causal, timestep=timestep_embed ) # Reshape the hidden states to be (batch_size, frames * height * width, channel) batch_size, channel, frames, height, width = hidden_states.shape hidden_states = hidden_states.view( batch_size, channel, frames * height * width ).transpose(1, 2) if attention.use_tpu_flash_attention: # Pad the second dimension to be divisible by block_k_major (block in flash attention) seq_len = hidden_states.shape[1] block_k_major = 512 pad_len = (block_k_major - seq_len % block_k_major) % block_k_major if pad_len > 0: hidden_states = F.pad( hidden_states, (0, 0, 0, pad_len), "constant", 0 ) # Create a mask with ones for the original sequence length and zeros for the padded indexes mask = torch.ones( (hidden_states.shape[0], seq_len), device=hidden_states.device, dtype=hidden_states.dtype, ) if pad_len > 0: mask = F.pad(mask, (0, pad_len), "constant", 0) hidden_states = attention( hidden_states, attention_mask=( None if not attention.use_tpu_flash_attention else mask ), ) if attention.use_tpu_flash_attention: # Remove the padding if pad_len > 0: hidden_states = hidden_states[:, :-pad_len, :] # Reshape the hidden states back to (batch_size, channel, frames, height, width, channel) hidden_states = hidden_states.transpose(-1, -2).reshape( batch_size, channel, frames, height, width ) else: for resnet in self.res_blocks: hidden_states = resnet( hidden_states, causal=causal, timestep=timestep_embed ) return hidden_statesclass DepthToSpaceUpsample(nn.Module): def init( self, dims, in_channels, stride, residual=False, out_channels_reduction_factor=1 ): super().init() self.stride = stride self.out_channels = ( np.prod(stride) * in_channels // out_channels_reduction_factor ) self.conv = make_conv_nd( dims=dims, in_channels=in_channels, out_channels=self.out_channels, kernel_size=3, stride=1, causal=True, ) self.residual = residual self.out_channels_reduction_factor = out_channels_reduction_factor
def forward(self, x, causal: bool = True): if self.residual: # Reshape and duplicate the input to match the output shape x_in = rearrange( x, "b (c p1 p2 p3) d h w -> b c (d p1) (h p2) (w p3)", p1=self.stride[0], p2=self.stride[1], p3=self.stride[2], ) num_repeat = np.prod(self.stride) // self.out_channels_reduction_factor x_in = x_in.repeat(1, num_repeat, 1, 1, 1) if self.stride[0] == 2: x_in = x_in[:, :, 1:, :, :] x = self.conv(x, causal=causal) x = rearrange( x, "b (c p1 p2 p3) d h w -> b c (d p1) (h p2) (w p3)", p1=self.stride[0], p2=self.stride[1], p3=self.stride[2], ) if self.stride[0] == 2: x = x[:, :, 1:, :, :] if self.residual: x = x + x_in return xclass LayerNorm(nn.Module): def init(self, dim, eps, elementwise_affine=True) -> None: super().init() self.norm = nn.LayerNorm(dim, eps=eps, elementwise_affine=elementwise_affine)
def forward(self, x): x = rearrange(x, "b c d h w -> b d h w c") x = self.norm(x) x = rearrange(x, "b d h w c -> b c d h w") return xclass ResnetBlock3D(nn.Module): r""" A Resnet block.
Parameters: in_channels (`int`): The number of channels in the input. out_channels (`int`, *optional*, default to be `None`): The number of output channels for the first conv layer. If None, same as `in_channels`. dropout (`float`, *optional*, defaults to `0.0`): The dropout probability to use. groups (`int`, *optional*, default to `32`): The number of groups to use for the first normalization layer. eps (`float`, *optional*, defaults to `1e-6`): The epsilon to use for the normalization. """ def __init__( self, dims: Union[int, Tuple[int, int]], in_channels: int, out_channels: Optional[int] = None, dropout: float = 0.0, groups: int = 32, eps: float = 1e-6, norm_layer: str = "group_norm", inject_noise: bool = False, timestep_conditioning: bool = False, ): super().__init__() self.in_channels = in_channels out_channels = in_channels if out_channels is None else out_channels self.out_channels = out_channels self.inject_noise = inject_noise if norm_layer == "group_norm": self.norm1 = nn.GroupNorm( num_groups=groups, num_channels=in_channels, eps=eps, affine=True ) elif norm_layer == "pixel_norm": self.norm1 = PixelNorm() elif norm_layer == "layer_norm": self.norm1 = LayerNorm(in_channels, eps=eps, elementwise_affine=True) self.non_linearity = nn.SiLU() self.conv1 = make_conv_nd( dims, in_channels, out_channels, kernel_size=3, stride=1, padding=1, causal=True, ) if inject_noise: self.per_channel_scale1 = nn.Parameter(torch.zeros((in_channels, 1, 1))) if norm_layer == "group_norm": self.norm2 = nn.GroupNorm( num_groups=groups, num_channels=out_channels, eps=eps, affine=True ) elif norm_layer == "pixel_norm": self.norm2 = PixelNorm() elif norm_layer == "layer_norm": self.norm2 = LayerNorm(out_channels, eps=eps, elementwise_affine=True) self.dropout = torch.nn.Dropout(dropout) self.conv2 = make_conv_nd( dims, out_channels, out_channels, kernel_size=3, stride=1, padding=1, causal=True, ) if inject_noise: self.per_channel_scale2 = nn.Parameter(torch.zeros((in_channels, 1, 1))) self.conv_shortcut = ( make_linear_nd( dims=dims, in_channels=in_channels, out_channels=out_channels ) if in_channels != out_channels else nn.Identity() ) self.norm3 = ( LayerNorm(in_channels, eps=eps, elementwise_affine=True) if in_channels != out_channels else nn.Identity() ) self.timestep_conditioning = timestep_conditioning if timestep_conditioning: self.scale_shift_table = nn.Parameter( torch.randn(4, in_channels) / in_channels**0.5 ) def _feed_spatial_noise( self, hidden_states: torch.FloatTensor, per_channel_scale: torch.FloatTensor ) -> torch.FloatTensor: spatial_shape = hidden_states.shape[-2:] device = hidden_states.device dtype = hidden_states.dtype # similar to the "explicit noise inputs" method in style-gan spatial_noise = torch.randn(spatial_shape, device=device, dtype=dtype)[None] scaled_noise = (spatial_noise * per_channel_scale)[None, :, None, ...] hidden_states = hidden_states + scaled_noise return hidden_states def forward( self, input_tensor: torch.FloatTensor, causal: bool = True, timestep: Optional[torch.Tensor] = None, ) -> torch.FloatTensor: hidden_states = input_tensor batch_size = hidden_states.shape[0] hidden_states = self.norm1(hidden_states) if self.timestep_conditioning: assert ( timestep is not None ), "should pass timestep with timestep_conditioning=True" ada_values = self.scale_shift_table[ None, ..., None, None, None ] + timestep.reshape( batch_size, 4, -1, timestep.shape[-3], timestep.shape[-2], timestep.shape[-1], ) shift1, scale1, shift2, scale2 = ada_values.unbind(dim=1) hidden_states = hidden_states * (1 + scale1) + shift1 hidden_states = self.non_linearity(hidden_states) hidden_states = self.conv1(hidden_states, causal=causal) if self.inject_noise: hidden_states = self._feed_spatial_noise( hidden_states, self.per_channel_scale1 ) hidden_states = self.norm2(hidden_states) if self.timestep_conditioning: hidden_states = hidden_states * (1 + scale2) + shift2 hidden_states = self.non_linearity(hidden_states) hidden_states = self.dropout(hidden_states) hidden_states = self.conv2(hidden_states, causal=causal) if self.inject_noise: hidden_states = self._feed_spatial_noise( hidden_states, self.per_channel_scale2 ) input_tensor = self.norm3(input_tensor) batch_size = input_tensor.shape[0] input_tensor = self.conv_shortcut(input_tensor) output_tensor = input_tensor + hidden_states return output_tensordef patchify(x, patch_size_hw, patch_size_t=1): if patch_size_hw == 1 and patch_size_t == 1: return x if x.dim() == 4: x = rearrange( x, "b c (h q) (w r) -> b (c r q) h w", q=patch_size_hw, r=patch_size_hw ) elif x.dim() == 5: x = rearrange( x, "b c (f p) (h q) (w r) -> b (c p r q) f h w", p=patch_size_t, q=patch_size_hw, r=patch_size_hw, ) else: raise ValueError(f"Invalid input shape: {x.shape}")
return xdef unpatchify(x, patch_size_hw, patch_size_t=1): if patch_size_hw == 1 and patch_size_t == 1: return x
if x.dim() == 4: x = rearrange( x, "b (c r q) h w -> b c (h q) (w r)", q=patch_size_hw, r=patch_size_hw ) elif x.dim() == 5: x = rearrange( x, "b (c p r q) f h w -> b c (f p) (h q) (w r)", p=patch_size_t, q=patch_size_hw, r=patch_size_hw, ) return xdef create_video_autoencoder_config( latent_channels: int = 64, ): encoder_blocks = [ ("res_x", {"num_layers": 4}), ("compress_all_x_y", {"multiplier": 3}), ("res_x", {"num_layers": 4}), ("compress_all_x_y", {"multiplier": 2}), ("res_x", {"num_layers": 4}), ("compress_all", {}), ("res_x", {"num_layers": 3}), ("res_x", {"num_layers": 4}), ] decoder_blocks = [ ("res_x", {"num_layers": 4}), ("compress_all", {"residual": True}), ("res_x_y", {"multiplier": 3}), ("res_x", {"num_layers": 3}), ("compress_all", {"residual": True}), ("res_x_y", {"multiplier": 2}), ("res_x", {"num_layers": 3}), ("compress_all", {"residual": True}), ("res_x", {"num_layers": 3}), ("res_x", {"num_layers": 4}), ] return { "_class_name": "CausalVideoAutoencoder", "dims": 3, "encoder_blocks": encoder_blocks, "decoder_blocks": decoder_blocks, "latent_channels": latent_channels, "norm_layer": "pixel_norm", "patch_size": 4, "latent_log_var": "uniform", "use_quant_conv": False, "causal_decoder": False, "timestep_conditioning": True, }
def test_vae_patchify_unpatchify(): import torch
x = torch.randn(2, 3, 8, 64, 64) x_patched = patchify(x, patch_size_hw=4, patch_size_t=4) x_unpatched = unpatchify(x_patched, patch_size_hw=4, patch_size_t=4) assert torch.allclose(x, x_unpatched)def demo_video_autoencoder_forward_backward(): # Configuration for the VideoAutoencoder config = create_video_autoencoder_config()
# Instantiate the VideoAutoencoder with the specified configuration video_autoencoder = CausalVideoAutoencoder.from_config(config) print(video_autoencoder) video_autoencoder.eval() # Print the total number of parameters in the video autoencoder total_params = sum(p.numel() for p in video_autoencoder.parameters()) print(f"Total number of parameters in VideoAutoencoder: {total_params:,}") # Create a mock input tensor simulating a batch of videos # Shape: (batch_size, channels, depth, height, width) # E.g., 4 videos, each with 3 color channels, 16 frames, and 64x64 pixels per frame input_videos = torch.randn(2, 3, 17, 64, 64) # Forward pass: encode and decode the input videos latent = video_autoencoder.encode(input_videos).latent_dist.mode() print(f"input shape={input_videos.shape}") print(f"latent shape={latent.shape}") timestep = torch.ones(input_videos.shape[0]) * 0.1 reconstructed_videos = video_autoencoder.decode( latent, target_shape=input_videos.shape, timestep=timestep ).sample print(f"reconstructed shape={reconstructed_videos.shape}") # Validate that single image gets treated the same way as first frame input_image = input_videos[:, :, :1, :, :] image_latent = video_autoencoder.encode(input_image).latent_dist.mode() _ = video_autoencoder.decode( image_latent, target_shape=image_latent.shape, timestep=timestep ).sample # first_frame_latent = latent[:, :, :1, :, :] # assert torch.allclose(image_latent, first_frame_latent, atol=1e-6) # assert torch.allclose(reconstructed_image, reconstructed_videos[:, :, :1, :, :], atol=1e-6) # assert (image_latent == first_frame_latent).all() # assert (reconstructed_image == reconstructed_videos[:, :, :1, :, :]).all() # Calculate the loss (e.g., mean squared error) loss = torch.nn.functional.mse_loss(input_videos, reconstructed_videos) # Perform backward pass loss.backward() print(f"Demo completed with loss: {loss.item()}")Ensure to call the demo function to execute the forward and backward pass
if name == "main": demo_video_autoencoder_forward_backward()
- Assistant
The code snippet you provided is part of the implementation of the CausalVideoAutoencoder used for video processing, more specifically for tasks like encoding and decoding videos in a latent space. This model is part of the LTX-Video framework, which has specific requirements for loading and using models.
Let's summarize how you might address the previous
UnboundLocalErrorand provide guidance on executing this code:Overview of the Cause of the Error
The
UnboundLocalErroryou encountered initially infrom_pretrainedsuggests that theconfigvariable was referenced before being assigned.In your
from_pretrainedmethod, configuration loading occurs only under certain conditions (e.g., whether the path is a directory containing specific files). If none of these conditions are met,configmay remain uninitialized by the time you're trying to use it.Steps to Resolve the Error and Use the Code
- Check Configuration Loading:
Ensure that your model directory structure aligns with what's required in the
from_pretrainedmethod. It should contain either:autoencoder.pthandconfig.jsonorvae/diffusion_pytorch_model.safetensorsand its correspondingconfig.json.
Verify the existence of these files based on how you try to load your model.
- Modify the
from_pretrainedMethod: If you want to improve the resilience to not finding a config, you might add error handling or default configurations. Here’s an adjusted portion of the method:
if pretrained_model_name_or_path.is_dir() and (pretrained_model_name_or_path / "autoencoder.pth").exists(): config_local_path = pretrained_model_name_or_path / "config.json" if config_local_path.exists(): config = cls.load_config(config_local_path, **kwargs) else: raise FileNotFoundError("Configuration file not found.") # Proceed with loading the state_dict, etc.- Run the Code:
Ensure that you are executing the
demo_video_autoencoder_forward_backwardfunction as the main entry point of the script when running it. You can copy this part into a Python environment after your adjustments are made:
if __name__ == "__main__": demo_video_autoencoder_forward_backward()- Handle Missing Dependencies:
If any additional dependencies are required (like
diffusers), make sure to install them via pip:
pip install torch torchvision torchaudio diffusers einops safetensors-
Run in the Correct Environment: If you're working within Google Colab, make sure to select a GPU runtime for better performance during video processing. You can choose this in the menu:
Runtime > Change runtime type > Hardware accelerator > GPU. -
Test Your Setup: Once all is set up, you can test your model by running the complete script with a validation tensor to ensure that both encoding and decoding work as expected.
Conclusion
By modifying your configurations and ensuring the paths for model loading are correct, along with verifying environment settings, you should be able to resolve your errors and run the video autoencoder successfully. If further issues arise during execution, consider enabling more detailed logging or printing variable states for debugging your input and model.
- Check Configuration Loading:
Ensure that your model directory structure aligns with what's required in the
- User
2025-01-05 08:42:51.293729: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2025-01-05 08:42:51.312687: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2025-01-05 08:42:51.318597: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2025-01-05 08:42:51.332730: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2025-01-05 08:42:52.828604: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Running generation with arguments: Namespace(ckpt_path='PATH_TO_MODEL/ltx-video-2b-v0.9.safetensors', input_video_path=None, input_image_path='/content/LTX-Video/image (4).jpeg', output_path=None, seed=42, num_inference_steps=40, num_images_per_prompt=1, guidance_scale=3, stg_scale=1, stg_rescale=0.7, stg_mode='stg_a', stg_skip_layers='19', image_cond_noise_scale=0.15, height=512, width=768, num_frames=24, frame_rate=25, precision='bfloat16', decode_timestep=0.05, decode_noise_scale=0.025, prompt='Your prompt here', negative_prompt='worst quality, inconsistent motion, blurry, jittery, distorted', offload_to_cpu=False) Padded dimensions: 512x768x25 Traceback (most recent call last): File "/content/LTX-Video/inference.py", line 488, in main() File "/content/LTX-Video/inference.py", line 327, in main vae = CausalVideoAutoencoder.from_pretrained(ckpt_path) File "/content/LTX-Video/ltx_video/models/autoencoders/causal_video_autoencoder.py", line 115, in from_pretrained video_vae = cls.from_config(config) UnboundLocalError: local variable 'config' referenced before assignment
- Assistant
The error message you're encountering indicates that the variable
configis being referenced before it has been assigned a value in thefrom_pretrainedmethod of theCausalVideoAutoencoderclass. This typically happens when none of the conditional branches that assignconfigare executed.Here's a breakdown of how you might address this issue and ensure that your use of the
CausalVideoAutoencoderworks correctly.Understanding the Problem
The
from_pretrainedmethod expects certain files to exist at the specifiedckpt_path. If those files are not found, thenconfigwill not be defined, leading to theUnboundLocalError.Steps to Fix the Error
-
Ensure the Model Files are Present: Check the directory specified by
ckpt_pathand ensure the required files are there:autoencoder.pth(or other model weights files)config.json(model configuration file)- Possibly other files that might be expected based on your model's requirements.
-
Improve Error Handling: Enhance the
from_pretrainedmethod to better handle cases when the necessary files are not found. Here's how you might modify the method:
@classmethod def from_pretrained( cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], *args, **kwargs, ): pretrained_model_name_or_path = Path(pretrained_model_name_or_path) config = None # Initialize config to None if ( pretrained_model_name_or_path.is_dir() and (pretrained_model_name_or_path / "autoencoder.pth").exists() ): config_local_path = pretrained_model_name_or_path / "config.json" if config_local_path.exists(): config = cls.load_config(config_local_path, **kwargs) else: raise FileNotFoundError(f"Configuration file not found at {config_local_path}") model_local_path = pretrained_model_name_or_path / "autoencoder.pth" state_dict = torch.load(model_local_path, map_location=torch.device("cpu")) # Additional loading logic... elif pretrained_model_name_or_path.is_dir(): # Handle diffusers config loading logic... pass # Include your existing logic here and handle the config appropriately # Check if config is still None if config is None: raise RuntimeError("No valid configuration loaded. Please check your model files.") video_vae = cls.from_config(config) # Load state_dict and continue...- Run Your Code Again: After making those changes, try running your code again. Ensure that the path you provide to load the model is correct and that it contains the required files.
Confirm Dependencies Are Installed:
Additionally, ensure you have the required libraries installed in your environment. Sometimes missing packages can lead to unexpected errors. You can install any missing packages using
pip, for example:pip install torch torchvision torchaudio diffusers einops safetensorsCheck Runtime Compatibility:
Lastly, ensure that the runtime (e.g., Google Colab or your local machine) is equipped with a compatible version of CUDA and has the necessary GPU resources available.
Make sure to also check for TensorFlow warnings or errors related to missing components as they might not directly affect your current script but could indicate potential future issues.
Conclusion
By verifying the presence of appropriate files, improving the error handling in the code, and ensuring dependencies are installed, you should be able to resolve the
UnboundLocalErrorand proceed with using the LTX-Video framework effectively. If you encounter additional issues, please provide more context, and I can help troubleshoot further. -
- User
2025-01-05 09:09:16.034890: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2025-01-05 09:09:16.054320: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2025-01-05 09:09:16.060279: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2025-01-05 09:09:16.074312: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2025-01-05 09:09:17.276404: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Running generation with arguments: Namespace(ckpt_path='PATH_TO_MODEL/ltx-video-2b-v0.9.safetensors', input_video_path=None, input_image_path=None, output_path=None, seed=42, num_inference_steps=40, num_images_per_prompt=1, guidance_scale=3, stg_scale=1, stg_rescale=0.7, stg_mode='stg_a', stg_skip_layers='19', image_cond_noise_scale=0.15, height=512, width=768, num_frames=24, frame_rate=25, precision='bfloat16', decode_timestep=0.05, decode_noise_scale=0.025, prompt='a flower in mountain at sunset', negative_prompt='worst quality, inconsistent motion, blurry, jittery, distorted', offload_to_cpu=False) Padded dimensions: 512x768x25 Traceback (most recent call last): File "/content/LTX-Video/inference.py", line 488, in main() File "/content/LTX-Video/inference.py", line 327, in main vae = CausalVideoAutoencoder.from_pretrained(ckpt_path) File "/content/LTX-Video/ltx_video/models/autoencoders/causal_video_autoencoder.py", line 115, in from_pretrained video_vae = cls.from_config(config) UnboundLocalError: local variable 'config' referenced before assignment
- Assistant
The error message you're receiving indicates that within the
from_pretrainedmethod of theCausalVideoAutoencoder, the variableconfigis being referenced before it has been assigned a value. This occurs when the logic paths that would normally setconfigare not executed, typically because the expected configuration file or model weights are not present.Here’s how you can troubleshoot this issue and ensure that the model can be loaded correctly:
Steps to Fix the Issue
-
Verify Model Files: Ensure that the path you provide (
'PATH_TO_MODEL/ltx-video-2b-v0.9.safetensors') contains all required files. The directory should contain:autoencoder.pthor the relevant model weights file.config.json(configuration for the model).- Possibly other required files like
per_channel_statistics.json.
Make sure you are actually pointing to the correct path and that these files exist.
-
Modify the
from_pretrainedMethod: You can enhance the error handling in thefrom_pretrainedmethod by adding checks for the expected files and raising an appropriate error message when files are missing. Here’s an example modification:
@classmethod def from_pretrained( cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], *args, **kwargs, ): pretrained_model_name_or_path = Path(pretrained_model_name_or_path) config = None # Initialize config to None for later checks # Check for the model files if ( pretrained_model_name_or_path.is_dir() and (pretrained_model_name_or_path / "autoencoder.pth").exists() ): config_local_path = pretrained_model_name_or_path / "config.json" if config_local_path.exists(): config = cls.load_config(config_local_path, **kwargs) else: raise FileNotFoundError(f"Configuration file not found at {config_local_path}") model_local_path = pretrained_model_name_or_path / "autoencoder.pth" state_dict = torch.load(model_local_path, map_location=torch.device("cpu")) elif pretrained_model_name_or_path.is_file() and str(pretrained_model_name_or_path).endswith(".safetensors"): # Handle loading from .safetensors files with safe_open(pretrained_model_name_or_path, framework="pt", device="cpu") as f: for k in f.keys(): state_dict[k] = f.get_tensor(k) configs = json.loads(f.metadata().get("config", "{}")) config = configs.get("vae", None) # If config is still None, raise an error if config is None: raise RuntimeError("No valid configuration loaded. Please check your model files.") video_vae = cls.from_config(config) if "torch_dtype" in kwargs: video_vae.to(kwargs["torch_dtype"]) video_vae.load_state_dict(state_dict) return video_vae-
Testing Your Setup: After making these code changes, attempt to run your script again. If you continue to have issues, print out the specific path being used and the files present in that directory to ensure everything aligns.
-
Check TensorFlow Warnings: The additional warnings you're seeing about cuDNN, cuBLAS, and TensorRT typically don’t stop code execution, but they indicate that certain CUDA plugins are being registered multiple times. These could also be due to how TensorFlow and PyTorch interact when using XLA or GPU resources. If your code executes correctly despite these warnings, they might not require immediate action. However, it’s good to keep your libraries updated:
pip install --upgrade tensorflow torch torchvision torchaudio
Conclusion
By ensuring that you have the correct model files present, improving the error handling in the
from_pretrainedmethod, and continuously checking your paths and environment, you should be able to resolve theUnboundLocalErrorand successfully initialize your model. If you still encounter issues after these steps, please provide more context on any changes or results for further assistance. -
- User
{ "_class_name": "AutoencoderKLLTXVideo", "_diffusers_version": "0.32.0.dev0", "block_out_channels": [ 128, 256, 512, 512 ], "decoder_causal": false, "encoder_causal": true, "in_channels": 3, "latent_channels": 128, "layers_per_block": [ 4, 3, 3, 3, 4 ], "out_channels": 3, "patch_size": 4, "patch_size_t": 1, "resnet_norm_eps": 1e-06, "scaling_factor": 1.0, "spatio_temporal_scaling": [ true, true, true, false ] }
- Assistant
The JSON configuration you provided appears to be for a model called
AutoencoderKLLTXVideo, which is structured to define the characteristics of the autoencoder such as the architecture, normalization settings, and channel configurations. This configuration is crucial for initializing your model correctly in the LTX-Video framework.Key Fields in the Configuration
-
_class_name: Specifies the class that represents the model architecture. In this case, it's
AutoencoderKLLTXVideo, indicating the type of architecture. -
block_out_channels: List of output channels for each block in the model. This defines the number of filters applied at different stages of the encoder/decoder.
-
decoder_causal & encoder_causal: Boolean values indicating whether the decoder or encoder uses causal convolutions. Causal convolutions are often used in models where the output at a particular time step can only depend on previous time steps.
-
in_channels: The number of input channels for the model. For color images, this typically equals 3 (RGB).
-
latent_channels: Number of channels in the latent representation. This defines how compressed the input will be in the latent space.
-
layers_per_block: Indicates how many layers each block of the model will have.
-
out_channels: The number of output channels. Again, this is set to 3 for RGB output images.
-
patch_size: Size of the patch in which the input data is divided for processing. This affects the model's ability to capture features.
-
patch_size_t: Similar to the above but for the temporal (time) dimension.
-
resnet_norm_eps: Epsilon value used in normalization layers (like GroupNorm or BatchNorm) for stability.
-
scaling_factor: A scaling parameter that could be used during training, possibly for adjusting learning rates or preventing gradient explosion.
-
spatio_temporal_scaling: A boolean list that indicates whether scaling is applied in spatial and temporal dimensions for respective blocks.
Using This Configuration
If your goal is to use this configuration to initialize your model, ensure the following steps are taken in your Python script or Colab notebook:
-
Load the Configuration: Ensure you're loading this configuration correctly into your model. Replace the existing method for loading the configuration in your model class if required.
-
Initialization: When creating a model instance, use this configuration to initialize the model properly:
config = { "_class_name": "AutoencoderKLLTXVideo", "_diffusers_version": "0.32.0.dev0", "block_out_channels": [128, 256, 512, 512], "decoder_causal": False, "encoder_causal": True, "in_channels": 3, "latent_channels": 128, "layers_per_block": [4, 3, 3, 3, 4], "out_channels": 3, "patch_size": 4, "patch_size_t": 1, "resnet_norm_eps": 1e-06, "scaling_factor": 1.0, "spatio_temporal_scaling": [True, True, True, False] } # Assuming CausalVideoAutoencoder class has been defined accordingly video_autoencoder = CausalVideoAutoencoder.from_config(config) -
Loading the Weights: Once the model instance is created, you can then load the weights using the
load_state_dictfunction as you've designed. -
Error Handling: If you see that certain fields are not being handled or lead to errors, it may be worthwhile to check the specifics of the model's constructor and adapt it to ensure it matches what's defined in the configuration.
Final Notes
-
If you encounter further errors during the model initialization or inference processes, take a close look at the console output to identify which specific component may be causing issues. The structure of these configurations should align closely with how your model is expected to be defined.
-
Always ensure that the environment (like Google Colab or your local setup) supports the GPUs and CUDA versions required for the LTX-Video framework and its dependencies.
-
If the model relies on specific versions of libraries (such as diffusers, PyTorch, etc.), ensure you have those installed. You can specify version requirements in your
pipinstall commands to avoid conflicts.
If you have further questions about specific parts of this process or the configuration, feel free to ask!
-
- User
if no contains all models how automaticly download?
- Assistant
To automatically download the required model files if they are not present, you can modify your code to include a check that looks for the necessary files at the specified path. If the files are missing, you can then trigger a download from a remote source (like Hugging Face, where many model files are hosted).
Here's how you can implement this in your
from_pretrainedmethod:Implementation Steps
-
Check for Necessary Files: Before you attempt to load the model, check if the required files are present in the specified directory.
-
Download Missing Files: If the files are not found, use the Hugging Face Hub to download them automatically.
-
Handle Different Scenarios: This includes checking for model weights, configuration files, and possibly other necessary files.
Example Code
Here's an example of how you might implement this:
from huggingface_hub import hf_hub_download import os from pathlib import Path class CausalVideoAutoencoder(AutoencoderKLWrapper): @classmethod def from_pretrained( cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], *args, **kwargs, ): pretrained_model_name_or_path = Path(pretrained_model_name_or_path) # Define the expected files model_files = { 'weights': 'autoencoder.pth', 'config': 'config.json', 'statistics': 'per_channel_statistics.json', # Example additional file } # Check if the model directory already exists if not pretrained_model_name_or_path.exists() or not pretrained_model_name_or_path.is_dir(): raise ValueError(f"The specified path {pretrained_model_name_or_path} is not a valid directory.") missing_files = [file for file in model_files.values() if not (pretrained_model_name_or_path / file).exists()] if missing_files: print(f"Downloading missing files: {missing_files}") # Download each missing file from Hugging Face # Replace 'your_model_id' with the actual model ID on Hugging Face for key, file in model_files.items(): if file in missing_files: try: model_file_path = hf_hub_download(repo_id='your_model_id', filename=file, cache_dir=pretrained_model_name_or_path) print(f"Downloaded {file} to {model_file_path}") except Exception as e: raise RuntimeError(f"Unable to download {file}: {str(e)}") # Now proceed with loading the model config_path = pretrained_model_name_or_path / model_files['config'] if not config_path.exists(): raise FileNotFoundError(f"Configuration file not found: {config_path}") config = cls.load_config(config_path, **kwargs) model_weights_path = pretrained_model_name_or_path / model_files['weights'] state_dict = torch.load(model_weights_path, map_location=torch.device("cpu")) # Remainder of your existing code... # ... video_vae = cls.from_config(config) if "torch_dtype" in kwargs: video_vae.to(kwargs["torch_dtype"]) video_vae.load_state_dict(state_dict) return video_vaeKey Components of the Code
-
Path Checking: The
osandpathliblibraries are used to handle file paths and check for the existence of required files. -
Model Files List: A dictionary named
model_filesdefines the expected model files. -
Automatic Downloads: If any files are found to be missing, the script tries to download them from Hugging Face using
hf_hub_download. -
Error Handling: Proper error handling ensures that if a download fails or if the directory is malformed, a descriptive error message is generated.
Usage
When you call
CausalVideoAutoencoder.from_pretrained(path_to_model), it will:- Check if the specified directory exists.
- Identify any missing files.
- Download any missing files from the Hugging Face model repository automatically.
- Load the model once all necessary files are present.
Replace
'your_model_id'with the actual model repository ID from Hugging Face that contains the model files you need.Considerations
-
Hugging Face Authentication: If your model requires authentication (e.g., private models), ensure you have set up your Hugging Face authentication token appropriately in your environment.
-
Error Handling for Downloading: Adjust the error handling based on your needs to provide better feedback in case of failure.
By implementing these steps, your model loading process will be more robust and capable of automatically fetching necessary files as needed.
-
- User
NameError Traceback (most recent call last) in <cell line: 5>() 3 from pathlib import Path 4 ----> 5 class CausalVideoAutoencoder(AutoencoderKLWrapper): 6 @classmethod 7 def from_pretrained(
NameError: name 'AutoencoderKLWrapper' is not defined
- Assistant
The
NameError: name 'AutoencoderKLWrapper' is not definederror indicates that the classAutoencoderKLWrapperhas not been imported or defined in your current code context. Before you define or use theCausalVideoAutoencoderclass, you need to ensure that all dependencies and parent classes are correctly imported.Steps to Resolve the Issue
-
Import Required Classes: Make sure to import all necessary classes and functions at the top of your script. This includes
AutoencoderKLWrapper. -
Check Your Imports: If the
AutoencoderKLWrapperclass is part of another module, you'll need to import it correctly.
Here’s an example of how to set up your imports and define your
CausalVideoAutoencoderclass, assumingAutoencoderKLWrapperis defined in your codebase or within an external library.Example Code
Assuming you have the class defined elsewhere in your codebase, here’s how it might look:
# Ensure all necessary imports are made at the top of your script import os import torch from pathlib import Path from huggingface_hub import hf_hub_download # Assuming AutoencoderKLWrapper is defined in ltx_video.models.autoencoders.vae or a similar file from ltx_video.models.autoencoders.vae import AutoencoderKLWrapper class CausalVideoAutoencoder(AutoencoderKLWrapper): @classmethod def from_pretrained( cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], *args, **kwargs, ): pretrained_model_name_or_path = Path(pretrained_model_name_or_path) # Define the expected files model_files = { 'weights': 'autoencoder.pth', 'config': 'config.json', 'statistics': 'per_channel_statistics.json', # Example additional file } if not pretrained_model_name_or_path.exists() or not pretrained_model_name_or_path.is_dir(): raise ValueError(f"The specified path {pretrained_model_name_or_path} is not a valid directory.") missing_files = [file for file in model_files.values() if not (pretrained_model_name_or_path / file).exists()] if missing_files: print(f"Downloading missing files: {missing_files}") # Download each missing file from Hugging Face # Replace 'your_model_id' with the actual model ID on Hugging Face for key, file in model_files.items(): if file in missing_files: try: model_file_path = hf_hub_download(repo_id='your_model_id', filename=file, cache_dir=pretrained_model_name_or_path) print(f"Downloaded {file} to {model_file_path}") except Exception as e: raise RuntimeError(f"Unable to download {file}: {str(e)}") config_path = pretrained_model_name_or_path / model_files['config'] if not config_path.exists(): raise FileNotFoundError(f"Configuration file not found: {config_path}") config = cls.load_config(config_path, **kwargs) model_weights_path = pretrained_model_name_or_path / model_files['weights'] state_dict = torch.load(model_weights_path, map_location=torch.device("cpu")) video_vae = cls.from_config(config) if "torch_dtype" in kwargs: video_vae.to(kwargs["torch_dtype"]) video_vae.load_state_dict(state_dict) return video_vaeTips
-
Install Necessary Libraries: Make sure you have all required libraries installed using pip. If you're using the LTX-Video framework repository, ensure any dependencies are also installed:
pip install torch huggingface_hub -
Model File Management: Ensure the specific path of where your model files are located is correct and accessible.
-
Running in a Notebook: If you're running this in a Jupyter Notebook or Google Colab, ensure that all cells up to this point have executed successfully to avoid missing context.
Conclusion
By ensuring the necessary imports and correcting any issues regarding the absence of required classes, you should be able to resolve the
NameErrorand proceed with using theCausalVideoAutoencoderas expected. If you encounter further issues, please provide additional context or error messages for more targeted assistance. -
- User
Error importing huggingface_hub.hf_api: cannot import name 'is_tqdm_disabled' from 'huggingface_hub.utils.tqdm' (/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/tqdm.py)
ImportError Traceback (most recent call last) in <cell line: 8>() 6 7 # Assuming AutoencoderKLWrapper is defined in ltx_video.models.autoencoders.vae or a similar file ----> 8 from ltx_video.models.autoencoders.vae import AutoencoderKLWrapper 9 10 class CausalVideoAutoencoder(AutoencoderKLWrapper):
8 frames /usr/local/lib/python3.10/dist-packages/huggingface_hub/lfs.py in 40 from .utils._lfs import SliceFileObj 41 from .utils.sha import sha256, sha_fileobj ---> 42 from .utils.tqdm import is_tqdm_disabled 43 44
ImportError: cannot import name 'is_tqdm_disabled' from 'huggingface_hub.utils.tqdm' (/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/tqdm.py)
NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the "Open Examples" button below.
- Assistant
The error message you're encountering indicates that there is an import issue with the
huggingface_hubpackage, specifically with the functionis_tqdm_disabled. This can happen for a few reasons, such as compatibility issues between different versions of libraries.Steps to Resolve the ImportError
-
Update the
huggingface_hubLibrary: First, try updating thehuggingface_hublibrary to ensure you have the latest compatible version. You can do this usingpip:!pip install --upgrade huggingface_hub -
Check Compatibility with Other Libraries: If you are using other libraries that depend on
huggingface_hub, such astransformersordiffusers, ensure they are also updated:!pip install --upgrade transformers diffusers -
Restart Your Runtime: After updating the libraries, it's often helpful to restart your runtime or Jupyter notebook kernel to ensure that the changes take effect. In Google Colab, you can do this by clicking on "Runtime" > "Restart runtime".
-
Re-run Your Code: After restarting, re-run the cells from the beginning, including the imports and your model class definition.
Example Code for Importing and Defining Your Class
Make sure your import statements are structured correctly:
# Import necessary libraries import os import torch from pathlib import Path from huggingface_hub import hf_hub_download # Ensure that the correct module path for AutoencoderKLWrapper is provided # Adjust this based on your environment setup try: from ltx_video.models.autoencoders.vae import AutoencoderKLWrapper except ImportError as e: print("Failed to import AutoencoderKLWrapper. Ensure it is defined in the correct module.") raise e # Define your CausalVideoAutoencoder class class CausalVideoAutoencoder(AutoencoderKLWrapper): @classmethod def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], *args, **kwargs): # Your implementation goes here ...Conclusion
By following these steps, you should be able to resolve the
ImportErrorand use thehuggingface_hublibrary effectively in your project. If you continue to encounter issues, please provide further details on your project setup, including versions of relevant packages, so that more specific guidance can be provided. -
- User
Error importing huggingface_hub.hf_api: cannot import name 'is_tqdm_disabled' from 'huggingface_hub.utils.tqdm' (/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/tqdm.py) Failed to import AutoencoderKLWrapper. Ensure it is defined in the correct module.
ImportError Traceback (most recent call last) in <cell line: 9>() 11 except ImportError as e: 12 print("Failed to import AutoencoderKLWrapper. Ensure it is defined in the correct module.") ---> 13 raise e 14 15 # Define your CausalVideoAutoencoder class
9 frames /usr/local/lib/python3.10/dist-packages/huggingface_hub/lfs.py in 40 from .utils._lfs import SliceFileObj 41 from .utils.sha import sha256, sha_fileobj ---> 42 from .utils.tqdm import is_tqdm_disabled 43 44
ImportError: cannot import name 'is_tqdm_disabled' from 'huggingface_hub.utils.tqdm' (/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/tqdm.py)
NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the "Open Examples" button below.
- Assistant
The error you're encountering indicates that there is an issue with the
huggingface_hubpackage related to theis_tqdm_disabledimport. This is likely due to a version mismatch where that particular function does not exist in the currently installed version ofhuggingface_hub.Here's how to address this:
-
Uninstalling and Reinstalling
huggingface_hub: Sometimes, simply updating the package may not resolve issues caused by installation corruption or incompatible versions. You can uninstall and then reinstall thehuggingface_hubpackage completely.Run the following commands in your notebook:
!pip uninstall -y huggingface_hub !pip install huggingface_hub -
Install Specific Version: If the issue persists, you can install a specific version of
huggingface_hubknown to work without this import error. As of now, a commonly stable version is0.13.3, though you may need to adjust it based on your other libraries:!pip install huggingface_hub==0.13.3 -
Upgrade Related Libraries: Upgrade any related libraries such as
transformers,diffusers, and others that might depend onhuggingface_hub:!pip install --upgrade transformers diffusers -
Restart Runtime: After making these changes, remember to restart the kernel or runtime in your environment (in Google Colab, click on "Runtime" -> "Restart runtime") to apply the updates correctly.
-
Check for Compatibility Issues: If you are using a specific repository like LTX-Video, make sure that its dependencies are compatible with the versions you installed. Refer to its documentation or requirements file if available.
Sample Code to Check Imports
After reinstalling and updating, use the following sample code to check if imports are successful:
# Ensure to import necessary functions try: from huggingface_hub import hf_hub_download, HfApi except ImportError as e: print("Error importing huggingface_hub. Please ensure it's installed correctly.") raise e # Check if AutoencoderKLWrapper can be imported try: from ltx_video.models.autoencoders.vae import AutoencoderKLWrapper except ImportError as e: print("Failed to import AutoencoderKLWrapper. Ensure it is defined in the correct module.") raise e # If imports are successful, you can define your class class CausalVideoAutoencoder(AutoencoderKLWrapper): # Your implementation goes here ...Conclusion
By uninstalling and reinstalling the
huggingface_hublibrary or installing a specific version, you should be able to resolve the import issues regardingis_tqdm_disabled. If problems persist, further investigate compatibility with other libraries or check for updates from the authors of the libraries involved. If you have further questions or issues, feel free to ask! -