Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-2024-07-18· OpenAIAi2 WildChat-4.8M (AI2)

How can I install this in colab? <p align="center"> <h2 align="center">InFusion:…

  1. User

    How can I install this in colab?

    News

    • [2024.4.18] 🔥 Release paper, inference code, and pretrained checkpoint.
    • [On-going] Clean and organize the masks corresponding to the dataset used in the experiments.
    • [On-going] Scale up the model with more training data and release stronger models as the foundation model for downstream tasks.
    • [To-do] Release training code.

    Installation

    Install with conda:

    conda env create -f environment.yaml
    conda activate infusion
    
    • 🛠️ For rendering depth, we use the diff-gaussian-rasterization-confidence from FSGS, thanks to their work! :)

    Download Checkpoints

    Download the Infusion checkpoint and put it in the 'checkpoints' folder:

    Data Preparation

    Our experiments are conducted on the datasets provided by Mip-NeRF, Instruct-NeRF2NeRF, and SPIn-NeRF. We will upload the masks used in the experiments and the challenge scene we shot ourselves in a few days.

    Taking "garden" in Mip-NeRF as an example, each scene folder should be organized as follows.

    garden
    ├── images # RGB data
    │   ├── DSC07956.JPG
    │   ├── DSC07957.JPG
    │   └── ...                   
    ├── seg # Mask 
    │   ├── DSC07956.JPG
    │   ├── DSC07957.JPG
    │   └── ... 
    │   # The part that needs to be inpainted is white
    └── sparse # Colmap
    │   └── 0
    │       └── ...
    
    • 🛠️ You can prepare your own data according to such a structure. In addition, accurate masks are very important. Here, we recommend two image segmentation tools: Segment and Track Anything and Grounded SAM.

    • 🛠️ To obtain camera parameters and initial point cloud, please refer to 'convert.py' in Gaussian-Splatting :)

    Instructions

    The entire pipeline is divided into three stages:

    • Train the Incomplete Gaussians.
    • Inpaint Gaussians via Diffusion Prior.
    • Combine Inpainted Gaussians and Incomplete Gaussians.

    🌺 Stage 1

    Use pre-annotated masks to train incomplete Gaussians.

    cd gaussian_splatting
    # Train incomplete Gaussians
    python train.py -s <path to scene folder> -m <path to output folder> -u nothing --mask_training
    #--color_aug
    
    # Obtain c2w matrix, intrinsic matrix, incomplete depth, and rgb rendering image
    python render.py -s <path to scene folder> -m <path to output folder> -u nothing
    
    • 🛠️ Tip: Sometimes, the rendered depth has too many empty points. Maybe you can use --color_aug during training, which will randomly select the background color when rendering depth, which may make the depth map more reliable.

    • 🛠️ Recently, some works focused on how to segment Gaussians. This is not the focus of this work, so a relatively simple method was chosen. :)

    🌺 Stage 2

    Inpaint Gaussians using depth inpainting model.

    • 📢 You need to select a single image and in 'path to output folder/train/ours_30000/renders' and mark the area that needs to be inpainted and save it as 'mask.png'. (It doesn’t have to be precise but it needs to cover all the missing parts.)

    • 📢 Next, you need to inpaint a single image. Here are some great tools to inpaint a single image: Stable Diffusion XL Inpainting and Photoroom. Here is an example:

    # Assume that the selected single image is named "DSC07956.JPG".
    cd depth_inpainting/run
    input_rgb_path=<path to inpainted single image>
    input_mask_path=<path to 'mask.png'>
    input_depth_path=<path to output folder/train/ours_30000/depth_dis/DSC07956.npy>
    c2w=<path to output folder/train/ours_30000/c2w/DSC07956.npy>
    intri=<path to output folder/train/ours_30000/intri/DSC07956.npy>
    model_path=</path to depth_inpainting model checkpoint>  # absolute path
    output_dir=<path to output folder>
    
    
    CUDA_VISIBLE_DEVICES=0 python run_inference_inpainting.py \
                --input_rgb_path $input_rgb_path \
                --input_mask $input_mask_path \
                --input_depth_path $input_depth_path \
                --model_path $model_path \
                --output_dir $output_dir \
                --denoise_steps 20 \
                --intri $intri \
                --c2w $c2w \
                --use_mask\
                --blend  # Whether to use 'Blended Diffusion (https://arxiv.org/abs/2111.14818)' during inference. 
    
    • 🛠️ Tip: If you feel that the depth map obtained by one inference is not satisfactory, you can use the newly obtained output_dir/<inpainted_image_name>_depth_dis.npy as the new $input_depth_path and loop two or three times to get better results.

    🌺 Stage 3

    Combine inpainted Gaussians and incomplete Gaussians and quickly fine-tune on inpainted single image.

    # Assume that the selected single image is named "DSC07956.JPG".
    origin_ply="path to output folder/point_cloud/iteration_30000/point_cloud.ply"
    supp_ply="path to output folder/DSC07956_mask.ply"
    save_ply="path to output folder/point_cloud/iteration_30001/point_cloud.ply"
    # Combine inpainted Gaussians and incomplete Gaussians.
    python compose.py --original_ply $origin_ply  --supp_ply $supp_ply --save_ply $save_ply --nb_points 100 --threshold 1.0
    # Fine-tune on an inpainted single image for 150 iterations.
    python train.py -s <path to scene folder> -m <path to output folder> -u DSC07956.JPG -n <path to inpainted single image> --load_iteration 30001 --iteration 150
    # Render
    python render.py -s <path to scene folder> -m <path to output folder> -u nothing --iteration 150
    
    • 🛠️ The two parameters --nb_points and --threshold are used to remove floaters. Increasing their values will remove more surrounding points. Removing floaters is very important for the final rendering results. Here, we need to find the most suitable parameters for removing floaters from the scene.

    • 🛠️ As explicit points, Gaussian can be directly edited and cropped in actual applications, such as KIRI Engine

    Acknowledgements

    This project is developed on the codebase of Gaussian-Splatting, Marigold and Magicboomliu. We appreciate their great works!

    Citation

    If you find this repository useful in your work, consider citing the following papers and giving a ⭐ to the public repository to allow more people to discover this repo:

    @article{liu2024infusion,
      title={InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior},
      author={Liu, Zhiheng and Ouyang, Hao and Wang, Qiuyu and Cheng, Ka Leong and Xiao, Jie and Zhu, Kai and Xue, Nan and Liu, Yu and Shen, Yujun and Cao, Yang},
      journal={arXiv preprint arXiv:2404.11613},
      year={2024}
    }
    
    
  2. Assistant

    To install and run the InFusion project in Google Colab, you'll need to adapt the installation instructions provided for a conda environment. Google Colab does not support conda natively, but you can install the necessary packages using pip and set up everything in your notebook.

    Here's how you can do it step-by-step:

    Step 1: Setting Up Environment

    1. Open a Google Colab Notebook.
    2. Install the required libraries: You can install the required packages using pip commands. If there’s a specific set of dependencies listed in environment.yaml, you can find equivalent pip packages.
    # First, install necessary packages
    !pip install numpy scipy matplotlib opencv-python  # add other necessary packages as per 'environment.yaml'
    

    Step 2: Clone the Repository

    You then need to clone the Infusion repository and navigate within it:

    # Clone the repository
    !git clone https://github.com/Johanan528/Infusion.git
    %cd Infusion
    

    Step 3: Install Other Dependencies

    If the project has additional dependencies listed in requirements.txt or similar, install those as well. This is a common practice in many repositories:

    # If there's requirements.txt available
    !pip install -r requirements.txt
    

    Step 4: Download Checkpoints

    You need to download the pretrained checkpoints into the correct folder within your Colab environment. You can do this using commands like wget or curl, or by mounting Google Drive if the checkpoints are too large.

    # Download checkpoints from HuggingFace or another location
    !mkdir -p checkpoints
    !wget <CHECKPOINT_URL> -O checkpoints/infusion_checkpoint.pth  # Replace <CHECKPOINT_URL> with the actual URL
    

    Step 5: Preparing Data

    Prepare your data in the format mentioned in the documentation. Since you’re using Colab, you may want to upload files via the Colab interface or mount Google Drive.

    from google.colab import drive
    drive.mount('/content/drive')
    
    # Now copy your data to the correct format in the Colab environment
    !mkdir -p garden/images
    !mkdir -p garden/seg
    !mkdir -p garden/sparse/0
    # Upload your images, masks, and Colmap data to the appropriate directories
    

    Step 6: Running the Pipeline

    You can then run the Python scripts as mentioned in the documentation. Here’s an example of how you could run the steps in the pipeline:

    # Example: Train incomplete Gaussians
    !python gaussian_splatting/train.py -s <path to scene folder> -m <path to output folder> -u nothing --mask_training
    
    # Example: Inpaint Gaussians
    !python depth_inpainting/run/run_inference_inpainting.py --input_rgb_path <path to inpainted single image> --input_mask <path to mask.png> --input_depth_path <depth> --model_path <model_path> --output_dir <output_dir>
    

    Notes

    • The paths need to be appropriately set in Colab, replacing placeholders with the actual paths of your files.
    • A lot of the commands above use placeholder paths. You will need to replace <path to ...> style text with your actual directory paths based on the structure you set up.
    • If performance issues arise because of Colab's GPU/Resource limits, consider breaking tasks into smaller chunks or use local machine setups for heavier operations.

    Remember, you would need to adapt based on dependencies and requirements of the specific project. Always consult the official project documentation for up-to-date installation and usage instructions.

  3. User

    ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

  4. Assistant

    If you're encountering the error indicating that the requirements file cannot be found (requirements.txt), it could be because a requirements.txt file does not exist in the Infusion repository or it is located in a different path or under a different name.

    Here are the steps you can take to proceed:

    Step 1: Check the Repository Structure

    First, you can check the folder structure of the cloned Infusion repository to see if there are any files that might contain dependency information.

    # List the contents of the cloned Infusion directory
    !ls Infusion
    

    Step 2: Install Dependencies Manually

    If there is no requirements.txt, you can manually install the necessary dependencies. The typical approach would be to check the import statements in the Python scripts to identify the libraries used.

    Here are some common libraries you might need based on typical deep learning and image processing workflows, and those might include libraries like:

    • NumPy
    • SciPy
    • OpenCV
    • Matplotlib
    • PyTorch
    • etc.

    You can install these libraries using pip as follows:

    # Sample installations; modify based on your actual needs
    !pip install numpy scipy matplotlib opencv-python torch torchvision torchaudio  # Add other libraries as needed
    

    Step 3: Try to Find Additional Dependencies

    If you have specific dependencies that the code might need, check for documentation within the repository, such as README files or comments in the code itself, which often list required libraries.

    Step 4: Modify the Script According to Your Needs

    If you identify that additional libraries are needed based on missing imports or errors during execution, you can install them directly:

    !pip install <missing-library-name>
    

    Step 5: Run the Application

    Once all relevant dependencies are installed, you can proceed with running the scripts as previously defined, ensuring that all your specified paths in the commands are valid.

    Summary

    • If the requirements.txt file doesn't exist, you can check for other files that may serve similar purposes or inspect the code to identify any other required libraries manually.
    • Install the identified libraries using pip based on your findings.
    • Continue with running the scripts once all necessary libraries are installed.

    If you encounter specific import errors during running the code, please provide those errors, and I’d be glad to help debug those issues specifically!

  5. User
        <div class="is-size-5 publication-authors">
          <span class="author-block"><sup>1</sup>USTC</span>&ensp;
          <span class="author-block"><sup>2</sup>HKUST</span>&ensp;
              <span class="author-block"><sup>3</sup>Ant Group</span>&ensp;
              <span class="author-block"><sup>4</sup>Alibaba Group</span>
            </div>
            <!-- <div class="is-size-5 publication-authors"> -->
          
    
            </div>
          </div>
    
          <div class="column has-text-centered">
            <div class="publication-links">
              <!-- PDF Link. -->
              <span class="link-block">
                <a href="http://arxiv.org/abs/2404.11613"
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="fas fa-file-pdf"></i>
                  </span>
                  <span>Paper</span>
                </a>
              </span>
              <!-- Code Link. -->
              <span class="link-block">
                <a href="https://github.com/ali-vilab/infusion"
                    class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="fab fa-github"></i>
                  </span>
                  <span>Code</span>
                  </a>
              </span>
              <!-- Video Link. -->
              
            </div>
        </div>
      </div>
    </div>
    
    <!-- <div class="container">
    <div id="results-carousel1" class="carousel results-carousel">
    </div>
    </div> -->
    
    
    <div class="container">
      <div id="results-carousel1" class="carousel results-carousel">
        <div class="item g21">
          <video poster="" id="g21" autoplay controls muted loop playsinline height="100%">
            <source src="./static/video/second/grass.mp4"
                    type="video/mp4">
          </video>
        </div>
        <div class="item g22">
          <video poster="" id="g22" autoplay controls muted loop playsinline height="100%">
            <source src="./static/video/second/fence.mp4"
                    type="video/mp4">
          </video>
        </div>
        <div class="item g23">
          <video poster="" id="g23" autoplay controls muted loop playsinline height="100%">
            <source src="./static/video/second/book.mp4"
                    type="video/mp4">
          </video>
        </div>
        <div class="item g24">
          <video poster="" id="g24" autoplay controls muted loop playsinline height="100%">
            <source src="./static/video/second/kettle.mp4"
                    type="video/mp4">
          </video>
        </div>
      </div>
    </div>
    
    
    <br>
    
            <!-- 模块2 -->
            <div class="item columns is-centered">
              <!-- 图片 -->
              <div class="column is-half">
                <figure class="image">
                  <img src="static/images/room.png" alt="描述性文本">
                </figure>
              </div>
              <!-- 3D模型 -->
              <div class="column is-half model-viewer-container" style="text-align: center;">
                <div id="viewer2_mesh" style="width: 100%;">
                  <model-viewer src="./glb/bag_mesh.glb" shadow-intensity="0" camera-controls touch-action="auto" style="width: 100%; height: 400px;">
                  </model-viewer>
                </div>
                <div id="viewer2_point" style="display: none;">
                  <model-viewer src="./glb/bag_point.glb" shadow-intensity="0" camera-controls touch-action="auto" camera-orbit="0deg 0deg 3m" style="width: 100%; height: 400px;">
                  </model-viewer>
                </div>
                <button class="toggleView">Point Cloud</button>
              </div>
            </div>
    
            <div class="item columns is-centered">
              <!-- 图片 -->
              <div class="column is-half">
                <figure class="image">
                  <img src="static/images/grass.png" alt="描述性文本">
                </figure>
              </div>
              <!-- 3D模型 -->
              <div class="column is-half model-viewer-container" style="text-align: center;">
                <div id="viewer2_mesh" style="width: 100%;">
                  <model-viewer src="./glb/grass_mesh.glb" shadow-intensity="0" camera-controls touch-action="auto" style="width: 100%; height: 400px;">
                  </model-viewer>
                </div>
                <div id="viewer2_point" style="display: none;">
                  <model-viewer src="./glb/grass_point.glb" shadow-intensity="0" camera-controls touch-action="auto" style="width: 100%; height: 400px;">
                  </model-viewer>
                </div>
                <button class="toggleView">Point Cloud</button>
              </div>
            </div>
          </div>
        </div>
      </div>
    </div>
    
    <section class="hero is-light is-small">
    
      
      <div class="container">
        <!-- Abstract. -->
        <div class="columns is-centered has-text-centered">
          <div class="column is-full">
            <br>
            <h2 class="title is-3">Pipeline</h2>
            <img src='static/images/overall.png' id="framework"><br><br>
            <div class="content has-text-justified">
              <!-- <p>
                3D Gaussians have recently emerged as an efficient representation for novel view synthesis.This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering.
                Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions.
                To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image.
                Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from large-scale diffusion prior.
                Thanks to the more accurate depth completion, our approach, dubbed <b>Infusion</b>, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios.
                We further demonstrate the effectiveness of \method with several practical applications, such as inpainting with user-specific texture or with novel object insertion.
              </p> -->
              <p>
                <i>Top</i>: To remove a target from the optimized 3D Gaussians, our <b><i>Infusion</i></b> first inpaints a selected one-view RGB image and applies the proposed diffusion model for depth inpainting to
                the depth projection of the targeted 3D Gaussians. The progressive scheme addresses
                view-dependent occlusion issues by utilizing other unobstructed viewpoints.
              </p>
              <p>
                <i>Bottom</i>: A detailed view of the training pipeline for the depth inpainting U-Net is presented.
                We employ a mask-driven denoising diffusion for training of the U-Net, which utilizes
                a frozen latent tokenizer by taking the RGB image and depth map as inputs.
              </p>
            </div>
          </div>
        </div>
        <br>
      </div>
    </section>
    <br><br>
    

    toggleButton.addEventListener('click', function() { var container = this.closest('.model-viewer-container'); var viewerPoint = container.querySelector('[id="point"]′);varviewerMesh=container.querySelector(′[id="_point"]'); var viewerMesh = container.querySelector('[id="_mesh"]');

    if (viewerPoint.style.display !== 'none') {
      viewerPoint.style.display = 'none';
      viewerMesh.style.display = 'block';
      this.innerText = 'Point Cloud';
    } else {
      viewerPoint.style.display = 'block';
      viewerMesh.style.display = 'none';
      this.innerText = 'Mesh';
    }
    

    });

    // 为按钮添加悬停效果 toggleButton.onmouseover = function() { this.style.backgroundColor = '#e0e0e0'; // 悬停时稍微暗一点的色调 };

    toggleButton.onmouseout = function() { this.style.backgroundColor = '#f0f0f0'; // 原背景色 }; });

  6. Assistant

    The HTML code you've posted appears to be part of a web page designed to showcase a research project titled "InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior." This page includes various sections such as publication details, video demonstrations, a gallery of 3D models, a pipeline explanation, and a BibTeX citation format.

    If your goal is to host or edit this web page, there are several steps you can follow based on common requirements for web development. Here’s a breakdown of what you might consider doing:

    1. Hosting the HTML Page

    To host this HTML page:

    • Using GitHub Pages: If you have a GitHub account, you can create a repository and upload this HTML file. You can enable GitHub Pages through the repository settings to host the page.
    • Using Netlify or Vercel: These platforms allow for easy deployment of static sites. You can create a new site and upload your HTML file.
    • Local Hosting: If you want to view it locally, save the code as an .html file and open it in a web browser.

    2. Edit Content

    • Modify Text and Links: You can change the project details, authors, video links, images, etc., directly in the HTML code to reflect your specific project's information.
    • Update Media Sources: If the source paths (like src="./static/video/...) are incorrect, make sure to update them based on where the files will be hosted or ensure that the media files are correctly placed relative to the HTML file.

    3. Interactivity and Styles

    The HTML file uses various JavaScript libraries such as Bulma for styling and model-viewer for displaying 3D models. Make sure you have the following in place:

    • Style Sheets: Ensure all CSS files (like bulma.min.css, index.css, etc.) are available on your hosting server or change their paths to load from a CDN or directly from a repository.
    • JavaScript Libraries: Similarly, ensure that all required JS files are loaded correctly. If using a CDN, ensure the links are up-to-date.

    4. Testing and Debugging

    Once deployed:

    • Test the Web Page: Open the page in different browsers to ensure all functionalities, such as video playback and 3D model rendering, function correctly.
    • Debugging: Use browser developer tools (usually accessed by pressing F12) to troubleshoot any issues with CSS not loading or JS errors.

    Example Modification

    Here's an example of what you might do if you wanted to change the title:

    <h1 class="title is-1 publication-title" style="font-size: 40px; line-height: 1.2;">
      InFusion - A New Title Here
    </h1>
    

    Note Please

    If your intention was different, such as needing help with something specific from this HTML code (like improving the structure, adding features, etc.), please clarify your objectives so I can provide more tailored assistance!

  7. User

    /bin/bash: line 1: path: No such file or directory /bin/bash: -c: line 1: syntax error near unexpected token newline' /bin/bash: -c: line 1: python depth_inpainting/run/run_inference_inpainting.py --input_rgb_path --input_mask --input_depth_path --model_path <model_path> --output_dir <output_dir>'

  8. Assistant

    The error messages you're encountering suggest there are issues with the command you are trying to run in your environment. Specifically, it seems that placeholders (e.g., <path to inpainted single image>, <model_path> etc.) were not replaced with actual file paths before executing the command. Below is a step-by-step guide to help you resolve these issues and successfully run the Python script.

    Step 1: Identify and Replace Placeholders

    To run the command correctly, you need to replace all placeholders in your command with actual paths that exist in your environment. Here’s an example command to help demonstrate how to do this:

    python depth_inpainting/run/run_inference_inpainting.py \
        --input_rgb_path '/path/to/inpainted_single_image.jpg' \
        --input_mask '/path/to/mask.png' \
        --input_depth_path '/path/to/depth.npy' \
        --model_path '/path/to/depth_inpainting_model_checkpoint.pth' \
        --output_dir '/path/to/output_directory'
    

    Make sure that:

    • /path/to/inpainted_single_image.jpg points to the correct path of your inpainted single image.
    • /path/to/mask.png is the path to the mask you created.
    • /path/to/depth.npy should point to the corresponding depth data file.
    • /path/to/depth_inpainting_model_checkpoint.pth should be the model file you downloaded.
    • /path/to/output_directory is the directory where you want to save output.

    Step 2: Verify the Paths

    Before running the command, ensure that:

    • All specified paths are valid. You can check in your filesystem or Colab.
    • The files are located at the paths you are specifying. For example, you can use the following command to list files in a directory in Colab:
    !ls /path/to/directory
    

    Replace /path/to/directory with the actual path you want to check.

    Step 3: Execute the Command

    Once you have replaced the placeholders with valid paths, you can then try executing the command again:

    !python depth_inpainting/run/run_inference_inpainting.py \
        --input_rgb_path '/content/path/to/inpainted_single_image.jpg' \
        --input_mask '/content/path/to/mask.png' \
        --input_depth_path '/content/path/to/depth.npy' \
        --model_path '/content/path/to/depth_inpainting_model_checkpoint.pth' \
        --output_dir '/content/path/to/output_directory'
    

    Sample Command with Actual Paths

    Assuming you uploaded your files to the /content/Infusion/ directory in Google Colab, a full command may look like this:

    !python depth_inpainting/run/run_inference_inpainting.py \
        --input_rgb_path '/content/Infusion/your_inpainted_image.jpg' \
        --input_mask '/content/Infusion/mask.png' \
        --input_depth_path '/content/Infusion/depth.npy' \
        --model_path '/content/Infusion/checkpoints/depth_inpainting_model_checkpoint.pth' \
        --output_dir '/content/Infusion/output_directory'
    

    Troubleshooting

    • Confirm File Existence: If you still receive a No such file error, the path you provided may not exist. Double-check that you've uploaded the files to Google Colab and the paths match exactly.
    • Avoid Special Characters: Ensure there are no special characters or spaces in your file paths. If your paths contain spaces, wrap them in quotes (as shown above).

    If you follow these steps and correctly input your paths, the command should execute without the errors you're currently encountering. If you need any further assistance or have additional error messages, feel free to share!