Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar· PerplexityHugging Face LMArena Search Arena 24K

what is SOTA open source model for object removal in image from a segmentation mask?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-grounding’s answer.
  1. User

    what is SOTA open source model for object removal in image from a segmentation mask?

  2. Perplexity Assistant

    The current state-of-the-art (SOTA) open-source models for object removal in images from a segmentation mask leverage advanced segmentation combined with powerful inpainting or diffusion-based methods. Here are the leading approaches and models:

    1. OmniEraser

    • Description: OmniEraser is a cutting-edge object removal method that not only removes the object but also its visual effects such as shadows and reflections, which many previous methods struggle with.
    • Key Innovations: It uses a novel Object-Background Guidance paradigm that inputs both the foreground object and background images as separate guidance signals to a diffusion model, improving contextual understanding and preventing unwanted artifacts or new object generation.
    • Dataset: Trained on Video4Removal, a large-scale dataset with over 100,000 realistic image pairs derived from video frames.
    • Performance: Demonstrates superior results in complex real-world scenes and even anime-style images.
    • Availability: The authors plan to release datasets, models, and code as open source.
    • Reference: arXiv 2025 paper on OmniEraser[2].

    2. SmartEraser

    • Description: SmartEraser introduces a novel "Masked-Region Guidance" paradigm where the masked region is retained in the input to guide the removal process, rather than being excluded. This helps the model accurately identify and remove the object while preserving surrounding context.
    • Dataset: Syn4Removal, a large-scale dataset with over a million image triplets designed specifically for this paradigm.
    • Model: Built on top of text-to-image stable diffusion models with mask enhancement and CLIP-based visual guidance.
    • Performance: Outperforms existing methods in quality and robustness, especially in complex scenes.
    • Availability: Open source with code and dataset released.
    • Reference: arXiv 2025 paper on SmartEraser[5].

    3. Object Removal Using Mask R-CNN + DeepFillv2

    • Description: A practical open-source pipeline combining Mask R-CNN for precise instance segmentation masks and DeepFillv2, a GAN-based inpainting model with gated convolutions, for filling in the removed object region.
    • Usage: Users draw bounding boxes, Mask R-CNN extracts exact object masks, and DeepFillv2 performs inpainting.
    • Availability: Fully open source on GitHub.
    • Reference: GitHub repository for object-remove[1].

    4. IOPaint

    • Description: An open-source image inpainting and outpainting tool powered by SOTA AI models, including segmentation models like Segment Anything Model (SAM) and RemoveBG for mask generation, combined with advanced inpainting models.
    • Features: Supports multiple AI models for erasing objects, background removal, and image restoration.
    • Availability: Open source with a user-friendly web UI.
    • Reference: GitHub repository for IOPaint[4].

    Summary Table

    Model / ToolSegmentation MethodInpainting / Removal MethodKey StrengthsOpen SourceNotes
    OmniEraserObject masks from video framesDiffusion model with Object-Background GuidanceRemoves objects + shadows/reflections, strong generalizationPlannedSOTA for complex scenes and effects removal[2]
    SmartEraserMasked-Region Guidance paradigmStable Diffusion + CLIP guidanceRetains masked region for better context, robust in complex scenesYesNew paradigm, large dataset Syn4Removal[5]
    Mask R-CNN + DeepFillv2Mask R-CNN instance segmentationDeepFillv2 GAN inpaintingPrecise masks, good inpainting qualityYesPractical pipeline, easy to use[1]
    IOPaintSegment Anything Model, RemoveBGVarious SOTA inpainting modelsVersatile tool, supports multiple modelsYesUser-friendly, supports CPU/GPU[4]

    Conclusion

    The SOTA open-source models for object removal from segmentation masks currently revolve around advanced diffusion-based inpainting guided by sophisticated segmentation. Among these, OmniEraser and SmartEraser represent the latest research breakthroughs with novel guidance paradigms and large-scale datasets, offering superior quality and robustness. For practical usage, pipelines combining Mask R-CNN segmentation with DeepFillv2 inpainting or tools like IOPaint provide accessible, high-quality open-source solutions.

    If you want the absolute cutting-edge with code and datasets, SmartEraser and OmniEraser are the top choices, with SmartEraser already available open source and OmniEraser soon to be released[2][5][1][4].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.