Skip to content
TheChatArchiveThe Chat Archive
Googlegemma-3-27b-it· GoogleHugging Face LMArena Human Preference 140K

please check online llm based open source test automation with vision tool

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwq-32b’s answer.
  1. User

    please check online llm based open source test automation with vision tool

  2. Google Assistant

    Okay, let's break down the landscape of open-source test automation tools leveraging LLMs (Large Language Models) with vision capabilities. This is a rapidly evolving area, so I'll give you a current overview as of late 2023/early 2024. I'll categorize tools and provide details, including strengths, weaknesses, and links. I'll also discuss the challenges.

    Understanding the Core Concepts

    • LLM-Based Test Automation: Traditionally, test automation relies on precise locators (XPath, CSS Selectors, etc.) to identify UI elements. LLMs change this. They allow you to describe elements in natural language ("Click the button that says 'Submit'"), and the LLM figures out how to find it. This makes tests more resilient to UI changes.
    • Vision-Based Test Automation: This uses image recognition to identify UI elements. It's useful when elements lack good locators (e.g., in dynamically generated content, or when dealing with images themselves). It's also good for visual regression testing (detecting unintended visual changes).
    • Combining LLMs and Vision: The most powerful approach. LLMs can interpret the context of the screen (using vision) and then generate more robust and adaptable test steps. For example, "Click the 'Add to Cart' button near the product image" – the LLM uses vision to find the product image and then locate the button relative to it.

    I. Promising Open-Source Tools & Frameworks

    Here's a breakdown, categorized by maturity and focus. I'll also indicate the level of LLM/Vision integration.

    A. Mature & Actively Developed (Good Starting Points)

    • Applitools Eyes (Open Source SDK): (https://applitools.com/open-source/)
      • Focus: Primarily visual regression testing, but increasingly incorporating AI.
      • LLM/Vision: Applitools uses AI (including visual AI) to detect visual differences. They've been adding LLM capabilities to help explain those differences and make it easier to triage failures. It's not full LLM-driven test generation yet, but it's moving in that direction. The open-source SDK allows you to integrate with various test frameworks.
      • Strengths: Excellent visual comparison algorithms, good reporting, integrates with many frameworks.
      • Weaknesses: Can be expensive for large-scale use (the cloud service has costs). The open-source SDK requires more setup.
    • Playwright with AI Support (Microsoft): (https://playwright.dev/)
      • Focus: End-to-end testing for web applications.
      • LLM/Vision: Playwright is actively integrating LLMs. They've released features like:
        • playwright.generate(): Generates tests from browser interactions. You record your actions, and it creates Playwright code. This is powered by LLMs.
        • toText(): Extracts text from elements, making it easier to assert content.
        • Visual Comparison: Playwright has built-in visual comparison capabilities.
      • Strengths: Fast, reliable, cross-browser, excellent documentation, strong community, actively developed AI features.
      • Weaknesses: LLM integration is still relatively new, so it's evolving. Vision capabilities are present but not as advanced as dedicated visual testing tools.
    • Cypress with AI (Beta): (https://www.cypress.io/)
      • Focus: End-to-end testing for web applications.
      • LLM/Vision: Cypress is also adding AI features, including:
        • AI-Powered Test Generation: Similar to Playwright's generate(), Cypress is working on generating tests from user interactions.
        • AI-Driven Flakiness Detection: Helps identify tests that fail intermittently due to timing issues or other factors.
        • Visual Testing: Cypress has plugins for visual regression testing.
      • Strengths: Developer-friendly, excellent debugging tools, large community.
      • Weaknesses: Limited to JavaScript/TypeScript. AI features are in beta and evolving.

    B. Emerging & Promising (More Experimental)

    • Testim.io (Open Source Core): (https://www.testim.io/)
      • Focus: AI-powered test automation.
      • LLM/Vision: Testim uses AI to stabilize locators and reduce test flakiness. It also has visual validation features. They have an open-source core, but the full platform is commercial.
      • Strengths: Good at handling dynamic UIs, AI-powered stabilization.
      • Weaknesses: The open-source core is limited. The full platform is expensive.
    • Locust with AI Plugins: (https://locust.io/)
      • Focus: Load testing.
      • LLM/Vision: While Locust itself doesn't have built-in LLM/Vision, there are emerging plugins that are starting to integrate these technologies to dynamically generate load test scenarios based on UI analysis. This is a very new area.
      • Strengths: Scalable, easy to use for load testing.
      • Weaknesses: LLM/Vision integration is experimental.
    • Automa: (https://automa.ai/)
      • Focus: No-code/low-code automation.
      • LLM/Vision: Automa is built around LLMs and computer vision. You can describe what you want to automate in natural language, and it will attempt to create the automation flow. It's very visual and user-friendly.
      • Strengths: Easy to learn, powerful for simple automations.
      • Weaknesses: May not be suitable for complex test scenarios. Relatively new.

    C. Research Projects & Libraries (Building Blocks)

    • Selenium with LLM/Vision Integrations: (https://www.selenium.dev/)
      • Focus: Web browser automation.
      • LLM/Vision: Selenium is the foundation for many test automation frameworks. You can integrate it with LLMs (using APIs like OpenAI's) and computer vision libraries (like OpenCV) to create custom solutions. This requires more programming effort.
      • Strengths: Extremely flexible, widely used, large community.
      • Weaknesses: Requires significant coding. LLM/Vision integration is not built-in.
    • OpenCV (Computer Vision Library): (https://opencv.org/)
      • Focus: Computer vision tasks (image recognition, object detection, etc.).
      • LLM/Vision: OpenCV is a powerful library for image analysis. You can use it to identify UI elements based on their visual appearance. It's often used in conjunction with Selenium or other automation frameworks.
      • Strengths: Very powerful, widely used in computer vision.
      • Weaknesses: Requires strong programming skills. Not a complete test automation solution.

    II. Challenges & Considerations

    • Cost of LLM APIs: Using LLMs like OpenAI's GPT models can be expensive, especially for large-scale test execution. Consider using open-source LLMs (see below) or carefully managing your API usage.
    • Reliability of LLMs: LLMs are not perfect. They can sometimes generate incorrect or ambiguous test steps. Thorough validation is crucial.
    • Vision Accuracy: Computer vision can be affected by factors like lighting, image quality, and UI variations. Robust image processing techniques are needed.
    • Maintenance: Even with LLMs, tests still require maintenance. UI changes will eventually break tests, so you need a strategy for updating them.
    • Open-Source LLMs: Consider using open-source LLMs like:
    • Data Privacy: Be mindful of data privacy when using LLM APIs. Avoid sending sensitive data to external services.

    III. Recommendations

    • Start with Playwright or Cypress: These frameworks have the most active LLM integration and are relatively easy to learn.
    • Consider Applitools for Visual Testing: If visual regression testing is a priority, Applitools is a good choice.
    • Experiment with Selenium + OpenCV: If you need maximum flexibility and control, you can build a custom solution using Selenium and OpenCV.
    • Explore Automa for No-Code Automation: If you want a quick and easy way to automate simple tasks, Automa is worth a look.
    • Keep Learning: This field is evolving rapidly. Stay up-to-date with the latest developments in LLMs and computer vision.

    To help me refine this further, could you tell me:

    • What type of application are you testing? (Web, mobile, desktop)
    • What is your programming experience level?
    • What are your primary testing goals? (Functional testing, visual testing, load testing, etc.)
    • What is your budget? (Are you willing to pay for commercial tools or services?)

    I hope this comprehensive overview is helpful! Let me know if you have any other questions.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt