Skip to content
TheChatArchiveThe Chat Archive
Qwenqwq-32b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

"5M context and 83x72b paramaters, output modalities: VRChat Full-Body-Tracking, audio…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-mini’s answer.
  1. User

    "5M context and 83x72b paramaters, output modalities: VRChat Full-Body-Tracking, audio (stereo), text, images , VRChat facial tracking data, object and scene interaction data (like it can press buttons) input modalities: vision (in realtime allows it to see the other participants and respond), audio (stereo), text, images (videos included), discord custom emoji & stickers, tactile feel, context thoughts, taste & scents"

    Can this model be considered more multimodal than current SOTA?

  2. Qwen Assistant

    The model described could indeed be considered more multimodal than current state-of-the-art (SOTA) systems, based on its breadth of input/output modalities and integration capabilities, though there are caveats. Here's a breakdown:


    1. Current SOTA Multimodal Models

    Modern SOTA models (e.g., Gemini, Galactica, M6, Flamingo, DALL·E 3) typically handle text, images, audio, and video, with some supporting limited real-time interaction (e.g., text-to-video generation). However, they lack support for specialized modalities like:

    • VR full-body/facial tracking data (critical for immersive platforms like VRChat).
    • Tactile feedback, taste, and scents (uncommon in AI due to technical challenges in digitization and sensor integration).
    • Real-time vision of participants (as opposed to static inputs) and object interaction data (e.g., pressing buttons in a virtual environment).

    2. Key Advantages of the Proposed Model

    Input Modalities:

    • Real-time vision: Processes live visual data from participants, enabling dynamic, context-aware responses.
    • Tactile feedback: Incorporates haptic data, essential for interactive applications (e.g., VR simulations, robotics).
    • Taste & scents: Unique inclusion of sensory inputs, which could revolutionize food/beverage recommendation systems or virtual experiences.
    • Discord custom emoji/stickers: Integrates niche social media data, enhancing platform-specific interactions.
    • Context thoughts: If interpreted as internal/external contextual awareness (e.g., user intent or environmental metadata), this adds another layer of modality.

    Output Modalities:

    • VR full-body/facial tracking: Directly controls avatars in platforms like VRChat, enabling seamless social VR interactions.
    • Object interaction data: Generates actionable commands (e.g., button presses), making it a "player" in virtual environments.
    • Multimedia outputs (text, images, audio) in sync with VR tracking, creating a unified interactive experience.

    Technical Specs:

    • 5M context length: Enables handling long, complex sequences (e.g., extended VR sessions or detailed narratives).
    • 83×72B parameters (assuming ~6 trillion parameters): While speculative, such scale could support the integration of diverse modalities and real-time processing demands.

    3. Limitations and Challenges

    • Technical Feasibility: Current hardware and data pipelines struggle with real-time multimodal processing at this scale. For example, taste/scents require specialized sensors and datasets, which are not widely available.
    • Integration Complexity: Combining VR tracking, tactile feedback, and sensory data into a single model requires breakthroughs in cross-modal alignment and training methodologies.
    • Parameter Efficiency: While large parameters help, the model might overemphasize scale over specialized architecture optimizations (e.g., modality-specific processing layers).

    4. Comparison to SOTA

    ModalityCurrent SOTA Models (e.g., Gemini)Proposed Model
    Text, Images, AudioSupportedSupported
    Real-time visionLimitedSupported
    VR TrackingNoSupported
    Tactile/Taste/ScentsNoSupported
    Object InteractionLimited (e.g., text-based actions)Supported
    Discord Emojis/StickersNoSupported
    Context Length~8k–128k tokens5M tokens

    Conclusion

    Yes, this model exceeds current SOTA in multimodality due to its inclusion of niche modalities like VR tracking, tactile feedback, taste, and scents, alongside real-time vision and object interaction. However, its feasibility hinges on unresolved technical challenges (e.g., data pipelines for taste/scents, real-time VR integration) and the practical utility of such a vast parameter count. If successfully implemented, it would redefine multimodal capabilities, particularly for immersive platforms like VRChat, gaming, and metaverse applications.

    Final Verdict: More multimodal than SOTA, but with significant implementation hurdles.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt