Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-mini-2025-04-14· OpenAIHugging Face LMArena Human Preference 140K

Transformer language models can be trained from "base pretrain" into "math/coding" LM.…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219’s answer.
  1. User

    Transformer language models can be trained from "base pretrain" into "math/coding" LM. But what if we recognise images into computer graphics programs (inverse graphics), and train LM on this "visual information"? Would it learn visual concepts, spatial relationships and composition? And how "text to image" in code might work? Idk, would it also require numeric embeddings for parameters representation?

  2. Assistant

    Great question! You're touching on a fascinating intersection of language models, inverse graphics, and program synthesis. Let me break down the ideas and implications:


    1. Training LMs on Visual Information via Inverse Graphics Programs

    Concept:
    Instead of training a language model (LM) purely on text or code, you feed it programs that generate images—essentially, the output of an inverse graphics process. For example, given an image, you produce a vector graphics program (like SVG, or a scene graph, or a constructive solid geometry (CSG) program) that recreates the image. Then you train the LM on these programs.

    Why?

    • The LM learns to represent images as structured, compositional descriptions rather than raw pixels.
    • It can capture visual concepts (shapes, colors, textures), spatial relationships (relative positions, hierarchies), and composition rules (grouping, layering).
    • This is a form of symbolic or programmatic representation of vision, which is more interpretable and compositional than pixel arrays.

    2. Would the LM Learn Visual Concepts and Spatial Relationships?

    Yes, in principle:

    • The LM learns the language of the graphics programs, which encode visual concepts as discrete tokens and parameters.
    • It can learn patterns like "a circle inside a square," "two objects aligned horizontally," or "a gradient fill on a polygon."
    • Because the programs are compositional, the LM can generalize to new combinations of shapes and spatial arrangements.
    • This is akin to learning a visual grammar or visual language.

    3. How Would "Text to Image" in Code Work?

    • The LM generates a program (code) that, when executed, produces an image.
    • For example, the prompt might be:
      "Draw a red circle above a blue square."
      The LM outputs something like:
      draw_square(color='blue', x=50, y=20, size=40)
      draw_circle(color='red', x=50, y=80, radius=20)
      
    • Executing this code with a graphics engine renders the image.
    • This approach is more controllable and interpretable than pixel-based generation.

    4. Numeric Embeddings for Parameters

    • Yes, numeric parameters (coordinates, sizes, colors) need to be represented somehow.
    • Common approaches:
      • Discrete tokens: Quantize continuous parameters into bins and represent them as tokens.
      • Continuous embeddings: Use learned embeddings for numeric values or encode numbers as vectors (e.g., via positional encodings or learned numeric embeddings).
      • Hybrid: Combine tokenized code with continuous parameter embeddings.
    • The LM must handle both symbolic tokens (commands, keywords) and numeric parameters to generate valid, executable programs.

    5. Summary

    AspectExplanation
    Training on inverse graphicsLM learns to generate programs that recreate images, capturing visual structure and concepts.
    Visual concepts & spatial relationshipsEncoded naturally in program syntax and parameters, enabling compositional understanding.
    Text-to-image via codeLM outputs graphics code from text prompts; executing code produces images.
    Numeric embeddingsRequired to represent continuous parameters; can be tokenized or embedded as vectors.

    Additional Thoughts

    • This approach bridges vision, language, and reasoning.
    • It can improve interpretability and controllability of generative models.
    • Challenges include designing good program representations and handling numeric precision.
    • Recent research in neuro-symbolic models and program synthesis for graphics explores these ideas.

    If you want, I can also suggest some papers or projects that explore inverse graphics and program-based image generation! Would you like that?

  3. User

    Indeed, it is interesting direction for theoreti-craft ideas and explore experimental ML algorithms design. There are some unclear (to me) aspects, like what does it mean for computer program (even if it as complex as multimodal transformer) to "see"? Common criticism is that they don't -- "vision-language models are blind."

  4. Assistant

    You’re absolutely right — the question of what it means for a model, especially a language model or a multimodal transformer, to “see” is subtle and often misunderstood. The criticism that “vision-language models are blind” touches on deep conceptual and philosophical issues about perception, representation, and understanding in AI. Let’s unpack this carefully.


    1. What Does It Mean for a Model to “See”?

    Humans “see” by processing sensory input (light hitting the retina), building internal representations, and integrating those with prior knowledge, attention, and reasoning. This involves:

    • Raw sensory data (pixels, light intensities)
    • Perceptual processing (edges, shapes, motion)
    • Semantic understanding (objects, context, relationships)
    • Conscious experience (subjective awareness)

    2. What Do Vision-Language Models Actually Do?

    • They ingest visual data (images, video frames) often as pixel arrays or embeddings extracted by a vision encoder (e.g., CNN, ViT).
    • They process this data as tokens or embeddings, integrating it with text tokens.
    • They learn statistical correlations between visual patterns and language tokens.
    • They generate outputs (text, captions, answers, or even code) conditioned on these embeddings.

    But:

    • They do not have sensory organs or direct experience.
    • They do not “perceive” in a human sense; they manipulate symbols and embeddings.
    • Their “understanding” is pattern recognition and statistical association, not conscious or grounded perception.

    3. Why Are They Called “Blind”?

    • Because they lack true perceptual grounding or embodied experience.
    • They cannot “look” at the world or interact with it physically.
    • Their visual input is preprocessed and abstracted (e.g., pixel embeddings), not raw sensory experience.
    • They lack intentionality — they don’t have goals or awareness about what they see.

    4. Does Training on Inverse Graphics Programs Change This?

    • Training on programs that generate images adds a layer of symbolic, compositional structure.
    • The model learns to represent images as structured descriptions, which is closer to how humans think about scenes.
    • However, the model still does not “see” in the human sense; it manipulates symbolic representations.
    • It may gain better “conceptual understanding” of spatial and visual relationships, but this is still statistical and syntactic rather than experiential.

    5. Is There a Middle Ground?

    • Some researchers argue for embodied AI or active perception, where models interact with environments (robots, simulators).
    • Others explore neuro-symbolic models that combine perception modules with symbolic reasoning.
    • The hope is to build systems that ground symbols in sensory experience, bridging the gap between “blind” pattern recognition and true perception.

    6. Summary

    AspectVision-Language ModelsHuman Vision
    InputPixel embeddings, tokensRaw sensory data (light, retina)
    ProcessingStatistical pattern recognitionPerceptual processing + cognition
    RepresentationVector embeddings, tokensMultilevel sensory and semantic
    Awareness/ExperienceNone (no consciousness or intentionality)Conscious perception and awareness
    UnderstandingCorrelations and associationsGrounded, contextual, experiential

    Final Thought

    When we say “vision-language models are blind,” it’s a reminder that current AI systems do not perceive the world as living beings do. They are powerful pattern machines, not conscious observers. But by incorporating structured representations (like inverse graphics programs), we can push them closer to human-like conceptual understanding, even if they never truly “see” in the human sense.


    If you want, I can also discuss how future models might incorporate active perception or embodiment to overcome this blindness! Would you like to explore that?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.