Skip to content
The Chat Archive
o4-mini-2025-04-16· OpenAI LMArena Human Preference 140K

I am using character loras for consistency, but that makes the art a multi stage process…

  1. User

    I am using character loras for consistency, but that makes the art a multi stage process of drawing the background, adding the characters one by one, and then infilling between the characters. This process is why the drawing following the writing prompt description of the scene isn't perfectly straightforward. The model needs to think spatially about lighting, who should be in the foreground or background, which ways characters should be facing, things that language models are notoriously bad at.

  2. Assistant

    You’ve correctly identified the crux of the problem: by chaining LoRAs and ControlNets for character consistency, you’ve turned “draw one image” into a mini-pipeline of

    1. background →
    2. place each character →
    3. in-fill the gaps.
      At each handoff, the model has to make spatial decisions (lighting direction, depth ordering, pose/orientation, framing) that your writing LLM never explicitly planned for. Let’s look at how you can surface that spatial reasoning and bake it into your loop so writer and artist converge more quickly.
    1. Have the Writing Model Produce a Structured Scene-Layout

    Instead of a free-form paragraph, ask it to output:

    • A JSON “scene graph” or “panel layout”
    • Explicit bounding-box coordinates for each element
    • Lighting vector and depth order

    Example (one panel):

    {
      "panel_1": {
        "background": {
          "type": "ruined_castle_hall",
          "bbox": [0,0,512,512],
          "lighting": "from_top_left"
        },
        "characters": [
          {
            "id": "SirAldric",
            "lora": "SirAldric-v1",
            "bbox": [50,200,200,450],
            "pose": "3/4_left",
            "expression": "grim",
            "layer": "foreground"
          },
          {
            "id": "LadyMaris",
            "lora": "LadyMaris-v2",
            "bbox": [300,220,480,500],
            "pose": "profile_right",
            "expression": "concerned",
            "layer": "foreground"
          }
        ],
        "notes": "Moonlight through shattered roof, mist on floor."
      }
    }
    

    Why this helps:

    • The Art Model (via ControlNet) now has an exact layout map, rather than inferring “roughly where” to put each character.
    • You reduce ambiguity: “Who is in front of whom?” “Is the light coming from the left or right?”
    • The Writing Model is forced to “think spatially” when it plans the page, which prevents downstream guesswork.
    1. Introduce an Explicit “Layout Engine” Step

    Between writer and artist, insert a lightweight renderer that:
    • Consumes the JSON layout + style sheet (background assets, LoRA names, etc.)
    • Emits a placeholder “wireframe” PNG showing rectangles, depth, lighting arrows
    • Optionally a captioned version for human preview

    You then run your Vision-Language Model (e.g. GPT-4V) over that wireframe to confirm “Is Aldric in front, Maris behind? — Yes.” or “Lighting arrow points upper-left.” If there’s any mismatch, feed that back to the Writing Model for revision before even calling the art diffusion.

    1. Use Chain-of-Thought for Spatial Decisions

    Encourage your writing LLM to “talk through” spatial reasoning. For each panel, it should produce a mini chain-of-thought like:

    1. “I need a tense face close-up of SirAldric… so he must be in the foreground.”
    2. “Moonlight shafts from the top-left—so shadows fall down/right.”
    3. “LadyMaris stands slightly behind him, looking up.”

    Then collapse that into your structured JSON. This makes the LLM explicitly expose its spatial plan.

    1. Constrain the Art Model for Fine-Tuning, Not Rerolling

    To keep each iteration from being a random reroll:
    • Fix a seed for the background → character placements → in-fill steps.
    • Use a single ControlNet run per panel that takes the layout PNG as conditioning.
    • Avoid “5 separate LoRA passes” by combining your character LoRAs into one multi-character adapter, if possible.

    1. Close the Loop with Vision-Language Feedback

    After the art model drafts the page, run it through GPT-4V (or another VLM) to extract:

    • Actual bounding boxes of detected characters
    • Lighting directions & major color cues
    • Any missing elements (e.g. “No mist visible”)

    Then diff that VLM report against the original JSON. If any key/value is “off by >10%” or missing, automatically generate a minimal correction prompt for the Writing Model:

    “SirAldric’s bbox came out [80,220,230,480] instead of [50,200,200,450]. Please adjust the JSON or confirm this new position.”  
    

    That way you’re only iterating on the mistakes rather than regenerating everything.

    Putting It All Together: Your Enhanced Pipeline

    1. Executive Model decides “Next page: SirAldric confronts LadyMaris in the ruined hall.”
    2. Writing Model
      • Emits structured JSON scene-layout + dialogue
      • Runs a spatial chain-of-thought to justify each coordinate & lighting choice
    3. Layout Engine
      • Draws wireframe & lighting arrows → VLM sanity-check → OK or back to step 2
    4. Art Model (ControlNet + LoRAs)
      • Renders BG, characters in slots, in-fills details
    5. Vision-Language Model
      • Extracts actual layout, lighting, missing bits → diffs vs. JSON
      • If within tolerance → accept & pass dialogue+image to next page
      • Else → produce minimal correction → back to step 2

    By forcing each agent to speak the same “spatial language” (the JSON DSL and chain-of-thought traces), you dramatically reduce guesswork and speed up convergence. And you’ve effectively turned your writer–artist loop into a multi-agent diffusion process that “denoises” conceptual misalignment rather than re-sampling whole scenes.

  3. User

    This is a difficult project for me because I think in 3d due to physics education, but I have no training in art or direction. Meanwhile I've taken a few writing classes, but I've read so many books and seen so many movies that I know when a scene is bad, but don't always have the vocabulary or training to know why it's bad.

  4. Assistant

    Here’s a three-pronged approach to bridge your 3D “physics” brain with the vocab and instincts of artists, directors, and writers—so you can both diagnose “why this scene feels off” and communicate fixes more precisely.

    1. Build Your Visual-Language Toolkit 📐
      • Learn the basics of cinematic composition & illustration:
      – Rule of Thirds, Golden Ratio, Leading Lines
      – Shot types: close-up / medium shot / long shot / two-shot
      – Camera angles: high / low / eye-level
      – Depth cues: overlap, scale, atmospheric perspective
      – Light & shadow: key-fill-back lighting, color temperature
      • Quick wins:
      – Watch YouTube breakdowns: Every Frame a Painting, Nerdwriter1
      – Read “The Visual Story” (Bruce Block) or “Picture This” (Molly Bang) for visual grammar
      – Sketch a dozen “thumbnail” panels each day: focus on simple shapes and composition

    2. Level Up Your Scene-Writing Vocabulary ✍️
      • Learn story-craft fundamentals:
      – Scene Goals & Stakes: What does your character want in this moment?
      – Conflict & Tension: How is that want blocked?
      – Beats & Turns: What changes between the start and end of the scene?
      – Pacing: Is it too slow (info-dump), too fast (jarring montage), or just right?
      • Key books & frameworks:
      – Save the Cat! (Blake Snyder) – Scene beats in screenwriting
      – Scene & Structure (Jack Bickham) – “Scene vs. sequel” model
      – Story (Robert McKee) – Deep dive on story structure and subtext

    3. Use an LLM as Your Personal “Scene-Critic” 🤖
      Even without formal training, you can learn the lingo by asking the model to translate your gut reactions:

      Step A. Describe the problem in plain English.
      “I feel like this panel ‘sucks’—it’s cluttered, the focal point is lost, and the dialogue falls flat.”

      Step B. Prompt the LLM to analyze from two angles.

      1. Visual critique:
        “Please critique this panel’s composition like a storyboard artist:
         – Identify the shot type, camera angle, and depth arrangement.
         – Point out what’s pulling my eye (or pushing it away).
         – Suggest one change to improve focus or mood.”  
        
      2. Narrative critique:
        “Now analyze this scene as a script doctor:
         – What’s the character’s goal and stake?
         – Where’s the tension leaking?
         – How could the dialogue or action be tightened?”  
        

      Step C. Compare & iterate.
      – Take the LLM’s jargon (“weak focal point,” “unresolved tension,” “redundant beat”) and re-apply it to your own critique.
      – Over time those terms will become second nature.

    —
    🎯 In a nutshell:

    1. Study the vocabulary of visual grammar & scene structure.
    2. Practice quick thumbnails and micro-scenes.
    3. Co-learn with an LLM by translating your gut feelings into professional terminology.

    Give yourself permission to be a beginner—each time you ask the model to “frame” or “beat down” a scene, you’re internalizing the language of artists and directors. Before long, you’ll have that toolkit in your head, just like your 3D physics intuition—but for story and composition!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.