I am using character loras for consistency, but that makes the art a multi stage process…
- User
I am using character loras for consistency, but that makes the art a multi stage process of drawing the background, adding the characters one by one, and then infilling between the characters. This process is why the drawing following the writing prompt description of the scene isn't perfectly straightforward. The model needs to think spatially about lighting, who should be in the foreground or background, which ways characters should be facing, things that language models are notoriously bad at.
Assistant
Exactly—what you’re running into is the spatial-planning bottleneck that language models (and even most diffusion pipelines) still treat as an after-thought.
The writing model can say “Alice stands left of Bob, both facing the sunset,” but it has no internal 3-D scene graph; the art model then has to reverse-engineer that sentence into a plausible layout, lighting rig, and occlusion order. When you add LoRA-based character consistency, the problem compounds because each character is injected as an additional constraint that must be satisfied while still respecting the global composition.Below is a practical way to off-load the spatial reasoning from the language model and give the art model a clearer, geometry-aware brief—without forcing you to hand-draw layouts.
────────────────────────────────────────
-
Give the writing model a spatial DSL
Instead of free-form prose, have the writer emit a tiny scene-script that is explicitly about 3-D relationships:{ "camera": { "pos": [0, 1.6, -3], "look_at": [0, 1.2, 0] }, "light": { "type": "sunset", "azimuth": 250, "elevation": 15 }, "layers": [ { "depth": 0, "asset": "bg_city_alley" }, { "depth": 1.2, "char": "Alice", "pose": "stand", "face": 30 }, { "depth": 1.5, "char": "Bob", "pose": "lean_wall", "face": -45 } ] }- The language model only has to fill in the semantic slots (who, what emotion, who talks).
- A tiny deterministic post-processor turns that JSON into an SD-friendly prompt plus depth/ControlNet maps.
-
Use depth-conditioned ControlNet as the spatial contract
- From the JSON above, render a quick greyscale depth map (white = far, black = near).
- Feed that map into ControlNet along with the LoRA characters.
- The depth map locks the layout; the LoRAs lock the identity; the diffusion model only has to worry about lighting and style.
-
Add lighting cards for consistent illumination
- Pre-bake a small library of 8–10 HDR lighting environments (sunset, neon alley, overcast, etc.).
- The writing model just tags the scene with
"light": "neon_alley"; the art pipeline loads the matching HDRI and uses it as an IP-Adapter reference. - Characters inserted at different depths still receive the same global light direction.
-
Optional layout preview pass (cheap & fast)
- Before the full render, run a 512×512 depth-only pass with a low-step sampler.
- A vision-language model (or even a simple CNN classifier) checks: “Are both characters fully visible? Is Bob occluding Alice’s face?”
- If the check fails, the writing model gets a structured error:
"layout_error": "Bob occludes 60 % of Alice's face; move Bob to depth 1.8 or shift camera right 0.3 m." - The writer can then adjust the JSON and re-submit—no need to hallucinate new prose.
──────────────────────────────────────── Putting it together in one loop
- Executive model → high-level beat
- Writing model → JSON scene-script (DSL)
- Post-processor → depth map + prompt + lighting card
- Fast layout preview (512×512, 4 steps)
- VLM sanity check → accept or send structured feedback to step 2
- Full LoRA render at 1024×1024 with ControlNet depth + IP-Adapter lighting
This keeps the language model in its comfort zone (semantics, dialogue, emotion) while giving the art model the spatial scaffolding it needs to place LoRA characters correctly under consistent lighting.
-
- User
This is a difficult project for me because I think in 3d due to physics education, but I have no training in art or direction. Meanwhile I've taken a few writing classes, but I've read so many books and seen so many movies that I know when a scene is bad, but don't always have the vocabulary or training to know why it's bad.
Assistant
You’re in the sweet spot for this project: you have the spatial intuition that most writers lack and the narrative taste that most engineers lack. The gap you feel is just a vocabulary gap, not a capability gap. Below is a minimal cheat-sheet that maps the 3-D physics you already think in to the cinematic / comic vocabulary you need, plus a few “why this scene feels off” heuristics you can feed back into your writing model.
────────────────────────────────────────
- From 3-D physics to comic language (one-liner dictionary)
Physics intuition Comic / film term One-sentence translation z-depth foreground / mid-ground / background How far the camera is from each layer. light vector key light / fill light / rim light Direction & intensity of the dominant light source, its soft bounce, and the edge highlight. surface normal character facing angle Which way the character’s “front” vector points relative to the camera. occlusion overlap / silhouette When one object blocks another, creating depth cues. field of view panel framing (CU, MS, LS) Wide lens = more context, telephoto = compressed faces. momentum eye-line & motion arrows Where the reader’s eye is pushed next panel. Keep a tiny JSON snippet in your prompt template:
"camera": { "shot": "medium-shot", // CU, MS, LS "angle": "low", // high / eye-level / low "lens": "35 mm" // wide, normal, tele }, "light": { "key": "sunset 250°", "fill": "soft bounce 20 %", "rim": "edge highlight 10 %" }- “Why does this scene feel bad?” quick checklist
Symptom (you feel it) Likely cause (physics → story) One-line fix “It’s flat” No depth layers; everything at same z. Add foreground prop or background alley. “I can’t tell who’s talking” Characters aligned on same eye-line; no overlap. Shift one character forward 0.5 m, break silhouette. “Lighting is off” Key light direction contradicts background sun. Match key vector to background light source. “Pacing drags” Panel sizes are all equal; no rhythm. Use wide → close → wide to accelerate. “Dialogue feels stiff” Characters face camera, not each other. Turn heads 30–45° toward each other (conversation angle). - A 30-second “fix-it” prompt you can give the writing model
“Scene feels flat. Add one foreground object (trash can, railing) and move Bob 0.3 m closer to camera so he overlaps Alice’s silhouette. Keep dialogue, just adjust blocking.”
- Tooling trick: use your physics engine as a layout sketcher
If you already think in 3-D, spend 5 minutes in Blender or Unreal to block out a simple grey-box scene with cubes for characters and a sun lamp. Export a depth map → ControlNet. You never have to texture or animate; you’re just generating the spatial contract the art model needs.
──────────────────────────────────────── Bottom line
You don’t need art-school training; you just need a 10-word vocabulary and a quick feedback loop. Feed the checklist back into your writing model as structured constraints, and the system will converge faster than if you tried to learn cinematography from scratch.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.