Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-2.1· AnthropicHugging Face Arena Human Preference 55K

Why are generative models so bad with generating scenes with several figures?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4-0613’s answer.
  1. User

    Why are generative models so bad with generating scenes with several figures?

  2. Claude Assistant

    I do not have enough context to definitively state why generative models struggle with generating scenes with multiple figures. However, some potential reasons include:

    • Model architecture limitations. Many generative models like GANs and VAEs generate images pixel-by-pixel or patch-by-patch. Capturing complex relationships between multiple figures may require more sophisticated architectures.

    • Lack of training data. Generating coherent multi-figure scenes requires seeing many examples to learn all the possible interactions and configurations. Many datasets are still limited in the diversity of multi-figure scenes.

    • Evaluation challenges. It can be hard to systematically evaluate quality issues with generating multiple figures as opposed to simpler metrics for single objects or people. Without proper evaluation, it's difficult to make progress.

    • Compositional complexity. The possible combinations and articulations grow exponentially as you add more figures. Even if individual figures look good, the joint scenes may suffer from odd poses or placements as complexity increases.

    • Entanglement issues. Features between figures may get entangled within generative models, leading to issues if one figure changes in appearance or pose that improperly affects other figures. Disentangling within scenes is an open problem.

    But there are also efforts to improve multi-figure generation using techniques like attention mechanisms, structured/compositional approaches, and better benchmarks and evaluation metrics that may lead to progress in the future. The challenges remain open research questions in generative modeling.

  3. User

    Would not generating intermediate 3d composition and than using it as controlnet leads to better results?

  4. Claude Assistant

    Yes, using an intermediate 3D composition step could potentially improve multi-figure scene generation results for generative models. Some key advantages of generating an intermediate 3D scene include:

    • Better handling of occlusions and depths between multiple figures. The 3D scene explicitly models occlusion, depth, and figure-to-figure interactions.

    • Consistent lighting and shadows. A 3D scene solver will produce lighting and shadows that are physically coherent between all the figures. This can improve realism.

    • Explicit control over positioning and poses. Manipulating posed 3D characters allows explicitly defining the positioning and articulations of all figures rather than relying on a model to get this right.

    • Re-rendering from multiple views. Having an intermediate 3D representation allows re-rendering the final result from multiple camera angles in a consistent way.

    Using the intermediate 3D scene as control input into an image generation model like a GAN could help overcome some of the compositional and coherence challenges faced when generating complex multi-figure scenes directly from scratch. The model can focus on rendering, textures, colors and fine details rather than getting the structure right. Progress in areas like neural rendering and novel view synthesis supports the promise of this hybrid 3D-to-2D generation approach for improving result quality.

  5. User

    We can generate 3d composition using neural net from text too. Also it could potentially help with issues like number of hands and legs and so on - in 2d picture hand or leg can be hidden, which confuse neural net during training. In the 3d composition there always should be 2 of each, even if they are not visible in the resulting 2d picture, which should led to better training results

  6. Claude Assistant

    Excellent point! Yes, using a text-to-3D-scene generator to create the intermediate 3D composition could also help alleviate coherence issues for multi-figure scene generation. As you noted:

    • Generating the 3D scene directly from text descriptions allows leveraging language model capabilities to get semantic scene understanding and high-level composition right.

    • Having an explicit 3D model means things like correct limb counts are inherently enforced even if not visible in the final rendered 2D image. This acts as an inductive bias for downstream training.

    • Text conditions allow easy experimentation and scaling to new scenes by generating new text prompts.

    This addresses many of the entanglement issues in purely 2D generation. By factorizing the problem into discrete steps - text-to-3D-scene-to-images - it becomes more modular and amenable to evaluation and tweaking at each stage.

    Recent work in areas like text-to-3D-graphics shows promising results for this generation pipeline. Combining the strengths of language, 3D structuring, and 2D rendering via neural graphics pipelines is likely to give superior coherent multi-figure results compared to end-to-end 2D-only approaches. The explicitly imposed 3D consistency, especially around body/limb counts, is a great advantage.

  7. User

    Even if we want anime or comix as final result, it should be possible just to apply needed style when we are generating image from 3d model

  8. Claude Assistant

    You're absolutely right. Using a 3D intermediate scene representation could benefit even stylized multi-figure generative outputs like anime or comics:

    • The 3D scene provides underlying pose, shape and composition consistency even if the final output is non-photorealistic in style.

    • Complex effects like cell-shaded outlines or manga shading patterns still respect physical interactions between figures grounded in the 3D space. Lighting and occlusions translate over.

    • Can leverage recent advances in neural stylization and rendering that transform photorealistic 3D to painterly/comic styles while preserving important spatial structure.

    • The 3D model acts as a "common canvas" to apply different rendering styles - anime, line drawings, painterly etc. Flexibly generates outputs in desired styles.

    • Stylization trained on existing anime/comics may fail to extrapolate to new poses or angles. 3D helps generalization by providing novel views.

    So while the final output style may be non-photorealistic, leveraging an intermediate coherent 3D foundation allows better generalization and consistency in the style translation process. It provides a strong prior for appearance variations or abstraction while preserving structural and compositional integrity. This will be key for robust generation of multi-figure stylized scenes.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.