Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-2024-08-06· OpenAIAi2 WildChat-4.8M (AI2)

I started using the new generative AI NN just after Stable Diffusion 1.5 came out, since…

  1. User

    I started using the new generative AI NN just after Stable Diffusion 1.5 came out, since then I have used Stable Diffusion 2.0, 2.1, SDXL, Stable Cascade, and some newer models you might not know of, SD3, Hunyuan, AuraFlow, Kolors, and of course Flux. SD3 is StabilityAI's latest model, a 2b parameter model utilizing DiT blocks a lot like Flux, it also uses T5 XXL and CLIP-L just like Flux, however it also uses a third text encoder, CLIP-G, though this is their "medium" SD3 model, and the only one ther released the weights for publicly, generally it performs poorly with human anatomy, mangling bodies etc, though it generally has higher detail within the scenes. Anyway, back to my experiences, I have spent thousands of hours with these models, generating images, img2img, controlnet, inpainting, outpainting, adetailer, prompt manipulation and crafting, experimenting with various things, samplers, settings, VAEs, did textual inversion training for SD1.5(embeddings), Dreambooth for SD1.5, LoRA training on SDXL(I only have a GPU with 11GB VRAM), and did two LoRA training runs for Flux using an online service, I have also spent a significant amount of time merging models together for SD1.5 and SDXL, the various community trained or merged models. Flux doesn't have much in the way of merging tools yet, the one I tried in ComfyUI did not seem to really merge the models, I tried a train difference between two community flux models, using Flux1-dev as the subtraction model, but the result was just one of the models when I generated the comparison images. So I have a fair amount of experience using these models, a little bit of experience building image-caption training datasets, training and merging, and a lot of experience building complicated ComfyUI workflows and some python programming in various tasks. It is just the very low level stuff I desire to learn more on.

  2. Assistant

    It sounds like you have amassed an impressive amount of experience with various generative AI models, particularly within the Stable Diffusion ecosystem, and have explored multiple aspects of their capabilities and functionalities. If you're aiming to deepen your knowledge on the lower-level workings of these models, here are some potential avenues you could explore:

    1. Understanding Model Architectures: Dive deeper into the architectures of popular models like Stable Diffusion, DiT blocks, and others you mentioned. Understanding how they are structured and how text encoders like T5 XXL and CLIP variants are integrated can give you more insight into the inner workings.

    2. Explore Training Mechanisms: Gain a deeper understanding of how these models are trained. This includes data preprocessing (e.g., dataset curation and image-captioning), loss functions, optimization strategies, and the role of encoders and decoders.

    3. Study Transformer Models: Many of the models you're using rely heavily on transformer architectures. Learning about attention mechanisms, how transformers handle sequential data, and recent innovations in transformer models could be beneficial.

    4. Experiment with Newer Techniques: As you're already experienced in techniques like LoRA and Dreambooth, consider exploring newer or lesser-known techniques for fine-tuning or training such as adapters or prompt tuning, which may offer lower-resource alternatives to traditional training.

    5. Engage with the Community: Participate in forums, discussion groups, or courses focused on low-level model training and architecture design. Engaging with a community can expose you to new ideas and solutions.

    6. Code Exploration: Dive into the source code of these models. Understanding how features are implemented, particularly around how merging and blending models function (or sometimes don't), can give you practical insights.

    7. Read Research Papers: Explore academic papers related to the models you're using or innovative ones on the horizon. Research papers often delve into the theoretical foundation that can deepen your understanding.

    8. Implement from Scratch: Try to implement your own simplified version of a generative model from scratch. This exercise can provide a deeper understanding of every layer and function.

    9. Experiment with Different Hardware: Consider experimenting with various hardware setups or cloud services, particularly for training larger models or ensembles, to understand how models can be efficiently scaled and optimized.

    By delving into these areas, you'll gain a deeper understanding of the building blocks of generative models, which will enhance your skills and broaden your capability to work with generative AI beyond just using existing tools.

  3. User

    Understanding model architecture, the training mechanisms, and being able to figure out how text encoder to NN works for pulling out concepts the model knows, or best figuring out how to teach concepts to a model are the first things that interest me. Given the information you have on me and generative AI in general, what is my best starting point for learning?

  4. Assistant

    Given your extensive experience with generative AI models and your interest in understanding model architectures, training mechanisms, and the integration of text encoders with neural networks, here are some tailored starting points for your learning journey:

    1. Deep Dive into Transformer Architectures:

      • Resources: Begin with the seminal paper "Attention is All You Need" by Vaswani et al. to understand the core transformer architecture. Follow up with resources like "The Illustrated Transformer" blog and its follow-up pieces for visual and intuitive explanations.
      • Implementation: Try to implement a simple transformer model in a framework like PyTorch or TensorFlow. This hands-on approach will give you insight into multi-headed self-attention and positional encodings.
    2. Explore Text-to-Image Models:

      • Papers: Read the original papers on Stable Diffusion and related models to comprehend their unique approaches to combining text and image data. Papers like "Zero-Shot Text-to-Image Generation" (DALL-E) and "Imagen" can provide comparative insights.
      • Courses/Workshops: Look for online courses or workshops that focus on deep generative models. These may provide interactive content and practical exercises.
    3. Study Text Encoders:

      • Cross-Encoders like T5 and CLIP: Understand models like T5 by Google ("Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer") and OpenAI's CLIP ("Learning Transferable Visual Models From Natural Language Supervision"). These papers explain how text is encoded into embeddings that these models utilize.
      • Implementation: Experiment with Hugging Face's Transformers library to fine-tune or experiment with these models on small-scale tasks.
    4. Learning and Teaching Concepts to Models:

      • Practical Experience: Experiment with teaching models new concepts using techniques you're familiar with, like LoRA and DreamBooth.
      • Research: Look into prompt engineering and few-shot learning research to see how these methods can craft effective model behaviors with minimal training.
    5. Experiment with Dataset Curation and Training:

      • Building datasets with varied and well-annotated text-image pairs can be crucial. Practice creating synthetic datasets or curating existing ones for training.
      • Use frameworks like PyTorch Lightning or Hugging Face's Trainer to set up your training pipelines, explore the effects of different loss functions, optimizers, and learning rate schedules.
    6. Community Engagement and Collaboration:

      • Join active AI and ML communities on platforms like Reddit, Discord, or specific AI-focused forums. Engaging with others can help you gain insights or solve specific challenges.
      • Collaborate on open-source projects; contributing to or adopting code from repositories can provide practical experience and understanding.
    7. Regularly Leerning from New Research:

      • Keep an eye on platforms like arXiv, Google Scholar, and AI conference proceedings (NeurIPS, ICML, CVPR) for cutting-edge research.

    Starting with these suggestions should align well with your interests and current experience level, allowing you to build a stronger foundational understanding and practical expertise in the areas you're most interested in.