remove the enghanced prompt code from this: --- Step 0: Install Required Libraries ---…
- User
remove the enghanced prompt code from this:
--- Step 0: Install Required Libraries ---
!pip install -q transformers diffusers accelerate torch safetensors sentencepiece gradio
--- Step 1: Sample Descriptions from Cookie Theft Dataset ---
descriptions = [ "the boy is stealing cookies the jar and he on a stool that falling down and the girl is laughing...", "children are getting into the cookie jar which is on the shelf...", "a boy is handing a cookie to the girl while the stool tips over..." ]
--- Step 2: Use FLAN-T5 to Enhance Prompts ---
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM import torch
model_name = "google/flan-t5-base" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to("cuda" if torch.cuda.is_available() else "cpu")
def clean_prompt(text): prompt = f"Make this prompt detailed and grammatically correct: {text}" inputs = tokenizer(prompt, return_tensors="pt", truncation=True).to(model.device) outputs = model.generate(**inputs, max_new_tokens=128) return tokenizer.decode(outputs[0], skip_special_tokens=True)
Pre-process the example descriptions
cleaned_descriptions = [clean_prompt(d) for d in descriptions] for i, desc in enumerate(cleaned_descriptions): print(f"Prompt {i+1}:", desc)
--- Step 3: Generate 1024x1024 Image using Stable Diffusion XL ---
from diffusers import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, use_safetensors=True ) pipe = pipe.to("cuda" if torch.cuda.is_available() else "cpu")
def generate_image(prompt): image = pipe(prompt=prompt[:300]).images[0] # Limit input for SDXL return image
--- Step 4: Gradio Interface ---
import gradio as gr
def infer(prompt): cleaned = clean_prompt(prompt) image = generate_image(cleaned) return image, cleaned
gr.Interface( fn=infer, inputs=gr.Textbox(label="Enter Cookie Theft Scene Description", lines=4), outputs=[ gr.Image(label="Generated Image (1024x1024)"), gr.Textbox(label="Enhanced Prompt") ], title="Cookie Theft Scene Generator", description="Uses FLAN-T5 for prompt enhancement and SDXL for 1024x1024 image generation." ).launch(share=True)
- Assistant
Here's the code with the enhanced prompt code removed. The prompts are used as-is without FLAN-T5 enhancement:
# --- Step 0: Install Required Libraries --- !pip install -q transformers diffusers accelerate torch safetensors sentencepiece gradio # --- Step 1: Sample Descriptions from Cookie Theft Dataset --- descriptions = [ "the boy is stealing cookies the jar and he on a stool that falling down and the girl is laughing...", "children are getting into the cookie jar which is on the shelf...", "a boy is handing a cookie to the girl while the stool tips over..." ] # --- Step 2: (Removed) -- Using raw prompts directly --- # --- Step 3: Generate 1024x1024 Image using Stable Diffusion XL --- from diffusers import StableDiffusionXLPipeline import torch pipe = StableDiffusionXLPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, use_safetensors=True ) pipe = pipe.to("cuda" if torch.cuda.is_available() else "cpu") def generate_image(prompt): image = pipe(prompt=prompt[:300]).images[0] # Limit input for SDXL return image # --- Step 4: Gradio Interface --- import gradio as gr def infer(prompt): # Use raw prompt directly without enhancement image = generate_image(prompt) return image, prompt gr.Interface( fn=infer, inputs=gr.Textbox(label="Enter Cookie Theft Scene Description", lines=4), outputs=[ gr.Image(label="Generated Image (1024x1024)"), gr.Textbox(label="Input Prompt (no enhancement)") ], title="Cookie Theft Scene Generator", description="Generates 1024x1024 images from Cookie Theft scene descriptions using Stable Diffusion XL." ).launch(share=True)Changes made:
- Removed FLAN-T5 imports and the
clean_promptfunction. - In
infer(), pass the prompt directly togenerate_image. - Outputs show the original prompt without enhancement.
- Removed FLAN-T5 imports and the
- User
we need to find a way to fetch the main characters out of the text that are mentioned in the input prompt. the stable diffusion then uses that prompt to output the image.
- Assistant
To extract the main characters mentioned in the prompt, you can use a simple Named Entity Recognition (NER) or keyword extraction approach. Since your example descriptions are short and about people ("boy", "girl", "children", etc.), a lightweight approach is to use a small NLP model to extract the key noun phrases or character mentions.
Here are two approaches you can try:
Approach 1: Use spaCy for Named Entity and noun chunk extraction
- spaCy supports NER models and noun chunk extraction.
- You can extract PERSON entities or use noun chunks to pick main characters.
Approach 2: Use a prompt-based completion model (e.g., FLAN-T5 or GPT) to extract characters
- Use a prompt like: "Extract the main characters from the text: {input}"
- The model outputs a list of characters. (This requires a hosted model and might be slower.)
I'll provide a straightforward example using spaCy, which you can extend or replace with a transformer-based NER if you want accuracy on more complex descriptions.
Code example with spaCy to extract main characters:
- Install spaCy and English model:
!pip install -q spacy !python -m spacy download en_core_web_sm- Updated code snippet integrating extraction and using those characters to guide Stable Diffusion prompt:
# --- Step 0: Install Required Libraries --- !pip install -q diffusers accelerate torch safetensors gradio spacy !python -m spacy download en_core_web_sm # --- Step 1: Sample Descriptions --- descriptions = [ "the boy is stealing cookies the jar and he on a stool that falling down and the girl is laughing...", "children are getting into the cookie jar which is on the shelf...", "a boy is handing a cookie to the girl while the stool tips over..." ] # --- Step 2: Extract main characters with spaCy --- import spacy nlp = spacy.load("en_core_web_sm") def extract_characters(text): doc = nlp(text) characters = set() # Collect PERSON entities for ent in doc.ents: if ent.label_ == "PERSON": characters.add(ent.text.lower()) # If no PERSON entities found, fallback to noun chunks that sound like people if not characters: for chunk in doc.noun_chunks: # Very rough heuristic: include words like boy, girl, children if any(w.lower_ in ['boy', 'girl', 'children', 'child', 'kid', 'kids', 'man', 'woman', 'men', 'women'] for w in chunk): characters.add(chunk.text.lower()) return list(characters) # --- Step 3: Generate image with characters highlighted in prompt --- from diffusers import StableDiffusionXLPipeline import torch pipe = StableDiffusionXLPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32, use_safetensors=True ) pipe = pipe.to("cuda" if torch.cuda.is_available() else "cpu") def generate_image(prompt): image = pipe(prompt=prompt[:300]).images[0] return image # --- Step 4: Gradio Interface --- import gradio as gr def infer(prompt): characters = extract_characters(prompt) # Optionally add extracted characters info to the prompt to help SDXL if characters: enhanced_prompt = f"{prompt} Main characters: {', '.join(characters)}." else: enhanced_prompt = prompt image = generate_image(enhanced_prompt) return image, enhanced_prompt gr.Interface( fn=infer, inputs=gr.Textbox(label="Enter Cookie Theft Scene Description", lines=4), outputs=[ gr.Image(label="Generated Image (1024x1024)"), gr.Textbox(label="Prompt used for generation") ], title="Cookie Theft Scene Generator with Character Extraction", description="Extracts main characters from prompt and uses them to enhance Stable Diffusion XL generation." ).launch(share=True)
How this works:
extract_characters()usesspaCyto find PERSON entities.- If no PERSON entities found, it heuristically checks noun chunks for common character words (boy, girl, children, etc.)
- These extracted characters are appended to the prompt before generating the image to help the model know the main subjects.
Possible Improvements:
- Use a more advanced/custom NER model tuned for your domain.
- Use transformer models for better extraction.
- Add adjective or appearance descriptors if you want richer images.
- Use prompt engineering or negative prompts to further refine the image output.
If you want, I can help you add one of these more advanced methods!