I want you to make that I first have a tab where I can chat with iamges. and then also a…
- User
I want you to make that I first have a tab where I can chat with iamges. and then also a other tab where I can chat wiht llama3.1 with duckduckgo tool search and a generate_images tool where it can generate images (demo funktion where it just uploads a file called iamge.png). and output the whole code: import gradio as gr from langchain_ollama import ChatOllama from langchain_core.messages import HumanMessage, AIMessage, SystemMessage from langchain_core.tools import tool from langchain_community.tools import DuckDuckGoSearchRun import base64 from io import BytesIO from PIL import Image import os
Define the web search tool using the @tool decorator
@tool def search_web(query: str) -> str: """Use this tool to search the web for real-time information based on a query.""" search = DuckDuckGoSearchRun() return search.run(query)
Initialize the models
llama3.2-vision for image inputs (does not support tools)
llm_image = ChatOllama( model='llama3.2-vision', temperature=0.8, # Other parameters as needed )
llama3.1 with tool calling capabilities for text inputs
llm_text = ChatOllama( model='llama3.1', temperature=0.8, # Other parameters as needed ).bind_tools([search_web])
def convert_image_to_base64(image): """ Convert a PIL Image to a base64 encoded string. """ buffered = BytesIO() image.save(buffered, format="JPEG") img_str = base64.b64encode(buffered.getvalue()).decode("utf-8") return img_str
def chatbot_response(message, history): # Determine if the user provided an image or text has_image = False has_text = False
if isinstance(message, dict): has_image = message.get("image") is not None or message.get("files") is not None has_text = message.get("text") is not None and message.get("text").strip() != "" elif isinstance(message, str): has_text = message.strip() != "" elif isinstance(message, Image.Image): has_image = True # Choose the appropriate model if has_image: llm = llm_image else: llm = llm_text # Prepare the conversation history for the model model_messages = [] def create_content_parts(msg): content_parts = [] if isinstance(msg, dict): if msg.get("text") and msg.get("text").strip() != "": content_parts.append({"type": "text", "text": msg["text"]}) if msg.get("image"): # Handle image as PIL Image object image_b64 = convert_image_to_base64(msg["image"]) content_parts.append({ "type": "image_url", "image_url": f"data:image/jpeg;base64,{image_b64}", }) if msg.get("files"): for file_path in msg["files"]: # Open the image file try: if os.path.exists(file_path): image = Image.open(file_path) image_b64 = convert_image_to_base64(image) content_parts.append({ "type": "image_url", "image_url": f"data:image/jpeg;base64,{image_b64}", }) else: print(f"File not found: {file_path}") except Exception as e: print(f"Error opening image file {file_path}: {e}") elif isinstance(msg, str): if msg.strip() != "": content_parts.append({"type": "text", "text": msg}) elif isinstance(msg, Image.Image): # msg is an image image_b64 = convert_image_to_base64(msg) content_parts.append({ "type": "image_url", "image_url": f"data:image/jpeg;base64,{image_b64}", }) return content_parts # Convert history to the format expected by the model for user_msg, bot_msg in history: # Handle user's previous messages user_content_parts = create_content_parts(user_msg) if user_content_parts: model_messages.append(HumanMessage(content=user_content_parts)) # Handle bot's previous messages if isinstance(bot_msg, str): bot_content_parts = [{"type": "text", "text": bot_msg}] model_messages.append(AIMessage(content=bot_content_parts)) elif isinstance(bot_msg, dict) or isinstance(bot_msg, Image.Image): bot_content_parts = create_content_parts(bot_msg) if bot_content_parts: model_messages.append(AIMessage(content=bot_content_parts)) # Handle the latest message from the user user_content_parts = create_content_parts(message) if user_content_parts: model_messages.append(HumanMessage(content=user_content_parts)) # If using llm_text, add a system message to instruct the assistant if llm == llm_text: system_prompt = ( "You are a helpful assistant. " "You have access to the following tool:\n\n" "1. **search_web**: Use this tool to search the web for real-time information based on a query.\n\n" "Use this tool whenever the user asks for current events or information not stored in your training data. " "Provide accurate and up-to-date answers by utilizing this tool when necessary." ) model_messages.insert(0, SystemMessage(content=system_prompt)) # Generate the response using the model and stream it response = "" for chunk in llm.stream(model_messages): if hasattr(chunk, 'content'): response += chunk.content yield response # Stream the response to Gradio else: continueCreate the Gradio ChatInterface with both text and image inputs
demo = gr.ChatInterface( fn=chatbot_response, multimodal=True, title="Multimodal Chatbot with Tool Support", description=( "Type a message and/or upload an image. The chatbot uses llama3.2-vision when an image is uploaded, " "and llama3.1 with internet access when only text is provided." ), )
Launch the Gradio app
if name == "main": demo.launch()
- Assistant
Here is the modified code that creates two tabs: one for chatting with images using
llama3.2-vision, and another for chatting withllama3.1equipped with both the DuckDuckGo search tool and agenerate_imagestool that, for demonstration purposes, simply uploads a file calledimage.png.import gradio as gr from langchain_ollama import ChatOllama from langchain_core.messages import HumanMessage, AIMessage, SystemMessage from langchain_core.tools import tool from langchain_community.tools import DuckDuckGoSearchRun import base64 from io import BytesIO from PIL import Image import os # Define the web search tool using the @tool decorator @tool def search_web(query: str) -> str: """Use this tool to search the web for real-time information based on a query.""" search = DuckDuckGoSearchRun() return search.run(query) # Define the generate_images tool (demo function) @tool def generate_images(prompt: str) -> str: """Generates an image based on the given prompt. (Demo function that uploads image.png)""" image_path = "image.png" if os.path.exists(image_path): # Return the image in base64 image = Image.open(image_path) image_b64 = convert_image_to_base64(image) # Return the image as a data URL return f"data:image/png;base64,{image_b64}" else: return "Image not found." # Function to convert images to base64 strings def convert_image_to_base64(image): """ Convert a PIL Image to a base64 encoded string. """ buffered = BytesIO() image.save(buffered, format="PNG") img_str = base64.b64encode(buffered.getvalue()).decode("utf-8") return img_str # Initialize the models # llama3.2-vision for image inputs (does not support tools) llm_image = ChatOllama( model='llama3.2-vision', temperature=0.8, # Other parameters as needed ) # llama3.1 with tool calling capabilities for text inputs llm_text = ChatOllama( model='llama3.1', temperature=0.8, # Other parameters as needed ).bind_tools([search_web, generate_images]) # Chatbot response function for the image chat (first tab) def chatbot_image_response(message, history): # Prepare the conversation history for the model model_messages = [] # Convert history to the format expected by the model for user_msg, bot_msg in history: # Handle user's previous messages model_messages.append(HumanMessage(content=user_msg)) # Handle bot's previous messages model_messages.append(AIMessage(content=bot_msg)) # Handle the latest message from the user model_messages.append(HumanMessage(content=message)) # Generate the response using the model and stream it response = "" for chunk in llm_image.stream(model_messages): if hasattr(chunk, 'content'): response += chunk.content yield response # Stream the response to Gradio # Chatbot response function for the text chat with tools (second tab) def chatbot_text_response(message, history): # Prepare the conversation history for the model model_messages = [] # Convert history to the format expected by the model for user_msg, bot_msg in history: # Handle user's previous messages model_messages.append(HumanMessage(content=user_msg)) # Handle bot's previous messages model_messages.append(AIMessage(content=bot_msg)) # Handle the latest message from the user model_messages.append(HumanMessage(content=message)) # Add a system message to instruct the assistant system_prompt = ( "You are a helpful assistant. " "You have access to the following tools:\n\n" "1. **search_web**: Use this tool to search the web for real-time information based on a query.\n" "2. **generate_images**: Use this tool to generate an image based on a prompt.\n\n" "Use these tools whenever appropriate to provide accurate and helpful responses." ) model_messages.insert(0, SystemMessage(content=system_prompt)) # Generate the response using the model and stream it response = "" for chunk in llm_text.stream(model_messages): if hasattr(chunk, 'content'): response += chunk.content yield response # Stream the response to Gradio # Create the ChatInterface for the image chat (first tab) chat_with_images = gr.ChatInterface( fn=chatbot_image_response, multimodal=True, title="Chat with Images", description=( "Chatbot using llama3.2-vision. You can type messages and/or upload images. " "The assistant can understand and discuss the content of the images." ), ) # Create the ChatInterface for the text chat with tools (second tab) chat_with_tools = gr.ChatInterface( fn=chatbot_text_response, title="Chat with Tools", description=( "Chatbot using llama3.1 with internet access and image generation tools. " "You can ask questions that require up-to-date information or request image generation." ), ) # Combine them into tabs demo = gr.TabbedInterface([chat_with_images, chat_with_tools], ["Chat with Images", "Chat with Tools"]) # Launch the Gradio app if __name__ == "__main__": demo.launch()Explanation:
-
Tools Defined:
search_web: Uses DuckDuckGo to search the web based on a query.generate_images: A demo function that returnsimage.pngas a base64 encoded string if it exists.
-
Models Initialized:
llm_image: Usesllama3.2-visionfor the image chat interface.llm_text: Usesllama3.1with bothsearch_webandgenerate_imagestools bound for the text chat interface.
-
Chatbot Response Functions:
chatbot_image_response: Handles the conversation in the first tab, usingllm_image.chatbot_text_response: Handles the conversation in the second tab, usingllm_textwith the tools.
-
Gradio Interfaces:
chat_with_images: The first tab where users can chat and upload images.chat_with_tools: The second tab where users can chat with access to web search and image generation.
-
Tabbed Interface:
- Combines both chat interfaces into a single Gradio app with two tabs.
Notes:
- Make sure to have an
image.pngfile in the same directory as your script for thegenerate_imagestool to work. - Ensure all necessary dependencies are installed and properly configured.
- The
multimodal=Trueparameter enables both text and image inputs in the first tab. - Adjust the model parameters (
temperature, etc.) as needed for your specific use case.
Usage:
-
First Tab ("Chat with Images"):
- You can type messages, upload images, or both.
- The assistant will process the inputs using
llama3.2-vision.
-
Second Tab ("Chat with Tools"):
- Interact with the assistant using text.
- The assistant can perform web searches and generate images upon request.
- Example prompts:
- "What's the latest news on renewable energy?"
- "Generate an image of a futuristic city skyline."
Launching the App:
Run the script, and Gradio will launch the web interface where you can interact with both chatbots.
python your_script_name.pyReplace
your_script_name.pywith the actual name of your Python script file. -