Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

I want you to make that I first have a tab where I can chat with iamges. and then also a…

  1. User

    I want you to make that I first have a tab where I can chat with iamges. and then also a other tab where I can chat wiht llama3.1 with duckduckgo tool search and a generate_images tool where it can generate images (demo funktion where it just uploads a file called iamge.png). and output the whole code: import gradio as gr from langchain_ollama import ChatOllama from langchain_core.messages import HumanMessage, AIMessage, SystemMessage from langchain_core.tools import tool from langchain_community.tools import DuckDuckGoSearchRun import base64 from io import BytesIO from PIL import Image import os

    Define the web search tool using the @tool decorator

    @tool def search_web(query: str) -> str: """Use this tool to search the web for real-time information based on a query.""" search = DuckDuckGoSearchRun() return search.run(query)

    Initialize the models

    llama3.2-vision for image inputs (does not support tools)

    llm_image = ChatOllama( model='llama3.2-vision', temperature=0.8, # Other parameters as needed )

    llama3.1 with tool calling capabilities for text inputs

    llm_text = ChatOllama( model='llama3.1', temperature=0.8, # Other parameters as needed ).bind_tools([search_web])

    def convert_image_to_base64(image): """ Convert a PIL Image to a base64 encoded string. """ buffered = BytesIO() image.save(buffered, format="JPEG") img_str = base64.b64encode(buffered.getvalue()).decode("utf-8") return img_str

    def chatbot_response(message, history): # Determine if the user provided an image or text has_image = False has_text = False

    if isinstance(message, dict):
        has_image = message.get("image") is not None or message.get("files") is not None
        has_text = message.get("text") is not None and message.get("text").strip() != ""
    elif isinstance(message, str):
        has_text = message.strip() != ""
    elif isinstance(message, Image.Image):
        has_image = True
    
    # Choose the appropriate model
    if has_image:
        llm = llm_image
    else:
        llm = llm_text
    
    # Prepare the conversation history for the model
    model_messages = []
    
    def create_content_parts(msg):
        content_parts = []
        if isinstance(msg, dict):
            if msg.get("text") and msg.get("text").strip() != "":
                content_parts.append({"type": "text", "text": msg["text"]})
            if msg.get("image"):
                # Handle image as PIL Image object
                image_b64 = convert_image_to_base64(msg["image"])
                content_parts.append({
                    "type": "image_url",
                    "image_url": f"data:image/jpeg;base64,{image_b64}",
                })
            if msg.get("files"):
                for file_path in msg["files"]:
                    # Open the image file
                    try:
                        if os.path.exists(file_path):
                            image = Image.open(file_path)
                            image_b64 = convert_image_to_base64(image)
                            content_parts.append({
                                "type": "image_url",
                                "image_url": f"data:image/jpeg;base64,{image_b64}",
                            })
                        else:
                            print(f"File not found: {file_path}")
                    except Exception as e:
                        print(f"Error opening image file {file_path}: {e}")
        elif isinstance(msg, str):
            if msg.strip() != "":
                content_parts.append({"type": "text", "text": msg})
        elif isinstance(msg, Image.Image):
            # msg is an image
            image_b64 = convert_image_to_base64(msg)
            content_parts.append({
                "type": "image_url",
                "image_url": f"data:image/jpeg;base64,{image_b64}",
            })
        return content_parts
    
    # Convert history to the format expected by the model
    for user_msg, bot_msg in history:
        # Handle user's previous messages
        user_content_parts = create_content_parts(user_msg)
        if user_content_parts:
            model_messages.append(HumanMessage(content=user_content_parts))
    
        # Handle bot's previous messages
        if isinstance(bot_msg, str):
            bot_content_parts = [{"type": "text", "text": bot_msg}]
            model_messages.append(AIMessage(content=bot_content_parts))
        elif isinstance(bot_msg, dict) or isinstance(bot_msg, Image.Image):
            bot_content_parts = create_content_parts(bot_msg)
            if bot_content_parts:
                model_messages.append(AIMessage(content=bot_content_parts))
    
    # Handle the latest message from the user
    user_content_parts = create_content_parts(message)
    if user_content_parts:
        model_messages.append(HumanMessage(content=user_content_parts))
    
    # If using llm_text, add a system message to instruct the assistant
    if llm == llm_text:
        system_prompt = (
            "You are a helpful assistant. "
            "You have access to the following tool:\n\n"
            "1. **search_web**: Use this tool to search the web for real-time information based on a query.\n\n"
            "Use this tool whenever the user asks for current events or information not stored in your training data. "
            "Provide accurate and up-to-date answers by utilizing this tool when necessary."
        )
        model_messages.insert(0, SystemMessage(content=system_prompt))
    
    # Generate the response using the model and stream it
    response = ""
    for chunk in llm.stream(model_messages):
        if hasattr(chunk, 'content'):
            response += chunk.content
            yield response  # Stream the response to Gradio
        else:
            continue
    

    Create the Gradio ChatInterface with both text and image inputs

    demo = gr.ChatInterface( fn=chatbot_response, multimodal=True, title="Multimodal Chatbot with Tool Support", description=( "Type a message and/or upload an image. The chatbot uses llama3.2-vision when an image is uploaded, " "and llama3.1 with internet access when only text is provided." ), )

    Launch the Gradio app

    if name == "main": demo.launch()

  2. Assistant

    Here is the modified code that creates two tabs: one for chatting with images using llama3.2-vision, and another for chatting with llama3.1 equipped with both the DuckDuckGo search tool and a generate_images tool that, for demonstration purposes, simply uploads a file called image.png.

    import gradio as gr
    from langchain_ollama import ChatOllama
    from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
    from langchain_core.tools import tool
    from langchain_community.tools import DuckDuckGoSearchRun
    import base64
    from io import BytesIO
    from PIL import Image
    import os
    
    # Define the web search tool using the @tool decorator
    @tool
    def search_web(query: str) -> str:
        """Use this tool to search the web for real-time information based on a query."""
        search = DuckDuckGoSearchRun()
        return search.run(query)
    
    # Define the generate_images tool (demo function)
    @tool
    def generate_images(prompt: str) -> str:
        """Generates an image based on the given prompt. (Demo function that uploads image.png)"""
        image_path = "image.png"
        if os.path.exists(image_path):
            # Return the image in base64
            image = Image.open(image_path)
            image_b64 = convert_image_to_base64(image)
            # Return the image as a data URL
            return f"data:image/png;base64,{image_b64}"
        else:
            return "Image not found."
    
    # Function to convert images to base64 strings
    def convert_image_to_base64(image):
        """
        Convert a PIL Image to a base64 encoded string.
        """
        buffered = BytesIO()
        image.save(buffered, format="PNG")
        img_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
        return img_str
    
    # Initialize the models
    # llama3.2-vision for image inputs (does not support tools)
    llm_image = ChatOllama(
        model='llama3.2-vision',
        temperature=0.8,
        # Other parameters as needed
    )
    
    # llama3.1 with tool calling capabilities for text inputs
    llm_text = ChatOllama(
        model='llama3.1',
        temperature=0.8,
        # Other parameters as needed
    ).bind_tools([search_web, generate_images])
    
    # Chatbot response function for the image chat (first tab)
    def chatbot_image_response(message, history):
        # Prepare the conversation history for the model
        model_messages = []
    
        # Convert history to the format expected by the model
        for user_msg, bot_msg in history:
            # Handle user's previous messages
            model_messages.append(HumanMessage(content=user_msg))
            # Handle bot's previous messages
            model_messages.append(AIMessage(content=bot_msg))
    
        # Handle the latest message from the user
        model_messages.append(HumanMessage(content=message))
    
        # Generate the response using the model and stream it
        response = ""
        for chunk in llm_image.stream(model_messages):
            if hasattr(chunk, 'content'):
                response += chunk.content
                yield response  # Stream the response to Gradio
    
    # Chatbot response function for the text chat with tools (second tab)
    def chatbot_text_response(message, history):
        # Prepare the conversation history for the model
        model_messages = []
    
        # Convert history to the format expected by the model
        for user_msg, bot_msg in history:
            # Handle user's previous messages
            model_messages.append(HumanMessage(content=user_msg))
            # Handle bot's previous messages
            model_messages.append(AIMessage(content=bot_msg))
    
        # Handle the latest message from the user
        model_messages.append(HumanMessage(content=message))
    
        # Add a system message to instruct the assistant
        system_prompt = (
            "You are a helpful assistant. "
            "You have access to the following tools:\n\n"
            "1. **search_web**: Use this tool to search the web for real-time information based on a query.\n"
            "2. **generate_images**: Use this tool to generate an image based on a prompt.\n\n"
            "Use these tools whenever appropriate to provide accurate and helpful responses."
        )
        model_messages.insert(0, SystemMessage(content=system_prompt))
    
        # Generate the response using the model and stream it
        response = ""
        for chunk in llm_text.stream(model_messages):
            if hasattr(chunk, 'content'):
                response += chunk.content
                yield response  # Stream the response to Gradio
    
    # Create the ChatInterface for the image chat (first tab)
    chat_with_images = gr.ChatInterface(
        fn=chatbot_image_response,
        multimodal=True,
        title="Chat with Images",
        description=(
            "Chatbot using llama3.2-vision. You can type messages and/or upload images. "
            "The assistant can understand and discuss the content of the images."
        ),
    )
    
    # Create the ChatInterface for the text chat with tools (second tab)
    chat_with_tools = gr.ChatInterface(
        fn=chatbot_text_response,
        title="Chat with Tools",
        description=(
            "Chatbot using llama3.1 with internet access and image generation tools. "
            "You can ask questions that require up-to-date information or request image generation."
        ),
    )
    
    # Combine them into tabs
    demo = gr.TabbedInterface([chat_with_images, chat_with_tools], ["Chat with Images", "Chat with Tools"])
    
    # Launch the Gradio app
    if __name__ == "__main__":
        demo.launch()
    

    Explanation:

    • Tools Defined:

      • search_web: Uses DuckDuckGo to search the web based on a query.
      • generate_images: A demo function that returns image.png as a base64 encoded string if it exists.
    • Models Initialized:

      • llm_image: Uses llama3.2-vision for the image chat interface.
      • llm_text: Uses llama3.1 with both search_web and generate_images tools bound for the text chat interface.
    • Chatbot Response Functions:

      • chatbot_image_response: Handles the conversation in the first tab, using llm_image.
      • chatbot_text_response: Handles the conversation in the second tab, using llm_text with the tools.
    • Gradio Interfaces:

      • chat_with_images: The first tab where users can chat and upload images.
      • chat_with_tools: The second tab where users can chat with access to web search and image generation.
    • Tabbed Interface:

      • Combines both chat interfaces into a single Gradio app with two tabs.

    Notes:

    • Make sure to have an image.png file in the same directory as your script for the generate_images tool to work.
    • Ensure all necessary dependencies are installed and properly configured.
    • The multimodal=True parameter enables both text and image inputs in the first tab.
    • Adjust the model parameters (temperature, etc.) as needed for your specific use case.

    Usage:

    • First Tab ("Chat with Images"):

      • You can type messages, upload images, or both.
      • The assistant will process the inputs using llama3.2-vision.
    • Second Tab ("Chat with Tools"):

      • Interact with the assistant using text.
      • The assistant can perform web searches and generate images upon request.
      • Example prompts:
        • "What's the latest news on renewable energy?"
        • "Generate an image of a futuristic city skyline."

    Launching the App:

    Run the script, and Gradio will launch the web interface where you can interact with both chatbots.

    python your_script_name.py
    

    Replace your_script_name.py with the actual name of your Python script file.