Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· openaiHugging Face Suzhen/CodeChat

I am build a chatbot on huggingface:"from huggingfacehub import InferenceClient import…

  1. User

    I am build a chatbot on huggingface:"from huggingface_hub import InferenceClient import gradio as gr

    client = InferenceClient( "mistralai/Mixtral-8x7B-Instruct-v0.1" )

    def format_prompt(message, history): prompt = "" for user_prompt, bot_response in history: prompt += f"[INST] {user_prompt} [/INST]" prompt += f" {bot_response} " prompt += f"[INST] {message} [/INST]" return prompt

    def generate( prompt, history, system_prompt, temperature=0.9, max_new_tokens=256, top_p=0.95, repetition_penalty=1.0, ): temperature = float(temperature) if temperature < 1e-2: temperature = 1e-2 top_p = float(top_p)

    generate_kwargs = dict(
        temperature=temperature,
        max_new_tokens=max_new_tokens,
        top_p=top_p,
        repetition_penalty=repetition_penalty,
        do_sample=True,
        seed=42,
    )
    
    formatted_prompt = format_prompt(f"{system_prompt}, {prompt}", history)
    stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
    output = ""
    
    for response in stream:
        output += response.token.text
        yield output
    return output
    

    additional_inputs=[ gr.Textbox( label="System Prompt", max_lines=1, interactive=True, ), gr.Slider( label="Temperature", value=0.9, minimum=0.0, maximum=1.0, step=0.05, interactive=True, info="Higher values produce more diverse outputs", ), gr.Slider( label="Max new tokens", value=256, minimum=0, maximum=1048, step=64, interactive=True, info="The maximum numbers of new tokens", ), gr.Slider( label="Top-p (nucleus sampling)", value=0.90, minimum=0.0, maximum=1, step=0.05, interactive=True, info="Higher values sample more low-probability tokens", ), gr.Slider( label="Repetition penalty", value=1.2, minimum=1.0, maximum=2.0, step=0.05, interactive=True, info="Penalize repeated tokens", ) ]

    examples=[["I'm planning a vacation to Japan. Can you suggest a one-week itinerary including must-visit places and local cuisines to try?", None, None, None, None, None, ], ["Can you write a short story about a time-traveling detective who solves historical mysteries?", None, None, None, None, None,], ["I'm trying to learn French. Can you provide some common phrases that would be useful for a beginner, along with their pronunciations?", None, None, None, None, None,], ["I have chicken, rice, and bell peppers in my kitchen. Can you suggest an easy recipe I can make with these ingredients?", None, None, None, None, None,], ["Can you explain how the QuickSort algorithm works and provide a Python implementation?", None, None, None, None, None,], ["What are some unique features of Rust that make it stand out compared to other systems programming languages like C++?", None, None, None, None, None,], ]

    gr.ChatInterface( fn=generate, chatbot=gr.Chatbot(show_label=False, show_share_button=False, show_copy_button=True, likeable=True, layout="panel"), additional_inputs=additional_inputs, title="Mixtral 46.7B", examples=examples, concurrency_limit=20, ).launch(show_api=False)" I want to delete the additional input part while make sure there is no error and the chatbot can run well and set my own system prompt :"I am a helpful chatbot".

  2. Assistant

    To remove the additional inputs and hardcode a system prompt, you can simplify your code by removing the additional_inputs list and modifying the generate function to use your fixed system prompt. Here's how you can do it:

    import gradio as gr
    from huggingface_hub import InferenceClient
    
    client = InferenceClient("mistralai/Mixtral-8x7B-Instruct-v0.1")
    
    def format_prompt(message, history):
        prompt = "<s>"
        for user_prompt, bot_response in history:
            prompt += f"[INST] {user_prompt} [/INST]"
            prompt += f" {bot_response}</s> "
        prompt += f"[INST] {message} [/INST]"
        return prompt
    
    def generate(prompt, history, temperature=0.9, max_new_tokens=256, top_p=0.95, repetition_penalty=1.2):
        system_prompt = "I am a helpful chatbot"  # Fixed system prompt
        temperature = float(temperature)
        top_p = float(top_p)
    
        generate_kwargs = dict(
            temperature=temperature,
            max_new_tokens=max_new_tokens,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            do_sample=True,
            seed=42,
        )
    
        formatted_prompt = format_prompt(f"{system_prompt}, {prompt}", history)
        stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
        output = ""
    
        for response in stream:
            output += response.token.text
            yield output
        return output
    
    
    examples = [
        ["I'm planning a vacation to Japan. Can you suggest a one-week itinerary including must-visit places and local cuisines to try?", None],
        ["Can you write a short story about a time-traveling detective who solves historical mysteries?", None],
        ["I'm trying to learn French. Can you provide some common phrases that would be useful for a beginner, along with their pronunciations?", None],
        ["I have chicken, rice, and bell peppers in my kitchen. Can you suggest an easy recipe I can make with these ingredients?", None],
        ["Can you explain how the QuickSort algorithm works and provide a Python implementation?", None],
        ["What are some unique features of Rust that make it stand out compared to other systems programming languages like C++?", None],
    ]
    
    gr.ChatInterface(
        fn=generate,
        chatbot=gr.Chatbot(show_label=False, show_share_button=False, show_copy_button=True, likeable=True, layout="panel"),
        title="Mixtral 46.7B",
        examples=examples,
        concurrency_limit=20,
    ).launch(show_api=False)
    

    I've removed the additional_inputs altogether, set system_prompt within the generate function, and made sure that the input arguments for the generate function in the examples are now just two (the prompt and the history). Remember, after adjusting your code, re-run your Gradio app to make sure the changes take effect.

  3. User

    What are top_p=0.95, repetition_penalty=1.2 ?

  4. Assistant

    In language generation models like GPT-3 or other similar transformer-based models, top_p and repetition_penalty are parameters to control the generation process. Let me explain what they are:

    1. top_p (also known as nucleus sampling): This parameter is used for controlling the randomness of the predictions by limiting the possible next words to a subset from which the cumulative probability exceeds a certain threshold (top_p). When generating a token, instead of considering the full vocabulary, we only consider the smallest set of words whose combined probability is higher than top_p. This means if top_p is set to 0.95, we would be sampling from the top 95% probable next words at each step in the generation. It's designed to prevent very unlikely words from being chosen and is an alternative to top_k sampling, which limits the sample pool to the top-k most likely words. A higher top_p increases diversity in the generated text, at the expense of coherence and quality.

    2. repetition_penalty: This parameter is used to discourage the model from repeating the same words and phrases. When set to a value greater than 1.0, it penalizes words that have already been generated, making it less likely for the model to choose them again, which promotes more diverse and interesting outputs. Conversely, setting it to a value less than 1.0 would make repetitions more likely. A repetition penalty is particularly useful for longer generations where you want to avoid getting stuck in a loop of repeating content.

    Together, these parameters help fine-tune the response of the model to generate text that is diverse, interesting, and less repetitive. They allow developers and researchers to balance between randomness and determinism in the model's text generation, according to the desired application.

  5. User

    My system prompt is " system_prompt = "I'm an AI chatbot named ChatSherman designed by a super-intelligent student named ShermanAI at the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University to help you with your engineering questions. Also, I can assist with a wide range of topics and questions. I am now version 2.0, which is more powerful than version 1.0, able to do more complex tasks, and optimized for chat."
    " I want the chatbot itself to be ChatSherman, not the user

  6. Assistant

    If you want the chatbot to introduce itself as ChatSherman with that system prompt every time it generates a response, you must ensure that this introduction is factored into every generated response. You can do this by prepending your system prompt to the user message in the prompt formatting function. Here's how you could modify your generate function to reflect this:

    import gradio as gr
    from huggingface_hub import InferenceClient
    
    client = InferenceClient("mistralai/Mixtral-8x7B-Instruct-v0.1")
    
    def format_prompt(message, history):
        system_prompt = ("I'm an AI chatbot named ChatSherman designed by a super-intelligent student named ShermanAI "
                         "at the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University "
                         "to help you with your engineering questions. Also, I can assist with a wide range of topics and questions. "
                         "I am now version 2.0, which is more powerful than version 1.0, able to do more complex tasks, and optimized for chat.")  
        prompt = "<s>"
        for user_prompt, bot_response in history:
            prompt += f"[INST] {user_prompt} [/INST]"
            prompt += f" {bot_response}</s> "
        prompt += f"[INST] {system_prompt} {message} [/INST]"
        return prompt
    
    def generate(message, history=[], temperature=0.9, max_new_tokens=256, top_p=0.95, repetition_penalty=1.2):
        temperature = float(temperature)
        top_p = float(top_p)
    
        generate_kwargs = dict(
            temperature=temperature,
            max_new_tokens=max_new_tokens,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            do_sample=True,
            seed=42,
        )
    
        formatted_prompt = format_prompt(message, history)
        stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
        output = ""
    
        for response in stream:
            output += response.token.text
            yield output
        return output
    
    examples = [
        # Examples with specific user prompts; the chatbot will prepend the system prompt to each
    ]
    
    gr.Interface(
        fn=generate,
        inputs='text',
        outputs='text',
        examples=examples,
        title="ChatSherman 2.0"
    ).launch()
    

    With these changes, every time the generate function is called, it will format the prompt to include the system introduction as if the chatbot is saying it, followed by the user's input message. This way, ChatSherman presents itself accordingly with each interaction.

  7. User

    My full code:"import gradio as gr from huggingface_hub import InferenceClient

    client = InferenceClient("mistralai/Mixtral-8x7B-Instruct-v0.1")

    def format_prompt(message, history): prompt = "" for user_prompt, bot_response in history: prompt += f"[INST] {user_prompt} [/INST]" prompt += f"{bot_response}" prompt += f"[INST] {message} [/INST]" return prompt

    def generate(prompt, history, temperature=0.6, max_new_tokens=256, top_p=0.95, repetition_penalty=1.2): system_prompt = "I'm an AI chatbot named ChatSherman designed by a super-intelligent student named ShermanAI at the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University to help you with your engineering questions. Also, I can assist with a wide range of topics and questions. I am now version 2.0, which is more powerful than version 1.0, able to do more complex tasks, and optimized for chat." # Fixed system prompt temperature = float(temperature) top_p = float(top_p)

    generate_kwargs = dict(
        temperature=temperature,
        max_new_tokens=max_new_tokens,
        top_p=top_p,
        repetition_penalty=repetition_penalty,
        do_sample=True,
        seed=42,
    )
    
    formatted_prompt = format_prompt(f"{system_prompt}, {prompt}", history)
    stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
    output = ""
    for response in stream:
        output += response.token.text
        yield output
    return output
    

    examples = [ ["What is ChatSherman, and how does it work?", []], ["Is my personal information and data safe when I use the ChatSherman chatbot?", []], ["What are some common applications of deep learning in engineering?", []] ]

    gr.ChatInterface( fn=generate, chatbot=gr.Chatbot(show_label=False, show_share_button=False, show_copy_button=True, likeable=True, layout="panel"), title="Mixtral 46.7B", examples=examples, concurrency_limit=20, ).launch(show_api=False)" show full code of your modification

  8. Assistant

    Certainly! Below is your modified code where the chatbot, ChatSherman, introduces itself with each response using its hard-coded system prompt. I've adjusted the generate function to incorporate the system prompt directly when formatting the initial prompt.

    import gradio as gr
    from huggingface_hub import InferenceClient
    
    client = InferenceClient("mistralai/Mixtral-8x7B-Instruct-v0.1")
    
    def format_prompt(message, history):
        system_prompt = ("I'm an AI chatbot named ChatSherman designed by a super-intelligent student named ShermanAI "
                         "at the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University "
                         "to help you with your engineering questions. Also, I can assist with a wide range of topics and questions. "
                         "I am now version 2.0, which is more powerful than version 1.0, able to do more complex tasks, and optimized for chat.")
        prompt = "<s>"
        for user_prompt, bot_response in history:
            prompt += f"[INST] {user_prompt} [/INST]"
            prompt += f"{bot_response}</s> "
        prompt += f"[INST] {system_prompt} {message} [/INST]"
        return prompt
    
    def generate(prompt, history, temperature=0.6, max_new_tokens=256, top_p=0.95, repetition_penalty=1.2):
        temperature = float(temperature)
        top_p = float(top_p)
    
        generate_kwargs = dict(
            temperature=temperature,
            max_new_tokens=max_new_tokens,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            do_sample=True,
            seed=42,
        )
    
        formatted_prompt = format_prompt(prompt, history)
        stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
        output = ""
        for response in stream:
            output += response.token.text
            yield output
        return output
    
    
    examples = [
        ["What is ChatSherman, and how does it work?", []],
        ["Is my personal information and data safe when I use the ChatSherman chatbot?", []],
        ["What are some common applications of deep learning in engineering?", []]
    ]
    
    gr.ChatInterface(
        fn=generate,
        chatbot=gr.Chatbot(show_label=False, show_share_button=False, show_copy_button=True, likeable=True, layout="panel"),
        title="ChatSherman 2.0",
        examples=examples,
        concurrency_limit=20,
    ).launch(show_api=False)
    

    In this modification, notice that the system_prompt is declared inside the format_prompt function, and it is combined with the user message and existing history to form a single prompt string.

    Please also ensure that you properly account for the possibility of an initial empty history when invoking the generate function. This can happen when the chatbot first starts up or when the history is cleared.

    Keep in mind that even though the system prompt introduces the chatbot as an AI designed by "ShermanAI," this is not an actual entity, but a fictional example. You would replace this and other information with actual details relevant to your bot.

  9. User

    No need to introduces itself with each response, that's strange, just need to know about itself is called ChatSherman and its basic information is ok.

  10. Assistant

    Understood. If you want the chatbot to be aware of its identity as ChatSherman and its basic background information without repeating it in every response, you can simply store that information within the chatbot's code and ensure it's able to use this information when appropriate. You don't need to append this information to every prompt then.

    Here's how the modified code would look:

    import gradio as gr
    from huggingface_hub import InferenceClient
    
    client = InferenceClient("mistralai/Mixtral-8x7B-Instruct-v0.1")
    
    # Let's keep the bot's basic information here, but we won't use it in the prompt directly
    chatbot_info = ("I'm an AI chatbot named ChatSherman designed by a super-intelligent student named ShermanAI at the "
                    "Department of Electronic and Information Engineering at The Hong Kong Polytechnic University to help you with your "
                    "engineering questions. Also, I can assist with a wide range of topics and questions. I am now version 2.0, which is more powerful than version 1.0, "
                    "able to do more complex tasks, and optimized for chat.")
    
    def format_prompt(message, history):
        prompt = "<s>"
        for user_prompt, bot_response in history:
            prompt += f"[INST] {user_prompt} [/INST]"
            prompt += f"{bot_response}</s>"
        prompt += f"[INST] {message} [/INST]"
        return prompt
    
    def generate(message, history, temperature=0.6, max_new_tokens=256, top_p=0.95, repetition_penalty=1.2):
        temperature = float(temperature)
        top_p = float(top_p)
    
        generate_kwargs = dict(
            temperature=temperature,
            max_new_tokens=max_new_tokens,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            do_sample=True,
            seed=42,
        )
    
        formatted_prompt = format_prompt(message, history)
        stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
        output = ""
        for response in stream:
            output += response.token.text
            yield output
        return output
    
    examples = [
        ["What can you do?", []],
        ["Tell me about yourself.", []],
        ["What's special about version 2.0?", []]
    ]
    
    gr.Interface(
        fn=generate,
        inputs='text',
        outputs='text',
        examples=examples,
        title="ChatSherman 2.0"
    ).launch()
    

    In this version of the code:

    • chatbot_info is stored, which includes information about the chatbot. It's not included in each prompt but could be used whenever the bot needs to provide information about itself.
    • The format_prompt function no longer includes system_prompt as part of the prompt it constructs.
    • Example user prompts in examples are set up to encourage the user to ask about the chatbot's capabilities, its identity, or its version, and then the chatbot can use chatbot_info to respond accordingly if necessary.

    When deploying the chatbot, make sure it is trained to reference its own information based on user queries. This way, it can provide its background details when users ask questions like "Tell me about yourself" or "What can you do?" without repeating the intro for every interaction.

  11. User

    Can I just prompt it one time, at the beginning of every chat?

  12. Assistant

    Yes, you can prompt the chatbot with its introduction just once at the beginning of each chat session. To do this, you need to integrate the system prompt into the initial response generated when the chat session starts but not include it in subsequent messages.

    One strategy for achieving this is to check if the history is empty and prepend the introduction only then. Here's how you would implement this in your code:

    import gradio as gr
    from huggingface_hub import InferenceClient
    
    client = InferenceClient("mistralai/Mixtral-8x7B-Instruct-v0.1")
    
    def format_prompt(message, history):
        system_prompt = ("I'm an AI chatbot named ChatSherman designed by a super-intelligent student named ShermanAI "
                         "at the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University "
                         "to help you with your engineering questions. Also, I can assist with a wide range of topics and questions. "
                         "I am now version 2.0, which is more powerful than version 1.0, able to do more complex tasks, and optimized for chat.")
        prompt = "<s>"
        if not history:  # Check if the history is empty
            prompt += f"[INST] {system_prompt} {message} [/INST]"
        else:
            for user_prompt, bot_response in history:
                prompt += f"[INST] {user_prompt} [/INST]"
                prompt += f"{bot_response}</s> "
            prompt += f"[INST] {message} [/INST]"
        return prompt
    
    def generate(message, history, temperature=0.6, max_new_tokens=256, top_p=0.95, repetition_penalty=1.2):
        temperature = float(temperature)
        top_p = float(top_p)
    
        generate_kwargs = dict(
            temperature=temperature,
            max_new_tokens=max_new_tokens,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            do_sample=True,
            seed=42,
        )
    
        formatted_prompt = format_prompt(message, history)
        stream = client.text_generation(formatted_prompt, **generate_kwargs, stream=True, details=True, return_full_text=False)
        output = ""
        for response in stream:
            output += response.token.text
            yield output
        return output
    
    examples = [
        ["What is your name?", []],  # Example to initiate the chat with an introduction
    ]
    
    gr.Interface(
        fn=generate,
        inputs='text',
        outputs='text',
        examples=examples,
        title="ChatSherman 2.0"
    ).launch()
    

    In this modification:

    • The format_prompt function checks if the history is empty, which would indicate the beginning of a new chat session.
    • If the history is empty, the system_prompt introducing ChatSherman is included with the user's first message.
    • For subsequent messages within the same session (when history is not empty), the system_prompt is not included.

    Now, when a user starts a new session with a first message, the chatbot will include its introduction. But for all subsequent exchanges within the same session, the bot will not repeat the introduction. When the chat session is reset or when a new user starts a new session, the bot will reintroduce itself again.

    Please remember that Gradio sessions are stateless by default and may not retain history between different user inputs. If you would like to maintain history across different interactions in the same session, you might need to implement session management that varies by the tools and frameworks you are using. Gradio's ChatInterface might provide a history mechanism internally for chat-based interfaces. However, for a detailed state management system, you might need to implement additional logic or use other tools that offer session management features.