Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

import gradio as gr from transformers import pipeline, AutoTokenizer,…

  1. User

    import gradio as gr from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM import logging

    Set up logging logging.basicConfig(level=logging.INFO)

    Load the model and tokenizer model_name = "tiiuae/falcon-7b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name)

    embedder_path = "all-MiniLM-L6-v2"

    Initialize the text-generation pipeline text_generator = pipeline( "text-generation", model=model, tokenizer=tokenizer, device=0 # Use GPU if available; remove or set to -1 for CPU )

    Define the chatbot logic def respond(symptom_input, history: list[tuple[str, str]]): system_message = ( "You are a healthcare assistant chatbot. Your goal is to provide health-related information " "and suggest possible non-diagnostic insights. Encourage users to consult healthcare professionals for diagnosis." )

    max_tokens = 150 temperature = 0.7 top_p = 0.95

    Construct the prompt with the system message

    prompt = f"{system_message}\n\n"

    Append conversation history with clear roles

    for user_msg, assistant_msg in history: if user_msg: prompt += f"User: {user_msg}\n" if assistant_msg: prompt += f"Assistant: {assistant_msg}\n"

    Append the latest user input

    prompt += f"User: {symptom_input}\nAssistant:"

    logging.info(f"Constructed Prompt:\n{prompt}")

    try: # Generate a response from the model generated = text_generator( prompt, max_new_tokens=max_tokens, num_return_sequences=1, temperature=temperature, top_p=top_p, pad_token_id=tokenizer.eos_token_id ) response_content = generated[0]['generated_text'][len(prompt):].strip() logging.info(f"Model Response:\n{response_content}")

    # Post-processing
    if not response_content.endswith('.'):
        response_content += '.'
    if 'consult a healthcare professional' not in response_content.lower():
        response_content += " Please consult a healthcare professional for personalized advice."
    
    response = response_content
    

    except Exception as e: response = f"Error during text generation: {str(e)}" logging.error(f"Generation Error: {str(e)}")

    return response Define the Gradio Chat Interface demo = gr.ChatInterface( fn=respond, title="Healthcare Chatbot Assistant", description="Enter your symptoms, and the chatbot will provide health-related advice." )

    if name == "main": demo.launch(server_name="10.172.0.233", server_port=5001) # Adjust server_name as needed

    The response generation have this "User" Can you remove it

  2. Assistant

    It seems that the model is including the "User" label in its generated responses. This happens because the model is continuing the conversation format you've provided in the prompt, which includes "User:" and "Assistant:" labels. To prevent the model from generating "User" in its response, you can modify your code in a couple of ways:

    1. Modify the Prompt Structure: Adjust the prompt so that the model doesn't have "User:" and "Assistant:" labels to imitate. Instead, format the conversation in a way that the assistant can understand without these labels.

    2. Post-Process the Model's Output: Remove any occurrence of "User:" from the generated response after it's been produced.

    Below, I'll provide detailed instructions for both approaches.


    Option 1: Modify the Prompt Structure

    Since you're using an instruction-tuned model (tiiuae/falcon-7b-instruct), it's designed to follow direct instructions without needing conversational role labels like "User:" and "Assistant:". Here's how you can adjust your prompt:

    Modified Prompt Construction:

    # Construct the prompt without 'User:' and 'Assistant:' labels
    prompt = f"{system_message}\n\n"
    
    # Append conversation history without role labels
    for user_msg, assistant_msg in history:
        if user_msg:
            prompt += f"{user_msg}\n"
        if assistant_msg:
            prompt += f"{assistant_msg}\n"
    
    # Append the latest user input
    prompt += f"{symptom_input}\n"
    
    # Optionally, add a directive for the assistant
    prompt += "Please provide helpful health-related information based on the user's input above."
    

    Explanation:

    • Remove Role Labels: By omitting "User:" and "Assistant:", the model is less likely to include them in the output.
    • Direct Instruction: Adding a directive at the end can guide the model to generate the desired response.

    Updated respond Function with Modified Prompt:

    def respond(symptom_input, history):
        system_message = (
            "You are a healthcare assistant chatbot. Your goal is to provide health-related information "
            "and suggest possible non-diagnostic insights. Encourage users to consult healthcare professionals for diagnosis."
        )
    
        max_tokens = 150
        temperature = 0.7
        top_p = 0.95
    
        # Construct the prompt without role labels
        prompt = f"{system_message}\n\n"
    
        # Append conversation history
        for user_msg, assistant_msg in history:
            if user_msg:
                prompt += f"{user_msg}\n"
            if assistant_msg:
                prompt += f"{assistant_msg}\n"
    
        # Append the latest user input
        prompt += f"{symptom_input}\n"
    
        # Add a directive for the assistant
        prompt += "Please provide helpful health-related information based on the user's input above."
    
        logging.info(f"Constructed Prompt:\n{prompt}")
    
        try:
            # Generate a response from the model
            generated = text_generator(
                prompt,
                max_new_tokens=max_tokens,
                num_return_sequences=1,
                temperature=temperature,
                top_p=top_p,
                pad_token_id=tokenizer.eos_token_id
            )
            response_content = generated[0]['generated_text'][len(prompt):].strip()
            logging.info(f"Model Response:\n{response_content}")
    
            # Post-processing
            if not response_content.endswith('.'):
                response_content += '.'
            if 'consult a healthcare professional' not in response_content.lower():
                response_content += " Please consult a healthcare professional for personalized advice."
    
            response = response_content
    
        except Exception as e:
            response = f"Error during text generation: {str(e)}"
            logging.error(f"Generation Error: {str(e)}")
    
        return response
    

    Option 2: Post-Process the Model's Output

    If you prefer to keep the prompt structure as it is, you can remove any occurrence of "User:" from the generated response.

    Add Post-Processing to Remove "User":

    # After generating the response, add this post-processing step
    
    # Post-processing to remove 'User:' and any following text
    if 'User:' in response_content:
        response_content = response_content.split('User:')[0].rstrip()
    

    Updated respond Function with Post-Processing:

    def respond(symptom_input, history):
        system_message = (
            "You are a healthcare assistant chatbot. Your goal is to provide health-related information "
            "and suggest possible non-diagnostic insights. Encourage users to consult healthcare professionals for diagnosis."
        )
    
        max_tokens = 150
        temperature = 0.7
        top_p = 0.95
    
        # Construct the prompt with the system message
        prompt = f"{system_message}\n\n"
    
        # Append conversation history with clear roles
        for user_msg, assistant_msg in history:
            if user_msg:
                prompt += f"User: {user_msg}\n"
            if assistant_msg:
                prompt += f"Assistant: {assistant_msg}\n"
    
        # Append the latest user input
        prompt += f"User: {symptom_input}\nAssistant:"
    
        logging.info(f"Constructed Prompt:\n{prompt}")
    
        try:
            # Generate a response from the model
            generated = text_generator(
                prompt,
                max_new_tokens=max_tokens,
                num_return_sequences=1,
                temperature=temperature,
                top_p=top_p,
                pad_token_id=tokenizer.eos_token_id
            )
            response_content = generated[0]['generated_text'][len(prompt):].strip()
            logging.info(f"Model Response:\n{response_content}")
    
            # Post-processing to remove 'User:' and any following text
            if 'User:' in response_content:
                response_content = response_content.split('User:')[0].rstrip()
    
            # Additional Post-processing
            if not response_content.endswith('.'):
                response_content += '.'
            if 'consult a healthcare professional' not in response_content.lower():
                response_content += " Please consult a healthcare professional for personalized advice."
    
            response = response_content
    
        except Exception as e:
            response = f"Error during text generation: {str(e)}"
            logging.error(f"Generation Error: {str(e)}")
    
        return response
    

    Explanation:

    • Detect "User:": The code checks if "User:" appears in the generated response.
    • Remove Unwanted Text: It splits the response at "User:" and keeps only the text before it.
    • Clean Up: The rstrip() method removes any trailing whitespace.

    Additional Suggestions

    • Use Stop Sequences: Some text generation pipelines allow you to specify stop sequences that instruct the model where to stop generating text. However, this might require using lower-level APIs or custom generation loops.

    • Update Model and Tokenizer: Make sure you're using the latest versions of the Hugging Face Transformers library and that the model supports the features you need.

    • Experiment with Formatting: Different models may respond better to different prompt formats. Feel free to experiment with the structure to see what yields the best results.


    Final Notes

    Adjusting the prompt structure (Option 1) is often the most effective way since it guides the model to generate responses in the desired format. However, combining both approaches may yield the best results.

    Let me know if you need further assistance or clarification on these solutions!