import gradio as gr from transformers import pipeline, AutoTokenizer,…
- User
import gradio as gr from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM import logging
Set up logging logging.basicConfig(level=logging.INFO)
Load the model and tokenizer model_name = "tiiuae/falcon-7b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name)
embedder_path = "all-MiniLM-L6-v2"
Initialize the text-generation pipeline text_generator = pipeline( "text-generation", model=model, tokenizer=tokenizer, device=0 # Use GPU if available; remove or set to -1 for CPU )
Define the chatbot logic def respond(symptom_input, history: list[tuple[str, str]]): system_message = ( "You are a healthcare assistant chatbot. Your goal is to provide health-related information " "and suggest possible non-diagnostic insights. Encourage users to consult healthcare professionals for diagnosis." )
max_tokens = 150 temperature = 0.7 top_p = 0.95
Construct the prompt with the system message
prompt = f"{system_message}\n\n"
Append conversation history with clear roles
for user_msg, assistant_msg in history: if user_msg: prompt += f"User: {user_msg}\n" if assistant_msg: prompt += f"Assistant: {assistant_msg}\n"
Append the latest user input
prompt += f"User: {symptom_input}\nAssistant:"
logging.info(f"Constructed Prompt:\n{prompt}")
try: # Generate a response from the model generated = text_generator( prompt, max_new_tokens=max_tokens, num_return_sequences=1, temperature=temperature, top_p=top_p, pad_token_id=tokenizer.eos_token_id ) response_content = generated[0]['generated_text'][len(prompt):].strip() logging.info(f"Model Response:\n{response_content}")
# Post-processing if not response_content.endswith('.'): response_content += '.' if 'consult a healthcare professional' not in response_content.lower(): response_content += " Please consult a healthcare professional for personalized advice." response = response_contentexcept Exception as e: response = f"Error during text generation: {str(e)}" logging.error(f"Generation Error: {str(e)}")
return response Define the Gradio Chat Interface demo = gr.ChatInterface( fn=respond, title="Healthcare Chatbot Assistant", description="Enter your symptoms, and the chatbot will provide health-related advice." )
if name == "main": demo.launch(server_name="10.172.0.233", server_port=5001) # Adjust server_name as needed
The response generation have this "User" Can you remove it
- Assistant
It seems that the model is including the "User" label in its generated responses. This happens because the model is continuing the conversation format you've provided in the prompt, which includes "User:" and "Assistant:" labels. To prevent the model from generating "User" in its response, you can modify your code in a couple of ways:
-
Modify the Prompt Structure: Adjust the prompt so that the model doesn't have "User:" and "Assistant:" labels to imitate. Instead, format the conversation in a way that the assistant can understand without these labels.
-
Post-Process the Model's Output: Remove any occurrence of "User:" from the generated response after it's been produced.
Below, I'll provide detailed instructions for both approaches.
Option 1: Modify the Prompt Structure
Since you're using an instruction-tuned model (
tiiuae/falcon-7b-instruct), it's designed to follow direct instructions without needing conversational role labels like "User:" and "Assistant:". Here's how you can adjust your prompt:Modified Prompt Construction:
# Construct the prompt without 'User:' and 'Assistant:' labels prompt = f"{system_message}\n\n" # Append conversation history without role labels for user_msg, assistant_msg in history: if user_msg: prompt += f"{user_msg}\n" if assistant_msg: prompt += f"{assistant_msg}\n" # Append the latest user input prompt += f"{symptom_input}\n" # Optionally, add a directive for the assistant prompt += "Please provide helpful health-related information based on the user's input above."Explanation:
- Remove Role Labels: By omitting "User:" and "Assistant:", the model is less likely to include them in the output.
- Direct Instruction: Adding a directive at the end can guide the model to generate the desired response.
Updated
respondFunction with Modified Prompt:def respond(symptom_input, history): system_message = ( "You are a healthcare assistant chatbot. Your goal is to provide health-related information " "and suggest possible non-diagnostic insights. Encourage users to consult healthcare professionals for diagnosis." ) max_tokens = 150 temperature = 0.7 top_p = 0.95 # Construct the prompt without role labels prompt = f"{system_message}\n\n" # Append conversation history for user_msg, assistant_msg in history: if user_msg: prompt += f"{user_msg}\n" if assistant_msg: prompt += f"{assistant_msg}\n" # Append the latest user input prompt += f"{symptom_input}\n" # Add a directive for the assistant prompt += "Please provide helpful health-related information based on the user's input above." logging.info(f"Constructed Prompt:\n{prompt}") try: # Generate a response from the model generated = text_generator( prompt, max_new_tokens=max_tokens, num_return_sequences=1, temperature=temperature, top_p=top_p, pad_token_id=tokenizer.eos_token_id ) response_content = generated[0]['generated_text'][len(prompt):].strip() logging.info(f"Model Response:\n{response_content}") # Post-processing if not response_content.endswith('.'): response_content += '.' if 'consult a healthcare professional' not in response_content.lower(): response_content += " Please consult a healthcare professional for personalized advice." response = response_content except Exception as e: response = f"Error during text generation: {str(e)}" logging.error(f"Generation Error: {str(e)}") return response
Option 2: Post-Process the Model's Output
If you prefer to keep the prompt structure as it is, you can remove any occurrence of "User:" from the generated response.
Add Post-Processing to Remove "User":
# After generating the response, add this post-processing step # Post-processing to remove 'User:' and any following text if 'User:' in response_content: response_content = response_content.split('User:')[0].rstrip()Updated
respondFunction with Post-Processing:def respond(symptom_input, history): system_message = ( "You are a healthcare assistant chatbot. Your goal is to provide health-related information " "and suggest possible non-diagnostic insights. Encourage users to consult healthcare professionals for diagnosis." ) max_tokens = 150 temperature = 0.7 top_p = 0.95 # Construct the prompt with the system message prompt = f"{system_message}\n\n" # Append conversation history with clear roles for user_msg, assistant_msg in history: if user_msg: prompt += f"User: {user_msg}\n" if assistant_msg: prompt += f"Assistant: {assistant_msg}\n" # Append the latest user input prompt += f"User: {symptom_input}\nAssistant:" logging.info(f"Constructed Prompt:\n{prompt}") try: # Generate a response from the model generated = text_generator( prompt, max_new_tokens=max_tokens, num_return_sequences=1, temperature=temperature, top_p=top_p, pad_token_id=tokenizer.eos_token_id ) response_content = generated[0]['generated_text'][len(prompt):].strip() logging.info(f"Model Response:\n{response_content}") # Post-processing to remove 'User:' and any following text if 'User:' in response_content: response_content = response_content.split('User:')[0].rstrip() # Additional Post-processing if not response_content.endswith('.'): response_content += '.' if 'consult a healthcare professional' not in response_content.lower(): response_content += " Please consult a healthcare professional for personalized advice." response = response_content except Exception as e: response = f"Error during text generation: {str(e)}" logging.error(f"Generation Error: {str(e)}") return responseExplanation:
- Detect "User:": The code checks if "User:" appears in the generated response.
- Remove Unwanted Text: It splits the response at "User:" and keeps only the text before it.
- Clean Up: The
rstrip()method removes any trailing whitespace.
Additional Suggestions
-
Use Stop Sequences: Some text generation pipelines allow you to specify stop sequences that instruct the model where to stop generating text. However, this might require using lower-level APIs or custom generation loops.
-
Update Model and Tokenizer: Make sure you're using the latest versions of the Hugging Face Transformers library and that the model supports the features you need.
-
Experiment with Formatting: Different models may respond better to different prompt formats. Feel free to experiment with the structure to see what yields the best results.
Final Notes
Adjusting the prompt structure (Option 1) is often the most effective way since it guides the model to generate responses in the desired format. However, combining both approaches may yield the best results.
Let me know if you need further assistance or clarification on these solutions!
-