I am trying to build a chatbot using llama model that run in vs code notebook locally,…
- User
I am trying to build a chatbot using llama model that run in vs code notebook locally, my code: import torch from transformers import AutoModelForCausalLM, AutoTokenizer
Path to your locally downloaded model directory model_path = r"C:\Users\shermanhonsf\Desktop\Python\llama_develop\model\Llama3.1-1B-Instruct-HF-Model"
Load the model and tokenizer tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoModelForCausalLM.from_pretrained(model_path)
Check if CUDA (GPU) is available and move the model to GPU if so device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device)
Function to generate chatbot responses def generate_response(user_input, max_new_tokens=100, temperature=0.6, top_p=1):
Only pass the latest user input to the model inputs = tokenizer(f"User: {user_input}\nChatbot:", return_tensors="pt") input_ids = inputs["input_ids"].to(device) # Move input tensors to the GPU if available
Disable gradient calculation for faster performance since we are in inference mode with torch.no_grad(): output = model.generate( input_ids, max_new_tokens=max_new_tokens, temperature=temperature, #top_k=top_k, # Only consider the top_k options at each step top_p=top_p, # Nucleus sampling - the cumulative probability threshold pad_token_id=tokenizer.eos_token_id, # Token ID for padding do_sample=True # Use sampling for text generation )
Decode the generated tokens to text response = tokenizer.decode(output[:, input_ids.shape[-1]:][0], skip_special_tokens=True) return response.strip() # Strip any leading or trailing whitespace Chatbot conversation loop def chat(): print("Chatbot is ready! Type 'exit' to stop the conversation.")
while True:
Get user input
user_input = input("You: ")
if user_input.lower() == "exit": print("Goodbye!") break
Generate response from the model
response = generate_response(user_input) print(f"Chatbot: {response}") if name == "main": chat()
However, the vs code notebook do not show real time output, so the output is only : Chatbot is ready! Type 'exit' to stop the conversation. Without input option. How to fix it?
- Assistant
It looks like you are running into an issue where the script is not providing a real-time input prompt within a Jupyter notebook environment in VS Code. Jupyter notebooks do not support traditional blocking input/output functions like
input()in the same way as regular Python scripts. Here's how you can adjust your code to work in a Jupyter environment:Step-by-Step Solution
-
Setup a Function for Input/Output in Notebook Cells:
Instead of using
input()for getting inputs from the user, you can use Jupyter widgets to create a more interactive experience:from IPython.display import display import ipywidgets as widgets def chat(): # Create a text input widget for user input input_box = widgets.Text( description="You:", placeholder="Type your message here" ) # Create an output widget to display chat history output_area = widgets.Output() # Define an action function for processing input and generating output def on_submit(sender): user_input = input_box.value input_box.value = "" # Clear input box if user_input.lower() == "exit": print("Goodbye!") return response = generate_response(user_input) # Append text to the output widget with output_area: print(f"You: {user_input}") print(f"Chatbot: {response}") # Bind Enter key event to the input box input_box.on_submit(on_submit) # Display input box and output area display(input_box, output_area) # Run the chat function chat() -
Update Your TensorFlow or PyTorch Environment:
Make sure your PyTorch installation is up to date in your Jupyter environment to avoid any GPU-related issues. You can run the following command in a notebook cell to check if your environment is set up correctly:
import torch print("CUDA available:", torch.cuda.is_available()) -
Ensure GPU Allocation is Working:
Double-check that your notebook environment is capable of allocating GPU resources if you have configured your model to run on CUDA. This can sometimes require additional setup in Jupyter environments or server configurations.
By making these updates, you should be able to test your chatbot model in a Jupyter notebook and interact with it in real-time using more user-friendly methods like widgets. This approach will circumvent the limitations of using
input()andprint()directly in notebook environments. -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.