Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-2024-08-06· OpenAIAi2 WildChat-4.8M (AI2)

I am trying to build a chatbot using llama model that run in vs code notebook locally,…

  1. User

    I am trying to build a chatbot using llama model that run in vs code notebook locally, my code: import torch from transformers import AutoModelForCausalLM, AutoTokenizer

    Path to your locally downloaded model directory model_path = r"C:\Users\shermanhonsf\Desktop\Python\llama_develop\model\Llama3.1-1B-Instruct-HF-Model"

    Load the model and tokenizer tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoModelForCausalLM.from_pretrained(model_path)

    Check if CUDA (GPU) is available and move the model to GPU if so device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device)

    Function to generate chatbot responses def generate_response(user_input, max_new_tokens=100, temperature=0.6, top_p=1):

    Only pass the latest user input to the model inputs = tokenizer(f"User: {user_input}\nChatbot:", return_tensors="pt") input_ids = inputs["input_ids"].to(device) # Move input tensors to the GPU if available

    Disable gradient calculation for faster performance since we are in inference mode with torch.no_grad(): output = model.generate( input_ids, max_new_tokens=max_new_tokens, temperature=temperature, #top_k=top_k, # Only consider the top_k options at each step top_p=top_p, # Nucleus sampling - the cumulative probability threshold pad_token_id=tokenizer.eos_token_id, # Token ID for padding do_sample=True # Use sampling for text generation )

    Decode the generated tokens to text response = tokenizer.decode(output[:, input_ids.shape[-1]:][0], skip_special_tokens=True) return response.strip() # Strip any leading or trailing whitespace Chatbot conversation loop def chat(): print("Chatbot is ready! Type 'exit' to stop the conversation.")

    while True:

    Get user input

    user_input = input("You: ")

    if user_input.lower() == "exit": print("Goodbye!") break

    Generate response from the model

    response = generate_response(user_input) print(f"Chatbot: {response}") if name == "main": chat()

    However, the vs code notebook do not show real time output, so the output is only : Chatbot is ready! Type 'exit' to stop the conversation. Without input option. How to fix it?

  2. Assistant

    It looks like you are running into an issue where the script is not providing a real-time input prompt within a Jupyter notebook environment in VS Code. Jupyter notebooks do not support traditional blocking input/output functions like input() in the same way as regular Python scripts. Here's how you can adjust your code to work in a Jupyter environment:

    Step-by-Step Solution

    1. Setup a Function for Input/Output in Notebook Cells:

      Instead of using input() for getting inputs from the user, you can use Jupyter widgets to create a more interactive experience:

      from IPython.display import display
      import ipywidgets as widgets
      
      def chat():
          # Create a text input widget for user input
          input_box = widgets.Text(
              description="You:",
              placeholder="Type your message here"
          )
      
          # Create an output widget to display chat history
          output_area = widgets.Output()
      
          # Define an action function for processing input and generating output
          def on_submit(sender):
              user_input = input_box.value
              input_box.value = ""  # Clear input box
      
              if user_input.lower() == "exit":
                  print("Goodbye!")
                  return
      
              response = generate_response(user_input)
      
              # Append text to the output widget
              with output_area:
                  print(f"You: {user_input}")
                  print(f"Chatbot: {response}")
      
          # Bind Enter key event to the input box
          input_box.on_submit(on_submit)
      
          # Display input box and output area
          display(input_box, output_area)
      
      # Run the chat function
      chat()
      
    2. Update Your TensorFlow or PyTorch Environment:

      Make sure your PyTorch installation is up to date in your Jupyter environment to avoid any GPU-related issues. You can run the following command in a notebook cell to check if your environment is set up correctly:

      import torch
      print("CUDA available:", torch.cuda.is_available())
      
    3. Ensure GPU Allocation is Working:

      Double-check that your notebook environment is capable of allocating GPU resources if you have configured your model to run on CUDA. This can sometimes require additional setup in Jupyter environments or server configurations.

    By making these updates, you should be able to test your chatbot model in a Jupyter notebook and interact with it in real-time using more user-friendly methods like widgets. This approach will circumvent the limitations of using input() and print() directly in notebook environments.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.