import gradio as gr from huggingfacehub import InferenceClient import os Initialize…
- User
import gradio as gr from huggingface_hub import InferenceClient import os
Initialize Hugging Face Inference Client
client = InferenceClient( model="meta-llama/Meta-Llama-3.1-8B-Instruct", token=os.getenv("HUGGINGFACEHUB_API_TOKEN") # Token is retrieved from the Hugging Face secret )
Function to interact with the model
def ask_model(question): responses = [] for message in client.chat_completion( messages=[{"role": "user", "content": question}], max_tokens=1000, stream=True, ): responses.append(message.choices[0].delta.content)
return ''.join(responses)Gradio Interface
interface = gr.Interface( fn=ask_model, inputs=gr.Textbox(lines=2, placeholder="Ask a question..."), outputs="text", title="Llama3.1 Based AI Chatbot | Interface by Thay Chansy", description="Ask the model any question and get a response." )
Launch Gradio app
interface.launch(share=True)
- Assistant
The script you've provided describes a simple Gradio interface for interacting with a language model hosted on Hugging Face using the
huggingface_hublibrary. Let me break down the process:-
Importing Required Libraries:
gradiois used to create an easy-to-use web interface that can be shared with others.InferenceClientfromhuggingface_hubfacilitates interaction with models on the Hugging Face Hub.osis used to access environment variables, which is where the Hugging Face API token is stored.
-
Initialize the Inference Client:
- The client is set up to communicate with the language model "Meta-Llama-3.1-8B-Instruct."
- An API token is retrieved from an environment variable named
HUGGINGFACEHUB_API_TOKEN, which must be set before running the script.
-
Function to Ask the Model:
ask_modeltakes a question, sends it to the model, and constructs a response from the streaming output.- The model's streaming output is collected and concatenated to form the complete response.
-
Creating a Gradio Interface:
- The
interfaceobject is defined to map theask_modelfunction into a Gradio interface. - It consists of a textbox for user input (
gr.Textbox) and displays the model's response as text. - Meta information such as
titleanddescriptionare provided to give context on what the tool does.
- The
-
Launching the Gradio App:
- The
interface.launch()function deploys the interface, allowing you to interact with it via a web browser. - The
share=Trueparameter generates a public URL, making the app accessible over the internet for others to use.
- The
Important Considerations:
-
API Token:
- Ensure you have a valid Hugging Face API token set in your environment to authenticate requests.
-
Model Availability:
- Ensure that "Meta-Llama-3.1-8B-Instruct" is available and you have access rights.
-
Streaming Outputs:
- The implementation involves collecting streaming outputs from the model, which might not be natively supported by all models or may require configuring stream settings properly.
-
Security:
- Be cautious when sharing the application publicly as it can expose interfaces to unexpected users.
This setup provides a user-friendly way to test and deploy language model interaction using Gradio and huggingface_hub, making AI capabilities accessible for a wide range of applications.
-