How should I run LLM evals using langsmith?
- User
How should I run LLM evals using langsmith?
Assistant
Running LLM Evaluations with LangSmith
LangSmith is a platform that helps you evaluate, monitor, and debug LLM applications. Here's how to run evaluations using LangSmith:
1. Set Up LangSmith
import os from langsmith import Client # Set your API key os.environ["LANGCHAIN_API_KEY"] = "your_api_key_here" client = Client()2. Create a Dataset
You need a dataset to evaluate against. You can:
# Create a new dataset dataset = client.create_dataset( "My Evaluation Dataset", description="Dataset for testing my LLM" ) # Add examples client.create_example( inputs={"question": "What is the capital of France?"}, outputs={"answer": "Paris"}, dataset_id=dataset.id ) # Or load from existing data from langsmith.schemas import Example examples = [ Example( inputs={"question": "What is the capital of France?"}, outputs={"answer": "Paris"} ), # More examples... ] dataset = client.create_dataset("My Dataset", description="...", examples=examples)3. Run Evaluations
Basic Evaluation
from langchain.chat_models import ChatOpenAI from langchain.smith import RunEvalConfig, run_on_dataset # Define your model model = ChatOpenAI(temperature=0) # Run evaluation eval_config = RunEvalConfig( evaluators=["criteria", "embedding_distance"], # Built-in evaluators ) results = run_on_dataset( dataset_name="My Evaluation Dataset", llm_or_chain=model, evaluation=eval_config, )Custom Evaluation Metrics
from langchain.evaluation import StringEvaluator class MyCustomEvaluator(StringEvaluator): def evaluate_strings(self, prediction, reference): # Your custom evaluation logic score = calculate_score(prediction, reference) return {"score": score} # Add to your eval config eval_config = RunEvalConfig( custom_evaluators=[MyCustomEvaluator()], )Using LLM-based Evaluators
from langchain.evaluation.criteria import LLMCriteriaEvaluator evaluator = LLMCriteriaEvaluator( criteria={ "accuracy": "Is the response factually accurate?", "helpfulness": "Is the response helpful and addressing the question?", } ) eval_config = RunEvalConfig( custom_evaluators=[evaluator], )4. Analyze Results
After running evaluations, you can:
- View results in the LangSmith UI dashboard
- Analyze programmatically:
# Get feedback from runs feedback = client.list_feedback(dataset_id=dataset.id) # Analyze results import pandas as pd feedback_df = pd.DataFrame([f.dict() for f in feedback])5. Continuous Evaluation
For production systems, set up continuous evaluation:
# Track runs in production from langchain.callbacks.tracers import LangChainTracer tracer = LangChainTracer(project_name="My Production App") # Use in your chain chain.invoke(input, {"callbacks": [tracer]})This provides a comprehensive workflow for evaluating LLMs using LangSmith, from dataset creation to analyzing results.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.