Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219· AnthropicHugging Face LMArena Human Preference 140K

How should I run LLM evals using langsmith?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro-preview-03-25’s answer.
  1. User

    How should I run LLM evals using langsmith?

  2. Claude Assistant

    Running LLM Evaluations with LangSmith

    LangSmith is a platform that helps you evaluate, monitor, and debug LLM applications. Here's how to run evaluations using LangSmith:

    1. Set Up LangSmith

    import os
    from langsmith import Client
    
    # Set your API key
    os.environ["LANGCHAIN_API_KEY"] = "your_api_key_here"
    client = Client()
    

    2. Create a Dataset

    You need a dataset to evaluate against. You can:

    # Create a new dataset
    dataset = client.create_dataset(
        "My Evaluation Dataset",
        description="Dataset for testing my LLM"
    )
    
    # Add examples
    client.create_example(
        inputs={"question": "What is the capital of France?"},
        outputs={"answer": "Paris"},
        dataset_id=dataset.id
    )
    
    # Or load from existing data
    from langsmith.schemas import Example
    examples = [
        Example(
            inputs={"question": "What is the capital of France?"},
            outputs={"answer": "Paris"}
        ),
        # More examples...
    ]
    dataset = client.create_dataset("My Dataset", description="...", examples=examples)
    

    3. Run Evaluations

    Basic Evaluation

    from langchain.chat_models import ChatOpenAI
    from langchain.smith import RunEvalConfig, run_on_dataset
    
    # Define your model
    model = ChatOpenAI(temperature=0)
    
    # Run evaluation
    eval_config = RunEvalConfig(
        evaluators=["criteria", "embedding_distance"],  # Built-in evaluators
    )
    
    results = run_on_dataset(
        dataset_name="My Evaluation Dataset",
        llm_or_chain=model,
        evaluation=eval_config,
    )
    

    Custom Evaluation Metrics

    from langchain.evaluation import StringEvaluator
    
    class MyCustomEvaluator(StringEvaluator):
        def evaluate_strings(self, prediction, reference):
            # Your custom evaluation logic
            score = calculate_score(prediction, reference)
            return {"score": score}
    
    # Add to your eval config
    eval_config = RunEvalConfig(
        custom_evaluators=[MyCustomEvaluator()],
    )
    

    Using LLM-based Evaluators

    from langchain.evaluation.criteria import LLMCriteriaEvaluator
    
    evaluator = LLMCriteriaEvaluator(
        criteria={
            "accuracy": "Is the response factually accurate?",
            "helpfulness": "Is the response helpful and addressing the question?",
        }
    )
    
    eval_config = RunEvalConfig(
        custom_evaluators=[evaluator],
    )
    

    4. Analyze Results

    After running evaluations, you can:

    1. View results in the LangSmith UI dashboard
    2. Analyze programmatically:
    # Get feedback from runs
    feedback = client.list_feedback(dataset_id=dataset.id)
    
    # Analyze results
    import pandas as pd
    feedback_df = pd.DataFrame([f.dict() for f in feedback])
    

    5. Continuous Evaluation

    For production systems, set up continuous evaluation:

    # Track runs in production
    from langchain.callbacks.tracers import LangChainTracer
    
    tracer = LangChainTracer(project_name="My Production App")
    
    # Use in your chain
    chain.invoke(input, {"callbacks": [tracer]})
    

    This provides a comprehensive workflow for evaluating LLMs using LangSmith, from dataset creation to analyzing results.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.