What safety benchmark would you create for a multimodality LLM model to localize abusive…
- User
What safety benchmark would you create for a multimodality LLM model to localize abusive segments in a video? Given a list of policies? Ideally, I should be able to retrieve the timestamps of all the abusive segments and an explanation of why they are abusive.
Assistant
I'd create a benchmark called "Temporal Abuse Localization Test" (TALT) with these key components:
-
Dataset Construction
- Diverse video corpus with precise human-annotated timestamps of policy violations
- Range of violation types, subtlety levels, and multimodal abuse (visual, audio, textual)
- Inclusion of edge cases where context determines violation status
-
Evaluation Metrics
- Temporal precision: How accurately timestamps capture abusive segments
- Policy classification accuracy: Correctly identifying which policies were violated
- Explanation quality: Relevance and specificity of reasoning
- False positive/negative rates across different abuse categories
-
Implementation Design
- Input: Video + list of applicable policies
- Output: JSON with timestamps, violated policies, and explanations
- Required format:
{start_time: float, end_time: float, policies_violated: [string], explanation: string}
-
Challenging Elements
- Implicit violations requiring contextual understanding
- Cross-modal violations (e.g., benign visuals with harmful audio)
- Cultural nuance detection
- Temporal context where meaning changes over time
This benchmark would evaluate both technical accuracy in localization and reasoning quality in explaining why specific content violates particular policies.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.