Skip to content
TheChatArchiveThe Chat Archive
Metallama-2-70b-chat· MetaHugging Face Arena Human Preference 55K

what are good questions to ask LLMs in order to reverse-engineer them?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-instant-1’s answer.
  1. User

    what are good questions to ask LLMs in order to reverse-engineer them?

  2. Meta Assistant

    Reverse-engineering an LLM (LL.M.) program can be a challenging task, as it requires understanding the program's inner workings, algorithms, and decision-making processes. However, asking the right questions can help you gain valuable insights into how the program works and what factors it considers when making predictions or recommendations. Here are some questions to ask an LLM program to help you better understand its workings:

    1. What is the main objective of the program? Understanding the program's primary goal will help you identify the types of algorithms and techniques it uses to achieve that goal. For example, if the program is designed to predict stock prices, it may use time-series analysis, machine learning algorithms, or statistical models.
    2. What data sources does the program use? LLM programs rely on vast amounts of data to train their models and make predictions. Asking about the data sources used by the program can help you understand the types of inputs it considers and how it processes that data. For instance, a program that uses financial data may rely on historical stock prices, economic indicators, or financial news.
    3. What features does the program extract from the data? Features are the specific variables or characteristics that the program uses to make predictions or recommendations. Asking about the features used by the program can help you understand how it processes the data and what factors it considers most important. For example, a program that predicts stock prices may extract features such as moving averages, trend indicators, or sentiment analysis.
    4. What algorithms or techniques does the program use? LLM programs use a variety of algorithms and techniques to analyze data and make predictions. Asking about the specific algorithms or techniques used by the program can help you understand how it arrives at its conclusions. For instance, a program that uses natural language processing may employ techniques such as sentiment analysis, topic modeling, or text classification.
    5. How does the program handle missing or incomplete data? Missing or incomplete data can be a significant challenge for LLM programs. Asking about the program's strategies for handling such data can help you understand how it deals with gaps in the data and how it affects the program's accuracy. For example, a program may use imputation techniques, such as mean or median substitution, or machine learning algorithms, such as decision trees or neural networks, to handle missing data.
    6. What are the program's performance metrics? Understanding the program's performance metrics can help you evaluate its accuracy and effectiveness. Asking about the program's metrics can help you identify the specific areas where the program excels or struggles. For example, a program that predicts stock prices may use metrics such as mean absolute error (MAE), mean squared error (MSE), or root mean squared error (RMSE) to evaluate its performance.
    7. Can the program provide explanations or interpretations for its predictions or recommendations? Some LLM programs can provide explanations or interpretations for their predictions or recommendations, while others may rely solely on statistical models. Asking about the program's ability to provide explanations can help you understand how it arrives at its conclusions and whether it can provide insights into the decision-making process.
    8. How does the program handle outliers or unusual data points? Outliers or unusual data points can significantly impact an LLM program's accuracy. Asking about the program's strategies for handling such data can help you understand how it deals with unexpected events or data points. For example, a program may use techniques such as z-score normalization, robust regression, or machine learning algorithms to handle outliers.
    9. Can the program adapt to new data or changing conditions? LLM programs must be able to adapt to new data or changing conditions to remain effective. Asking about the program's ability to adapt can help you understand whether it can learn from new data and whether it can adjust its models or algorithms accordingly. For instance, a program that uses machine learning algorithms may be able to learn from new data and adjust its weights or biases accordingly.
    10. How does the program handle conflicting or contradictory data? Conflicting or contradictory data can be a challenge for LLM programs. Asking about the program's strategies for handling such data can help you understand how it resolves conflicts and arrives at its conclusions. For example, a program may use techniques such as weighted averaging, ensemble methods, or conflict resolution algorithms to handle conflicting data.

    By asking these questions, you can gain a deeper understanding of how an LLM program works, what factors it considers when making predictions or recommendations, and how it handles various challenges associated with data analysis and decision-making.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.