Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-instant-1· AnthropicHugging Face Arena Human Preference 55K

When is the Fisher Information matrix equal to the Hessian?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-3.5-turbo-1106’s answer.
  1. User

    When is the Fisher Information matrix equal to the Hessian?

  2. Claude Assistant

    The Fisher information matrix and the Hessian are equal under certain conditions:

    • The Fisher information matrix is equal to the Hessian of the log-likelihood function when the observed data is distributed according to the assumed probability distribution.

    • Specifically, if the observed data is distributed according to a distribution from the exponential family, then the Fisher information matrix is equal to the negative of the expected value of the Hessian of the log-likelihood function with respect to the natural parameters.

    • For example, if the observed data is distributed as a multivariate Gaussian distribution with mean μ and covariance Σ, then the Fisher information matrix is equal to the negative Hessian of the log-likelihood with respect to μ and the elements of Σ.

    • So in summary, the Fisher information matrix and Hessian will be equal when:

    1. The assumed probability distribution of the observed data is from the exponential family.

    2. The Fisher information matrix is taken with respect to the natural parameters of the distribution.

    3. The Hessian is of the log-likelihood function with respect to the same natural parameters.

    So whenever these conditions hold, the Fisher information matrix and Hessian matrix will be numerically equal.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.