Skip to content
The Chat Archive
Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written) OpenAssistant Conversations (oasst2)

Why do artificial neurons use a bias term? Shouldn't only using weights be enough?

In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
  1. User

    Why do artificial neurons use a bias term? Shouldn't only using weights be enough?

  2. Assistant

    This is important to introduce biases because it allows for the model to have adjustable thresholds values. Considering for instance a neuron with a ReLU activation function, the fact that this neuron outputs in the flat region of its activation function or not may only depend on a scale invariant condition on its input vector, which is not true in the presence of a bias. Hence we can see that the non linearity of biased neurons can extend further than that of unbiased ones, and that biased neurons may learn more objective criteria, like "this combination of my inputs is greater than 3" while unbiased neurons may only learn relative criteria, like "this first combination of my inputs is greater than this second one" which would remain unchanged if the whole input vector was multiplied by a positive factor.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.