Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0613· OpenAIHugging Face Arena Human Preference 55K

Why is Nesterov not always better than classical in practice, although there is a…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Metallama-2-70b-chat’s answer.
  1. User

    Why is Nesterov not always better than classical in practice, although there is a theoretical analysis showing Nesterov outperforms classical momentum ?

  2. Assistant

    Theoretically, the Nesterov Accelerated Gradient (NAG) method is shown to have better convergence rates than classical Momentum in convex optimization problems. However, in actual practice, there are several reasons why Nesterov might not always outperform classical momentum.

    1. Non-Convexity: Machine Learning problems often deal with non-convex optimization problems where the global minimum is hard to find. The theoretical advantages of Nesterov are mostly applicable to convex problems, and may not hold in non-convex settings.

    2. Hyperparameters: Both NAG and Momentum have hyperparameters that need to be tuned. If these are not optimally set, the performance of the algorithms can vary. It's possible that classical Momentum could perform better with optimal hyperparameters than Nesterov with non-optimal ones.

    3. Noise: In stochastic settings where the gradient is estimated using a mini-batch of data, there can be a lot of noise. Nesterov's method, which relies on future gradient estimates, can be particularly sensitive to this noise.

    4. Computational Overhead: Nesterov requires more computation per step than classical Momentum. In situations where computational resources are limited, this could make a difference in the practical effectiveness of the algorithm.

    5. Problem Specific: The performance of optimization algorithms can depend heavily on the specific problem at hand. While Nesterov may outperform classical Momentum on some problems, the opposite may be true for others.

    Therefore, despite the theoretical advantages, the practical performance of Nesterov and classical Momentum can depend on many factors, and one may not always outperform the other.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.