why is it that gradient descent works well for cos function when we feature engineer…
- User
why is it that gradient descent works well for cos function when we feature engineer upto x13 and not if it is upto around x4?
x = np.arange(0,20,1) y = np.cos(x/2)
X = np.c_[x, x2, x3,x4, x5, x6, x7, x8, x9, x10, x11, x12, x13] X = zscore_normalize_features(X)
model_w,model_b = run_gradient_descent_feng(X, y, iterations=1000000, alpha = 1e-1)
plt.scatter(x, y, marker='x', c='r', label="Actual Value"); plt.title("Normalized x x2, x3 feature") plt.plot(x,X@model_w + model_b, label="Predicted Value"); plt.xlabel("x"); plt.ylabel("y"); plt.legend(); plt.show()
- Assistant
The behavior you're observing with gradient descent in fitting a polynomial regression model to a cosine function is likely due to the complexity of the polynomial used and how well it can approximate the target function, ( \cos(x/2) ). Here’s why using a higher-degree polynomial (like up to ( x^{13} )) works better:
-
Function Complexity and Polynomial Degree:
- The cosine function is periodic and non-linear. Approximating it with lower-degree polynomials (like up to ( x^4 )) would fail to capture the periodic nature effectively. These polynomials aren't flexible enough to model the oscillations, leading to high bias.
-
Fourier Series and Polynomial Approximation:
- Trigonometric functions like cosine can be approximated by a Fourier series, which is inherently a sum of sine and cosine terms. High-degree polynomials are better at mimicking such complex, periodic structures because they have more terms to adjust to the curvature and oscillations.
-
Overfitting and High Bias:
- A lower-degree polynomial has fewer parameters, meaning it has a simpler representation and is more likely to underfit the data (high bias) when the data generation process is complex, as with ( \cos(x/2) ).
-
Capacity to Capture Oscillations:
- Higher-degree polynomials increase the capacity of the model to capture rapid changes and oscillations in the data. This allows gradient descent to find parameter values that better optimize the function fit.
-
Z-score Normalization:
- By normalizing the features using Z-score normalization, you make the gradient descent optimization process more stable and efficient, as it mitigates issues associated with feature scaling. This especially benefits higher-degree terms, which can have large values otherwise.
However, bear in mind that with higher-degree polynomials, there is also a risk of overfitting if there were noise in your data — though in your case ( y = \cos(x/2) ), the data follows a deterministic function without noise.
In conclusion, the higher-degree polynomial provides sufficient flexibility to closely fit the ( \cos(x/2) ) function, capturing its periodic behavior more effectively than a lower-degree polynomial. This is why gradient descent is able to find a better fit in the higher-dimensional feature space.
-