1. What is a gradient in machine learning?
A gradient is the vector of partial derivatives of a loss function with respect to model parameters. It points in the direction of the steepest increase of the function. Optimization algorithms update parameters in the negative gradient direction to minimize loss.
2. Difference between Gradient Descent, Stochastic Gradient Descent, and Mini-Batch Gradient Descent
- Gradient Descent: Uses the entire dataset to compute gradient (slow but stable).
- Stochastic Gradient Descent (SGD): Uses one sample at a time (fast but noisy).
- Mini-Batch Gradient Descent: Uses a subset (batch) → balance of speed and stability.
Noise helps escape local minima and enables parallelism on GPUs.
3. What is the vanishing gradient problem?
Gradients become extremely small when propagated backward through deep networks. This prevents early layers from learning effectively.
Main causes:
- Sigmoid or tanh activation saturation
- Poor initialization
Solutions: ReLU, batch normalization, residual connections, Xavier/He initialization.
4. What is the exploding gradient problem?
Gradients become excessively large and cause unstable training.
Solutions: Gradient clipping, lower learning rates, proper initialization.
5. Why does cross-entropy loss produce better gradients than MSE for classification?
MSE with sigmoid often produces tiny gradients because:
&hat;y (1 - &hat;y) ≈ 0
Cross-entropy simplifies the gradient to:
∂L/∂z = &hat;y - y
This avoids vanishing gradients and allows much faster training.
6. What is gradient clipping?
Gradient clipping limits the magnitude of gradients to avoid exploding gradients. Common in RNNs and very deep networks.
7. What is the role of gradient noise in SGD?
Noise from mini-batch sampling helps:
- Escape local minima
- Avoid saddle points
- Improve generalization
8. Why do we use backpropagation?
Backprop efficiently applies the chain rule through layers. It reduces complexity to O(L) instead of exponential. Frameworks reuse intermediate results to compute gradients efficiently.
9. What are saddle points and why do they matter?
A saddle point has zero gradient but is not a minimum. High-dimensional loss surfaces contain many saddle points. SGD helps escape them due to gradient noise.
10. How does learning rate affect gradients?
- Too high → diverges
- Too low → slow convergence
Schedulers like cosine decay, warmup, etc., improve gradient stability.
11. What is gradient flow?
Gradient flow refers to how gradients propagate through the layers during backprop. Poor flow leads to vanishing or exploding gradients.
12. What is automatic differentiation?
Frameworks like PyTorch and TensorFlow automatically compute gradients using reverse-mode differentiation. Efficient for functions with many inputs but few outputs (e.g., loss).
13. Analytical vs Numerical Gradients
Analytical: Computed by calculus formulas (exact).
Numerical: Computed with finite differences:
(f(x + ε) - f(x - ε)) / (2ε)
Used for debugging gradient implementations (gradient checking).
14. What is gradient checking?
It compares analytical gradients to numerical gradients to verify correctness of backprop. Done on small models due to computational cost.
15. How do optimizers modify gradients?
- SGD: Uses raw gradient.
- Momentum: Adds velocity to smooth updates.
- Adam: Adaptive learning rate + momentum.
- RMSProp: Divides gradient by moving average of squared gradients.