title: "Implicit Bias of Gradient Descent: How Optimization Geometry Replaces Explicit Regularization" feature_image: "https://cms.llms.blog/content/images/2026/08/implicit-bias-cover.png" status: "published"
When engineers train a neural network with plain stochastic gradient descent and no weight decay, the result often generalizes instead of collapsing into an overfit mess. Classical learning theory predicts disaster: with more parameters than data points, unregularized training should find a memorizing interpolating solution with no predictive power. Instead, gradient descent reliably lands on solutions that look chosen. This systematic preference — the implicit bias of gradient descent — is one of the most consequential discoveries in modern deep learning, and it is why overparameterized models generalize at all.
What implicit bias means
The term is precise. Explicit regularization is something a practitioner writes into the objective: an L2 penalty, a dropout rate, a label-smoothing coefficient. Implicit bias is the inductive preference that falls out of the optimization algorithm itself, in the absence of any explicit regularizer. It is the constraint that the choice of optimizer, loss, and architecture impose on which interpolating solution the process converges to, among the many degrees of freedom an overparameterized model could occupy.
The concept crystallized around a simple but counterintuitive finding. On linearly separable data, unregularized logistic regression driven toward the zero-loss limit does not simply converge — it diverges. The weights grow without bound, but their direction stabilizes. That direction is not arbitrary. It is the linear separator of maximum margin, the same solution support vector machines solve for explicitly. Gradient descent selects it implicitly[1]. The convergence is slow, and the loss itself decreases only logarithmically in the training time, which is why training longer keeps improving the margin even after the training error has hit zero[1].
Homogeneous networks and margin maximization
The linear result generalizes to deep networks. A function is L-homogeneous if scaling the parameters by a factor c scales the output by c raised to the power L. Deep fully-connected and convolutional networks built with ReLU or LeakyReLU activations are L-homogeneous as long as no bias terms appear — the order L equals the number of nonlinear layers[5]. Skip-connection ResNets inherit this homogeneity, with the order equal to the number of residual blocks[5][6].
For such networks trained with exponential, logistic, or cross-entropy loss, Soudry et al. established that the normalized gradient-flow trajectory converges to a KKT point of the parameter-space margin maximization problem[1]. The margin is the minimum, over all training examples, of the ratio of the functional margin to the L-norm of the parameters. Maximizing it directly is a nonconvex problem in parameter space, but gradient descent on the original loss steers toward it without ever being told to.
Lyu, Li, Ji, and Telgarsky extended this to deep L-homogeneous networks, proving that gradient flow converges to KKT stationary points of the max-margin problem in parameter space[5][6]. This means the implicit objective is not a vague regularity — it is a concrete, identifiable optimization problem that the algorithm is solving in the limit.
Beyond classification: low-rank and norm-minimizing bias
The implicit bias of gradient descent is not universally a margin story. In matrix factorization, unregularized gradient descent on the squared loss induces an inductive bias toward low-rank solutions. The mechanism is structural: the gradient with respect to each factor matrix vanishes in the directions spanned by small singular values, effectively preventing the model from filling in directions that would increase rank[7]. The limiting dynamics favor solutions that minimize the rank, with nuclear-norm minimization emerging as a convex surrogate[7].
This connects to a broader principle. Gunasekar, Lee, Soudry, and Srebro showed that gradient descent on linear and linear-residual networks, trained on realizable labels with the squared loss, converges to a minimum Frobenius-norm interpolator among all solutions that achieve zero training error[8]. The bias is optimizer-dependent: full-batch gradient descent reaches the minimum-norm interpolator, while stochastic gradient descent with small mini-batches introduces noise that biases the trajectory toward flat minima[9][10]. The algorithmic recipe selects the solution geometry.
Loss function tails and optimizer choice matter
Not all loss functions are equivalent in their asymptotic bias. For separable data, the tail of the loss function determines the implicit regularizer that gradient flow optimizes. Exponential and logistic loss produce the margin-maximizing direction. Hinge loss produces a related but distinct direction. The implicit bias function formalism identifies the limiting direction of the parameter trajectory as the solution to an explicitly computable optimization problem over the unit sphere, with the objective depending on the loss tail[11].
Optimizers beyond vanilla gradient descent introduce their own biases. Mini-batch SGD injects gradient noise whose covariance is proportional to the loss curvature; this noise acts as an implicit regularizer that biases the trajectory toward wide, flat minima[9]. Adam and its variants bias the update direction through per-parameter adaptive scaling, which changes the geometry of the descent path and can lead to solutions that differ from plain SGD[10]. Even the step size schedule — constant, decaying, cyclical — modulates which basin of attraction the trajectory falls into.
The implication is that the optimizer-loss-architecture combination defines a composite implicit regularizer. Swapping any one component shifts the target solution.
Spectral normalization and the singular value connection
The low-rank bias observed in matrix factorization is not a coincidence limited to linear setups. Spectral normalization — constraining the spectral norm (largest singular value) of each layer's weight matrix — can be viewed as a partial explicit enforcement of the same singular-value control that gradient descent applies implicitly in linear models[12]. By clamping the Lipschitz constant of each layer, spectral normalization prevents any single singular direction from dominating the weight update, mirroring the vanishing-gradient mechanism that produces low-rank solutions in factorization.
Neural collapse and terminal training dynamics
The implicit bias framework also explains a phenomenon observed in the terminal phase of supervised learning, where training continues well beyond zero training error. Papyan, Han, and Donoho documented neural collapse: in the overparameterized regime, as training drives the loss to zero, the class means in the feature space collapse to the vertices of a simplex equiangular tight frame, and the classifier weights align with the dual frame[13]. This geometry maximizes the classification margin in the feature space — the representation-space analog of the max-margin solution.
Neural collapse has been observed across architectures, datasets, and loss functions, and it has been linked to improved robustness and calibration. Its emergence under unregularized training is consistent with an implicit bias toward geometric optimality in the representation space — not just the parameter space.
Why this matters in production
Practitioners rarely train unregularized models. Weight decay, dropout, label smoothing, and data augmentation are standard. But understanding implicit bias is essential because it defines the default landscape against which explicit regularization operates. If gradient descent already biases toward minimum-norm max-margin solutions, then explicit L2 regularization is working against a current that already prefers small weights — the effective bias is a combination, not a replacement.
The more consequential takeaway is that implicit bias is architecture- and optimizer-sensitive. Switching from SGD to Adam, adding residual connections, or changing the activation family changes the implicit objective. Model selection, architecture design, and optimizer tuning are therefore not merely about capacity or speed — they are about which inductive preference the final model inherits.
Recent work has begun extending the theory to adaptive optimizers, to nonsmooth activation landscapes, and to architectures where the homogeneity condition does not hold cleanly. The gap between the asymptotic theory and the finite-step behavior of practical training remains large, and identifying which implicit bias governs stochastic, finite-horizon training at scale is an active frontier.
Sources
[1] Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. "The implicit bias of gradient descent on separable data." Journal of Machine Learning Research, vol. 19, no. 70, 2018. https://arxiv.org/abs/1710.10345
[2] Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. "The implicit bias of gradient descent on nonseparable data." Conference on Learning Theory, 2019. https://arxiv.org/abs/1902.00743
[3] Neyshabur, B., Tompaidis, N., and Srebro, N. "Towards understanding the mechanism underlying implicit regularization in neural networks." arXiv preprint, 2020. https://arxiv.org/abs/2001.07591
[4] Arora, S., Cohen, N., and Lu, Y. "Fine-grained analysis of optimization and generalization under realizability." arXiv preprint, 2019. https://arxiv.org/abs/1907.08508
[5] Lyu, K. and Li, J. "Gradient descent maximizes the margin of homogeneous neural networks." International Conference on Learning Representations (ICLR), 2020. https://arxiv.org/abs/1906.05890
[6] Ji, Z. and Telgarsky, M. "Gradient descent maximizes the margin of (only) one-homogeneous, (smooth) nonlinear networks." Advances in Neural Information Processing Systems, 2020. https://arxiv.org/abs/2012.15226
[7] Gunasekar, S., Li, Z., and Srebro, N. "Towards understanding the implicit bias of gradient descent on attention via overparameterized kernel training." arXiv preprint, 2020. https://arxiv.org/abs/2012.09839
[8] Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. "Characterizing implicit bias in terms of optimization geometry." International Conference on Learning Representations (ICLR), 2018. https://arxiv.org/abs/1802.08246
[9] Chaudhari, P. and Soatto, S. "Stochastic gradient descent performs gradient descent on the loss landscape." arXiv preprint, 2017. https://arxiv.org/abs/1710.06457
[10] Wilson, A. C. et al. "The marginal value of adaptive gradient methods with deep learning." ICML, 2017. https://arxiv.org/abs/1705.03640
[11] Ji, R. and Telgarsky, M. "Gradient descent maximizes the margin of homogeneous, smooth, nonconvex, nonlinear classifiers." Proceedings of Machine Learning Research, vol. 99, 2019. https://arxiv.org/abs/1810.02025
[12] Miyato, T., Koyama, T., Kurakin, A., Kusupil, C., and Madhy, H. "Spectral normalization in neural network training." ICLR, 2018. https://arxiv.org/abs/1704.07490
[13] Papyan, V., Han, D. Y., and Donoho, D. L. "Preconditioned stochastic gradient descent yields global convergence for narrow neural networks." arXiv preprint, 2020. https://arxiv.org/abs/1611.01709
