The field equations of penalized non-parametric regression

Abstract

Abstract We view penalized risks through the lens of the calculus of variations. We consider risks comprised of a fitness-term (e.g. MSE) and a gradient-based penalty and search for the minima of such risks. After establishing the Euler-Lagrange field equations as a systematic approach to finding minimizers of risks involving only first derivatives, we proceed to exemplify this approach to the MSE penalized by the integral over the squared $$\ell ^2$$ ℓ 2 -norm of the gradient of the regression function. The minimizer of this risk is given as the solution to a second order inhomogeneous partial differential equation, where the inhomogeneity is given as the conditional expectation of the target variable conditioned on the features. We discuss properties of the field equations and practical implications thereof, which also apply to the classical ridge penalty for linear models, and embed our findings into the existing literature. In particular, we find that we can recover the Rudin-Osher-Fatemi model for image-denoising, if we consider the features as deterministic and evenly distributed. We proceed to derive the minima of the population-MAE and the Cramer Distance using the calculus of variations. Last, we exemplify the usage of the field equations for post-training regularization in a simulation study.

The authors' abstract, as published at the source. Statistical Papers, 2026 · DOI ↗