SIAM Journal on Applied Mathematics· 2026Q1
Correcting Autodifferentiation in Neural ODE Training
- 1citations
- Q1SCImago
- 2026year
Short summary
Brute-force autodifferentiation in Neural ODEs using high-order methods (like leapfrog or ERK) can introduce gradient oscillations that hinder convergence; simple postprocessing techniques can correct these gradients.
AI-generated from the title and abstract; the full text is not read.
Key points
- Autodifferentiation in Neural ODEs with high-order methods can produce spurious gradient oscillations.
- These oscillations prevent convergence during model training.
- Simple postprocessing techniques are proposed to correct gradients for leapfrog and 2-stage ERK methods.
- Corrected gradients lead to accurate updates and enable convergence.
AI-generated from the title and abstract; the full text is not read.
Abstract
Abstract. Does the use of autodifferentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by autodifferentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as linear multistep methods or explicit Runge–Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force autodifferentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of leapfrog and 2-stage ERK, we propose simple postprocessing techniques that effectively eliminate these oscillations, correct the gradient computation, and thus return the accurate updates.
The authors' abstract, as published at the source. SIAM Journal on Applied Mathematics, 2026 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Statistical and Nonlinear Physics
Statistical and Nonlinear PhysicsPhysics and Astronomy