Module 7 of 14 · one week
A time-dependent PDE gives three ways to pose the learning problem, and the choice is not a matter of taste. Each formulation has a failure mode with a number attached, and the arithmetic of error accumulation says which one you can afford. This module derives that arithmetic, reads it off a published table, gives the two stabilizers that work, and states the wall none of them get past.
Two credible positions on this question contradict each other in public. The seed episode takes the whole-rollout view: predict the entire space-time block at once. The FNO paper agrees for its own experiments and says why. Meanwhile the weather models that run in production do the opposite. FourCastNet is trained to predict six hours ahead and then rolled out for months with, in Anandkumar's words, "a little bit of multi-step fine-tuning", a result she calls "very surprising" (earlier episode 01:00:03 to 01:00:40, research_notes/company-episode.md:62).
Both are right for their own problem, and the quantity that decides which is a property of the dynamics, not of the architecture. This module derives that quantity and then shows you can read it directly off MeshGraphNets' results table.
Learn \(F:u(t)\mapsto u(t+\Delta t)\), possibly from \(\ell\) past steps (PDEBench's \(\mathring F_\theta\), research_notes/pdf-mgn-pdebench.md:158-162), then iterate at test time. FNO-2D on Navier-Stokes does 2D convolution in space with an RNN structure in time, mapping ten prior steps to the next, in 414,517 parameters. It can run to any horizon. Its error compounds, and the next section says how fast.
Learn \(u_0\mapsto u(\cdot,t)\) for all \(t\in[0,T]\) as a single space-time function. FNO-3D convolves in space-time with 6,558,537 parameters and is "more expressive and easier to train" (ICLR p.8). It is fixed to \([0,T]\) but free in the time discretization inside it. No compounding, and no extrapolation either.
Learn \(u_0\mapsto u(\cdot,T)\) for one \(T\). Lab 01 run B is this, with \(T=1\). It is the cleanest operator-learning problem there is, and it answers no question at all about what happens between 0 and \(T\).
| FNO ICLR Table 1, 64×64, N = 1000 unless stated | Params | s/epoch | ν = 1e-3, T = 50 | ν = 1e-4, T = 30 | ν = 1e-4, N = 10,000 | ν = 1e-5, T = 20 |
|---|---|---|---|---|---|---|
| FNO-3D (whole trajectory) | 6,558,537 | 38.99 | 0.0086 | 0.1918 | 0.0820 | 0.1893 |
| FNO-2D (autoregressive) | 414,517 | 127.80 | 0.0128 | 0.1559 | 0.0834 | 0.1556 |
| U-Net | 24,950,491 | 48.67 | 0.0245 | 0.2051 | 0.1190 | 0.1982 |
| TF-Net | 7,451,724 | 47.21 | 0.0225 | 0.2253 | 0.1168 | 0.2268 |
| ResNet | 266,641 | 78.47 | 0.0701 | 0.2871 | 0.2311 | 0.2753 |
Read that table by column rather than by row (research_notes/pdf-fno-jmlr.md:131-146). Where data is abundant relative to difficulty, at ν = 1e-3 with N = 1000 or ν = 1e-4 with N = 10,000, the whole-trajectory model wins: 0.0086 and 0.0820. Where data is scarce relative to difficulty, at ν = 1e-4 or 1e-5 with N = 1000, every method is above 15 percent and the autoregressive FNO-2D is the least bad, at 0.1559 and 0.1556. The autoregressive model also does this on one sixteenth of the parameters, which is the usual advantage of not having to represent time explicitly.
To convert viscosities into something physical, the paper estimates the Reynolds number as \(\mathrm{Re} = \sqrt{0.1}/(\nu(2\pi)^{3/2})\), citing Chandler and Kerswell (JMLR p.33). Since \((2\pi)^{3/2} = 15.75\), that gives Re ≈ 20 at ν = 1e-3, Re ≈ 201 at ν = 1e-4 and Re ≈ 2008 at ν = 1e-5. So the often-quoted summary "under 1 percent at Re 20, 8 percent at Re 200" is the 0.0086 and the 0.0820, and the second of those needed ten times the training data.
The objection to autoregressive rollout is that errors accumulate. That is true and unhelpfully vague, because the rate of accumulation varies by orders of magnitude between problems, and it is a property of the physics.
Claim. Let the learned one-step map \(F_\theta\) satisfy \(\|F_\theta(v)-F_\theta(w)\|\le L\|v-w\|\) and let its one-step error be at most \(\varepsilon\), meaning \(\|F_\theta(u)-F(u)\|\le\varepsilon\) for every state \(u\) on the true trajectory. Write \(e_n\) for the error after \(n\) rollout steps. Then \[e_n \;\le\; \varepsilon\,\frac{L^n-1}{L-1} \quad (L\ne1), \qquad e_n \le n\varepsilon \quad (L=1).\]
Proof. Split the step into the two things that go wrong, using \(u_{n+1}=F(u_n)\) for the true state and \(\hat u_{n+1}=F_\theta(\hat u_n)\) for the predicted one: \[e_{n+1} = \|F_\theta(\hat u_n) - F(u_n)\| \le \underbrace{\|F_\theta(\hat u_n)-F_\theta(u_n)\|}_{\le\,Le_n} + \underbrace{\|F_\theta(u_n)-F(u_n)\|}_{\le\,\varepsilon}.\] So \(e_{n+1}\le Le_n+\varepsilon\) with \(e_0=0\), and unrolling that recursion gives \(e_n \le \varepsilon(1+L+\cdots+L^{n-1})\), which is the geometric sum above. \(\square\)
The three regimes. If \(L<1\) the dynamics contracts, the sum converges, and \(e_n \le \varepsilon/(1-L)\) for every \(n\): the rollout is stable forever, however long. If \(L=1\) the error grows linearly, \(n\varepsilon\), which is slow enough to be workable. If \(L>1\) it grows like \(L^n\), and no achievable \(\varepsilon\) buys many steps: to hold \(e_n\) below a tolerance \(\delta\) you need \(L^n \lesssim \delta(L-1)/\varepsilon\), so halving \(\varepsilon\) extends the horizon by \(\log 2/\log L\) steps, a fixed number that does not grow however small \(\varepsilon\) already is. At \(L = 2\) that is exactly one step. Chaotic systems have \(L>1\) by definition, since a positive Lyapunov exponent is exactly the statement that nearby states separate exponentially.
The useful part is that a results table with both a one-step and a rollout column tells you which regime a problem is in, without any extra experiment. Compare the ratio of rollout-50 error to one-step error against 50, which is what pure linear accumulation would give. Using MeshGraphNets Table 1 (Module 6 has it in full):
| Dataset | 1-step ×10⁻³ | rollout-50 ×10⁻³ | ratio | vs 50 (linear) | Regime |
|---|---|---|---|---|---|
| FlagSimple (cloth) | 1.08 | 92.6 | 86 | 1.7× faster | amplifying, \(L>1\) |
| FlagDynamic (cloth, remeshed) | 1.57 | 72.4 | 46 | about linear | marginal |
| SphereDynamic | 0.292 | 11.5 | 39 | about linear | marginal |
| DeformingPlate | 0.25 | 1.8 | 7.2 | 7× slower | contracting |
| CylinderFlow | 2.34 | 6.3 | 2.7 | 19× slower | strongly contracting |
| Airfoil | 314 | 582 | 1.9 | 27× slower | strongly contracting |
The pattern is exactly what the physics predicts. One step of care first: the \(L\) in the claim is the Lipschitz constant of the learned map \(F_\theta\), while contraction and chaos are properties of the true map \(F\). The two are tied only because a model that approximates \(F\) well on the relevant states inherits its stretching behaviour, so \(L\) tracks the true constant. A badly fitted model can be expansive on a contracting problem, which is one of the things a rollout curve detects. Viscous flow settling toward a periodic wake is contracting, so CylinderFlow accumulates 19 times more slowly than linear and its rollout is cheap. A hyperelastic plate relaxing to equilibrium is contracting too. Cloth in the wind is chaotic, and FlagSimple's error grows 1.7 times faster than linear over the first 50 steps, which is the visible face of \(L>1\). Over the full 400 steps FlagSimple's ratio reaches only 129, far short of the thousands that \(L^{400}\) would suggest, because the state space is bounded: the flag cannot be more wrong than a completely different flag. The error saturates rather than diverging, which is why MeshGraphNets reports rollout-50 as the discriminating number and attributes the rest to decoherence after roughly 50 steps (research_notes/pdf-mgn-pdebench.md:129).
What that buys you before you train anything. Run the solver twice from slightly perturbed initial conditions and watch how the difference grows. If it shrinks, an autoregressive model with a stabilizer will roll out a long way and a whole-trajectory model is wasted capacity. If it grows exponentially, no rollout length is safe and the honest deliverable is a statistical one: the right question becomes whether the model reproduces the attractor's statistics, not whether it tracks a trajectory. That question is Module 12's, where the weather models live.
The bound above assumed the one-step error \(\varepsilon\) holds "for every state on the true trajectory", and that is precisely the assumption training does not deliver. A one-step model is trained on clean states drawn from the data and tested on its own outputs, which drift off that distribution as soon as the first error is made. Both known fixes attack that mismatch rather than the accumulation.
During training, add zero-mean Gaussian noise of a tuned scale to the most recent input state and correct the target so the network learns to remove the perturbation instead of ignoring it. The model therefore sees, at training time, the kind of slightly-wrong state it will meet at rollout time. The scale is tuned per dataset on a log grid with two values per decade, centred on the natural one-step error: FlagSimple position 1e-3, FlagDynamic 3e-3, CylinderFlow momentum 2e-2, Airfoil momentum 1e1 and density 1e-2 (research_notes/pdf-mgn-pdebench.md:50). For second-order systems such as cloth the target correction is ill-posed, and a mixing parameter \(\gamma=0.1\) worked best. The payoff is that a model trained only on one-step supervision rolls out 250 to 600 steps.
Unroll two steps during training, backpropagate only through the second, and treat the first step's output as an adversarial perturbation that the gradient is not allowed to remove (research_notes/foundation.md:172-183). The effect is the same as noise injection in spirit, but the perturbation is the model's own error distribution, not a Gaussian guess at it. On Burgers at \(n_x=100\), the time before error exceeds 10 percent roughly doubles, from about 2 to 3 seconds to about 5 to 6. In their comparison an unrolled FNO-RNN reaches error 29.98 while FNO with pushforward plus temporal bundling reaches 0.51, against WENO5's 6.23 in the same units: a factor of 59.
The two stabilizers are not interchangeable, and one paper reports the other failing. Brandstetter et al. find that plain Gaussian noise injection "actually performs worse than no stabilization" in their experiments. That is not a contradiction of MeshGraphNets, and the difference is instructive: MeshGraphNets tunes the noise scale per dataset on a log grid, and Brandstetter's baseline does not. A stabilizer with an untuned hyperparameter is a different method from the same stabilizer with a tuned one. PDEBench landed on pushforward for its U-Net baseline after naive autoregressive training proved unstable, and reports that it gave "the best stability for all problems and the best accuracy over longer prediction horizons" (Section 4.3, pdf-mgn-pdebench.md:261).
Both stabilizers keep a rollout stable inside the horizon the training data covered. Neither gets past the end of it, and that limit is reported consistently across the literature.
PDEBench trains FNO on half the time steps of 1D advection and 1D Burgers and evaluates beyond them. Error "increases monotonically for \(t\) beyond the training window", and the paper's conclusion is unhedged: "present ML models cannot properly capture the dynamic behavior of the PDEs, and it is a challenging task to provide reliable predictions beyond the time experienced during training" (p.9, :263). DPOT, a 0.5 billion parameter pretrained model, still shows 33.5 percent full-trajectory error on long Kolmogorov turbulence rollouts, improved from 82.2 percent without pretraining (foundation.md:55). That is a real improvement and it is not a solution.
The rules this leaves you with. Use a whole-trajectory model when you have the data and a fixed horizon. Use an autoregressive model with a tuned stabilizer when the horizon must be open. Measure the growth rate of a perturbation in the solver before choosing, because that is what decides whether either will work. Always report rollout error against horizon as a curve, never a single number, since a good one-step model can still drift. And keep "resolution transfer" and "time extrapolation" as separate lines in every report; they are different questions with different answers, and the project kit's evaluation section says so (../05_physics_foundation_model_study_kit.html).
LAB Lab 01 run A is a whole-trajectory model. It maps \(u_0\) on 16 points to \(u(t,x)\) on a 17 by 16 space-time block and reaches test relative \(L^2\) 0.0044 (../labs/01_burgers/results_a.json). It has no rollout and no horizon beyond \(t=1\), so it cannot answer this module's question on its own. Three exercises turn it into the experiment.
research_notes/pdf-fno-jmlr.md:84-89, :131-146. Reynolds estimate \(\sqrt{0.1}/(\nu(2\pi)^{3/2})\) citing Chandler and Kerswell, read from 21-1524.pdf p.33; notes at :592-595.2010.03409v4.pdf), error methodology Appendix A.5.2: research_notes/pdf-mgn-pdebench.md:50, :78-89, :129.:261-263.research_notes/foundation.md:172-183.foundation.md:45-56.research_notes/company-episode.md:62.../labs/01_burgers/results_a.json.You can now derive how a rollout's error grows from the one-step error and a Lipschitz constant, diagnose which regime a problem is in from a published results table, name the two stabilizers and the condition under which one of them fails, and say what none of them fix.