Module 8 of 14 · one week
RMSE is not a report card. PDEBench's own headline finding is that it "is not a good proxy for evaluating ML models, in particular in turbulent and non-smooth regimes". This module defines the metric suite, shows what each metric caught in the benchmark, and then computes the whole suite on this course's two labs, where it finds a grid-scale oscillation and a broken boundary condition that the field error hides completely.
The problem takes one sentence, using this course's own results. Lab 01's relative \(L^2\) at 1024 points is 0.035, the picture of the prediction is visually identical to the truth, and the output nevertheless carries an odd-even oscillation of amplitude 0.014 that neither the number nor the picture separates from anything else (Module 2). Lab 02 is worse: its field error at 128² is twelve times its error at 64², and the quantity a thermal engineer actually needs is not the field at all but the peak temperature and where it sits.
A metric that cannot fail a model is not a metric, it is a formality. The teaching survey found no course that covers evaluation and rollout metrics for neural PDE solvers (research_notes/teaching.md:215), which is why this module exists and why it comes before the scaling and foundation-model modules, not after them. You cannot audit a claim about scale without a measurement that can come back negative.
PDEBench's suite splits into metrics computed on the data and metrics computed on the physics, and the split is the useful part: the first group asks whether the numbers are close, the second asks whether the answer is a solution of anything.
| Scope | Name | Definition (PDEBench Appendix B) | What it can fail |
|---|---|---|---|
| Data | RMSE | root mean square error | bulk error, scale-dependent |
| Data | nRMSE | \(\|u_{\text{pred}}-u_{\text{true}}\|_2/\|u_{\text{true}}\|_2\) | bulk error, scale-free; this is the "relative \(L^2\)" of every FNO table in the course |
| Data | max error | largest single pointwise error | local blow-up; a stability proxy |
| Physics | cRMSE | \(\|\sum u_{\text{pred}}-\sum u_{\text{true}}\|_2/N\), on a quantity the PDE conserves | a violated conservation law |
| Physics | bRMSE | RMSE restricted to the boundary points | a boundary condition the model never learned |
| Physics | fRMSE, banded | \(\sqrt{\sum_{k=k_{\min}}^{k_{\max}}|F(u_{\text{pred}})-F(u_{\text{true}})|^2/(k_{\max}-k_{\min}+1)}\), with low \(k\in[0,4]\), mid \([5,12]\), high \([13,\infty)\); in 2D and 3D integrate out the angle and sum radially | error hidden inside one band: Gibbs oscillation at shocks, over-smoothed fine scales, grid-scale oscillations |
The banded fRMSE is the one to understand instead of memorise, because it is the only member of the suite that can localise an error in frequency, and every defect this course has found so far lives at one end of the spectrum. Note one property of the definition that matters when reading the numbers below: it divides by the number of bins in the band, so it is a per-bin average and not a total. That makes it comparable across bands of different widths but means a single badly wrong bin gets diluted by however many quiet bins sit beside it.
Two more metrics come from the rollout literature. Error against horizon, reported as a curve, with the first 50 steps given separately when decoherence sets in around there (MeshGraphNets, research_notes/pdf-mgn-pdebench.md:129), and error bars: standard error over 100 test trajectories, plus min, median and max over seeds. Two more come from this course's own labs, because the benchmark suite has no equivalent: the error in the peak value, and the error in the peak's position, both in physical units.
Split rule. Split the data by initial-condition family and coefficient regime, never by taking random frames out of one trajectory, because consecutive frames of a trajectory are nearly the same field and a random-frame split leaks the answer into the training set. Hold out an entire physics family only after the shared input representation is fixed, or you are measuring the representation instead of the transfer (project kit, evaluation section). Inverse problems need a second layer: the error of the estimated initial condition and the error of the forward prediction made from it are different numbers, and the second is always smaller, because high-frequency errors in the initial condition are damped by the forward map (PDEBench Appendix B.1, :221).
Each of the following is a result that a single nRMSE column would have concealed or reversed.
pdf-mgn-pdebench.md:246). The bulk error moves far less. Table 7 gives the bulk numbers for context: at ν = 0.001, RMSE is 1.3e-1 for U-Net, 9.6e-3 for FNO and 2.2e-1 for a PINN; at ν = 1.0 the three are 2.7e-2, 1.2e-3 and 1.1e-2 (:255).:247, :256). This is the one benchmark in the course where FNO loses on a headline number, and it is a property of the metric as much as of the models.:241). A metric that can produce evidence for a mechanism, not only against one, is doing more work than a scalar.:259). Any architecture comparison quoted without its regime is close to meaningless.research_notes/pdf-fno-jmlr.md:662-687).Every number above compares a model against a reference solution, and the reference is itself a numerical approximation. PDEBench Appendix H makes that concrete in an uncomfortable way. Its 2D diffusion-reaction data is generated at 512×512 and downsampled to 128×128 to serve as truth, and the authors compare that against data generated directly at 128×128. The coarse solver's own error against the fine reference is 1.8e-1, which is larger than the FNO's 8.1e-3 error against the same reference. The surrogate is more faithful to the fine solution than the coarse solver is. The authors add, correctly, that further work is needed to establish whether 512×512 is fine enough to count as truth at all (pdf-mgn-pdebench.md:286).
That is the reason both labs in this course carry a solver check that runs before any model is trained and aborts the script on failure. Lab 01 halves \(\Delta t\) and sees the solution move by \(1.9\times10^{-12}\), and doubles the grid to 2048 points and sees \(9\times10^{-12}\). Lab 02 solves an analytic sine load and matches it to \(1.25\times10^{-5}\), and confirms a 15 W block load between a 128 and a 256 grid to \(8.3\times10^{-5}\). The first version of Lab 01's solver failed its own check at \(2\times10^{-5}\), traced to single-precision wave numbers, and was fixed before any model saw its output. A surrogate cannot be more accurate than the data it is fitted to, so the check is not optional bookkeeping.
LAB ../labs/01_burgers/metrics.py loads the saved weights of Lab 01 run B at both values of \(k_{\max}\), and of Lab 02, and writes metrics.json. For Burgers it computes nRMSE, max error, cRMSE on the mean of \(u\), and banded fRMSE at 64 and 1024 points. For the thermal problem it computes nRMSE and max error in kelvin, bRMSE on the outer ring of cells, and banded radial fRMSE at 64² and 128², alongside the peak and hot-spot metrics that the training script already recorded.
| Model, grid | nRMSE | max error | cRMSE on mean of \(u\) | fRMSE low 0-4 | fRMSE mid 5-12 | fRMSE high 13-∞ |
|---|---|---|---|---|---|---|
| \(k_{\max}=8\), 64 | 0.0224 | 0.886 | 0.0042 | 4.96e-3 | 2.98e-3 | 6.02e-4 |
| \(k_{\max}=8\), 1024 | 0.0349 | 1.303 | 0.0039 | 4.91e-3 | 3.17e-3 | 1.00e-3 |
| \(k_{\max}=16\), 64 | 0.0226 | 0.911 | 0.0043 | 4.97e-3 | 2.83e-3 | 6.44e-4 |
| \(k_{\max}=16\), 1024 | 0.0264 | 1.153 | 0.0036 | 4.57e-3 | 3.00e-3 | 6.25e-4 |
Four things come out of that table, and none of them is visible in the nRMSE column alone.
The error the training grid could see does not change. Low-band and mid-band errors are the same at 64 and at 1024 points for both models, to within a few percent. Whatever goes wrong at an unseen grid does not go wrong in the modes the loss was computed on. That is a reassuring result and it is also the reason the bulk number rises so modestly: most of the signal lives in those bands.
The high band is where the two models part company. Going from 64 to 1024 points, the high-band error rises 66 percent for \(k_{\max}=8\), from 6.02e-4 to 1.00e-3, and does not rise at all for \(k_{\max}=16\), 6.44e-4 to 6.25e-4. Module 2 identified the culprit as a single Fourier coefficient of magnitude \(1.44\times10^{-2}\) at the Nyquist bin. Check that it is big enough to explain this: the high band at 1024 points averages over about 500 bins, so one bin at \(1.44\times10^{-2}\) contributes \(\sqrt{(1.44\times10^{-2})^2/500} = 6.4\times10^{-4}\) on its own. That is the largest single contribution in the band, and it is comparable to the entire band's value at the training grid. The \(k_{\max}=16\) model's Nyquist coefficient is \(5.64\times10^{-3}\), which contributes only \(2.5\times10^{-4}\), and its high band accordingly does not move.
The worst point is off by half the range. Max error is 0.886 to 1.303 while the field error is two to three percent. The true solution at \(t=1\) has root-mean-square amplitude 0.644 over the test set and its largest value anywhere in the test set is 2.70, so the worst point of the prediction is wrong by roughly half of the largest value in the data. That point sits on the steepest front, which is exactly where the truth's curvature is largest and where a truncated spectral representation has the least to work with. If the downstream use is a safety margin rather than an average, this column is the one that matters.
The model does not conserve what the PDE conserves. Periodic Burgers preserves the spatial mean of \(u\) exactly, since \(\partial_t\int u = -\int\partial_x(u^2/2) + \nu\int u_{xx} = 0\) when both integrands are derivatives of periodic functions. The FNO's mean is off by 0.0042 against a typical true mean of 0.2906, which is 1.4 percent, at both grids and for both values of \(k_{\max}\). Nothing in the loss asked for conservation and the model did not learn it for free. Detecting that is precisely cRMSE's job, and no amount of nRMSE would have raised the question.
| Grid | nRMSE | max error (K) | bRMSE (K) | true boundary mean (K) | fRMSE low (K) | fRMSE mid (K) | fRMSE high (K) | peak error mean (K) |
|---|---|---|---|---|---|---|---|---|
| 64² (trained) | 0.0065 | 3.45 | 0.165 | 0.760 | 0.060 | 0.022 | 0.0079 | 0.20 |
| 128² (unseen) | 0.0814 | 13.76 | 0.651 | 0.377 | 0.258 | 0.030 | 0.213 | 1.55 |
The high-band error at 128² is 27 times its value at 64² while the mid band moves by a factor of 1.4. That is the stripe pattern of Module 6's figure, converted into a number that can be tracked, compared and regressed against. It is also the two-dimensional counterpart of the Burgers Nyquist coefficient, so the same defect has now been caught twice by the same metric in two different dimensions.
The boundary row is the one that should stop a reader. At 128² the model's error on the boundary ring is 0.651 K against a true boundary mean of 0.377 K. The boundary condition is \(T=0\) there, so the true values are small numbers from the discretization, and the model is wrong by more than the entire quantity. A field metric cannot see this, because the boundary is a one-dimensional set inside a two-dimensional domain and contributes almost nothing to the bulk norm. A thermal engineer would see it immediately, because a die model that does not respect its own edge condition is not a die model.
The last column against the first completes the picture. At 128², the field is 8.1 percent wrong, the boundary is 173 percent wrong, and the peak is 2.6 percent wrong, 1.55 K on a mean peak of 58.6 K with a worst case of 6.21 K. Those three statements are all true about the same predictions, and they support three different decisions. Reporting one number would have hidden two of them, and which one you happened to report would have decided whether the model looked usable.
research_notes/pdf-mgn-pdebench.md:201-221; Section 4.2 and Tables 5 to 13: :239-259; Appendix H on ground truth: :286; conclusions: :298-310.:127-129.21-1524.pdf p.47: research_notes/pdf-fno-jmlr.md:662-687.../labs/01_burgers/metrics.json (all sixteen Burgers entries and both thermal entries), ../labs/02_thermal/results_thermal.json (peak and hot-spot columns), ../labs/01_burgers/spectrum_b_m16.npz and spectrum_b_m32.npz (the Nyquist coefficients used in the high-band check), burgers_nu0.1.pt (the rms and maximum of the true solution).../05_physics_foundation_model_study_kit.html.You can now write down a metric suite that is capable of failing a model, say which member of it catches which defect, and check a banded number against a spectrum to confirm that the two agree about the mechanism.