Kinetic Surrogates

Fast ability transforms for systems in which thousands of latent stochastic processes compete to produce the first, fastest, weakest, or otherwise extremal event.

A chemical system chooses one reaction pathway out of thousands. A stressed material fails at one bond out of millions. A glass relaxes through whichever local rearrangement happens first. In each case an expensive physical process is, abstractly, a race: many latent processes compete, and we observe the winner — sometimes only the winner. This project asks whether the machinery built for inferring the relative abilities of racehorses from betting odds — the fast ability transform — can turn expensive kinetic simulations into cheap, transferable surrogate models.

The connecting mathematics is a pair of leave-one-out identities. Both say the same thing: one global computation is a compressed representation of the entire family of single-deletion systems. Compute the field once, and every “what if competitor $i$ were absent?” question is answered by a cheap downdate rather than a fresh solve. In racing this is the trick that makes the ability transform $O(N)$; in statistical mechanics it is the cavity method's central identity, read as an algorithm.

The two identities

Multiplicative (races). If the survival function of the whole field is the product $S_{\text{field}} = \prod_j S_j$, then the field faced by competitor $i$ is recovered by division: $S_{-i} = S_{\text{field}} / S_i$. One pass over the field prices every competitor — this is what makes the ability transform of Cotton (2021) $O(N)$ instead of $O(N^2)$.

Rank-one (quadratic systems). If $G = A^{-1}$ is the inverse (Green function, susceptibility, resolvent) of a coupled system, the inverse with site $i$ deleted is a Schur-complement downdate:

One $O(n^3)$ factorization contains all $n$ cavity systems; the subtracted rank-one term is exactly the interaction between the remaining degrees of freedom that was mediated through $i$. Statistical mechanics knows this identity intimately — it is the cavity Green function — but typically uses it site by site, not as a way to harvest the whole deletion ensemble from a single solve. The introduction develops both identities and their correspondence; the research program lists the questions they open.

Status

This is an exploratory research repository with four experiments run so far (experiments directory), all against real simulated physics:

• Narrow escape (Brownian first passage): diffusive kinetics violates IIA geometrically — blocked flux is inherited by neighboring windows — and no model could recover this from winner frequencies alone: substitution structure is unidentified in winner-only data (the runner-up principle, Q3 of the program).
• Correlated race: a geometry-informed correlation corr$(i,j)=e^{-d_{ij}/\ell}$, with $\ell$ calibrated on a held-out intervention, cuts the counterfactual error ~9× (TV 0.009 vs 0.081) — the first positive result for Q6 of the program.
• Disordered elastic network: the entire single-defect ensemble (784 sites, ~870× speedup) and 4,000 pair-defect interactions from one inverse in under 50 ms.
• Kramers temperature transfer: Arrhenius rescaling halves the transfer error at kT $1.0 \to 0.7$ and $\to 0.55$; the noise law is second-order. A follow-up showed the residual is not correlational: the correlated race helps deletion counterfactuals, not global parameter shifts — two orthogonal failure axes for kinetic surrogates.
• The fast correlated transform: fit $\Sigma \approx VV^\top + D$; conditionally on the factors the multiplicative cavity applies at every quadrature node — the two identities compose. Exact given the factor model, deletion ensemble intact, and the correlated-race counterfactual reproduced with no Monte Carlo anywhere (TV 0.017 vs 0.082 for IIA).

The applications page ranks the domains to try next.

Further reading

• Introduction — the race model, the two leave-one-out identities, and the dictionary between racing and kinetics.
• Cotton, P. (2021). “Inferring Relative Ability from Winning Probability in Multientrant Contests.” SIAM Journal on Financial Mathematics 12(1), 295–317 — the fast ability transform.
• The thurstone package — the maintained implementation (docs and interactive demos).
• Mézard, M., Parisi, G., and Virasoro, M. A. (1987). Spin Glass Theory and Beyond — the cavity method, whose central identity is the rank-one downdate above.

Cite

Cotton, P. (2021). “Inferring Relative Ability from Winning Probability in Multientrant Contests.” SIAM Journal on Financial Mathematics 12(1), 295–317.

@article{cotton2021inferring,
  author  = {Cotton, Peter},
  title   = {Inferring Relative Ability from Winning Probability
             in Multientrant Contests},
  journal = {SIAM Journal on Financial Mathematics},
  volume  = {12},
  number  = {1},
  pages   = {295--317},
  year    = {2021}
}