Research program
The questions this repository exists to answer, roughly in order of attack.
The organizing question is narrow and, we think, fertile:
Which expensive “remove one component and recompute” calculations in computational kinetics and statistical mechanics are currently performed explicitly, even though one full-system solve already contains all the leave-one-out answers through a Schur-complement downdate?
With one standing caveat: “remove” means different things — scratching a competitor (a marginal), pinning a coordinate (a conditional), deleting a bond (a Woodbury update), forming a vacancy (a block update plus relaxation). The semantics of deletion page fixes the vocabulary; each question below names its intervention.
Q1. The deletion-ensemble audit survey
Catalogue remove-one-and-recompute workflows across computational statistical mechanics: vacancy and defect formation energies, Hessian-based stability analyses, elastic and resistor network response, spin-glass cavity iterations, Gaussian-field conditioning. For each: is the recomputation a fresh solve today, and does the rank-one (or leave-$k$-out) downdate apply exactly, approximately, or not at all? The deliverable is a table with speedup estimates — $n$ solves collapsed to one solve plus downdates — and the failure modes (nonlinearity, relaxation after deletion, non-quadratic energy landscapes).
Q2. Thurstone races as kinetic surrogates core
Kinetic Monte Carlo is an exponential race; real first-event data often is not. Train the fast ability transform on first-event observations from a ground-truth simulator (synthetic KMC first, then non-exponential waiting-time models) and measure: does a Thurstone race with a fitted base density predict counterfactual races — channels blocked, fields merged, temperature shifted — better than the Harville/exponential surrogate? Two corrections learned along the way: the right null model is common-clock proportional hazards (any common time change of exponential clocks is still Luce — exponentiality is sufficient, not necessary), and experiments 1–5 showed that independent races of any noise law fail on deletion counterfactuals, whose information is correlational (Q6) and, more fundamentally, not present in winner-only data at all (Q3).
Q3. The runner-up principle theorem + experiment
Winner-only data does not identify deletion counterfactuals — and the failure is exact, not statistical. After scratching $i$, $q_j^{(-i)} = p_j + p_i M_{ij}$ where $M_{ij} = P(\pi_2 = j \mid \pi_1 = i)$ is completely unconstrained by the winner probabilities $p$: any row-stochastic redistribution is consistent. This is the finite-choice face of classical competing-risks nonidentifiability, and it reframes experiment 1: no model — independent or otherwise — can learn substitution structure from an $N$-vector. The data hierarchy: winner-only (bounds only) → +winning time → +runner-up identity (identifies every singleton scratch) → top-$(k{+}1)$ ranking ($k$-deletions) → observed blocked-set interventions. The runner-up is worth more than the winning margin. The independent inversion is then honestly described as a canonical completion of $p$ into a ranking model, validated only by held-out interventions — which is exactly how experiment 3 earned its correlations (geometry supplied the structure; a held-out intervention calibrated it). Confirmed on real physics in experiment 8 (ranked narrow escape): winner-only models degrade with deletion depth (TV 0.038 → 0.069) while the empirical runner-up kernel, from pairs data alone, sits at the sampling-noise floor for singles, pairs, AND triples — the Markov substitution resolvent $q_A = p_A + p_B(I-M_{BB})^{-1}M_{BA}$ composes essentially exactly on diffusive first-passage geometry.
Q4. Leave-$k$-out: the compressed defect ensemble extension
The identity $G^{(S)} = G_{\bar S,\bar S} - G_{\bar S,S} G_{S,S}^{-1} G_{S,\bar S}$ prices every small deletion cluster at a $k \times k$ inversion. Defect pairs and triples — interactions between vacancies, cooperative rearrangements — are the physically interesting objects. How far does the compressed representation reach before conditioning or fill-in erodes the advantage, and can the ensemble of all $\binom{n}{2}$ pair deletions be screened lazily (compute the cheap scalar first, expand only promising pairs)?
Q5. Dynamic fields and streaming maintenance extension
Propensities drift: temperature ramps, ageing glasses, evolving stress fields. The
thurstone package already tracks time-varying abilities with a Kalman
layer. The open question is joint maintenance — keeping the global field (or
global inverse) current under both parameter drift and structural events (channels
appearing and disappearing), so the leave-one-out ensemble stays a downdate away at
all times. This mirrors the streaming covariance problem solved for portfolios in
allocation.
Q6. Where the two identities meet first positive result
The multiplicative cavity (divide out a survival function) and the rank-one cavity (subtract a mediated interaction) are the independent-competitor and Gaussian-coupled faces of the same deletion question. Correlated races — competitors whose performances share a Gaussian copula — need both at once: the field product for the extremal structure and the Schur downdate for the coupling. Evidence this is the right target: in the narrow-escape experiment, real diffusive kinetics violates IIA through geometry (walkers turned away from a blocked window are absorbed nearby) and no independent race captures it — while in experiment 3 a geometry-informed correlated race, with corr$(i,j) = e^{-d_{ij}/\ell}$ and $\ell$ calibrated on a held-out intervention, cuts the counterfactual error ~9× (TV 0.009 vs 0.081), to the simulation noise floor. The fast transform for correlated fields now exists (experiment 6): fit $\Sigma \approx VV^\top + D$ by factor analysis; conditionally on the factors the competitors are independent, so the multiplicative cavity applies at every quadrature node — the two identities literally compose. Exact given the factor model, the deletion ensemble intact (1e-16 agreement from one pass), and the exp03 counterfactual reproduced with no Monte Carlo anywhere (TV 0.017 vs 0.082 for IIA). Remaining open: kinked (Markov) kernels at large $N$ want sequential/Vecchia-style conditioning rather than global factors — the same Schur machinery from the other end. One caution from the semantics page: this composition conditions on latent factors and scratches competitors; it is not a claim that scratching itself invokes a Schur complement. And one immediate export: the correlated re-race step of Thurstone portfolios (long-only weights as winning probabilities, currently priced by sampling) can use this transform directly — equity risk models already arrive in $VV^\top + D$ form, and the deletion ensemble prices every single-name exclusion from one pass.
Q7. Softmax as homogenized kinetics theorem verified
The deepest reframing: model the race as channels firing at intensities $\lambda_i(Y_t)$ driven by a fast hidden Markov state $Y$ with generator $\mathcal{L}/\varepsilon$. Conditional on $Y$ it is ordinary KMC; marginally the waiting times are dependent and non-exponential; the win probability solves a killed-resolvent equation $u_i^A = (\Lambda_A - \mathcal{L}/\varepsilon)^{-1} \lambda_i$. Homogenizing: at leading order $p_i^A = \bar\lambda_i / \bar\Lambda_A + O(\varepsilon)$ — softmax is the homogenized choice law of fast hidden kinetics — and the first correction is a Green–Kubo term built from integrated rate autocovariances $K_{jk} = \int_0^\infty \mathrm{Cov}_\pi(\lambda_j(Y_0), \lambda_k(Y_t))\,dt$. The same $(\bar\lambda, K)$ answers every blocked-set counterfactual by restricting sums to the survivors — the true statistical analogue of “one global computation encodes all deletions”; common-mode fluctuations cancel exactly; low-rank environmental loadings give $K = B\Gamma B^\top$, a compressed correlated race with physical provenance. Verified exactly on finite-state chains in experiment 7: convergence slopes 0.99/1.99 (theory 1/2), one $(\bar\lambda, K)$ pair corrects all 41 tested blocked subsets, rank-$r$ loadings give rank-$r$ $K$ exactly, and common-mode fluctuation gives Luce exactly at every $\varepsilon$ (it is a common time change) — stronger than the theorem. Next: reversible diffusions; narrow escape (where the disk's boundary operator diagonalizes in Fourier modes) is the analytic testbed. Target paper: Softmax from Fast Mixing: Green–Kubo Corrections for Counterfactual Races. The unifying object across Q6/Q7 and the quadratic side is then a transfer resolvent — $(\Lambda_A - \varepsilon^{-1}\mathcal{L})^{-1}$, $(I - M_{BB})^{-1}$, $A^{-1}$ — whose response to suppression is Schur/Woodbury theory.
Method of work
Each question gets a numbered experiment directory with a runnable script, synthetic ground truth, and a short write-up; results that survive get promoted to the papers page. The first experiments verify the two identities and their advertised complexities, so that later claims stand on checked ground.