Research
My research has three parts, and each answers part of one question: how can a learning system improve itself and stay correct? The first is about structure that makes prediction cheap, the second about simulations that can be forked and compared, and the third about checks that a self-modifying system cannot bypass.
Learning structure
A model that matches the structure of its problem needs fewer parameters and less data. I work on extracting that structure from a problem description and building it into the network, for example as the attention mask of a transformer.
Papers
- Graphically Structured Diffusion Models
- Structured Conditional Continuous Normalizing Flows for Efficient Amortized Inference in Graphical Models
- Flexible Diffusion Modeling of Long Videos
- Trans-Dimensional Generative Modeling via Jump Diffusion Models
- Scaling Graphically Structured Diffusion Models
- Inferring the Structure of Ordinary Differential Equations
- PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI
Software
Inference over forkable worlds
A simulation lets a system test a decision before making it. I work on inference methods that fork a running simulation at each random choice and weigh the forks against data, so that nothing is recomputed from the start.
defn model []:
spin:
mu := sample(dist/normal(0.0, 1.0), :id, :mu)
observe(dist/normal(mu, 1.0), 1.0, :id, :y)
mu
end
end
;; each sample and observe site is a savepoint;
;; particles fork from it, and resampling prunes and copies them
def posterior: sp/with-context(sp/create-execution-context(),
@infer/smc-infer(model(), 1000))
(defn model []
(spin
(let [mu (sample (dist/normal 0.0 1.0) :id :mu)]
(observe (dist/normal mu 1.0) 1.0 :id :y)
mu)))
;; each sample and observe site is a savepoint;
;; particles fork from it, and resampling prunes and copies them
(def posterior
(sp/with-context (sp/create-execution-context)
@(infer/smc-infer (model) 1000)))
Papers
- All-in-one Simulation-Based Inference
- Daphne: Multi-Pass Compilation of Probabilistic Programs into Graphical Models and Neural Networks
- Sequential Core-Set Monte Carlo
- Planning as Inference in Epidemiological Dynamics Models
- Time-sensitive probabilistic inference for the edge
- Prospective Messaging: Learning in Networks with Communication Delays
Software
Further reading
Self-improvement under verification
A system that modifies itself needs a fork on which to try a change and a check it cannot alter. I build both. Every level of the stack can be forked, and programs in Klara adopt an optimisation only after a small kernel has checked it. Simmis applies the same rule to teams of people and agents. Most of this is research software, and the proofs cover a core calculus, not the whole implementation.
;; the meaning
defn power [n x]:
if n = 0 :
1
else:
x * power(dec(n), x)
end
end
;; a program that writes a program
defn power-gen [n x]:
if n = 0 :
code(1)
else:
code(esc(x) * esc(power-gen(dec(n), x)))
end
end
;; use the generator as an optimiser; Klara checks it against power
compiler/prepare(code(x -> power(3, x)),
{:recipe fn [_]: code(x -> esc(power-gen(3, code(x)))) end})
;; => {:status :ready ...}
;; a wrong optimiser is rejected
compiler/prepare(code(x -> power(3, x)), {:recipe fn [_]: code(x -> x * x) end})
;; => {:status :rejected :rule :L4.check.not-equivalent ...}
;; the meaning
(defn power [n x]
(if (= n 0) 1 (* x (power (dec n) x))))
;; a program that writes a program
(defn power-gen [n x]
(if (= n 0)
(code 1)
(code (* (esc x) (esc (power-gen (dec n) x))))))
;; use the generator as an optimiser; Klara checks it against power
(compiler/prepare (code (fn [x] (power 3 x)))
{:recipe (fn [_] (code (fn [x] (esc (power-gen 3 (code x))))))})
;; => {:status :ready ...}
;; a wrong optimiser is rejected
(compiler/prepare (code (fn [x] (power 3 x)))
{:recipe (fn [_] (code (fn [x] (* x x))))})
;; => {:status :rejected :rule :L4.check.not-equivalent ...}
Papers
- Decoupling Conflict Resolution with CDVCS
- If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History
Software
Further reading
Doctoral thesis
My thesis at the University of British Columbia, with Frank Wood in the PLAI group, shows how structural knowledge about an inference problem can be mapped automatically into the mechanism that solves it: from a compiler that extracts a faithful inverse graph, through structured normalizing flows, to graphically structured diffusion models that scale to Sudoku, sorting and matrix factorisation, and connect to simulation-based inference.
Talks
- Rationalizing a Society of Learners
- Programming as and for inference
- Partially evaluating higher-order probabilistic programs without stochastic recursion to graphical models
- Flexible diffusion models for videos, molecules, and inference in structured simulators
- Video and graphically structured diffusion models
- Graphically structured diffusion models
- Anglican: Turing-complete functional probabilistic programming as a domain-specific language
- Datalog as a distributed systems language
- Let's replicate state together
- Machine learning workshop
Full list of publications on Google Scholar.