#161 Amortized Inference & Neural Processes, with Luigi Acerbi

Support & Resources
→ Support the show on Patreon
→ Bayesian Modeling Course (first 2 lessons free)

Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work

Takeaways:
Q: What is Variational Bayesian Monte Carlo (VBMC) and how is it different from Bayesian optimization?
A: VBMC borrows the machinery of Bayesian optimization but aims at a different target. Bayesian optimization fits a Gaussian process surrogate to an expensive function and uses it to hunt for the optimum. VBMC instead treats the log-posterior as the function to model, evaluates it at a few carefully chosen points, and keeps the whole reconstructed shape rather than just its peak. That gives you the full posterior, not a single best-fit value. Where MCMC might need tens of thousands to millions of evaluations, VBMC often reconstructs a good posterior approximation from a few hundred, which matters when each evaluation is slow.

Q: When should you reach for PyVBMC, and when is it the wrong tool?
A: Two symptoms tell you PyVBMC might help. First, speed: if a single evaluation of your log density takes on the order of a second, running MCMC over tens of thousands of evaluations becomes painful, and PyVBMC's few-hundred-evaluation budget pays off. Second, dimensionality: because it leans on a Gaussian process surrogate, it works well up to roughly 10 to 15 parameters and degrades beyond that. If your model already runs fine in Stan or PyMC, you do not need it. It shines for expensive, low-dimensional models common in science and engineering, where you are modeling a process rather than composing nice distributions.

Full takeaways here

Chapters:
00:18:13 What is Variational Bayesian Monte Carlo (VBMC) and how does it differ from Bayesian optimization?
00:30:21 When should you use VBMC versus BADS in practice?
00:31:20 What is Bayesian Adaptive Direct Search (BADS) and how does its hybrid optimization strategy work?
00:39:18 What are neural processes, and why are transformers a natural neural process architecture?
00:45:54 What is the Amortized Conditioning Engine (ACE) and what problem does it unify?
00:55:42 What do PriorGuide and the new autoregressive buffer paper solve for amortized inference?
01:02:03 How does the new autoregressive buffer speed up predictions in transformer probabilistic models?
01:06:11 What is Luigi Acerbi's vision for a foundation model for inference?
01:09:26 What is ALINE and how does it add active data acquisition to amortized inference?
01:12:43 How does Luigi Acerbi connect LLM agents, Bayesian decision theory, and the nature of intelligence?
01:18:44 For a PyMC, Stan, or NumPyro user, where should you start with VBMC, BADS, or BayesFlow?

Thank you to my Patrons for making this episode possible!

Links from the show here