AF - Calculating Natural Latents via Resampling by johnswentworth

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Calculating Natural Latents via Resampling, published by johnswentworth on June 6, 2024 on The AI Alignment Forum.
So you've read some of our
previous
natural latents
posts, and you're sold on the value proposition. But there's some big foundational questions still unanswered. For example: how do we find these natural latents in some model, if we don't know in advance what they are? Examples in previous posts conceptually involved picking some latents out of the ether (like e.g. the bias of a die), and then verifying the naturality of that latent.
This post is about one way to calculate natural latents, in principle, when we don't already know what they are. The basic idea is to resample all the variables once simultaneously, conditional on the others, like a step in an MCMC algorithm. The resampled variables turn out to be a competitively optimal approximate natural latent over the original variables (as we'll prove in the post).
Toward the end, we'll use this technique to calculate an approximate natural latent for a normal distribution, and quantify the approximations.
The proofs will use the graphical notation introduced in
Some Rules For An Algebra Of Bayes Nets.
Some Conceptual Foundations
What Are We Even Computing?
First things first: what even is "a latent", and what does it even mean to "calculate a natural latent"? If we had a function to "calculate natural latents", what would its inputs be, and what would its outputs be?
The way we use the term, any conditional distribution
(λ,xP[Λ=λ|X=x])
defines a "latent" variable Λ over the "observables" X, given the distribution P[X]. Together P[X] and P[Λ|X] specify the full joint distribution P[Λ,X]. We typically think of the latent variable as some unobservable-to-the-agent "generator" of the observables, but a latent can be defined by any extension of the distribution over X to a distribution over Λ and X.
Natural latents are latents which (approximately) satisfy some specific conditions, namely that the distribution P[X,Λ] (approximately) factors over these Bayes nets:
Intuitively, the first says that Λ mediates between the Xi's, and the second says that any one Xi gives approximately the same information about Λ as all of X. (This is a stronger redundancy condition than we used in previous posts; we'll talk about that change below.)
So, a function which "calculates natural latents" takes in some representation of a distribution (xP[X]) over "observables", and spits out some representation of a conditional distribution (λ,xP[Λ=λ|X=x]), such that the joint distribution (approximately) factors over the Bayes nets above.
For example, in the last section of this post, we'll compute a natural latent for a normal distribution. The function to compute that latent:
Takes in a covariance matrix ΣXX for X, representing a zero-mean normal distribution P[X].
Spits out a covariance matrix ΣΛΛ for Λ and a cross-covariance matrix ΣΛX, together representing the conditional distribution of a latent Λ which is jointly zero-mean normal with X.
… and the joint normal distribution over Λ,X represented by those covariance matrices approximately factors according to the Bayes nets above.
Why Do We Want That, Again?
Our
previous
posts talk more about the motivation, but briefly: two different agents could use two different models with totally different internal (i.e. latent) variables to represent the same predictive distribution P[X]. Insofar as they both use natural latents, there's a correspondence between their internal variables - two latents over the same P[X] which both approximately satisfy the naturality conditions must contain approximately the same information about X.
So, insofar as the two agents both use natural latents internally, we have reason to expect that the internal latents of one can be faithfully translated int...