Reparameterization Trick

Check my VAE notebook.

Diagonal matrices

When you read that a paper has diagonal matrices parameterizing distributions, it means that each distribution is independent! This makes sense since the covariance between all the distributions is 0 (they don’t interact). This makes it so that instead of having n^2 things to model, it’s just n. You’ll find these absolutely everywhere. Think of it as a simplifying assumption of how the world works that makes things much more mathematically and computationally tractable.