Deep Bayes: Discrete Latent Variables

Introduction:

这篇笔记会记录一些离散隐变量模型，转载请注明。
Reference：Deep Bayes

Motivation

1. Easier to interpret discrete categories than continuous spectrum
   example: discrete variational autoencoder
2. Allow the model to make a discrete choice
   example: hard attention
   An attention module generates binary mask of where to look at
   The network classifies masked images
   We want attention module to attend only important areas of the image.

Reinforce Estimator

However, this typically has large variance
Requires sophisticated Variance Reduction methods
Just taking bigger M gives only a modest improvement.


Idea: Relax the objective over discrete random samples z into an objective oven continuous random samples during training and use the reparametrization trick:

Gumbel-Max trick

Some ideas about Gumbel Distribution：
https://qinqianshan.com/math/probability_distribution/gumbel-distribution/

Variance Reduction

Control Variates
Consider some with tractable expectation . Then



Simple Baselines:
Constant baseline

Variance Minimization:

Gumbel-Relaxed Baselines: