Classification of Divergences

Def Divergence The divergence is a real-valued function $\mathcal{D}(p:q)$ with $p$ and $q$ be two probability measures defined on the same measurable space. It satisfies

• $\mathcal{D}(p:q)\ge 0$ with equality holds iff $p=q$.
• $\mathcal{D}$ may not be symmetric: $\mathcal{D}(p:q)\ne \mathcal{D}(q:p)$ in general.
• $\mathcal{D}$ may not satisfy the triangle inequality. Therefore, $\mathcal{D}$ is not a metric distance.

Def Locally-Rao A divergence $\mathcal{D}$ is called locally-Rao, if$\mathcal{D}(\theta :\theta +\text{dd}\theta )=\frac{1}{2}\text{dd}{\theta }^{\top }I(\theta )\text{dd}\theta +O(\Vert \text{dd}\theta {\Vert }^3).$

Def $f$-Divergence For probability distribution $p\ll q$, we define:${\mathcal{D}}_f\left(p:q\right)=\int q(x)f\left(\frac{p(\chi )}{q(\chi )}\right)\text{dd}\nu$ where $f:(0,\infty )\to \mathbb{R}$ with properties that $f(1)=0$ and $f^{\prime \prime }(x)\ge 0$. Clearly $f$ is convex and we have ${\text{E}}_p(f(x))\ge f({\text{E}}_p[x])$, yielding that ${\mathcal{D}}_f(p:q)\ge 0$.

Thrm ${\mathcal{D}}_f(p:q)$ is convex w.r.t. $p$.

Prop $f$-divergence is locally-Rao.

Def $\alpha$-Divergence Suppose $\alpha \in \mathbb{R}\setminus \{-1,1\}$, plug $f_{\alpha }(x)=\frac{4}{1-{\alpha }^2}\left(1-x^{\frac{1+\alpha }{2}}\right)$into the definition of $f$-divergence, we get $D_{\alpha }(p:q)=\frac{4}{1-{\alpha }^2}\left(1-\int p^{\frac{1+\alpha }{2}}(x)q^{\frac{1-\alpha }{2}}(x)\text{dd}\nu \right)$ Prop

• $f$-divergence with $f(t)=t\log t$ is exactly the KL divergence.
• $f$-divergence with $f(t)=-\log t$ is the reverse KL divergence.
• ${\lim }_{\alpha \to 1}{\mathcal{D}}_{\alpha }(p:q)={\mathcal{D}}_{KL}(p:q)$.

Prop KL Divergence of Exponential Family The KL divergence of exponential family $p$ and $q$ is${\mathcal{D}}_{KL}(p:q)=\phi ({\eta }_p)-{\eta }_p^{\top }{\theta }_q+\psi ({\theta }_q)$