Monotonicity of Fisher-Rao Metric

Stochastic Map

A stochastic map is a linear map $T$ from the space of probability measures on some set $A$ to the space of probability measures on some set $B$. Explicitly, we require$T:P(A)\to P(B)\mu \mapsto {\sum }_{a\in A}\mu (a)\cdot T({\delta }_a)$where ${\delta }_a$ is the point measure at $a$.

Theorem

The relative entropy $S$ is monotone. i.e. for any stochastic map $T:P(X)\to P(Y)$, we have $S(Tp:Tq)\le S(p:q)$

Proof $\begin{array}{rl}S(Tp:Tq)& =\sum _{y\in Y}(Tp)(y)\log \frac{(Tp)(y)}{(Tq)(y)}\\ & =\sum _{y\in Y}\left(\sum _{x\in X}p(x)(T{\delta }_x)(y)\right)\log \frac{{\sum }_{x\in X}p(x)(T{\delta }_x)(y)}{{\sum }_{x\in X}q(x)(T{\delta }_x)(y)}\\ & =\sum _{y\in Y}\left(\sum _{x\in X}p(x)(T{\delta }_x)(y)\right)\log \frac{{\sum }_{x\in X}p(x)(T{\delta }_x)(y)}{{\sum }_{x\in X}p(x)\frac{q(x)}{p(x)}(T{\delta }_x)(y)}\end{array}$By joint convexity of relative entropy, we have $\begin{array}{rl}& \sum _{y\in Y}\left(\sum _{x\in X}p(x)(T{\delta }_x)(y)\right)\log \frac{{\sum }_{x\in X}p(x)(T{\delta }_x)(y)}{{\sum }_{x\in X}p(x)\frac{q(x)}{p(x)}(T{\delta }_x)(y)}\\ \le & \sum _{x\in X}p(x)\left(\sum _{y\in Y}(T{\delta }_x)(y)\log \frac{(T{\delta }_x)(y)}{\frac{q(x)}{p(x)}\cdot (T{\delta }_x)(y)}\right)\\ =& \sum _{x\in X}p(x)\left(\sum _{y\in Y}(T{\delta }_x)(y)\log \frac{p(x)}{q(x)}\right)\\ =& \sum _{x\in X}p(x)\log \frac{p(x)}{q(x)}\\ =& S(p:q)\end{array}$which is the desired result. $\square$