Posterior Inference

Loss Function

Def Loss Function For a decision $d\in \mathcal{D}$, a loss function $L(\theta ,d)$ defines the penalty in taking decision $d$ given parameter value $\theta$. In Bayesian scenario, we have $d=\hat{\theta }$ to be the point estimate.

Def Quadratic Loss We define the quadratic loss as$L(\theta ,d)=(\theta -d{)}^2$Prop Quadratic loss is minimized when $d=\mathbb{E}[\theta \mid X]$.

Def Linear Loss We define the linear loss:$L(\theta ,d)=\{\begin{array}{ll}g(d-\theta )& \mathrm{if}d>\theta \\ h(\theta -d)& \mathrm{if}d<\theta \end{array}$For given scalars $g$ and $h$. Prop Linear loss is minimized at $d=q$, where $q$ is the $\frac{h}{g+h}$ quantile of the posterior.

Def Absolute Error Loss Absolute error loss is a special case of the linear loss when $g=h=1$:$L(\theta ,d)=|\theta -d|$Prop The posterior median minimizes absolute error loss.

Def 0-1 Loss The 0-1 loss is defined as follows:$L(\theta ,d)=\{\begin{array}{ll}0& \mathrm{if}|d-\theta |\le ϵ\\ 1& \mathrm{if}|d-\theta |>ϵ\end{array}$Prop $\mathbb{E}[L(\theta ,d)]=\text{pr}(|\theta -d|>ϵ)$ Prop The posterior mode minimizes 0-1 loss when choosing $ϵ$ arbitrarily small.

Predictive Inference

Def Predictive Density Function Predictive density function of a future observation $y$ is$f(y\mid X)=\int f(y\mid \theta )\pi (\theta \mid x)\text{dd}\theta$e.g. Suppose we have $x\sim \text{Binomial}(n,\theta )$ with conjugate prior $\theta \sim \text{Beta}(a,b)$. Then we know that $\theta \mid x\sim \text{Beta}(a+x,b+n-x)$. Now suppose we intend to make $N$ further observations, let $y$ be the number of successes, $y\mid \theta \sim \text{Binomial}(N,\theta )$. Hence$f(y\mid \theta )=\left(\genfrac{}{}{0ex}{}{N}{y}\right){\theta }^y(1-\theta {)}^{N-y}$So for $y=0,1,...,N$,$\begin{array}{rl}f(y|x)& ={\int }_0^1\left(\genfrac{}{}{0ex}{}{N}{y}\right){\theta }^y(1-\theta {)}^{N-y}\times \frac{{\theta }^{a+x-1}(1-\theta {)}^{b+n-x-1}}{B(a+x,b+n-x)}\text{dd}\theta \\ & =\left(\genfrac{}{}{0ex}{}{N}{y}\right)\frac{\text{Beta}(y+a+x,N-y+b+n-x)}{\text{Beta}(a+x,b+n-x)}\end{array}$e.g. Suppose $y\mid \theta \sim \mathcal{N}(\theta ,{\sigma }_y^2)$ and $\theta \mid x\sim \mathcal{N}({\mu }_{\theta },{\sigma }_{\theta }^2)$, then we can derive that $y\mid x\sim \mathcal{N}({\mu }_{\theta },{\sigma }_{\theta }^2+{\sigma }_y^2)$.

Algorithm Sampling from Posterior Predictive Distributions

1. Obtain posterior samples ${\theta }_i\sim \pi (\theta |x)$
2. For each ${\theta }_i$ we can generate $y_i\sim f(y|{\theta }_i)$
3. This gives us joint samples $({\theta }_i,y_i)\sim f(y|\theta )\pi (\theta |x)$
4. To obtain samples from $f(y|x)$, integrate out $\theta$ (discard the ${\theta }_i$ values), leave $y_i\sim f(y|x)$ only.