MLE with normally distributed model

Setup

We have a set of observations $(x_i,y_i)$, and we want to find the parameters $a,b,\sigma ,\mu$ such that $Y=aX+b+ϵ$, where $ϵ\approx \mathcal{N}(0,\sigma )$ is normally distributed.

Solution

We will apply Maximum Likelihood Estimation. Applied to this problem, we want to find the $argmax$ of the expression below:

$$\begin{array}{rll}\prod \mathbb{P}[y_i|x_i]& =\log \left(\prod \mathbb{P}[y_i|x_i]\right)& \text{As log(∏zi)=∑log(zi)}\\ & =\sum \log \left(\mathbb{P}[y_i|x_i]\right)& \text{As P[yi∣xi]=σ2π1e−21(σaxi+b−yi)2}\\ & =\sum \log \left(\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}(\frac{a{x}_i+b-y_i}{\sigma }{)}^2}\right)& \text{As log(∏zi)=∑log(zi)}\\ & =-N\log (\sigma )-\frac{N}{2}\log (2\pi )-\frac{1}{2}\sum (\frac{a{x}_i+b-y_i}{\sigma }{)}^2& \text{As −2Nlog(2π) is a constant}\\ & =-N\log (\sigma )-\frac{1}{2}\sum (\frac{a{x}_i+b-y_i}{\sigma }{)}^2& \text{As −2Nlog(2π) is a constant}\end{array}$$

In order to find the $argmax$ we equal all derivatives to zero:

$$\begin{array}{rlc}0& =\frac{\partial }{\partial b}\prod \mathbb{P}[y_i|x_i]& \\ & =\frac{\partial }{\partial b}\left(-N\log (\sigma )-\frac{1}{2}\sum (\frac{a{x}_i+b-y_i}{\sigma }{)}^2\right)& \text{Removing terms without b and constants}\\ & =\frac{\partial }{\partial b}\left(\sum (\frac{a{x}_i+b-y_i}{\sigma }{)}^2\right)& \\ & =\sum \frac{\partial }{\partial b}\left((a{x}_i+b-y_i{)}^2\right)& \\ & =\sum 2(a{x}_i+b-y_i)& \text{Dividing both sides by 2}\\ & =\sum a{x}_i+b-y_i& \\ & \Downarrow & \\ a\sum x_i+Nb& =\sum y_i& \end{array}$$

Doing the same for $a$:

$$\begin{array}{rlc}0& =\frac{\partial }{\partial a}\prod \mathbb{P}[y_i|x_i]& \\ & =\frac{\partial }{\partial a}\left(-N\log (\sigma )-\frac{1}{2}\sum (\frac{a{x}_i+b-y_i}{\sigma }{)}^2\right)& \text{Removing terms without a and constants}\\ & =\frac{\partial }{\partial a}\left(\sum (\frac{a{x}_i+b-y_i}{\sigma }{)}^2\right)& \\ & =\sum \frac{\partial }{\partial a}\left((a{x}_i+b-y_i{)}^2\right)& \\ & =\sum 2x_i(a{x}_i+b-y_i)& \text{Dividing both sides by 2}\\ & =\sum a{x}_i^2+x_{i}b-x_i{y}_i& \\ & \Downarrow & \\ a\sum x_i^2+b\sum x_i& =\sum x_i{y}_i& \end{array}$$

Writing the 2 equations above in matrix form:

$$\begin{array}{rl}\left(\begin{array}{cc}\sum x_i& N\\ \sum x_i^2& \sum x_i\end{array}\right)\left(\begin{array}{c}a\\ b\end{array}\right)& =\left(\begin{array}{c}\sum y_i\\ \sum x_i{y}_i\end{array}\right)\\ & \Downarrow \\ \left(\begin{array}{c}a\\ b\end{array}\right)& ={\left(\begin{array}{cc}\sum x_i& N\\ \sum x_i^2& \sum x_i\end{array}\right)}^{-1}\left(\begin{array}{c}\sum y_i\\ \sum x_i{y}_i\end{array}\right)\\ & ={\left((\sum x_i{)}^2-N\sum x_I^2\right)}^{-1}\left(\begin{array}{cc}\sum x_i& -N\\ -\sum x_i^2& \sum x_i\end{array}\right)\left(\begin{array}{c}\sum y_i\\ \sum x_i{y}_i\end{array}\right)\end{array}$$

Applying the formula above, we get the solution for the linear regression as a function of our observations. We can also calculate $\sigma$ if we want to know the estimated error of our predictions, but it is not needed to build the predictor itself