Least squares
Least squares estimates parameters by making model predictions close to observed values in squared-error distance.
For observations \(y_i\) and predictions \(f(t_i;\theta)\), define residuals
\[r_i=y_i-f(t_i;\theta).\]The ordinary least-squares estimate minimises
\[S(\theta)=\sum_{i=1}^{n}r_i^2.\]Weighted least squares
If observations have different variances, a weighted criterion can be used:
\[S_w(\theta)=\sum_i w_i r_i^2.\]The choice of weights should reflect the observation model rather than being arbitrary.
Key idea. Least squares measures fit through residuals, but its statistical interpretation depends on assumptions about measurement errors.