7.4. A Statistical Framework for Parameter Estimation and the Maximum Likelihood Method

So far we have not appealed to any statistical arguments for the estimation of θ. In fact, our framework of fitting models to data makes sense regardless of a stochastic setting of the data. It is, however, useful and instructive at this point to briefly describe basic aspects of statistical parameter estimation and relate them to our framework.

Estimators and the Principle of Maximum Likelihood

The area of statistical inference, as well as that of system identification and parameter estimation, deals with the problem of extracting information from observations that themselves could be unreliable. The observations are then described as realizations of stochastic variables. Suppose that the observations are represented by the random variable y^N = (y(1), y(2), ..., y(N)) that takes values in R^N. The probability density function (PDF) of y^N supposed to be