15.10. Troubleshooting Nonlinear Mixed Model Fitting

15.10.1. General Considerations

Numerical fitting of nonlinear models is often much less stable than fitting of linear models, and so do not be discouraged if your attempts at fitting a nonlinear model fail at first. Nonlinear mixed models are even more challenging because the likelihood function is an integral over the random effects. All of the standard caveats for nonlinear model fitting apply (see, for example, Gallant 1987, Bates and Watts 1988, Davidian and Giltinan 1995, Vonesh and Chinchilli 1997, Seber and Wild 2003). Please also keep in mind that if your data are really noisy, or if your nonlinear model is not appropriate for the data, your chances for model fitting failures increase considerably. Ideally successful convergence will be indicated by small gradients for every parameter (preferably less than 1E–3), a positive definite Hessian matrix at the solution, and small overall values for convergence criteria like GCONV in PROC NLMIXED.

One common source of instability is parameters with widely varying scales. If the scaling of your parameters varies by more than a few orders of magnitude, the numerical stability of the optimization problem can be seriously reduced and result in computational difficulties. A simple remedy is to rescale each parameter so that its final estimated value has a magnitude near 1.