Nonlinear Regression

Overview Nonlinear regression refers to algorithms for fitting complex and even arbitrary curves to one's data using iterative estimation. Simple curves can be implemented in general linear models (GLM) and OLS regression (ex., polynomial models may be fitted by adding the square or cube of an independent to the model) and in models supported by the generalized linear modeling family (ex., GZLM will implement logistic, probit, or gamma regression, among others, where the raw values of the dependent are nonlinearly related to the linear predictor (the terms on the right-hand side of the equation) because the dependent is transformed by some nonlinear link function). Nonlinear regression may be used to fit curves not possible by these methods. Common models for nonlinear regression, adapted from the SPSS manual, include these: Asymptotic Regression/Growth Model b1 + b2 * exp(b3 * x) Logistic Population Growth Model b1/(1+exp(b2+b3*x)) Asymptotic Regression/Decay Model b1 – (b2 * (b3 ** x)) Saturation Model b1*exp(b2*x) Density (b1 + b2 * x) ** (–1 / b3) Gauss b1 * (1 – b3 * exp(–b2 * x ** 2)) Gompertz b1 * exp(–b2 * exp(–b3 * x)) Johnson-Schumacher b1 * exp(–b2 / (x + b3)) Log-Modified (b1 + b3 * x) ** b2 Log-Logistic b1 – ln(1 + b2 * exp(–b3 * x)) Metcherlich Law of Diminishing Returns b1 + b2 * exp(–b3 * x) Michaelis Menten b1 * x / (x + b2) Morgan-Mercer-Florin (b1 * b2 + b3 * x ** b4) / (b2 + x ** b4) Peal-Reed b1 / (1+ b2 * exp(–(b3 * x + b4 * x **2 + b5 * x ** 3))) Ratio of Cubics (b1 + b2 * x + b3 * x ** 2 + b4 * x ** 3) / (b5 * x ** 3) Ratio of Quadratics (b1 + b2 * x + b3 * x ** 2) / (b4 * x ** 2) Richards b1 / ((1 + b3 * exp(–b2 * x)) ** (1 / b4)) Verhulst b1 / (1 + b3 * exp(–b2 * x)) Von Bertalanffy (b1 ** (1 – b4) – b2 * exp(–b3 * x)) ** (1 / (1 – b4)) Weibull b1 – b2 * exp(–b3 * x ** b4) Yield Density (b1 + b2 * x + b3 * x ** 2) ** (–1) Regression and GLM are preferred when the model is that the untransformed dependent variable equals some additive combination of parameters times variables (plus an error term), even if the variables are transformations of independents (ex., squares, logs). GZLM and its constituent models are preferred when some link function (transformation, such as logit or Poisson) of the dependent variable equals some additive combination of parameters times variables (plus error). In both situations, even though the variables themselves may be nonlinear in their relationship, the models (functions) are linear in their parameters (ex., logistic regression relates the independents linearly to the logit transformation of the dependent). However, some models are intrinsically nonlinear because the parameters relating the dependent to the independents are nonlinear and nonlinear modeling approaches are required. This happens, for instance, in certain exponential decay curves which level off to some unknown value: y=a*exp(-b*x)+c. We can try to linearize such an equation by getting rid of the exp() function by taking the log of each side of the equation, but because of the c term, one would get log(y-c) equals a linear function. There is no way to get rid of the c term in the usual way: the model is intrinsically nonlinear. It should be emphasized that the model should precede the method. Using brute force statistics to fit arbitrarily-selected nonlinear models to the data may lead to statistical good fit, but such a model is very apt to be unstable when applied to a new set of data (it is apt to be overfitted) and such a model is apt to contain unduly complex terms which cannot be explained by theory (it is apt to lack parsimony). In SPSS, select Analyze, Regression, Nonlinear from the menu system. Enter a numeric dependent variable; enter a model expression in the Model field; click Parameters to identify parameters for the model. If a segmented model is desired (one with different expressions for different parts of its range), use conditional logic in the Model statement. See below for further discussion.

Contents Key concepts and terms Statistics Assumptions Frequently asked questions Bibliography

Key Concepts and Terms

• Entering a model. The researcher should select the model based on theory and past experience in the field, not arbitrary selection or brute force shuffling through a large number of possible models. For instance, in demographics it has proved useful to model population growth using a logistic growth model, the equation for which is: b1/(1+exp(b2+b3*x)). In this equation, x is the time variable (ex., year) and b1 is the maximum value of the dependent range (the asymptote). A good starting value for estimating be would be somewhat more than the highest observed value for Y, which is population in this example. At time=0, which means x=0, the b3*x term drops out. To solve for b2, get rid of the exp() operator by taking the log of both sides and transpose, giving b2 = log((b1/Y₀)-1), where Y₀ is the population at time 0. If Y₀ is known, then b2 can be solved to get a good starting value. Using the original equation, the population at Y₁ = b1/(1+exp(b2+b3*1)), and b3 can be solved as b3 = log((b1/Y₁)-1)-b2), assuming the population at year 1 is also known. See Norusis, 2007: 300-302). In sum, the model is selected on theoretical grounds and known values are used to solve parameters to get useful starting points to enter in the Parameters dialog discussed below.
  • SPSS dialog. The main Nonlinear Regression dialog in SPSS, the researcher enters the dependent variable in the Dependent textbox and the model equation in the Model textbox, then clicks the Parameters button to give starting values for each parameter named in the equation.

  
  

• Parameters. Parameters are the terms in the model equation (entered in the Model textbox) which are to be estimated. In SPSS, once a model (equation) is entered and its name and starting values given as described below, the parameters will appear listed in the Parameters list on the left-hand side of the Nonlinear Regression dialog box.
  • Parameter names. Clicking on the "Parameters" button in the Nonlinear Regression dialog box lets the researcher enter parameter names and starting values. Names must be the same ones used in the Model expression. In the models shown in the table in the "Overview" section above, for instance, the "b" terms would be parameters and the "x" terms would be predictor variable names.

  • Starting values. Starting values should be as close as possible to the expected estimated values for the given parameter. Poor starting values can lead to suboptimal estimates (local optimums) or failure to converge on a solution at all. Poor starting values can also result in out-of-range "impossible" estimates. Sometimes starting values can be calculated by solving a set of simultaneous equations, as discussed above in the "Entering a model" section. Another strategy is to use estimates from an OLS linear regression model. The professional literature may report parameter estimates useful as starting values. There is an option to "Use starting values from previous analysis" which may be selected if the researcher has already run a nonlinear regression analysis from the same dialog box using the same model. If all else fails, "guesstimates" are better than random starting values. After specifying a name and starting value, click Add to add the parameter to the Parameters list. When all parameters are entered, click "Continue".

  • Parameter constraints and overflow/underflow errors. The "Constraints" button from the Nonlinear Regression dialog allows the researcher to restrict permissible parameter values used by the iterative algorithm in estimating parameters. Unconstrained is the default. For instance, b2 <= 2 requires the b2 parameter to be less than or equal to 2. Using conditional logic, (b2 >= 0 & b2<1) = 1 requires that b2 be greater than 0 and less than 1. If you get overflow or underflow errors during iteration, numbers have been generated in the iterative process too large or too small for computer capacity, possibly because the specified model required exponentiation of or by large data values. Using parameter constraints may avoid such errors. Other strategies include transforming raw values to be smaller in magnitude (though not all models are scale-invariant), or using a different estimation algorithm. Note that setting a parameter constraint resets the estimation algorithm to sequential quadratic programming (see below) as the default algorithm is only for unconstrained models.

  • Estimation algorithms. In SPSS, two estimation algorithms are available by pressing the Options button in the main Nonlinear Regression dialog. The default is Levenberg-Marquardt. The alternative is the sequential quadratic programming algorithm. If there are overflow/underflow errors and failure to converge, one may select the latter method. Which algorithm is "best" depends on the data. Note that other statistical packages may use other estimation algorithms not available in SPSS (as of SPSS 16 at least), such as the Quasi-Newton method, the Simplex procedure, Hooke-Jeeves Pattern Moves, and Rosenbrock pattern search. This may lead to differences in parameter estimates between statistical packages.

  • Other options. The "Options" button also enables the researcher to select "Bootstrap Estimates" (standard errors are estimated using repeated samples from the dataset, a method not requiring distributional assumptions). If bootstrap estimates are requested, the estimation algorithm should be set to sequential quadrativ programming (see above). Another option is to give a researcher-set value for the maximum number of iterations, the step limit, tolerance, and convergence criteria. Options vary according to the estimation method. If iteration stops only because the default maximum is reached (a footnote to the "Iterations History" table informs you), parameter estimates may be unreliable and it is advisable to set the maximum number of iterations to a higher value. Setting the step size larger will speed calculation but run the risk of skipping over the optimal parameter estimate, so this option is recommended only when convergence is not attained using the default. Smaller step size, of course, increases calculation time. For the default estimation method, the default convergence criterion is 1E(-08) Setting the criterion larger will requirer fewer iterations to convergence, smaller more.

  
  

• Segmented models. It is possible to specify a nonlinear model which has different equations for different ranges of its domain. The equations are specified as terms within conditional logic statements in the usual Model textbox used for simpler equations.
  • Conditional logic statements are equality and inequality statements which evaluate to 1 if true and 0 if false. For instance:
    1. (X<=0) is 1 if X is less than or equal to 0, otherwise the expression is 0
    2. (X>0 & X<1) is 1 if X is more than 0 and less than 1, otherwise the expression is 0
    3. (state = "NY") is 1 if the string variable state is "NY", otherwise the expression is 0
    Note in case (2) that double inequalities are specified with the ampersand (not "0< X <1"). Note in case (3) that the value of a string constant must be contained withing quotation marks or apostrophes.
    

  • Multiple conditions string conditional logic statements together using the "+" operator. Logic statements, which evaluate to 0 or 1, are multiplied by some appropriate value. For example, (X<=0*0)+(X>0 & X<1)*X+(X>1)*1 returns 0 when X is less than or equal to 0, 1 if X is more than 1, and otherwise returns X.

  • Alternative models as multiple conditions. In creating a segmented model, one constructs a series of conditional logic statements, one for each range of the dependent thought to be characterized by a different curve. Then in each conditional statement, the "True" (evaluates to 1) condition is multiplied by the equation which applies to that range. That is, the complex equation substitutes for "X" in the simple example above.

  
  

• Setting the loss function. The loss function is what the nonlinear regression algorithm is trying to minimize when it iteratively estimates parameters in the model . By default in SPSS the loss function is based on minimizing the sum of squared residuals, but the researcher may set a different loss function.
  • In SPSS click the "Loss" button on the main Nonlinear Regression dialog to enter a custom loss function, if desired. If you entered RESID_**2, you would be entering the default sum of squared residuals loss function. Conditional logic for segmented loss functions can be used, and also string variables, as discussed above in terms of segmented models. However, the default loss function is by far the most common choice in practice.

  • Other loss functions include minimizing absolute deviations, weighted least squares, maximum likelihood estimation with probit and logit models, and others.

  
  

• Statistical Output. In the SPSS nonlinear regression module, various statistics can be output:.
  • Parameter Estimates Table. This portion of the output gives estimated coefficients for each parameter in the model in the "Estimate" column, followed by the corresponding standard error and lower and upper 95% confidence interval bounds. If 0 is within these bounds, the parameter cannot be assumed to be different from 0. Note, however, that unlike OLS regression, these bounds are not exact but rather are approximations based on asymptotic (large sample) assumptions or, if requestion, on bootstrapped estimates based on multiple resampling of the dataset.

  • Correlation of Parameter Estimates Table. This gives the parameters as both rows and columns, with the cell entries being the correlations. If the correlations are very high, this may indicate too small a sample for the model or, put another way, too many parameters in the model for the sample size ("overparameterization"). A larger sample and/or a more parsimonious model may be called for.

  • ANOVA Table. The Anova table gives sums of squares, with degrees of freedom, for "Residual" (for residuals, representing error) and for "Corrected Total" (for deviations around the mean, representing maximum possible error). A footnote to this table gives R-Squared, a model fit measure, reminding you that R-squared = 1 - (Residual Sum of Squares) / (Corrected Sum of Squares). R-square is the percent that can be explained by the model of the variance of the dependent variable around its mean. The Anova table also gives summs of squares for "Regression" (for predicted values) and for "Uncorrected Total" (for the dependent variable).

  
  

• Saving output variables. Using the "Save" button on the main SPSS Nonlinear Regression dialog allows saving of residuals, predicted values, derivatives, and loss function values (only if a custom loss function is requested) for using with the Graph command or other SPSS procedures. SPSS names for these saved variables are resid, pred_, d followed by the first six characters of parameter names (for derivatives), and loss_ respectively.
  • Residual analysis, in particular, depends on saving the residuals and is critical to model diagnostics, identifying outliers, etc. A simple example would be plotting residuals on the Y axis against the dependent variable on the X axis, to see if error was heteroscedastic (varied by range of the dependent variable).

  
  

Assumptions

• Data level. In SPSS, all variables must be quantitative. This means any categorical variables must be recoded as binary variables (ex., 0-1 dummy variables, though other types of contrast may be coded).

• Proper specification. Coefficients are properly interpreted only if the correct nonlinear model has been fitted. The curve estimation procedure in SPSS (select Analyze, Regression, Curve Estimation) may assist in identifying useful models and may even lead to foregoing nonlinear regression in favor of GLM transformations or GZLM link functions. The Curve Estimation procedure in SPSS compares linear, logarithmic, inverse, quadratic, cubic, power, compound, S-curve, logistic, growth, and exponential models.

• Good starting points. As a technical consideration, note that for some models, a poor choice of starting values may lead to failure of the model to converge or to a locally optimal solution that is not globally optimal. That is, you may fail to get output or you may get output, but it may be misleading.

Frequently Asked Questions

• Does nonlinear regression significantly improve upon nonlinear transforms in GLM or link functions in GZLM?
  Obviously it depends, but one author comparing methods found only a 2% improvement in a few case studies. Some improvement is expected since one can fit a more complex, even arbitrary curve. This might reflect overfitting to the current data, suggesting the need for crossvalidation of the linear regression model with a hold-out validation dataset, to demonstrate stability. Rather than be motivated by the quest for the highest possible R-squared, choice of nonlinear modeling may be on sounder ground when selected because the model (equation) emerges from theory and practice in the literature, leading the researcher to wish to test this model against his or her own dataset. Working the other way, having data but no particular model in mind, is apt to lead to overly complex models which lack stability. Nonlinear regression is thus a doubtful exploratory technique but has value as in confirmatory research.

• Remind me about the rules for logarithmic transformations that might be needed when trying to linearize a model equation.
  Logarithms are nearly all either base 10 or natural logarithms to the base e. In nonlinear modeling, natural logarithms are more common and are signified by "ln", whereas base 10 is usually signified by "log", but the operations are the same.
  Multiplication rule: ln(XY) = ln(X) + ln(Y)
  Division rule: ln(X/Y) = ln(X) - ln(Y)
  Exponential rule: ln(X)^Y = Y* ln(X)
  Logarithm of 1.0 = 0
  The log of 0 or of a negative number is undefined.
  

Bibliography

• Bates, D. M. and Watts, D. G. (1988). Nonlinear regression analysis and its applications. New York: Wiley. Second ed., 2007.
• Seber, G. A. F. and Wild, A. J. (1989). Nonlinear regression.. New York: Wiley. Second ed., 2003.
• Huet,Sylvie; Anne Bouvier; Marie-Anne Poursat; & Emmanuel Jolivet (2003). Statistical tools for nonlinear regression: A practical guide with S-PLUS and R examples. NY: Springer.