Weighted Least Squares (WLS) Regression

Overview

One of the critical assumptions of OLS regression is homoscedasticity: that the variance of residual error should be constant for all values of the independent(s). If the independent(s) has/have different error variance at different ranges of their values, then the estimates of the regression coefficients will have unduly large standard errors for some ranges of the dependent and too small for other ranges. The power of significance tests will be reduced, which is to say regression estimates will be inefficient. Weighted least squares (WLS) regression compensates for violation of the homoscedasticity assumption by weighting cases differentially: cases whose value on the dependent variable corresponds to large variances on the independent variable(s) count less and those with small variances count more in estimating the regression coefficients. That is, cases with greater weights contribute more to the fit of the regression line. The result is that the estimated coefficients are usually very close to what they would be in OLS regression, but under WLS regression their standard errors are smaller.

Apart from its main function in correcting for heteroscedasticity, WLS regression is sometimes also used to adjust fit to give less weight to distant points and outliers, or to give less weight to observations thought to be less reliable.

SPSS: WLS in SPSS is a two-step process. There is a "Weight Estimation" module (Analyze, Regression, Weight Estimation) to calculate and save the optimal weights using the Save button. Then the OLS regression is run on the weighted cases (Analyze, Regression, Linear). Note the weights are entered in the "WLS weights" box of the linear regression dialog, not under Data, Weight Cases (which is ignored). Note also that the "Weight Estimation" routine will also generate the estimated regression coefficients and their standard errors, but by running the OLS regression on weighted cases, fuller output is obtained, including, for instance, residuals useful for analyzing patterns of error.

Key Terms and Concepts

• Violation of homoscedasticity occurs when error variance is correlated with the magnitude of the dependent. Put another way, violation of homoscedasticity occurs when the magnitude of the dependent is correlated with the variance of the independent. One possible cause of this might be a skewed rather than normally distributed dependent variable. When homoscedasticity is violated, a simple scatterplot of the independent, x, on the x axis, and the dependent, y, on the y axis, will typically display a "funnel shape," as in the figure below where age predicts preference.
  

  That is, the y-dimension spread of points may increase (or decrease), often exponentially, as x increases linearly. It may simultaneously be true that the x-dimension spread of points may increase (or decrease), often exponentially, as y increases linearly. In the figure above, the variance in Preference increases for higher values of Age.

  In more unusual cases, violation of homoscedasticity may display a "u-shaped" distribution (narrow in the middle, expanding to a funnel at both ends), or other non-homogenous distributions. A violation of homoscedasticity can be seen after running OLS regression: a plot of standardized predicted values on the x axis and studentized residuals on the y axis will also show a "funnel" or other heteroscedastic shape rather than the desired random cloud of residual (error) values all along the x axis. I

• Weighted cases. To reduce standard error associated with parameter (regression coefficient) estimates in OLS regression when homoscedasticity is violated, cases are weighted by the reciprocal of their estimated point variance. This causes cases where the independent(s) have large variance to count less in the parameter estimates.
  • Weighting with powers. The SPSS Weight Estimation procedure estimates case weights to use when compensating for violation of homoscedasticity. It does so by looking for the power of the independent variable which maximizes the log-likelihood of the dependent variable, then uses the reciprocal of this power of the independent to weight the cases. This procedure assumes that violation of homoscedasticity is characterized by the variance of the dependent increasing exponentially at some power function of the independent. In the Weight Estimation procedure, the researcher inputs a power range and increment (ex., -3.0 to 3.0 by steps of .2), and SPSS finds the power which maximizes the log likelihood function.

    
    • SPSS: As illustrated in the figure above, select Analyze, Regression, Weight Estimation; enter the dependent and the independent(s) in the dialog which appears; enter the weight variable (the variable that is the source of heteroscedasticity; SPSS accepts only one weight variable per run of the Weight Estimation procedure) and give a power range and step increment; check to save the "best weight as new variable" (SPSS will create a variable called WGT_n, where n is a number assigned to give the variable a unique name, starting with WGT_1). Note that the largest "log-likelihood function value" is shown in the "Model summary" table. The "Log-likelihood values" table shows the possible powers at all steps and the corresponding log-likelihood value, and in this table the one which maximizes the log-likelihood function is footnoted. The maximizing log-likelihood value is not necessarily the largest value in the "Log-likelihood values" table. Output is illustrated below:
      

      In this case the value which maximizes the likelihood function is 3.0. In the annotated output below, a finer set of steps reveals that 3.2 is the best value. In this run, however, 3.0 is used as the power value to generate the WGT_1 values, which become added to the dataset as illustrated below:

      

  • Weighting with replicates. A special case occurs when the values of independent fall into sets of replicates of the same value, as would happen when the independent has finite discrete values. In this situation, one can compute directly the variances of the independent for each replicate value and use the reciprocal of these variances as the values of the weighting variable. For two or more independents, one computes variances for each combination of replicate values across the set of independent variables (this may require a large sample size to have enough cases in each cell of the table of replicate combinations). The Weight Estimation procedure is not used but instead weights are entered directly into a weight variable, cases are weighted, and ordinary OLS regression is run.

• Weighted predicted/residual plots can be used to assess the goodness of fit of the weighted model. That is, weighted predicted is plotted on the x axis and weighted residual on the y axis. When there is good fit, the residuals will no longer form a funnel shape but instead be uniformly distributed around the 0.0 line of the y axis.
  • SPSS. Plots are not produced by the Plots button of the linear regression dialog. This is because when WLS weights are entered, SPSS does not compute standardized predicted and residual values. Instead one must use the Save button in the linear regression dialog to first save the unstandardized predicted and residual values (SPSS will add them to the end of the working data sheet as RES_n and PRE_n, where n = 1 for the first such request). This is illustrated in the figure below:
    

    After saving, use Transform, Compute, to create weighted predicted and weighted residual values: ex., WTDRESID = SQRT(WGT_1) * RES_1; and WTDPRED = SQRT(WGT_1) * PRE_1. That is, the unstandardized residual and predicted variables are multiplied by the square root of the WLS weights to get their weighted equivalents. (Of course, if you have run the regression a second time, you will be dealing with RES_2 and PRE_2, etc.)

    

    In the figure above, running linear regression with the Save option gives the PRE_1 and RES_1 columns for unstandardized residual and predicted values, and the Transform, Compute operations create the WTDRESID and WTDPRED columns.

    As a final step, one uses Graph, Scatterplot to plot WTDPRED on the x axis and WTDRESID on the y axis, illustrated below:

    

    Comparing this chart with the initial one above, it can be seen that the funnel shape has disappeared. Because of the reduction in heteroscedasticity, standard errors will be small but estimates will be very similar. The table below shows the estimated b coefficient for age and its standard error, with output above for OLS and below for WLS:

    

  
  

Assumptions

• Proper specification As pointed out by Downs & Rocke (1979), heteroscedasticity can indicate model misspecification. Simply adjusting for it through WLS rather than re-specifying the model may lead the researcher into serious errors in drawing causal inferences.

• Data level. The dependent and the weight variable (the independent variable which is the source of heteroscedasticity) must be interval. The other independents must be interval, dichotomous, or dummy variables coded from categorical variables.

• Multivariate normality. The dependent variable is assumed to be normally distributed for each independent variable in one-IV models or for each combination of values of the independent variables in multi-IV models.

• Linearity. The dependent is assumed to be linearly related to each independent.

• Independence. All observations are assumed to be independent of one another.

• Predictable variance. While homoscedasticity is not assumed, the manner in which the variance of the dependent variable changes with increases in magnitude of the independent(s) must be predictable (and reflected in the weight variable).

Example of SPSS WLS Regression Output

• WLS Output. Sample data predicting preferences from age.

Frequently Asked Questions

• Is WLS regression something that could be used with regression models other than OLS?
  Yes and no. By definition, "weighted least squares regression" is a form of weighting cases and running an OLS model. However, it is possible (and appropriate when homoscedasticity is violated) to use the Weight Estimation procedure to generate a weight variable, then run weighted variables on other models. Logistic regression is often used with dichotomous dependent variables, for instance. Cox regression, Kaplan-Meier survival analysis, and life table analysis are used with censored data (ex., time-to-event or survival data). If data are not related linearly (and transformation doesn't help), the Curve Estimation procedure may be used. If data are not independent, one may use the SPSS linear mixed models procedure, specifying repeated measures.

• Can you do things in syntax not available in the Weight Estimation and WLS dialogs?
  Yes. The WLS command syntax allows the researcher to specify not just a range of power values to test, but also a list, multiple ranges, or mixes of lists and ranges.

  
  

Bibliography

• Downs, George W. & Rocke, David M. (1979). Interpreting heteroscedasticity. American Journal of Political Science 23(4), 816-828.
• Farebrother, R. W. (1988). Linear least squares computations. Boca Raton, FL: CRC Press.
• Wolberg, John (2006). Data analysis using the method of least squares: Extracting the most information from experiments. NY: Springer.