Variance Components Analysis

Overview

Variance components models are a way to assess the amount of variation in a dependent variable that is associated with one or more random-effects variables. The central output is a variance components table which shows the proportion of variance attributable to the random variable's main effect and, optionally, the random variable's interactions with other factors. Random-effects variables are categorical factors whose categories (levels) are conceived as a random sample of all categories (not as all the categories/levels of interest). Examples might include id variables for observers taking measurements, for days of the week when measurements were taken, or for subjects. Variance components analysis will show whether such random observer effects, day-of-week effects, or subject effects are important or if they may be discounted.

Variance components analysis usually applies to a mixed effects model - that is, one in which there are random and fixed effects, differences in either of which might account for variance in the dependent variable. There must be at least one random effects variable. For instance, a researcher might study time-to-promotion for a random sample of fire stations, also looking at hours of training of the firemen. Stations would be a random effect. Training would be a fixed effect. Variance components analysis would reveal if the between-stations random effect accounted for an important or a trivial amount of the variance in time-to-promotion, based on a model which included random-effects variables, fixed-effects variables, covariates, and interactions among them. In general, variance components analysis is often used in split plot, univariate repeated measures, random block, and other mixed effects designs.

Actual hypothesis testing is normally done using linear mixed models (which also analyzes variance components), univariate general linear modeling (GLM), or some other procedure. The variance components procedure is often an adjunct to these procedures, particularly linear mixed modeling. Note that unlike the linear mixed models procedure in SPSS, the variance components procedure estimates only variance components, not model regression coefficients.

Variance components analysis is an adaptation of general linear model (GLM) univariate procedures, with the same models as in the GLM Univariate procedure in SPSS. GLM in SPSS does support analysis of variance associated with random effects but estimates their parameters as if they were fixed, calculating variance components based on expected mean squares. In contrast, the variance components procedure, like the linear mixed models procedure, uses maximum likelihood estimation to estimate these parameters. In fact, the variance components procedure supports four methods of estimation, each of which gives somewhat different estimates: analysis of variance (ANOVA), maximum likelihood (ML), restricted maximum likelihood (REML), and the minimum norm quadratic unbiased estimator (MINQUE) method. The Variance Components procedure in SPSS also contrasts in this way with the linear mixed modeling procedure, which only supports ML and REML estimation of variance components.

In SPSS, select Analyze, General Linear Model, Variance Components. One must have the Advanced Models option installed.

Key Concepts and Terms

• Variables

  • Variable entry in SPSS is accomplished directly via the main dialog for Variance Components. The dialog lists dataset variables on the left, while on the right there are places to move variables representing the dependent variable (only one is allowed), categorical fixed factors, categorical random factors, continuous covariates, and a WLS weighting variable (only one is allowed). The minimum model specification is one dependent and one random factor.
    

  • Dependent variable. The dependent variable must be quantitative.

  • Factors. Independent variables include categorical factors. SPSS supports numeric or string coding of factors.
    • Fixed effects factors. Fixed factors are categorical variables where the levels are exhaustive (are all the categories of interest are measured for that variable). The operative phrase is "of interest": there may be non-measured categories but if the measured ones are the only ones of interest, the variable is a fixed factor.

    • Random effects factors. There must be at least one random-effects factor, where levels of that factor are seen as a random sample of all possible levels in the population (only some of all possible categories for the variable are measured)..

  • Covariates. Optionally there may also be quantitative independent variables also correlated with the dependent variable. Generally, the factors are the variables of primary interest and covariates are control variables. Covariates in the variance components procedure must be considered fixed effects, unlike the more general mixed linear modeling (LMM) procedure, which allows random effects covariates.

  • Interactions. Models, discussed below, can be constructed using any combination of interactions between and among random effects variables, fixed effect variables, and covariates. A factor-covariate interaction implies the researcher believes there is a linear relationship between the covariate and the factor such that the relationship shifts for different levels of the factor. Which interactions are modeled will affect the estimates of variance components. Note, however, that the variance components table in output will show estimates only for the main effects of the random variable(s), interactions of the random variable(s) with one or more fixed factors, and interactions of random variables with each other, depending on what terms are entered into the model. That is, interactions of a random variable with a covariate may be entered in the model, but no variance component estimate will be calculated for these interactions.

  • Nested factors. In addition to being "crossed" (interactions), random factors can be nested. For instance, a political analyst may be interested in the effect on opinion poll points of inches of newspaper advertising. Newspapers would be a random variable because only a sample of newspapers would be tested. Advertising inches would also be random since only, say, three amounts from among all possible amounts would be tested. That is, inches would be random within newspapers, and this would be expressed as the nested factors, inches(newspaper). The model might be pollpoints = constant + newspaper + inches(newspaper).

  • Weighting. WLS weighting is an option for giving different cases different weights. For instance, cases with greater precision of measurement might be weighted more.

• Models
  • Model entry in SPSS is accomplished in the Model dialog, accessed via the Model button from the main Variance Components dialog screen. The Model dialog is the same as that found in univariate GLM (Anova).
    

  • Full factorial models. This is the default, specifying a model with all main effects of fixed and random factors, all covariate main effects, and all factor-by-factor interactions. Covariate interactions are not included in full factorial models.

  • Custom models. If a full factorial model is not wanted, the researcher can check the Custom radio button and proceed to specify exactly which main effects and interactions should be in the model. Interactions may be factor-by-covariate, factor-by-factor, or covariate-by-covariate. However, the model must contain a random-effects factor.

  • Balanced vs. unbalanced models. In experimental research most models are balanced, meaning that there are equal numbers of observations in each cell formed by categories of the factors. Unbalanced designs have unequal n's in the cells. As discussed below, this can make a difference in the options chosen in estimating variance components.

  • Multilevel models can be modeled in the variance components procedure (VARCOMP) as they can in the mixed linear models (MIXED) procedure in SPSS. Level 2 variables must be categorical and are modeled as random-effects factors. Note that the linear mixed models (MIXED) procedure in SPSS, in contrast, does not require high level variables to be categorical only for multilevel models.

  • Repeated measures models. Like the linear mixed models procedure, the variance components procedure can handle repeated measures, as in before-after studies and other designs where the same individual or case is measured at more than one point in time. In a repeated measures model, subject is treated as the random effects variable and it is nested within fixed between-subjects factors (ex., gender, which does not vary by time of measurement) but is not nested within fixed within-subjects factors (ex., year, which does vary by time of measurement since it dates the measurement). The model might be dependent = constant + gender + gender(subject) + year + year*gender, plus a residual term. For further explanation, see Norusis (2007: 184-190; Winer, Brown & Michels, 1991). Note, however, that the univariate or multivariate general linear model (GLM) module is more versatile for repeated measures designs.

  • Intercept. The Model dialog allows inclusion of an intercept in the model. Inclusion is the default. If the researcher feels the assumption is warranted that the model passes through the origin, the intercept may be deselected.

• Variance components
  • VARCOMP. In the SPSS variance components procedure, which in syntax is the VARCOMP procedure, the variance components are a partition of total variance in the dependent variable into components associated with the dependent variable. The partitions or components will be the main effect(s) of the random-effects variable(s), interactions of the random effects variables with fixed effects variables if specified, and a component associated with error, representing all other effects.

  • ANOVA. It should be noted that there is a more general use of the term "variance components" to refer to any procedure which partitions the variance of the dependent variable in any way, as is done, for example, in analysis of variance.

• Estimation
  • Specifying estimation options is done in SPSS in the Options dialog, accessed by clicking the Options button from the main Variance Components screen.

  • Methods. There are four methods:
    • ANOVA The analysis of variance method has the advantage that no assumption is required assuming that the random effect or the residual error are normally distributed. This method can sometimes yield negative variance estimates, an outcome which points to incorrect model specification, too small a sample, presence of outliers, variance in the random effect approaching zero, or that a different estimation method should be used, perhaps because of negative correlations in the data. The ANOVA method should not be used when the cells formed by the factors include empty cells as this may result in negative variance estimates. See Hocking (1985) for further discussion and additional reasons for negative variance estimates.
      • Sums of squares. If the ANOVA method is selected, the researcher must choose between estimation based on Type I or Type III sums of squares. Type III is the default and is appropriate for most models, including unbalanced models. Type I is used in hierarchical models, where the researcher is able to stipulate beforehand the order of model predictors, in a hierarchical design where main effects are specified before first-order interaction effects, and first-order interaction effects are specified before second-order interaction effects, etc. Type I is also used for purely nested models where a first effect is nested within a second effect, the second within a third, etc.; and is used in polynomial regression models where simple terms are specified before higher-order terms (ex., squared terms). Type I tests may be thought of as a test of sequential effects: if an effect is non-significant by Type I, this means it cannot be assumed to be different from 0 when that effect is entered into a model containing only effects that precede it in order of entry. Type I ANOVA is not appropriate for unbalanced models.

        There is a second reason to choose Type I sums of squares, however, and this reason is why Type I is the default in SPSS and SAS. Type I sums of squares, as the figure below illustrates, are additive, whereas Type III are not. Total and error sums of squares, and mean square error, are the same under Type I and Type III, but sums of squares for other model terms are not. The result is that the VARCOMP procedure will produce the same variance components estimate for error under Type I and III, but other random effect component estimates will differ, except for the highest-order term (here, storeid*gender*usecoup). This is because the highest order term in a factorial model is confounded with residual (random) error.

        

      • Display sums of squares and expected mean squares. If the ANOVA method is selected, in addition to selecting which type of sums of squares to apply, the researcher can ask for output of the sums of squares and expected mean squares. Expected mean squares are set to 0 for covariates, which are not included in the variance components estimates.

    • ML. Maximum likelihood estimation creates values for estimates make observed data most likely. The ML method requires assuming that the randome effect and the residual error are normally distributed. Violation of normality can lead to biased estimates, but this is not a problem with large samlpes as ML is asymptotically normal. Also, ML is efficient at recovering from initial parameter estimates which were inaccurate. ML output has the advantage of generating an asymptotic covariance matrix, which contains the estimated variance for the random factor. The square root of this variance is the standard error for the random factor and can be used to calculate confidence intervals on the estimate.
      • Criteria.. If the ML or REML method is selected, the researcher may specify the convergence criterion (1E-8 is default) and the maximum number of iterations (50 is default).

    • REML. Restricted maximum likelihood estimates also assumes normality. It is a variant on the ML method which incorporates model degrees of freedom in estimating variance components. REML usually leads to smaller standard errors than ML and is often preferred for balanced designs. For instance, in mixed linear modeling, REML is the most common choice.

    • MINQUE. The minimum norm quadratic unbiased estimator, MINQUE, is the default method in SPSS. MINQUE is robust against mild to moderate departures from normality and can fit a wide variety of models, including unbalanced designs (unequal numbers of observations in each cell formed by the factors). Like the ANOVA method, MINQUE can yield negative variance components estimates.
      • Random effect priors. If the MINQUE method is selected, the researcher must further select between uniform or zero random effect priors. That is, this method requires setting a priori values for the variance components. The MINQUE algorithm uses the proportions in the priors when calculating estimates of the variance components. Uniform priors, which is the default assumption, assumes that the collective effect of all random effects is equal to the effect of the residual term in the model. Zero priors implies that the variances of the random effects are zero.

  
  

• Output
  • Variance components estimates are output for any of the four variance components methods. By default, VARCOMP computes a full factorial model for whatever fixed and random effects and covariates are entered, but the variance estimates shown below are only for terms involving a random effects variable or the error variance. These estimates will differ, of course, if something other than a full factorial model is entered using the Custom radio button in the Model dialog, or if there are other factors or covariates in the model.

    

    In the example above, storeid is a random factor identifying stores selected at random from a population of stores. The dependent variable is amount spent. The fixed factors are gender and source of coupon used. There is no covariate. As can be seen, the four methods do not differ sharply on the estimate of the unexplained variance component, reflected in the Var(Error) term. In all four methods, the unexplained variance component is the largest component. The highest-order interaction term (here, storeid*gender*usecoup) is confounded with residual (random) error (or one may say residual error is confounded with the highest order interaction). The researcher may be interested, for instance, in whether the variation in estimates of the random effect variable's contribution to the random variation in the dependent variable are large or small compared to the error variance component and random error. In the example above, they are relatively small.

    However, in this example the partition of other variance components of the random-effects variable storeid differ considerably by method, and in the case of the ANOVA method even lead to a negative variance component estimate. When estimates contradict each other (ex., differ in sign), a common research strategy is to re-run the analysis using alternative methods, then discount the aberrant estimate.

    • Redundant estimates in ML and REML. ML and REML estimation methods constrain estimates not to be negative, causing such estimates to be set to 0 and labeled redundant.

    • Confidence limits on variance components, while not output by SPSS, can be computed easily. The standard error of the MINQUE estimate of var(storeid) is its positive square root = SQRT(307.103) = 17.524. The 95% confidence interval is then 307.103 plus or minus 1.96*17.524, which is the range from 272.756 to 341.451.

  • ANOVA table and expected mean squares are output if the ANOVA method is selected. These could be used to compute the variance components estimates above on a manual basis (as explained, for instance, in the online SPSS case tutorial).
    
    

  • Covariance matrix table. An asymptotic covariance matrix table is output if the ML or REML method is selected.

  • Iteration history table. An iteration history table is output if the ML or REML method is selected.

  • Saved variables. By clicking the Save button in the SPSS variance components dialog, one can save the variance components estimates regardless of which method was used. If the ML or REML methods was used, one can also save the component covariance or correlation matrix.

Assumptions

• Random effects. There must be at least one random effects variable.

• Homoscedastic random effects. It is assumed that the model parameters of a random effect have means of zero and have a constant variance across the range of the dependent variable. Random effects are assumed to be independently and identically distributed (iid).

• Uncorrelated random effects. If there are two or more random effects in the model, it is assumed that their effect parameters are uncorrelated. Knowing the effect for one level of the random-effects variable does not predict the effect for another level. However, observations may be correlated within any given level of a random factor.

• Uncorrelated residuals. It is assumed that residuals from one observation are independent of and uncorrelated with those from another. That is, knowing the residuals for one case does not help in estimating the residuals from another case. Residual error is uncorrelated with both fixed and random effects in the model.

• Well distributed residuals. Errors are assumed to be normally distributed around a mean of 0. Residuals should display a constant variance across the range of the dependent variable. Residuals should have a constant variance across categories formed by the factors (for design cells). The more unequal the cell sizes, the more important this assumption.

• Normality. ML and REML methods require that both model effects and residuals be normally distributed. ANOVA and MINQUE are robust for mild to moderate violations of normality assumptions.

• Linearity. As in other GLM models, factors and covariates are assumed to be linearly related to the dependent variable. Note that parallelism is not assumed: each factor level may have a different linear effect on the dependent variable. For covariates, within each combination of factor levels (within each design cell), the covariates are assumed to be linearly related with the dependent variable.

Frequently Asked Questions

• What is the syntax for the variance components procedure in SPSS?

   VARCOMP dependent variable BY factor list [WITH covariate list] 
    /RANDOM = factor [factor ...]
   [/METHOD = {MINQUE({1})**}]
              {0}
              {ML           }
              {REML         }
              {SSTYPE({3})  }
                      {1}
   [/INTERCEPT = {INCLUDE**}]
                 {EXCLUDE    }
   [/MISSING = {EXCLUDE**}]
               {INCLUDE  }
   [/REGWGT = varname]
   [/CRITERIA = [CONVERGE({1.0E-8**})] [EPS({1.0E-8**})] [ITERATE({50**})]]
                          {n       }        {n       }            {n   }
   [/PRINT = [EMS] [HISTORY({1**})] [SS]]
                            {n  }
   [/OUTFILE = [VAREST] [{COVB}] ('savfile'|'dataset') ]
                         {CORB}
   [/DESIGN = {[INTERCEPT] [effect effect ...]}]
  
  ** Default if the subcommand or keyword is omitted.
  
  Example:
  
  VARCOMP Y1 BY B C WITH X1 X2
   /RANDOM = C.
  

• How does variance components (VARCOMP) analysis compare to linear mixed models (LMM, aka MIXED) and general linear models (GLM) for the same data?
  This question is illustrated in the section on linear mixed models. In general, VARCOMP is a subset of MIXED in SPSS, but does support MINQUE and ANOVA estimation, which MIXED does not, and with ANOVA comes sums of squares, mean squares, and estimated mean squares not produced by the ML and REML methods used by MIXED.

Bibliography

• Corbeil, R. R. & Searle, S. R. (1976). Restricted maximum likelihood (REML) estimation of variance components in the mixed model. Technometrics 18, 31-38
• Giesbrecht, F. G. (1983). An efficient procedure for computing MINQUE of variance components and generalized least squares estimates of fixed effects. Communications in Statistics, Part A - Theory and Methods 12, 2169-2177.
• Hemmerle, W. J., & Harley, H. O. (1973). Computing maximum likelihood estimates for the mixed A.O.V. model using the W transformation. Technometrics 15, 819-831. Focuses in detail on the ML method.
• cThe analysis of linear models. Monterey, CA: Brooks/Cole.
• Jennrich, R. I., & Sampson P. F. (1976). Newton-Raphson and related algorithms for maximum likelihood variance component estimation. Technometrics 18, 11-17.
• Lynch M, & Walsh B. (1998). Genetics and analysis of quantitative traits. Sunderland, Mass: Sinauer Associates Inc.
• Norusis, Marija J. (2007). SPSS 15.0 Advanced Statistical Procedures Companion. Chicago, IL: SPSS, Inc.
• Planken, R. Nils ; Keuter, Xavier H. A. ; Hoeks, Arnold P. G. ; Kooman, Jeroen P. ; van der Sande, Frank M. ; Kessels, Alfons G. H. ; Leiner, Tim; & Tordoir, Jan H. M. (2006). Diameter measurements of the forearm cephalic vein prior to vascular access creation in end-stage renal disease patients: Graduated pressure cuff versus tourniquet vessel dilatation. Nephrology Dialysis Transplantation 21(3), 802-806. Example of use of variance components analysis.
• Rao, C. R. (1973). Linear statistical inference and its applications, Second ed.New York: John Wiley and Sons. Summarizes the MINQUE method.
• Rao, C. R., and J. Kleffe. (1988). Estimation of variance components and applications. Amsterdam: North-Holland. Focuses in depth on the MINQUE method.
• Searle, S. R., Casella, G. & McCulloch, C. E. (1992). Variance Components. New York: John Wiley and Sons.
• Speed, F. M. (1979). Choice of sums of squares for estimation of components of variance. Proceedings of the Statistical Computing Section. Alexandria, Va.: American Statistical Association. Discusses selection of the appropriate type of sum of squares for the ANOVA method.
• Winer, B. J.; Brown, R; & Michels, K. M. (1991). Statistical principles in experimental design, 3rd ed.. NY: McGraw-Hill.