Log-Linear, Logit, and Probit Models

Overview Log-linear, logit, and probit models are special cases of general linear models (GLM, which includes regression and ANOVA models) to better treat the case of dichotomous and categorical variables. Log-linear analysis deals with association of categorical or grouped data, looking at all levels of possible main and interaction effects, comparing this saturated model with reduced models, with the primary purpose being to find the most parsimonious model which can account for cell frequencies in a table. That is, log-linear analysis is a non-dependent procedure for accounting for the distribution of cases in a crosstabulation of categorical variables. Log-linear analysis is a type of multi-way frequency analysis (MFA) and sometimes log-linear analysis is labeled MFA. Logit modeling is similar to log-linear modeling, but explains one or more dependent categorical variables. When there is a dependent categorical variable, however, binary and multinomial logistic regression are more commonly used. Logistic regression is also used when the independents are continuous (forcing continuous variables into categories attenuates correlation and is not recommended). Conditional logit handles matched-pairs and panel data, and data for analyzing choices. Probit is a variant of logit modeling based on different data assumptions. Logit is the more commonly used, based on the assumption of equal categories. Probit may be the more appropriate choice when the categories are assumed to reflect an underlying normal distribution of the dependent variable, even if there are just two categories. Log-linear models were developed to analyze the conditional relationship of two or more categorical values. Log-linear analysis is different from logistic regression in four ways: The expected distribution of the categorical variables is Poisson, not binomial or multinomial. The link function is the natural log of the dependent, not the logit of the dependent as in logistic regression (the natural log of the odds, which is the probability the dependent equals a given value (usually 1, indicating an event has occurred or a trait is present) divided by the probability it does not). Predictions are estimates of the cell counts in a contingency table, not the logit of y. Logit and probit extend the log-linear model to allow a mixture of categorical and continuous independent variables to predict one or more categorical dependent variables. Both logit and probit usually lead to the same conclusions for the same data. Logit regression yields results equivalent to logistic regression, but with different output options. Many problems can be handled by either logit or logistic regression, though the latter has become more popular among social scientists. Note that generalized linear models, discussed separately, represent a more recent set of procedures which can also analyze categorical dependents and independents, and in this sense represent a different method of implementing log-linear, logit, probit, Poisson, and other models. See also the separate section on ordinal regression, which can also implement logit, probit, and other models. See also the separate section on probit response models, which additionally supports logit response models. Traditional approaches to categorical data relied on chi-square and other measures of significance to establish if a relationship existed in a table, then employed any of a wide variety of measures of association to come up with a number, usually between 0 and 1, indicating how strong the relationship was. Loglinear methods are similar in function but have the advantage of making it far easier to analyze multi-way tables (more than two categorical variables) and to understand just which values of which variables and which interaction effects are contributing the most to the relationship. For simple two-variable tables, traditional approaches may still be preferred, but for multivariate analysis of three or more categorical variables, log-linear analysis is preferred. Loglinear methods also differ from multiple regression in substituting maximum likelihood estimation of a link function of the dependent for regression's use of least squares estimation of the dependent itself. The link function transforms the dependent variable and it is this transform, not the raw variable, which is linearly related to the model (the terms on the right-hand side of the equation). The link function used in log-linear analysis is the log of the dependent, y. The function used in logit is the natural log of the odds ratio. The function used in probit is the inverse of the standard normal cumulative distribution function. There are several possible purposes for undertaking log-linear modeling, the primary being to determine the most parsimonious model which is not significantly different from the saturated model, which is a model that fully but trivially accounts for the cell frequencies of a table. Log-linear analysis is used to determine if variables are related, to predict the expected frequencies (table cell values) of a dependent variable, the understand the relative importance of different independent variables in predicting a dependent, and to confirm models using a goodness of fit test (the likelihood ratio). Residual analysis can also determine where the model is working best and worst. Often researchers will use hierarchical loglinear analysis (in SPSS, the Model Selection option under Loglinear) for exploratory modeling, then use general loglinear analysis for confirmatory modeling. SPSS supports these related procedures, among others: Hierarchical loglinear analysis (HILOG). Select Analyze, Loglinear, Model Selection. Often used for automatic selection of the best hierarchical model. General loglinear analysis (GENLOG). Select Analyze, Loglinear, General. Often used to refine the best hierarchical model to be more parsimonious by dropping terms. Poisson regression may be performed within the GENLOG procedure. Often used in event history analysis and other research involving rare events where assumptions of a normally distributed dependent do not apply and the researcher wishes to specify a categorical variable as the dependent variable in a model. Logit. Select Analyze, Loglinear, Logit. Used when the researcher wishes to specify a categorical variable as the dependent variable in a model (hierarchical and general loglinear analyses are non-dependent procedures). Conditional logit may be performed under Analyze, Survival, Cox regression. This is a dependent procedure used when observations are correlated rather than independent, as in before-after datasets. Probit response models. Select Analyze, Regression, Probit. Probit response models are used when the researcher wishes to specify a binary variable as the dependent in a model, and that binary variable is assumed to be a proxy for a true underlying continuous normal distribution. Generalized linear model. GZLM can now implement most of these models.

Contents Key concepts and terms Residual analysis Effect size measures Hierarchical loglinear modeling General loglinear modeling Logit regression Conditional logit Probit Tobit Poisson regression Assumptions Frequently asked questions Bibliography

Key Concepts and Terms

• Types of variables. In log-linear analysis, variables may be factors, dependents, covariates, contrasts, or cell structure variables. Not each type of log-linear procedure supports all types.
  1. Factors, also known as "response variables," are categorical variables which define rows and columns in a crosstabulation table.In hierarchical loglinear models in SPSS, 10 factors are possible and the researcher can specify which levels of each to analyse. In probit models, the factor's levels are used to define subgroups for analysis. Factors are input in the main log-linear dialog for general, hierarchical, logit, or probit models.

  2. Dependents pertain specifically to logit models, where one or more categorical variables may be specified as dependent. Probit models allow entry of a variable either as a response frequency variable (a variable with counts of individuals who responded when exposed to a particular stimulus level or stimuli levels) and a total observed variable (a variable containing the total number of individuals exposed to a particular stimulus level or combination of stimulus levels.)

  3. Covariates. These are continuous variables and the the mean covariate values of cases in any given cell are used to model cell frequencies. Usually covariates are conceptualized as confounding factors or control variables. General, logit, and probit models allow covariates but not hierarchical log-linear analysis.

  4. Cell structure variables are used to weight cells within the table. This is used for a variety of purposes, including forcing structural zeros (discussed in the FAQ section) and table standardization. Cell structure is an option in general and logit models, and hierarchical models has a "cell weight" variable entry option. The probit procedure does not support cell structure/weights.

  5. Contrast variables. You can specify a set of contrast variables to test the differences between model effects by computing generalized log-odds ratios. The values of the contrast variable are the coefficients for the linear combination of the logs of the expected cell counts. Contrast variables may be entered in general, hierarchical, and logit log-linear models, but not as such in probit models.

• Log-linear modeling is an analog to multiple regression for categorical variables. When used in contrast to log-linear regression models like logit and logistic regression, log-linear modeling refers to analysis of table frequencies without necessarily specifying a dependent. Rather the focus is in accounting for the observed distribution of cases.
  • Saturated models and effects. A saturated log-linear model is one which incorporates all possible effects: a 1-way effect for each variable, all 2-way interaction effects for models with two or more variables, all 3-way interaction effects for models with three or more variables, and so on. Overall, there will be (2^k - 1) terms plus a constant in a saturated model's equation predicting the log of an expected table frequency, where k is the number of variables. A saturated model imposes no constraints on the data and always reproduces the observed cell frequencies. As such the saturated model forms the "baseline" for log-linear analysis. The researcher tries to see if a model restricted to only some of the possible effects can explain the observed frequencies. In the SPSS, saturated models are obtained if one does not check Custom under the Model button and thus does not enter specific main and/or interaction effects; there is a Saturated model choice, which is selected by default, and this is true of general, hierarchical, and logit loglinear analysis.

  • Parsimonious models. Saturated models always have perfect goodness of fit to the data, but this is a trivial finding. The purpose of log-linear modeling is to eliminate some of the effects while still being able to achieve goodness of fit. A parsimonious model is the most incomplete model which still achieves a satisfactory level of goodness of fit. To put it another way, the researcher tests to see if a restricted model does not significantly differ from the saturated model. If there is no significant difference, then the researcher concludes that the effects dropped from the saturated model were not needed to explain the observed distribution of data in the table. The researcher explores in this manner until the most parsimonious model which still has acceptable fit is found. In SPSS, parsimonious models are obtained by checking Custom under the Model button and entering specific main and/or interaction effects as a design (or using the /DESIGN command with parameters in syntax mode; with no parameters, a saturated model is specified).

    As elaborated below in the section on effects, a saturated log-linear model takes the form: the natural log of the frequency for any cell equals the grand mean (the constant) plus the sum of the lambda parameter estimates for all 1-way, 2-way, 3-way, .... k-way interaction effects in a model with k variables. Depicted graphically, a saturated model with six variables (A through F) would show connecting lines from each variable to each other variable, for a total of (2^k - 1) = 127 effects and 16 lines. However, a parsimonious model such as that below might have far fewer connecting lines.

    

    The parsimonious model above has the form: ln(cell frequency) = + A + B + C + D + E + F + A *B + B*C + B*D + C*D + D*E + E*F + B*C*D, for a total of only 12 effects and 6 connecting lines. Each effect is reflected in a parameter estimate, discussed below.

  • Effects as categorical control variables.. In the traditional elaboration model using crosstabulation, one divided an original table (ex., literacy by race) into two or more subtables based on the control variable (ex., subtables for region = South or North). For instance, the overall table might show an association between race=black and literacy=not, but if the South tended to have more blacks than the North and more non-literates for both races, then the original relation might well be spurious because region was a control variable. In general, the control variable subtables might show the same relationship as the original table, no relationship, or a relationship for one subtable but not the other. This is illustrated below for a small sample for purposes of showing the corresponding parsimonious model effects when analyzing the same data using loglinear analysis:
    
    

    Original Table
    -	Black	White
    Not Literate	6	2
    Literate	2	6

    South - Black White Not Literate 3 1 Literate 1 3

    North - Black White Not Literate 3 1 Literate 1 3

    Model A: Race*Literacy

    South - Black White Not Literate 6 2 Literate 0 0

    North - Black White Not Literate 0 0 Literate 2 6

    Model B: Race*Region + Literacy*Region

    South - Black White Not Literate 2 2 Literate 2 2

    North - Black White Not Literate 4 0 Literate 0 4

    Model C: Race*Literacy + Race*Region + Literacy*Region

    The original table above is shown with three different possible splits by the control variable Region. In Model A, the split tables have the same relationship as the original table. There is no control effect and therefore the control variable, Region, is not part of the loglinear generating class reported by hierarchical loglinear modeling using backward elimination. In Model B, there is full explanation (total control by the control variable Region) and each component of the loglinear generating class is an interaction involving the control variable). In Model C, the original relationship disappears (is controlled) in the South region but is stronger than the original in the North region, showing the original table to be a misleading average. For Model C, the loglinear generating class contains all three interactions.

  • Convergence. In loglinear modeling, parameter estimates are calculated by an iterative process. SPSS's output log reports how many iterations were taken to reach a solution (convergence). If the model did not converge, the researcher could try increasing the number of iterations allowed. In SPSS, maximum iterations has a default of 20 but can be reset under the Options button. Most models converge under the default setting.

  • Goodness of fit is measured by the likelihood ratio, also known as likelihood ratio chi-square, deviance chi-square, L², or G², discussed below. For two-way tables, traditional Pearson chi-square may be used also. Both the likelihood ratio and chi-square are based on assessing the difference between observed cell frequencies and frequencies predicted by the model.
    • Pearson chi-square. For two-way tables, chi-square may be used to test association. A significant chi-square means that one rejects the null hypothesis that the two variables are independent. A significant likelihood ratio means that one rejects the null hypothesis that the researcher's restricted (parsimonious) model does not differ from the trivial saturated model. Testing the "independence model," discussed below, is equivalent to traditional chi-square tests for lack of association between variables.

    • Likelihood ratio. SPSS generates both chi-square and the likelihood ratio. Because the maximum likelihood estimation used in loglinear analysis minimizes a likelihood function, most researchers prefer to use the likelihood ratio rather than Pearson's chi-square as a goodness of fit test. When the likelihood ratio is not significant then the model being tested is a good fit to the data because this means the parsimonious model is not significantly worse than the perfect-fitting saturated model. This is the same as saying that a non-significant likelihood ratio means the model-predicted frequencies are not significantly different from the observed frequencies. The likelihood ratio is printed by SPSS in the "Goodness of fit" table in output. If a stepwise method has been selected, then at the end of each step, SPSS will print out the "Likelihood ratio chi square". Its P value will be >.05 for a well-fitting model. The larger the sample, the closer likelihood ratio chi-square and Pearson chi-square are apt to be.

    • Factor list warning: SPSS computes goodness of fit measures based on the number of cells created by a table based on all the variables in the factor list. This is not necessarily the same as all the variables in the model, if one asks for a "Custom" model and leaves out one or more factors in the factor list. Goodness of fit measures will differ for the same model, depending on variables in the factor list.

    
    

  • Residual analysis. Goodness of fit confirms/disconfirms the overall model, whereas residual analysis helps the researcher spot outlier cells - where the parsimonious model is not fitting as well, even if overall the model is a well-fitting one by the likelihood ratio chi-square test. In a well-fitting model, ideally, residuals should be small, non-significant positive and negative values which are evenly distributed across all the cells of the table. Note that when there are many cells, many standardized residuals are generated and one would expect 5 percent of the cells to have significant residuals simply due to chance: these are not usually considered outliers for this reason.

    Once the most parsimonious model is selected, SPSS can compute the expected frequencies. These expected frequencies can be subtracted from the observed cell frequencies to give the residuals. The smaller the residual, the better the model is working for that cell. Likewise, large residuals indicate marginal (row and column) conditions where the model is not working well. SPSS shows residuals in a table of "Cell Counts and Residuals." Note SPSS outputs four types of residuals and three types of plots:

    1. Residual: Observed minus expected frequency, aka "raw residual."

    2. Standardized residual: The raw residual divided by the estimated standard deviation of observed counts. Significant standardized residuals have values > 1.96 and such cells may be considered "model outliers." As a rule of thumb, more than one model outlier per 20 table cells may cause the researcher to seek a different model.

    3. Adjusted residual: The standardized residual divided by the estimated standard error. This gives residuals with a mean of 0 and a standard deviation of 1. Adjusted residuals penalize for the fact that large expecteds tend to have larger residuals. Looking at the cells with the largest adjusted residuals shows where the model is working least well. The adjusted residual is also called the studentized residual. Adjusted residuals are are preferred over the standardized residuals when assessing normality.

    4. Deviance: The deviance residual, also called the "studentized deviance residual," is a more accurate version of the adjusted residual. It is how much a cell contributes to the likelihood ratio. Likelihood ratio chi-square equals the sum of squared deviances. It also has a mean of 0 and a standard deviation of 1 for large samples.

    • Residual plots. If the researcher checks the "Adjusted residuals" option under the Plots button of the General Loglinear Analysis dialog in SPSS, a scatterplot matrix will be output. In a well-fitting model, the plot of model-expected and observed frequencies will approximate a straight diagonal line, indicating expected and observed frequencies are close.

    • Q-Q plots: In a well-fitting model, residuals will be normally distributed, with most near 0 and then trailing off in a bell-shaped curve of too-high and too-low values. The Q-Q plot of expected normal values and adjusted residuals should approximate a 45-degree angle when residuals are normally distributed.

    • Detrended Q-Q plots, In this type of plot, there should be no trend in how adjusted residuals deviate from a normal distribution (cases will be scattered randomly with respect the the horizontal 0 line).

    
    

  • Effect size measures. A saturated model has all possible effects and the strategy of log-linear analysis is to set as many of these effects to zero as possible in order to find the most parsimonious model. A one-way effect is the influence of a single row or column variable on cell counts (ex., gender on vote). A two-way effect is the interaction (joint) effect of a column and a row variable (ex., gender and income category on vote). A significant two-way effect means the two variables are related (are not statistically independent of one another). For three variables, there can be one 3-way effect, three 2-way effects,. and three 1-way effects. In general, for k variables, there are a maximum of (2^k - 1) effects. As in regression, a loglinear model also has a constant, which reflects the cell count when all other effects are zero.
    • Lambda parameter estimates. Lambda () is the usual designation for the effect coefficient. Mu () is the usual designation for the constant. These coefficients are obtained in SPSS by asking for Estimates under Options in the loglinear dialog. The loglinear model is one in which the natural log of the frequency for any cell is equal to a grand mean (the constant, mu) plus the lambda parameter estimate for the effect of the first independent, plus the lambda for each other independent, plus the lambdas for all 2-way, 3-way, or higher interaction effects, according to the number of independents. Thus for two categorical variables, A and B, the saturated model is:
      Ln(F_ij) = + _i^A + _j^B + _ij^AB
      where i and j are the row and column numbers, _i^A is the main effect for variable A (the row effect), _j^B is the main effect for variable B (the column effect), and _ij^AB is the interaction effect of A with B. However, this is the saturated model which is always fully predictive of the table frequencies, but trivial. The trick is to figure out how many lambdas can be constrained to 0 and still have acceptable estimates of the frequencies.

      SPSS prints out all these lambda parameter estimates in the "Parameter Estimates" table of the output. Lambdas appear as coefficients in the "Estimates" column of this table. While some authors label these estimates as "b" coefficients, they are not regression-type coefficients. Lambdas are effect estimates, not slopes.

    • Predicted frequencies. Thus the predicted frequency of any cell is exp(mu + lambda_1...n), where mu is the constant, lambda_1...n is the sum of the lambdas for all effects in the model, and the exp() function returns the base of the natural logarithm (e = 2.71828182845904) raised to the power of the sum of mu plus the lambdas. Note the lambdas are added to the constant: they are not like b regression coefficients, which must be multiplied by their respective variable values for a case when making a prediction.

    • Effect sizes may be measured by odds, odds ratios, and logits, all of which are discussed below. In SPSS output, "Parameter estimates" are effect sizes, discussed below. They may be expressed as unstandardized or standardized b coefficients. Standardized parameter estimates can be used to see just which values of which variables in the model are most or least important to the interactions in the given parsimonious model. The more positive (if significant) the parameter estimate for an effect, the more cases are predicted to be in a cell over and beyond those predicted by the constant and other effects. The more negative (if significant), the fewer cases are predicted. If the parameter estimate is non-significant, the effect is not associated with any change in cell frequencies which are predicted by the constant and other effects. In SPSS, parameter estimates are obtained simply by checking the Estimates box under the Options button in the general loglinear analysis procedure (Analyze, Loglinear, General).

      How to interpret a log-linear parameter estimate, b, using odds ratios. The odds ratio, discussed below, is Exp(b). That is, the odds ratio is the natural log base e raised to the power of b. There will be a b coefficient for each category of the categorical variable, except the reference category. The odds of a person in the given category of an independent variable (the category corresponding to b) also being associated with the reference category of the dependent variable (usually 1, corresponding to an event happening; or to the highest dependent category when the dependent has > 2 values; though the researcher can set the dependent reference category as desired) is Exp(b) times the odds of a person in the reference category of the independent variable, controlling for other variables in the model. Also, when the independent increases one unit, the odds of the dependent (usually 1 = event happening) increase by a factor of x.

      Consider an example looking at the effects of gender, party, and race upon one another. In this model, for the most parsimonious model (Design: Constant + Gender * Party + Race * Party), the Parameter Estimates table looks like that below. Note that because output is from Analyze, Loglinear, General, regression-type indicator (dummy) coding is used, where the last category becomes the left-out category. (The differences between indicator coding in General Loglinear Regression and deviance coding in Hierarchical Loglinear Regression are discussed later on.):

      
      

      For the data above, Party is coded 1=Democrat, 2=Independent, 3=Republican. Gender is coded 1=Male, 2=Female.Race is coded 1=White, 2=Hispanic, 3=Black.

      Here it can be seen that the (Party=3)*(Race=2) interaction, which corresponds to Republican Hispanics, is not significant. All of the other combinations of interacting values are significantly contributing to the explanation of the distribution of data in the table. The highest significant Z value for this example is 8.38, for Female Democrats. If we were to go back to the original cell counts and compare the expected cell counts, we would find that the sum of absolute residuals (observed minus expected) for Republican Hispanics was only 3.5, whereas the corresponding residual sum for Female Democrats was 38.5 - far larger.

      Thus parameter estimates can be used to explore the relative importance of the independent variables. The ratio of the absolute magnitudes of the standardized parameter estimates (labeled 'Z' in the Parameter Estimates table) for any two cells reflects the relative importance of those parameters in explaining the frequencies in the table. Standardized parameters are parameters divided by their standard errors and are shown in the Z column in the SPSS output table above.

      In the table above, the two-way effect [Party=1]*[Gender=1] has a parameter estimate of 1.329; e^1.329 = exp(1.329) = 3.777, which is the odds ratio. The odds ratio is also a measure of effect size, in this case for the Male*Democrat effect. Standardized parameters are preferred for comparing effects, however.

    • Reference categories. The output from Analyze, Loglinear, General, will show which values have been aliased as reference categories. For any given one-way effect, there will be a parameter estimate for each of its values except an omitted ("aliased") reference category (by default, the last category). The computed parameter estimates in log-linear analysis reflect the effect of moving from the reference category to the given category. Thus Gender with two values will have one parameter. Party, with three values (Democrat, Independent, Republican) will have two parameters. The model in the table above had no one-way effects, but if we consider the model Design: Constant + Gender + Party +Race * Party, the table looks like this, showing the one-way effects of Gender and Party:
      

    • Comparing effect sizes. One-way effects with large standardized parameters ( the Z column) flag the variable values which were most important in explaining the distribution of the data. For one-way effects with three levels or more, the ratio of standardized lambdas will indicate which values contributed the most to deviation from the expected (flat) distribution for that one-way effect (that variable). In this example, for instance, where Party = 1 for Democrat, 2 for Independent, and 3 for Republican, Republican is the omitted reference category, and being Democrat is roughly twice as important and in the opposite direction from being Independent, in explaining deviation from a flat distribution of Party.

      Two-way effects with large standardized parameters flag the most important two-way interactions, and so on for higher-way effects. For any given two-way effect, there will be a parameter estimate for each cell and the ratio of standardized lambdas will indicate which cells contributed the most to that effect. Higher-way effects are interpreted analogously to two-way effects.

  • Probability is the chance an event will occur. In a population with 60 Democratic males and 30 Republican males, the probability of a male Democrat, p(D) = 60/90 = .667. Likewise, p(R) = 1 - p(D) = .333. The probability a woman is a Democrat is 45/90 = .5.
    

  • Odds is the ratio of the probability an event occurs divided by the probability of the corresponding non-event. In the context of log-linear analysis, odds refers to the probability of a case being in a particular cell of a table divided by the probability of it not being in that cell. Thus for the event "being Democratic" for males in the example above, odds(D) = .667/.333 = 2:1 = 2. This means males are twice as likely to experience the event "being Democratic" as not to experience it. If in the same population there are an equal number of women and half of women are Democrat and half Republican, then for females, odds(D) = .5/.5 = 1:1 = 1. When odds are 1.0 for a group, a person in that group (here, women) is equally likely to experience the event as to experience the nonevent (that is, be D vs. being R). When odds are greater than 1.0, the event is more likely to happen than not for a person in the group being considered. For conversion, odds(event) = P(event)/(1 - P(event)), where P(event) is the probability of the event occurring. Therefore odds(nonevent) = P(nonevent)/P(event), where P(nonevent)= 1 - P(event). Note also, P(event) = odds(event)/(1 + odds(event)).

  • Odds Ratios. An odds ratio is the ratio of two odds. In the example above, for the event D (being Democratic), the odds ratio of Party for males = odds(D) for males/odds(D) for females = 2/1 = 2. Men have twice the odds of being Democratic as do women.
    

    For further discussion of the table above, see the section on relative risk and odds ratios as measures of association. (In SPSS, the table is obtained by selecting Analyze, Descriptive Statistics, Crosstabs, and then checking Risk under Statistics.)

    • An odds ratio of 1.0 means the independent has no effect on the dependent. That is, an odds ratio of 1.0 indicates the two variables are statistically independent.

    • An odds ratio above 1 indicates that the independent increases the likelihood of the event. That is, a unit increase in the independent variable is associated with an increase in the odds that the dependent equals 1 in binomial logistic regression, or being the highest value in the case of multinomial logistic regression. Put another way, an odds ratio > 1.0 means increasing the independent variable increases the odds that the dependent variable equals a given value (usually 1) by a factor of the odds ratio (ex., a factor of 2 might mean the odds increase from 1:1 to 2:1). The larger the positive difference between the observed odds ratio and 1.0, the stronger the relationship. In this example, the odds ratio of 1.333 indicates a moderate relationship.

    • An odds ratio below 1 indicates a that the independent decreases the likelihood of the event. That is, a unit change in the independent variable is associated with a decrease in the odds of the dependent being 1 in binomial logistic regression, or being the highest value in the case of multinomial logistic regression. Thus, an odds ratio < 1.0 signifies the odds the dependent variable equals a given value decreased by a factor of the odds ratio (ex., a factor of -2 might mean the odds decrease from 2:1 to 1:1).

    • Example. To take another example, given by SPSS senior statistician David Nichols, if in a drug example one group is given a drug and another group a placebo, and then all patients are assessed for recovery, the odds ratio measures the increase (or decrease) in odds of recovery for patients given the active drug relative to those given the placebo. An odds ratio of 1 represents no effect, while a ratio greater than 1 indicates that the drug increases the odds of recovery and a ratio less than 1 indicates that it dimishes the odds of recovery compared to the placebo group. See further discussion of odds ratios in the section on the relative risk coefficient.

  • Confidence interval on the odds ratio. If the 95% confidence interval on the odds ratio includes the value of 1.0, by convention the variable is not considered a useful predictor variable. When the odds ratio is 1, then a change in value of the independent variable is not associated with a change in the odds that the dependent variable equals a given value. Note in SPSS this is referenced as the confidence interval of Exp(B), where Exp(B) is the odds ratio. Significance testing of odds ratios is discussed separately.

  • Nice properties of odds ratios as measures of strength of relationship. Odds ratios can handle categorical data, not requiring interval data. Odds ratios also do not require variables be normally distributed or that relationships between variables be homoscedastic (the same along the entire length of the relationship). That is, using odds ratios as measures of strength would involve much more relaxed data assumptions than using, say, regression beta weights.

  • Odds ratios, MLE, and OLS. Regression coefficients are computed using an ordinary least squares (OLS) methodology, which minimizes the sum of squared distances of data points to the regression estimate. Logistic regression is among the techniques which compute coefficients using maximum likelihood estimation (MLE), meaning coefficients are calculated which maximize the odds that a dependent variable equals a given value (ex, that party = D). MLE estimates benefit from the relaxed data assumptions mentioned above and are now more popular where appropriate. In the case of multivariate logistic regression, the odds ratios are interpreted the same but one must add the caveat "when all other independent variables are held constant."

  • Delta. If you are counter-checking SPSS results with those from hand computation or another source, keep in mind that by default, SPSS adds .5 to all cell counts to avoid having any cells with 0 count. To obtain comparable results, you may need to suppress this by setting this "delta" factor to 0, which is done under Options, Criteria, in the log-linear dialog.

• Partial chi-square is also computed in SPSS hierarchical log linear (HILOG, not GENLOG). Partial chi-square can be used to further refine model parsimony. It may be, for instance, that not all 2-way effects need be modeled even though the likelihood ratio test showed 2-way effects should be retained. Two-way effects with non-significant partial chi-squares in the "Tests of Partial Associations" table may be dropped from the model. Note that because the likelihood ratio test takes precedence, higher-order effects with significant partial chi-squares are not included when these higher-order effects fail the likelihood ratio test (that is, if a three-way effect, for instance, has a significant partial association, it will still not be included in the parsimonious model if the likelihood ratio test found that 3-way associations were not significant). Note that lower-order effects which are part of a significant higher-order effect must be retained even if they fail the partial chi-square test (ex., a first-order effect such as Gender might have a non-significant partial chi-square but would be retained if the Gender*Role second-order effect was significant).

Hierarchical loglinear analysis

Hierarchical loglinear analysis (HILOGLINEAR, called HILOG in SPSS) is often used as a prelude to general loglinear analysis (GENLOG in SPSS). Hierarchical loglinear analysis supports a "Use backward elimination" stepwise procedure in SPSS which can be used for exploratory purposes to screen alternative models automatically. SPSS looks the highest-order interaction effect(s) needed in the model, then includes all hierarchically lower effects. For instance, if for three variables a 3-way effect is to be included, then all 2-way and 1-way effects will be included also. The output options differ from general loglinear analysis discussed below but include the same observed, expected, and residual frequencies, as well as the usual likelihood ratio and Pearson chi-square goodness of fit tests. Note that HILOG produces parameter estimates only for saturated models and there is no output of correlation or covariance matrices for estimates. It is common to select automatically a best model in HILOG, then refine this model in GENLOG to obtain parameter estimates for the best parsimonious model. HILOG models are constrained to be hierarchical, including all lower interaction and main effects implied by a higher order interaction, but GENLOG models can be more parsimonious because there is no hierarchical constraint and it is possible to omit some of the lower-order terms even though a higher-order term is specified in the model.
• SPSS. Hierarchical loglinear analysis is invoked in SPSS by selecting Analyze, Loglinear, Model Selection from the menus. After the factors are entered (ex., Race, Party, Gender) one must click the Define Range button to specify, for instance, that Party ranges from 0 to 2 (0=Democrat, 1=Independent, 2=Republican). Based on the factors the researcher enters, HILOG automatically creates the highest generating class of interactions and all subsidiary ones (ex., Race*Party*Gender and all subsidary 2-way interactions and the three main effects). To ask for partial chi-squares, click the Options button and check the "Association Table" box (note this is available only when running the saturated model, not when you have clicked Custom and are asking for a parsimonious model).

  The "Cell Counts and Residuals" table below is output from HILOG for the party-race-sex data discussed previously above. Delta is set to 0 (the SPSS default adding .5 to all cells is overridden). As can be seen, the saturated model explains cell frequencies perfectly, with 0 residuals.

  

• Generating class refers to the hierarchical nature of HILOG models, which automatically generate a saturated model with all possible effects. Specifically, the generating class is the set of highest-order terms in which factors appear. The hierarchical model contains the terms that define the generating class and all lower-order terms. Thus, for three factors, the generating class factor1*factor2*factor3 would cause SPSS to create a hierarchical model with this three-way interaction plus all subsidiary two-way interactions plus the three main effects of these three factors. For the example above, the highest generating class would be Gender*Race*Party.

• Parameter estimates are used by HILOG's algorithms to compute the expected cell frequencies implied by the model. As this is done automatically, in practice the "Parameter Estimates" table may well be largely ignored by the researcher. The parameter estimates which HILOG computes only for the saturated model, differ compared to those for the saturated model in GENLOG and must be interpreted differently (though if the researcher is using HILOG to select a best parsimonious model, there may be no reason to interpret parameter estimates anyway). In HILOG, estimates are deviation contrasts while in GENLOG they are based on indicator coding. Deviation contrasts are coefficients which estimate deviations of each category except the left-out category from the unweighted grand mean of all of the categories of a factor or interaction. The effect of deviation contrasts is to constrain parameter estimates so they must add to 0 for any given main or interaction effect.

  • Estimates and coding. HILOG output lists each main and interaction effect in the hierarchy of all effects generated by the highest-order interaction in the set of factors the researcher enters. For main effects, it will number the parameters 1....(p-1), where (p-1) is the number of categories minus the one left-out category. Warning: thus even if the variable Race is coded 0=White, 1 = Hispanic, 2= Black, Parameter 1 will be for White, 2 for Hispanic, and Black will be the left-out reference category. This not-printed parameter estimate for the left-out category is the negative of the sum of the printed parameter estimates (since the estimates must add to 0). Thus in the example below, the left-out category is Black, whose parameter is not shown but is -(.069 + .005) = -.074.
    

    In the "Parameter Estimates" table above, Gender has two categories while Race and Party have three. This is why the main effects for Gender, Race, and Party in the table above have 1, 2, and 2 parameters respectively, with the last in each case being the redundant reference category. The two-way interactions involving Gender thus have 1*2 = 2 parameters, while the Race*Party interaction has 2*2 = 4 parameters. The three way interaction has 1*2*2 = 4 parameters also.

  • Parameter numbering for interaction terms. For interaction effects, SPSS numbers the parameters starting with the first value of the first factor entered, then cycles the values of the second one (leaving out the reference categories), etc. First factors cycle slowest; later factors are cycled the fastest. Thus if factor1 has three categories and factor2 has two categories, there will be (3-1)*(2-1) = 2 parameters. Parameter 1 will be for the interaction factor1=1*factor2=1; parameter 2 will be for the interaction factor1=2*factor2=1. To compute the not-printed parameter estimates for the left-out categories, create a table of all possible estimates, then use the sum-to-zero rule to fill in the last row and last column. In the example above, Parameter 1 for Gender*Party = -.160, and this is the parameter for Gender=0*Party=0 (since the first category was coded 0 for each factor). Parameter 2 for Gender*Party = -.013, and this is the parameter for Gender=0*Party=1. The parameters for the remaining combinations of Gender*Party are redundant and not shown, but could be computed using the table method just described.

  • HILOG vs. GENLOG parameter coding. In contrast to HILOG's use of deviation coding, with the indicator coding used by GENLOG, the left-out category has a parameter of 0. Indicator coding is like dummy coding in linear regression, whereas deviation coding is like effects coding in linear regression. Re-run in GENLOG the best model selected under HILOG to get the more widely used indicator coding parameter estimates, which GENLOG will give for all models, not just the saturated model.

• Tests of k-way effects. The HILOG procedure outputs two tables of k-way tests: (1) Tests that K-way and higher order effects are zero; and (2) Tests that K-way effects are zero. Both rely on the significance of differences in the likelihood ratio (or of Pearson chi-square, which is also shown) when a set of terms of a given order are added to the model. "Order" is 1 for main effects (for ex., factor1, factor2), 2 for two-way or 2nd order effects(ex., factor1*factor2), etc. These tables give an initial idea of what order(s) of effects are or are not appropriate for the most parsimonious model.

  

  The "Tests that K-way and higher order effects are zero" table, illustrated above, shows the value of adding effects of a given order or higher to the model. The table will have rows for K= 1 up to p, where p is the highest order possible for the data at hand (in this example, 3, since there are 3 factors - Gender, Race, Party). If the "Sig" significance level for the K = 3 row is non-significant, as it is above, then the researcher would conclude 3-way interaction terms should not be in the model. If the "Sig" for the second row, which is K = 2 for this example, were non-significant, then the researcher would conclude neither 2-way nor 3-way terms should be in the model. However, since K = 2 is significant above, the researcher fails to reject that null hypothesis. Both likelihood ratio and Pearson chi-square tests of significance are available, but the former are generally preferred. In this example it makes no difference, which is usually the case.

  The "K-Way Effects" table is the lower half of the same table in SPSS output, as shown above. This tests if specific K-way effects are zero. The table shows the value of adding main, two-way, three-way, fourth-order, or higher effects to the model. The table will have rows for K=1 to p, where p is the highest order for the data at hand. The probability column ("Prob." ) for the likelihood ratio ("L. R. Chisq") shows the significance of adding the corresponding order of effects. For instance, if row 3 is non-significant, then adding 3rd-order effects (3-way interactions) to the model is rejected, as it would be in the example above. In the example, adding main and 2nd order effects in the model is warranted..

  In hierarchical models, if one has a higher-order term, one must have subsidiary lower ones. If one dropped a 3rd-order term, one could not retain a 4th-order term containing one of the elements of the 3rd-order term. For this reason, the "Tests that K-way and higher order effects are zero" table is the more relevant to modeling using HILOG.

• Tests of Partial Associations. Tests of partial association (aka, partial chi-square) are partial likelihood ratio (or partial Pearson chi-square) tests and are based on the difference in likelihood ratio (or Pearsonian) chi-square for the model with and without a given term. The "Tests of Partial Association" table in HILOG output is used to test individual terms in the model. Even though the k-way tests may suggest dropping a whole class of terms (ex., all 3rd-order terms), it may be that one or more individual terms in that class are significant. Likewise, the k-way test may suggest retaining a class of terms, yet one or more individual items in that class may be nonsignificant. Keep in mind, though, that regardless of the partial association test, one must retain even non-significant lower-order terms if they are components of a significant higher-order term which is to be retained in the hierarchical model. Thus in the example below, one would retain Gender and Race even though they are non-significant because they are terms in the two significant two-way interactions, Gender*Party and Race*Party. Thus the partial associations test suggest dropping only the Gender*Race interaction.
  

• Backward elimination. If "Use backward elimination" is checked (it is the default), the hierarchical loglinear procedure will automatically screen all possible models in a generating class hierarchy for the most parsimonious one. Starting with the saturation model is not required. The researcher can still enter a custom model using the Model button. Whatever model the researcher starts with, the backward elimination algorithm will drop the least useful term one step at a time. "Least useful" is operationalized as the term whose removal has the least effect on lowering the likelihood ratio chi-square (recall lower = more toward significance = bad fit).
  

  In the example above, the default of starting with the saturation model was take. Thus Step 0 is for Gender*Race*Party and all hierarchically subsidiary 2nd and 1st order terms. In Step 0, the backward elimination algorithm tests to see if the highest order (here, 3rd order) term may be dropped from the model as non-significant. At Sig. = .953, it is indeed non-significant and is dropped, leading to Step 1. Step 1 is the model with all 2nd order (two-way) terms and subsidiary 1st order terms. Since here three factors corresponds to three two-way interactions, each of the three is tested for possible dropping. It is found that Gender*Race is non-significant and may be dropped, but the other two 2nd order terms should be retained. In Step 2, Gender*Race is dropped and the remaining two 2nd order interactions are used as the generating class. This time no terms are found suitable for dropping (non are found to be non-significant). Step 3, the final step, merely lists the generating class for the most parsimonious hierarchical model.

  How it works: the backward elimination option calculates partial chi-square for every term in the generating class. Backward elimination deletes any term with a zero partial chi-square , then it sees which effect has the largest significance of change in chi-square if it is deleted (the default alpha significance level is .05). This gives a new model and a new generating class, which is tested in turn. The process continues until there is no significant gain in deleting further terms.

  In the final step output under backward elimination, SPSS will print the likelihood ratio chi square and its significance for the model as a whole. A non-significant likelihood ratio indicates a good fit, as is the case in this example. Keep in mind that in a hierarchical model, a higher-order term like factor2*factor3*factor4 includes subsidiary 2-way and 1-way effects such as factor2*factor3. If when the researcher goes back to GENLOG to enter a custom model, the researcher would enter the hierarchically-implied terms as well as the actual "final model" terms listed in the HILOG output. Of course, backward elimination does not guarantee the most parsimonious well-fitting model - researcher experimentation may still be called for. If one enters the example data into GENLOG (general loglinear modeling, discussed below) and asks for the best model emerging from HILOG (the "Model Selection" option in SPSS), one will get the goodness-of-fit table below, which has the same likelihood ratio goodness of fit as shown in the backward elimination table in HILOG (Sig.=.964, where non-significance corresponds to a well-fitting model). For more on computation of Pearson and likelihood ratio chi-square, click here.

  

General Loglinear Modeling

General loglinear modeling is the usual procedure to select when the researcher wishes to use loglinear analysis on categorical variables and wishes to test a model (common models are discussed below) determined in advance (ex., determined by hierarichal loglinear analysis) or wishes to search manually among a finite set of models to determine the most parsimonious one. There are three general approaches:
1. Theory. Ideally, theory and the research literature would lead the researcher to the specification of which effects should be included in the model to be tested.
2. Backward elimination. The researcher starts with the saturated model, which is always 100% well fitting, and deletes terms one at a time, starting with higher order interaction terms, until the likelihood ratio test shows the nested model is significantly different from the saturated one. If the researcher has already used hierarchical loglinear analysis, with its built-in backward elimination process, to arrive at a hierarchical parsimonious model, then in general loglinear analysis the researcher drops terms to see if there is an even more parsimonious non-hierarchical model.
3. Forward aggregation. Though uncommon, it is also possible for the researcher to start with the independence model (only main effects, no interaction effects) and add terms one at a time until the model is not significantly different from the saturated one by the likelihood ratio test.

In SPSS, select Analyze, Loglinear, General to select the GENLOG procedure, illustrated below. In the General Loglinear Analysis dialog box, move all the categorical variables of interest (ex., gender, race, and party in the tabled example below) to the Factors box. Clicking OK enters the saturated model by default. (If you click the Models button, you will see that "Saturated model" is checked by default.) There is also an Options button where you may check Frequencies, Residuals, Estimates, Criteria, plots, and more. You will normally want to select at least Estimates, which also gives significance of the estimates for each effect. Under Options, Criteria, you can set Delta=0 to suppress the default under which .5 is added to all cells to avoid having cells with zero count. Set the data distribution assumption (see below), Click Continue. OK.


• Data distribution assumptions. Note tht at the bottom of the main GENLOG dialog you may select between Poisson and multinomial distribution assumptions. Both distributions are commonly used to account for frequencies in a table. Both usually yield the same parameter estimates, but the interpretation is different. Poisson is the default. A Poisson distribution is used if total sample size is not fixed and the cell frequencies are independent. This would happen, for instance, if the research design were to sample all cases of a given nature, or all cases within a data-gathering period of a given duration. In either case, the sample size n would not be known in advance. Assuming data independence also, Poisson distributions are appropriate for these designs, as well as for designs where the sample size is fixed but analysis is not conditional on sample size. A multinomial distribution, in contrast, is used if the total sample size is fixed and the cell frequencies are not independent. This would happen in research designs where the sample size is a predetermined n, and the sample is stratified by row and/or column, meaning row and/or column totals are predetermined and sum to n, which in turn means cell frequencies are not independent. Note: When multinomial assumptions are selected, the constant (mu) in the loglinear equation is treated as a given and no confidence limits are computed. This is because the constant in a loglinear model is a function of the means, and the means are determined by the marginals, which are known.

• Covariates. In addition to moving categorical variables to the Factors box, interval-level variables can be added in the covariate box in the GENLOG dialog box. Covariates are usually conceptualized as confounding or control variables. For each cell in the table, SPSS automatically computes the mean values of the covariates and uses this mean value similar to a factor to model the cell counts in the table.
• Categorical Log-Linear Models. In exploratory log-linear analysis, one is looking for the most parsimonious model which can explain the distribution of observations in a table formed by categorical variables. However, in confirmatory log-linear analysis, one wishes to test a particular model based on theory. In this section various common possible types of log-linear models are discussed. The list of models is not comprehensive, as log-linear modeling supports testing an abundance of different types of models. To illustrate, consider the table above relating Race (0=white, 1=Hispanic, 2=Black) to Party (0=Democratic, 1=Independent, 2=Republican), controlling for Gender (0=Male, 1=Female):

  Looking at the significance of effects obtained by asking for Estimates under the Options button of the general loglinear dialog box, is a prime way of reducing the saturated model, eliminatiing non-significant effects.When dropping effects which are nonsignificant, it is best to drop one effect at a time to be sure lower-order non-significant effects don't become significant when a higher-order non-significant effect is dropped. When two or more effects are nonsignificant, start the reduction process by dropping the highest-order nonsignificant effect first, then proceed by dropping one term at a time on subsequent runs. To specify an unsaturated model, in the loglinear analysis dialog, click Model, Custom, and enter the effect terms you want (ex., race, gender, race*gender, highschool).

  1. The saturated model. This is obtained as described above simply by entering all categorical variables and clicking OK in the GENLOG dialog box. The saturated model, illustrated below, always explains the table's data distribution fully, though this is a trivial result. The saturation model is the baseline against which subsequent models are compared. The saturated model for this example is: Design: Constant + Gender + Race + Party + Gender * Race + Gender * Party + Race * Party + Gender * Race * Party. The likelihood ratio and the Pearson chi-square for the saturated model will be 0. It is a perfectly fitting model but "Sig." probability levels are not reported as there is always a 100% probability the saturation model will be well-fitting. For more on computation of Pearson and likelihood ratio chi-square, click here.
     

  2. The complete independence model. This model hypothesizes the factors are unrelated. That is, it hypothesizes that there are only main effects and no interaction effects. In this example, the hypothesis would be that race, party, and gender are all unrelated to each other. To create this and any model other than the saturated model, you must click the Model button in the General Loglinear Analysis dialog box, then click Custom. For the complete independence model, move all three factors (race, party, gender) to the Terms in Model box. Set Delta to 0. Click Continue, OK, to obtain output. The model is: Design: Constant + Gender + Race + Party. For these data, the likelihood ratio is 65.6 and the Pearson chi-square is 64.9. Both are significant at the .000 level. This means the complete independence model is significantly different from the saturated model. This further means that at least some of the terms in the saturated model beyond Gender + Race + Party are necessary to account for the distribution of data in the table. That is, at least one of these additional terms in the saturated model is necessary, but so far we do not know which: Gender * Race + Gender * Party + Race * Party + Gender * Race * Party.
     
     • Quasi-independence models. One can test to see if the independence model applies to a subset of cells in a table, ignoring other cells (ex., ignoring diagonal cells). This is discussed in the FAQ section below in the section on use of forced structural zeros.

  3. One factor independence models. These models hypothesize that one of the factors is unrelated to the others. For example, we could hypothesize that race and party are related, but gender is independent. Click the Model button in the General Loglinear Analysis dialog box, then click Custom. Move Gender over into the Terms in Model box. Then highlight both Race and Party in the Factors & Covariates box and then, with Interactions selected in the Build Terms box, click the right arrow to create the term Race*Party in the Terms in the Model Box. Continue, OK. The model is: Design: Constant + Gender + Race*Party. For the data in this example, the likelihood ratio has a probability of .040 and Pearson's chi-square has a probability of .042. This means that there is a significant difference between this particular one factor independence model and the saturated model, which means this model is not a good fit to the data.
     
     • There are two other one factor independence models for this example:
       Design: Constant + Race + Gender*Party, likehood ratio significance = .000; this model is also not a good fit to the data.
       Design: Constant + Party +Race*Gender, likehood ratio significance = .000; this model is also not a good fit to the data.

  4. Conditional independence models. These models hypothesize that only some of the two-way interaction effects are needed to account for the distribution of data in the table. For this example, there were three two-way interaction effects (Gender * Race + Gender * Party + Race * Party) in the saturated model. For instance, we might hypothesize that Gender*Party and Race*Party were important, but Gender*Race was not significant. This model is: Design: Constant + Gender * Party + Race * Party. In the modeling section, highlight all three variables, select All 2-way Interactions, click the right arrow, then delete the Gender*Race term, leaving the two others. Continue, OK. The significance level of the likelihood ratio for these data for this model is .964. This means this model is not significantly different from the saturated model in accounting for the distribution of data in the table. We accept this conditional independence model as a superior model to the saturated model because it is more parsimonious. Note, however, that other models may also be well-fitting.
     
     • There are two other one factor independence models for this example:
       Design: Constant + Gender * Race + Race * Party, likehood ratio significance = .013; this model is not a good fit to the data.
       Design: Constant + Gender * Party + Gender * Race, likehood ratio significance = .000; this model is also not a good fit to the data.

     • Comparison with the model generated by HILOG. Recall that the best model generated by the Model Selection procedure was the full factorial model minus the Gender*Race interaction for our example. When entered into the general loglinear procedure, this model generates an identical model fit as illustrated below. The HILOG best model is the hierarchically implied version of the conditional independence model. In GENLOG, the model fit is the same but the estimates of the parameters differ.
       

     • Parameter estimates for the best model. Looking at the significant parameter estimates, shown in red below, we can analyze the relative importance of different effects in the model. For these data, Party was coded 0, 1, 2 for Democratic, Independent, and Republican respectively. Race was coded 0, 1, 2 for White, Hispanic, Black respectively. Gender was coded 0 = Male, 1 = Female. Interpretation of parameter estimates tables is discussed above. The standardized parameter estimates, labeled Z below, show the relative importance of effects. Here the largest effect is for Race=2*Party=0, which is black Democrats. Thus we can say that the pattern of frequencies shown in the original table reveals the variables not to be independent of each other, but rather the interaction of Party with Race and Party with Gender both help explain why observed frequencies are different from what would be expected on the basis of the marginals, and the effect of black Democrats in particular is important to this difference.
       

  5. The homogenous association model. This model hypothesizes that all of the two-way interactions are needed to account for the data, leaving out all others. The model is: Design: Constant + Gender * Race + Gender * Party + Race * Party. In the modeling area, enter All 2-Way Interactions. The likehood ratio significance for the data in this example is .954, almost as high as for the conditional independence model. It is also a well-fitting model but would be rejected as being not as parsimonious as the conditional independence model above (Design: Constant + Gender * Party + Race * Party). In fact, since it was already known that a conditional independence model was acceptable and the one-factor gender*race model was not, the researcher probably would not have tested the homogenous association model.
     

• Conditional loglinear models.
  1. Symmetry models. A symmetry model controls for one of the main effects in a table. For instance, in a table formed by gender, party, and race, we might test to see if party and race have similar distributions (are symmetrical) by gender. The model is : Design = party race party*race, where gender, party, and race are in the factor list. Note that if gender were not in the factor list, we would have a perfectly-fitting saturation model of the 2-way party by race table. If the likelihood ratio goodness-of-fit is not significant, then the symmetry model is a good fit. That is, when we leave out gender as a main effect or any of its interactions with party or race, we can still do a good job of predicting the cell frequencies in the gender*party*race table. This is because when you do a good job predicting the party*race table for one gender, you've done it for the other gender, since the distributions are symmetrical. However, for these data the symmetry model is significant, which means it is not a good fit:
     

  2. Conditional symmetry models. Also called adjusted models of quasi-symmetry, these models test to see if the symmetry model is also conditional on a control variable. The model for our example is: Design= gender party race party*race, where gender, party, and race constitute the factor list. This uses gender as a control, which is to say that expected counts are constrained so it is as if there were an equal chance of being male or female. Then what is tested is whether in this constrained situation, it is equally probable that a case is in cell(i,,j) as in cell(j,i) - that is, if the distribution of party and race is symmetrical by gender, once frequencies are adjusted to equalize males and females. [Whereas the symmetry model tests if in the unconstrained, observed data, it is equally likely a case would fall in cell(i,j) or in cell(j,i), with no frequency adjustments for totals by gender]. For these data, the conditional symmetry model with gender as control is not a good fit since the likelihood ratio is significant.
     
     • Marginal homogeneity test. To test if marginals are the same among values of a control variable (gender in our example), subtract the goodness-of-fit chi-square for the conditional symmetry model from the chi-square for the symmetry model. Use degrees of freedom = (r-1) or (c-1), whichever is less, where r = number of rows and c= number of columns in the table without the control variable (in our example, the party*race table). If the difference is significant, then the researcher concludes the marginals are not the same for the various categories of the control variable (lack of marginal homogeneity). For these data, the chi-square difference = 16.991 - 16.024 = .967, and df = 3 - 1 = 2. In a chi square table for df = 2 and alpha = .05, the cutoff value is 5.99. Therefore the difference is not significant, leading the researcher to conclude the marginals are the same for the categories of gender.

• Ordinal models. If the row and/or column variable consists of ordered categories (ex., ascending income groups or age cohorts, or simply Likert-type scales) then the category codes may be used to achieve more parsimonious models than if order information were not known.
  1. Linear-by-linear association models assume both the row and column variable are ordinal. Let B = the estimated regression coefficient for the product of the category codes for the row and column variables. In a linear-by-linear association model, Design: Constant + B + rowvariable + columnvariable. In SPSS, Transform, Compute and set B = rowvariable*columnvariable; then in Analyze, Loglinear, General, enter rowvariable and columnvariable as Factors and enter B as a Cell covariate; click Model, Custom, and specify a model with Main effects for rowvariable, columnvariable, B.

     Consider the following table, in which TestRank is used to predict WorkRank:

     

     The estimate of the B regression coefficient is shown in the "Parameter Estimates" table, B row, Estimate column.

     

     If the likelihood ratio (or Pearson chi-square) is nonsignificant, there is goodness-of-fit achieved simply by adding the B linear-by-linear association (interaction) effect to the complete independence model (which would be Design: Constant + rowvariable + columnvariable). For these data, a finding of significance means the linear-by-linear interaction terms should not be added to the model.

     

  2. Row-effects models assume only the column variable is ordinal. It is called "row effects" because it adds to the complete independence model the effects of the interaction of the row variable with the ordinal column variable treated as a covariate (rather than entered as a simple rowvariable*columnvariable 2-way interaction). Let C = a copy of the column variable, created in SPSS by Transform, Compute, C = columnvariable (this copy is needed because the column variable can't be both a Factor and a Covariate). In a row effects model, Design: Constant + rowvariable + columnvariable + rowvariable*C. In SPSS, Analyze, Loglinear, General, enter rowvariable and columnvariable as Factors and enter C as a Cell covariate; click Model, Custom, and specify a model with Main effects for rowvariable, columnvariable; and specify an interaction effect for rowvariable*C. The "Parameter Estimates" table will show the row and column main effects, and the effects of the interaction of the row variable with the column variable used as a covariate.
     

     If the likelihood ratio (or Pearson chi-square) is nonsignificant, there is goodness-of-fit. Here that is not the case.

     

  3. Column-effects models assume only the row variable is ordinal. These are identical to row-effects models, except the row variable is copied and used as the covariate.

  
  
  

Logit Loglinear Models/Logit Regression

Logit loglinear analysis pertains to a class of loglinear models which are used for analyses in which there are one or more categorical dependent variables (whereas model comparison (HILOG) and general loglinear (GENLOG) procedures discussed above are non-dependent procedures). Multinomial logit analysis simply refers to the case in which one or more variables may be modeled as dependents. Results may be identical to those from multinomial logistic regression, discussed separately.
• The table being analyzed is defined as the cross-classifications of all the factors and dependent variable as specified on the main "Logit Loglinear Analysis" dialog box, whether or not all factors are modeled (under the Model button). That is, a factor may not be used in a model but if it is listed in the main dialog, it will help define the table whose frequencies are being explained.

• Logit models model the proportions of cases (usually people) in each category of the dependent(s) for each category of the independent(s). For instance, in a study of race causing party id, logit models would model the ratio of one party to another for each category of race. However, there could be multiple dependent variables and/or multiple independent variables. Logit regression can be seen as a subclass of general loglinear models, where the MODEL parameter is set to "multinomial" (rather than the usual Poisson) and only the main effect of the dependent(s) plus their interactions with independent(s) are modeled, as discussed below. Logit models normally do not include a constant as the constant cancels out in computation, though SPSS will print estimates for the constant, warning in a footnote that "Constants are not parameters under the multinomial assumption."

• Logit models under GENLOG. In SPSS, when one selects Analyze, Loglinear, Logit, one gets the GENLOG module. However, when invoked under the Logit option, GENLOG operates differently than if selected by Analyze, Loglinear, General:
  
  1. There is a BY statement in the factor list, with the dependent(s) before and the independent(s) after the BY.
  2. The MODEL will default to MULTINOMIAL.
  3. DESIGN will be of the format described below, which includes only estimates for effects involving the dependent variable(s). It does not include main effects for the independent(s) or interactions solely among the independent and/or covariate variables.
  4. Though SPSS will list a constant in the design and will compute constants, a footnote reminds the researcher that "Constants are not parameters under the multinomial assumption. " The Model button in SPSS contains a checkbox for "Include constant for dependent" to cause constants to be computed and output. For custom models, the researcher can suppress computation of the constant.

  The DESIGN takes these forms for the saturated model:
  

  1. One predictor x1 of the dependent y: DESIGN = y + y*x. This was the design for the output illustrated above for Party predicted by Income.
  2. Two predictors x1 and x2 of y: DESIGN = y + y*x1 + y*x2 + y*x1*x2
  3. When there are two or more dependents, the logit model can become complex, but the design still is limited to terms involving the dependent variables and does not include main effects for the independent variables nor interactions of just the independent variables. Logit loglinear design for two dependents and two independents, for the saturation model: dependent1 + dependent2 + dependent1 * dependent2 + dependent1 * independent1 + dependent1 * independent2 + dependent2 * independent1 + dependent2 * independent2 + dependent1 * dependent2 * independent1 + dependent1 * dependent2 * independent2 + dependent1 * independent1 * independent2 + dependent2 * independent1 * independent2 + dependent1 * dependent2 * independent1 * independent2.

  For unsaturated models, obtained under the Custom choice under the Model button, the design will include the main effect of the dependent plus the effect of the dependent interacting with whatever terms ar listed. If x1 is listed, the design will be y + y*x1. If x1 and x2 are listed, the design will be y + y*x1 +y*x2. Etc.

  

  Thus in the example above, Gender and Race are used to predict Party. The model is not saturated since Gender*Race is not modeled. The logit model thus includes Party (the main effect of the dependent) plus Party*Gender (the dependent's interaction with the first factor) plus Party*Race (the dependent's interaction with the second factor). Because the model is not saturated, it is possible for residuals to differ from 0 and for the goodness of fit to be computed. Because model fit is non-significant in this example, the model is considered well-fitting.

• Logits (parameter estimates) in logit loglinear models. Logit models deal with the odds that a person in a given cell has the dependent = the reference category (by default, the highest category). The parameter estimate for y=1, x=1 is the logit for that cell (the first cell in the table, in the upper left). The base of the natural log e to the power of the logit [exp(logit)] expresses the effect relative to the reference category of the dependent. Consider the following table (different from the example above) in which high/low Income causes Party = Republican/Democratic affiliation.

  	
  _____________________________________________________________________________							
  		Income							
  		Low=0	High=1		odds	odds ratio	ln(ratio)		
  Party	R = 0	400	500		0.667			-0.405		
  	D = 1	600	400		1.25			0.223			
  						0.533		-0.629	
               .223 = parameter estimate for party=0
  	-.629 = parameter estimate for party=0*income=0	
  _____________________________________________________________________________
  

  * The odds of being Republican compared to Democrat for low income people is 400/600 = .667

  * The odds of being Republican compared to Democrat for high income people is 500/400 = 1.25

  * Since Democrats is the reference category, we can take the log of the odds to get the parameter estimate using the reference row Party = 1 = Democrat. It is ln(1.25) = .223. In the Parameter Estimates table this will be listed as the estimate for Party = 0 (Republican). The estimate for Party = 1 (Democrat) will be 0, since it is the reference category.

  

  * The odds ratio is the ratio of the odds of being Republican for low income people (.667) to the odds of being Republican for high income people (1.25). In this example it is .667/1.25 = .5333. The log of the odds ratio, ln(.5333)= -.629 is the parameter estimate for the interaction of the independent and dependent. Specifically, it is the parameter estimate for party=0*income=0 in the output above.

  * The odds ratio, .5333, is easier to put into a sentence than is the corresponding parameter estimate of -.629. We can say that the odds of being a Republican if low income is .53 times the odds of being a Republican if high income, for the data in this example. Thus, the odds ratio of .533 = .667/1.25, which is the ratio of the two odds in the table above.

  * Tip: If replicating this in SPSS, set Delta=0 so as not to add .5 to each cell.

  • Positive vs. negative parameter estimates. When we use the exp() function to convert the parameter estimate back into an odds ratio, and the parameter estimate is negative, the odds ratio is the fraction (in this case .53) that the given cell is in relation to the reference cell for that category of the independent variable. That is, significant negative parameter estimates correspond to a decrease in the odds of being in the given category of the dependent variable compared to the the odds of being in that category for people who are in the reference category of the independent factor. For example, in an earlier example where the parameter estimate was -.629, we converted that to an odds ratio of .533 and said therefore the odds of being a Republican if low income was .53 times the odds of being a Republican if high income). If the parameter estimate is positive, the odds ratio will be the multiple that the given cell is in relation to the reference cell on that value of the independent variable. That is, parameter estimates with positive coefficients increase the likelihood of the given response category.

  • Reporting odds ratios. The general format of the statement based on the odds ratio is: the odds of someone in the given category of the independent(s) being in the given category of the dependent(s) compared to being in the reference category of the dependent(s) is a multiple (if the parameter estimate is positive) or a fraction (if the parameter estimate is negative) of the odds for a person in the reference category of the independent(s) being in the given category of the dependent(s).

  • Logits in logistic regression. Logits are the natural log of the odds ratios. Logits take the form logit(p), where p = the probability that y=1 in binomial logistic regression or y=the highest value in multinomial logistic regression. A positive logit means the independent variable has the effect of increasing the odds that the dependent variable equals a given value (usually 1 for binary dependents, usually the last value for multinomial dependents). A negative logit means the independent variable has the effect of decreasing the odds that the dependent variable equals the given value. For instance, in logistic regression, an odds ratio of 10 means that when the independent variable increases one unit, the odds that the dependent = 1 increase by a factor of 10. (Tip: if you are experimenting using your calculator, note that natural log transformations are very sensitive to rounding off. You cannot go from an odds ratio to a logit, round off to three places, and raise e to the rounded logit and expect to get back the original odds ratio, for instance).
    Why logits are used when odds ratios are more intuitive. Logits are used in log-linear analysis in preference to odds ratios because of their mathematical properties. Odds ratios are asymmetric in interpretation: an odds ratio of 3.0 indicates the same difference in odds as an odds ratio of .33. This asymmetry disappears when one takes the natural log of the odds ratios. Thus LN(3) = +1.0986 and LN(.333333)=-1.0986. That is, odds ratios asymmetrically vary from 0 to 1 on the negative side and 1 to infinity on the positive side. Logits vary symmetrically from 0 to minus infinity on the negative side and from 0 to plus infinity on the positive side. Logits and odds ratios contain the same information, but this difference in mathematical properties makes logits better building blocks for logistic regression and log-linear analysis.

  • Multiple independent variables. To estimate the effect of a combination of two independent variables, let L_d be the parameter estimate (lambda) for the dependent category of interest of interest and let L₁ be the estimate for the interaction of the first independent variable's category of interest with this level of the dependent, and let L₂ be a similar estimate for a second independent in the model. To get the odds for the joint effect of the two independent variables each at a given category of interest, calculate exp(L_d+L₁+L₂). The resulting odds ratio is the odds that a person with the given values of the independent variables is in the dependent variable category of interest compared to the same odds for a person with the reference category values on the independent variables. To generalize, the terms within the exp() function are the terms for the model used to generate the estimates.

  • Polytomous independent variables. If income had 5 categories rather than two, the interpretation of the parameter estimate for party=0*income=0 would still be the same, but made with respect to people in the highest income category, which would be the reference category. If one wanted to compare income=0 people with income=1 people rather than with the income=5 reference category, one would simply subtract the parameter estimate for income=1 from the parameter estimate for income=0, then convert to an odds ratio using the exp() function, concluding that the odds of being a Republican (party=0) is x times the odds of being a Republican if income = 1, where x is the odds ratio.

  • Polytomous dependent variables.. Polytomous (polychotomous) logit models handle the case of a dependent with more than two categories. The dependent is either (1) the odds of being in one dependent category (usually the first) compared to another category as the reference category (usually the last); or (2) in the baseline category approach, the dependent is the odds of being in one category (usually the first) compared to all others as the reference category. The SPSS Logit procedure defaults to having the last dependent category as the reference category.

  • Covariates. Continuous variables can be used as independents by entering them into the Cell Covariate(s) box in the main Logit loglinear dialog. Covariates cannot be used to calculate a saturated model. Instead, click the Model button in SPSS and select Custom, then enter a nonsaturated model which includes the covariate(s). SPSS calculates the mean value of the covariate(s) for all observations in the cell formed by the factors (independents) and uses that in calculating estimates. For a dependent with 5 levels, for instance, SPSS will calculate dependent=1*covariate through dependent=4*covariate parameter estimate. The dependent=5*covariate parameter estimate will be 0 since it is the reference category. Note that whereas independents entered as factors have parameter estimates calculated on a case by case basis, independents entered as covariates have parameter estimates calculated based on grouped data (weighted means within cells formed by the factors). There is the possibility that by using means rather than case data, variance will be reduced and the effect sizes reflected in the parameter estimates may be attenuated for covariates compared to factors.

  • Confidence limits on parameter estimates: SPSS defaults to 95% confidence limites on estimates, but this can be changed by the researcher under the Options button.

• Measures of association. Entropy and concentration measures appear in the SPSS logit loglinear output in the "Analysis of Dispersion" and the "Measures of Association" tables, as illustrated below for the earlier example of Race and Gender predicting Party. The latter table is the one of primary interest. The "Analysis of Dispersion" table contains the data on which the "Measures of Association" table is based. Thus, entropy in the "Measures of Association" table equals model entropy divided by total entropy in the "Analysis of Dispersion" table, and similarly for concentration.
  

  Though sometimes described as being similar to R-square in regression, these effect size coefficients may be small even when the relation between the independent and dependent is strong. Each estimates the percent of the dispersion in the dependent variable which is explained by the model, and both coefficients are usually but not always close to one another.

  • Entropy association is model entropy dispersion divided by total (model + residual = total) entropy dispersion. Entropy dispersion is measured by Shannon's entropy coefficient:
    H = -SUM(p_ilogp_i), where p_i is the probability of the dependent in cell i.

  • Concentration association is model concentration dispersion divided by total concentration dispersion. Concentration dispersion is measured by Gini's concentration coefficient:
    C = 1 - SUM(p²_i).

Conditional logit models

Conditional logit regression is also called conditional logistic regression, fixed effects logit for panel data, or Clogit and is not to be confused with conditional models in general loglinear modeling, discussed above. It deals with matched pairs or panel data situations, or with data derived from an instrument in which people are asked to rank choices (ex., preferences among investment alternatives, candidates, etc.). First choice can be predicted on the basis of independent factors and covariates. In some software packages, the first choice can be excluded and the second choice analyzed similarly, and so on. Specialized conditional logit regression software. such as Latent Gold Choice, include many more options than does SPSS. Conditional logit models can also be fitted in Stata and in SAS.
• Matched pairs or panel data. The data setup for a conditional logistic regression on matched pairs requires that there be these columns:
  1. A variable containing the subject id number, which will be the same for the case and its control
  2. A dichotomous dependent variable (response variable), coded 1 for the case and 2 for the control.
  3. A copy of the dichotomous dependent variable
  4. One or more continuous explanatory variables (covariates) and/or
  5. One or more categorical explanatory variables (categorical covariates)

  Every subject id will have two data rows: one for the case and one for its control.

  In SPSS, select Analyze, Survival, Cox Regression. In the Cox Regression dialog, let the "Time" variable be the dichotomous dependent variable (ex., type - the row is the subject, coded 1, or is the control, coded 2. This means 1 is the "event" condition and 2 is the "censored" condition for each matched pair. Let the "Status" variable be a copy of the dependent variable (ex., type2) and then click the Define Event button and select Single Value and set it to 1. This tells the program that the value 1 corresponds to the event occurring, that is, being the case rather than the control. Enter continuous explanatory variables in the Covariates box (there is an option to do so in blocks). Click the Categorical button and enter any categorical explanatory variables as covariates, choosing the default Indicator contrasts. Back in the main Cox Regression dialog, let the "Strata" variable be a variable giving the subject's id number

• Choice models. When conditional logit regression is applied to choices rather than matched pairs, the logic is similar. The dependent is still a dichotomy, but now represents 1=chosen vs. 2=not chosen (censored) for a given alternative. A copy of the dependent is still the Status variable, and it is still set to 1 (assuming 1 = chosen). Each subject id will have two or more rows, for as many rows as there are choices. Continuous or categorical explanatory variables are entered as in matched pairs, and the subject id is entered as the stratification (Strata) variable. Indicator coding is still used.

• Output
  • The likelihood ratio test appears in the "Omnibus Tests of Model Coefficients" table, in the "Change from previous block" and "Change from previous step" columns. If the explanatory variables have been input using the "Enter" method (the default), these will be the same and a significant likelihood ratio change means the given model is significantly better than the baseline model with only the constant (which is the main effect of the response variable).

  • Parameter estimates for each explanatory variable in the "Variables in the Equation" table are associated with the interactions between the dependent (case vs. control, choice vs. non-choice) and the explanatory variables. The researcher concludes that variables whose significance (in the "Sig. column) is equal to or better than .05 are related to the dependent variable.

  • Exp(b) in the "Variables in the Equation" table gives the odds ratios associated with each explanatory variable.

Probit

• Probit.Probit models, while closely analogous to logit regression models, are not implemented under Analyze, Loglinear, in SPSS. In SPSS the following modules, discussed separately, implement probit:
  1. Probit and logit response models
  2. Ordinal regression
  3. Generalized linear models

  Probit regression is an alternative log-linear approach to handling categorical dependent variables. Its assumptions are consistent with having a categorical dependent variable assumed to be a proxy for a true underlying continuous normal distribution. A typical use of probit is to analyze dose-response data in medical studies. Like logit or logistic regression, the researcher focuses on a transformation of the probability that Y, the dependent, equals 1. Where the logit transformation is the natural log of the odds ratio, the function used in probit is the inverse of the standard normal cumulative distribution function. Where logistic regression is based on the assumption that the categorical dependent reflects an underlying qualitative variable and uses the binomial distribution, probit regression assumes the categorical dependent reflects an underlying quantitative variable and it uses the cumulative normal distribution. As with logit regression, there are oprobit (ordinal probit) and mprobit (multinomial probit) options.

  In practical terms, probit models usually come to the same conclusions as logistic regression and have the drawback that probit coefficients are more difficult to interpret (there is no equivalent to logistic regression's odds ratios as effect sizes in probit), hence they are less used, though the choice is largely one of personal preference. Both the cumulative standard normal curve used by probit as a transform and the logistic (log odds) curve used in logistic regression display an S-shaped curve. Though the probit curve is slightly steeper, differences are small. Because of its basis on the standard normal curve, probit is not recommended when there are many cases in one tail or the other of a distribution. An extended discussion of probit is found in Pampel (2000: 54-68).

• Analyze, Regression, Probit in SPSS leads to a subtype of probit regression designed for analysis of grouped dose-response data, as in medicine. However, it is possible to use it for more general purposes. In this procedure's dialog boxes, one must input:
  1. Covariate(s): There must be at least one covariate (a continuous independent, not to be represented by a set of dummy variables)
  2. Factor: Optionally, there can be one (and only one) categorical independent. Note that if you specify a factor, PROBIT includes it in the equation with a dummy variable for each level of the predictor and eliminates the intercept, so that the coefficient estimates you get are the predicted values for each level of the factor with the covariates set to 0.
  3. Response count variable: The dependent is the "response count" variable, which must be coded 0 or 1
  4. Total observed variable: This is simply a variable which has a value of 1 in all cases. This can be created by the COMPUTE statement. PROBIT uses this to read the response proportions as 0 out of 1 or 1 out of 1.

  Note that the chi-square test of goodness of fit cannot be used with PROBIT because it is based on an n by 2 table with one observation per row, which cannot approximate the chi-square distribution even for large samples.

• Cumulative normal distribution. The S-shaped curve which results when you add up the bell-shaped normal curve, moving from z = -infinity to z=+infinity. The cumulative normal distribution is used by probit and not by logit.

• Probit coefficients correspond to the b coefficients in regression or the logit coefficients in logit or logistic regression. All are effect coefficients. Logit and probit analysis generally arrive at the same conclusions for the same data, but the logit and probit coefficients differ in magnitude. Logit coefficients are about 1.8 times the corresponding probit coefficients.

  • Meaning. In the case of probit coefficients, the coefficient is how much difference a unit change in the independent makes in terms of the cumulative normal probability of the dependent variable. This means the probit coefficient measures the effect of the independent on the Z scores of the dependent. Note that the probability of the dependent is not a linear function of Z, but rather is a cumulative normal function of Z. This means that the effect of a unit change in the independent on the probability of the dependent depends on the level of the independents. Therefore to assess the effect of probit coefficients it is necessary choose some level of the independents as a reference point and in particular the standard reference point is when all independents are at their sample means.

  • Use. One substitutes the sample means of the independents into the probit regression equation to get the estimated Z score, then one looks in a table of the standard normal distribution to find the corresponding probability level. This gives us the baseline statement that when all variables are at their sample means, the probability that the dependent will have a value of 1 is that probability. This baseline can then be used to understand the effects of one unit change in an independent, given its probit coefficient value. The answer will be the probability when the independent is at its sample mean plus one unit, minus the baseline probability, with all other independents held constant at their sample means. That is, one takes the baseline equation and simply substitutes the mean of the independent plus 1, then calculates the probability. Then one subtracts the baseline probability from this probability.

  • Elasticity. This calculated probability difference is called the elasticity of P(Y) with respect to the independent, when variables are held at their sample means, where Y is the dependent. The elasticity is the effect of a unit increase in the independent variable on the probability that the dependent=1, when all other independents are held constant at their mean values.

Tobit

• Developed by Tobin in the 1950s, tobit is a nonparametric alternative to OLS regression or ANOVA, often used when variables have extreme skews and thus do not meet parametric assumptions. Tobin, for instance, used his procedure in consumer research where for certain items (variables) the bulk of consumers made no purchase. Substance abuse and crime pattern research often covers equally skewed variables. Tobit uses maximum likelihood estimation (MLE) to yield coefficient estimates of the probability of being above the modal value (usually 0 = no purchase, no substance abuse, etc.). SPSS does not support tobit (apart from manual computation in the SPSS MATRIX module). However, LimDep, an econometrics package does. SPSS does support logistic regression, also a MLE approach, which may achieve some of the same purposes. That is, one may regress a given outcome (ex., 1 = purchase, 1 = substance abuse, etc.) for a set of predictors using logistic regression to assess the effect of each predictor and the significance of their collective effect on the outcome.

Poisson regression

• Poisson regression is a form of analysis common in event history analysis and other research involving rare events where assumptions of a normally distributed dependent do not apply. It is also implemented in generalized linear modeling (GZLM), discussed separately. In the context of loglinear analysis in SPSS, selecting the Poisson distribution is an option under Analyze, Loglinear, General. In fact, it is the default assumption under HILOG and GENLOG.

• Rate variables as dependents in loglinear Poisson regression. In a simple loglinear model involving counts as a dependent variable (ex., suicides) and various nominal variables as predictors (ex., region, religion), data are weighted by the count variable (Data, Weight Cases, in SPSS). What differentiates a loglinear model as Poisson regression is that the researcher is interested in predicting a rate, not a count (ex., rate of suicides, not count of suicides). In a Poisson regression involving rates rather than counts as a dependent variable, an additional variable representing the denominator must be added as a Cell Structure variable in the Analyze, Loglinear, General dialog box. For instance, for a model of suicides in various sampled cities, the additional variable might be city population. Thus the normal data setup for this example would be region category as the first column, religion category as the second column, number of suicides (for that region-religion combination) as the third column, and population size (for that region-religion combination). Then in Data, Weight Cases, the count variable number of suicides would be the weighting variable; and region and religion would be entered in the loglinear model as factors and population would be the Cell structure variable in the Analyze, Loglinear, General dialog.

• Predictor variables in loglinear Poisson regressionContinous predictor variables may be added as covariates in either count or rate Poisson regression models. If there are only continuous predictors and no other factors, a subject id variable is added with values which are unique to each case. In the Analyze, Loglinear, General dialog box, the continuous variables are specified as Cell covariates and the subject id variable is specified as a Factor. Any 0 marginals will have to be recoded to a very small positive number (ex., 1E-12) or GENLOG will refuse to compute maximum likelihood estimates.

• Discussion. Like other forms of loglinear analysis, Poisson regression is predicting the count or rate associated with each cell in the table formed by the factors (region and religion in this example). For the example of predicting suicide rate for each table cell, SPSS will produce a table of "Parameter Estimates," with one estimate for each cell, and it will produce a constant. These are the mu and lambda coefficients discussed above in the section on general loglinear modeling. The last category of region and of religion will be the reference category (hence it would have been best if the researcher had set up the data with the desired reference category being the highest-coded). If the parameters are negative, then that cell has a fraction of the suicide rate of the reference category. For instance, if the parameter for region=1 is -2.24, since exp(-2.24)=.106, region 1 has 10.6% the suicide rate of the reference region. The greater in a negative direction the estimate, the smaller the fraction. A positive parameter would have meant the region had a risk which was a multiple of the reference group. The greater in a positive direction the parameter, the greater the multiple. By way of comparison, logit coefficients tend to be about 1.7 times larger than probit coefficients, unless an independent variable has a large standard error.

  If you ask for Predictions under the Save button, SPSS will add a column of predicted counts (labeled PRE_1 on the first run) to the working dataset. In the example, this would be the predicted count of suicides for the given cell. The predicted rate can be calculated based on the Poisson regression model:
  

  ln(predicted count) - ln(number at risk) = mu + sum of lambdas
  which is equivalent to:
  ln(predicted count/number at risk) = mu + sum of lambdas
  Which in turn is equivalent to saying:
  ln(predicted rate) = mu + sum of lambdas
  So the predicted rate is the antilog:
  predicted rate = e to the power of (mu + sum of lambdas)
  

  
  

Assumptions

• Not assumed: like logistic regression, loglinear and logit methods have relaxed data assumptions compared to OLS regression. There is no assumption that the dependents and independents be related linearly. There is no assumption of homoscedasticity. The dependent need not be normally distributed. The independents need not be interval in level of measurement, nor need they be unbounded.

• Well-populated tables. Log-linear analysis focuses on categorical variables, whose intersections are tables. Sparse cells in the table may well render log-linear analysis unreliable. If many cells have small expected values, both likelihood ratio and Pearson chi-square may mis-estimate the true significance. A rule of thumb is that expected frequencies should be >= 1 for all cells and no more than 20% of cells may have frequencies <5. Note this assumption pertains to expected frequencies, not observed frequencies (even observed frequencies of 0). When cell frequencies are small, SPSS's default addition of .5 to all cells (the delta factor) can appreciably alter results obtained. Expected frequencies are calculated as the product of row and column marginals divided by n, the sample size. As this assumption is violated, the power of the log-linear procedure diminishes (more likelihood of Type II errors, thinking there is no relationship when there is a relationship). Signficance tests, which deal with Type I error, are usually not affected by violation of this assumption. Collapsing rows and columns to create better-populated tables also decreases power.
  • Small models with few variables. A corollary is that large models with many variables will usually fail to have well-populated tables and thus will not be suitable for log-linear analysis.

  • Adequate sample size. Well-populated tables also require adequate sample size. Moreover, both logit and probit models utilize maximum likelihood estimation methods, which require a larger sample size than the corresponding OLS regression methods. A rule of thumb is that sample size should be a bare minimum of five times the number of cells in the tables being analyzed. Inadequate sample size will lead to too many cells with zero count and log-linear analysis may fail to converge on a solution.

  • Adding .5 to each cell is a default strategy taken by SPSS and others, to avoid having 0-count cells. However, this strategy improves Type I error only minimally while reducing power (that is, Type II errors are increased). The .5 default value is "Delta" under the Options button in SPSS, where the researcher may reset it to 0. Resetting to Delta = 0 is widely recommended.

• Small residuals (no large outlier cells). Outlier frequences are observed cell frequencies greatly different from expected frequencies. Outlier cells are flagged in log-linear residual analysis by having standardized residuals > 1.96, for the alpha=.05 significance level. Unusually large outliers may mean that no model can achieve goodness of fit. Sometimes adding variables or collapsing rows or columns can improve fit.

• Normally distributed residuals. As detected in Q-Q plots discussed above, non-normal distribution of residuals indicates the loglinear model is under- and/or over-estimating certain ranges of cases. Residual analysis helps the researcher identify outlier cells in the table being analyzed, and this in turn may help the researcher respecify the loglinear model.

• Categorical data. If continuous data are categorized to make them "work" in log-linear analysis, information is lost and correlations may be attenuated, yielding misleading results.

• Evenly distributed categories. Logit loglinear analysis assumes evenly distributed categories. If the categorical dependent variable is assumed to be a proxy for an underlying latent variable which is normally disributed, probit may be a more appropriate choice. In general, however, logit and probit yield very similar substantive results.

• Independence. Observations are assumed to be independent. Knowing case n does not predict case (n+1). Thus, log-linear analysis is not appropriate for correlated observations as found in time series, before-after, panel study, and other within-groups designs. However, conditional logit models exist which will handle correlated data.

• Data distribution assumptions. The researcher has the option in SPSS general log-linear modeling to assume a Poisson distribution of cell frequencies, or a multinomial distribution. The usual choice is Poisson distribution, which is used to predict rare events and cell frequencies when neither sample size nor row and/or column totals are predetermined in advance. Multinomial distribution is assumed when sample size and row and/or column totals are predetermined (sampling has been stratified, for instance, in a race by religion table, the numbers of white Protestants, black Catholics, etc., are predetermined). Thus there are four possibilities:
  1. Sample size is not fixed, nor are row/column marginals. Use the Poisson distribution.
  2. Sample size is fixed, but cell frequencies are independent. This is the simple multinomial distribution but if analysis is not conditional on total sample size (the usual case), the Poisson distribution is selected.
  3. Sample size and row or column marginals are fixed in advance. This is a product multinomial distribution, and the researcher should select multinomial distribution.
  4. Sample size and both row and column marginals are fixed in advance. This is a hypergeometric distribution, and the researcher should select multinomial distribution.
  Logit loglinear models assume a multinomial distribution of counts within each combination of categories of independent variables.

• Appropriate dispersion. Expected and observed variances of the dependent should be checked for under- and over-dispersion, as in logistic regression. However, in loglinear analysis the expected variance(y) equals simply the mean of y. As in logistic regression, if there is great under- or overdispersion, then there is a problem with model specification or other severe design problems. If the dispersion is moderate, one corrects by using adjusted standard error = SE * SQRT(D/df), where D is scaled deviance, which for loglinear analysis equals G² and df = the number of terms in the model not counting the constant.

• See logistic regression section for other assumptions.

Frequently Asked Questions

• Why not just use regression with dichotomous dependents?
  When the dependent is binary, the distribution of residual error is heteroscedastic, which violates one of the assumptions of regression analysis. Likewise, when the dependent variable is binary it is not normally distributed, so OLS estimates of the sum of squares are misleading and therefore significance tests and the standard error of regression are wrong. Also, for a dependent which assumes values of 0 and 1, the regression model will allow estimates below 0 and above 1. Also, multiple linear regression does not handle non-linear relationships unless power and/or interaction terms are explicitly added to the model, whereas log-linear methods can. These objections to the use of regression with dichotomous dependents apply to polytomous dependents also.

• Why not just use crosstabulation and ordinal measures of association rather than ordinal log-linear analysis?
  Ordinal log-linear analysis is superior to use of crosstabulation with ordinal measures of association because ordinal log-linear methods test models, not just assess strength of relationship (although ordinal association may still be useful as a summary measure).

• What computer packages implement log-linear analysis?
  SPSS's GENLOG, HILOGLINEAR, and Generalize Linear Model options; SAS's CATMOD option; Stata, GLIM, CDAS, ANOAS, and GAUSS are some packages which can be used for implementation of log-linear analysis.

• What are second-order and partial odds ratios?
  Second-order odds ratios are a form of conditional odds ratio, dealing with three-variable relationships. They are simply odds ratios for three-way tables. If the third variable is level of education (college, high school, less than h.s.), then the second-order odds ratios would be the three odds ratios for political party and gender for each education level subtable (that is, for each value of the control variable, educational level). The analysis of contingency tables using conditional odds ratios is a focus of Rudas (1998).

  Partial odds ratios. Partial odds ratios, like partial correlation coefficients for interval data, indicate the strength of a relationship when other variables are controlled. Put another way, partial odds ratios are a measure of main and interaction effects in a model. The partial odds ratio is the geometric mean of second-order odds ratios (odds ratios for conditional odds ratios on a third variable, such as odds ratios for men and women on being Democrats, for levels of education as a third variable). The partial odds ratio for education as a control variable would be the geometric mean of the simple (marginal) odds ratios for each the three levels of education.

  The partial odds ratio and the marginal odds ratio usually differ. If the simple or marginal odds ratio of Democrat as the dependent variable and female as the independent is 1.50, then a unit increase (switching from male=0 to female=1) is associated with a 50% (1 - 1.5) increase in the odds of being a Democrat. If the partial odds ratio turns out to be, say 1.25, then a unit increase (switching from male=0 to female=1) is associated with a 25% (1 - 1.25) increase in the odds of being a Democrat when education is controlled.

• What are structural zeros and sampling zeros in the SPSS "Data Information" table?
  Structural zeros, also called fixed zeros, refer to cells for which the expected frequency is 0. Sampling zeros refer to cells for which the observed frequency = 0 but the expected frequency > 0. Structural zero cells are not used in fitting the loglinear model in SPSS and most other packages. Structural zeros occur when a row and/or column marginal is zero. Some researchers collapse such columns and/or rows. Note that when row or column codes are used for ranking or other substantive purposes, collapsing will distort subsequent computation and is not recommended.

  Structural zeros may also occur when the Cell structure option is used to weight cells, and the If button is used to set the weighting variable to 0 under certain conditions. This might be done by a researcher who wanted to see if a loglinear model was a good fit not only on all the cells in a table, but also on the table ignoring some of the cells. The to-be-ignored cells are set to structural zeros using the Cell structure and If options, thereby forcing the creation of structural zeros. Then the loglinear analysis is run normally, but SPSS will not use the structural zero cells. When the to-be-ignored cells are the diagonal cells, the test of quasi-independence uses this method to see if the independence model (constant and main effects only, no higher effects) is a good fit (nonsignificant on the likelihood ratio).

• Since logit and probit generally lead to the same statistical conclusions, when is one better than the other?
  In principle, one should use logit if one assumes the categorical dependent reflects an underlying qualitative variable (hence logit uses the binomial distribution), and use probit if one assumes the dependent reflects an underlying quantitative variable (hence probit uses the cumulative normal distribution). In practice, these alternative assumptions rarely make a difference in the conclusions, which will be the same for both logit and probit under most circumstances. Prime among these circumstances is the fact that logit regression is better if there is a heavy concentration of cases in the tails of the distributions.

• Do I really need to do multinomial logit (multinomial logistic regression) or multinomial probit? Could I just apply M different logit or probit models for a variable with M levels?
  No, because the models are not independent and one will not have corrected for “experiment-wise” error.

• What if my variables are multiple-response type?
  Erroneous inferences will ensue if ordinary loglinear analysis is applied to multiple-response items. Necessary analytic modifications for multiple-response data are discussed by DeCarlo et al. (2000).

• Explain "partial odds.".
  Partial odds are the odds controlling for a third variable (ex., the odds of a woman being a Democrat, controlling for levels of education). Partial odds are calculated as the geometric mean of second-order odds. If there are three levels of education, there will be three conditional odds of a woman being a Democrat - one for each level - and their geometric mean will be the partial odds. Recall that the geometric mean is an average calculated as the nth root of the product of n numbers, and it is used rather than the arithmetic mean (which deals with adding and differences) because geometric means are the measure of central tendency for products and division, which is what odds ratios involve. In general, however, log-linear analysis focuses on partial odds ratios (and their natural log transforms, the partial logits), not partial odds.

• Explain coding in saturated vs. nonsaturated models.
  Recall log-linear and logistic models estimate main (single-variable) effect parameters (ex., one parameter for the set of table cells for Education=college). However, they can also estimate interaction effect p