Overview Evaluation research seeks to provide objective assessments of past, present, or proposed programs of action. Though seeking to be objective, most evaluation research is sponsored by actors in the environment of the program, whether superordinate organizations, auditing organizations, peer organizations, donor organizations, or others. The methodology of evaluation research may be managerial, oriented to providing management with needs assessments, impact studies, cost-benefit information, or critical path analysis (ex., CPM, PERT) for planning purposes; experimental or quasi-experimental, oriented toward identifying causal processes; or intersubjective, oriented toward providing ethnographic, phenomenological, or other subjectively understood information, often employing a "bottom up" or "grounded theory" approach, though participant observation may be used as well.. See also: Program theory.

Contents Key concepts and terms Cost-benefit analysis Return on investment Performance reference model Program assessment rating tool Needs assessments Feasibility studies Frequently asked questions Bibliography




Key Concepts and Terms

• The evaluation research question. There are three basic types of question in evaluation research - descriptive, normative, and impact.
  • Descriptive studies. The researcher describes the goals, objectives, start-up procedures, implementation processes, and anticipated outcomes of a program, presenting the details of each. Types of questions include the following.. Are the goals articulated? Are the goals communicated throughout the organization? Are the objectives formulated in a clear and precise manner? Is responsibility for objectives assigned to specific individuals. Is there a good plan for assessing progress on objectives? What is the form of accountability (ex., performance contracts)?

  • Normative studies. The researcher evaluates the program's goals and objectives by multiple values. Types of questions include the following. Are the goals minimalist, challenging, or unrealistically high? Are goals short or long term? Is the articulation of goals continous or sporadic? Are objectives behavioral or attitudinal?

  • Impact studies. The research evaluates program goals and objectives in terms of outcomes. Types of questions include the following. What are the short-term, direct impacts on each client or stakeholder in the program? What are the long-term impacts? What are the indirect costs and benefits?

• Pilot studies are "reality check" trial runs of a full-blown evaluation research effort. They are strongly recommended as a way to test assessment instruments and determine if one's data collection methodology will work as expected.

• Research design. There are four major types of evaluation design, not necessarily mutually exclusive: survey research, case study, field experiment, and secondary analysis of archival data.
  • Survey research. Survey research is used primarily around descriptive and normative questions. Surveys based on random samples are far superior to samples of available people and are the only type of survey supporting statistical significance testing. The leading types are cross-sectional (one point in time), panel (same sample at multiple time points), and cohort (same population at multiple time points) sample surveys. See the separate section on survey research.

  • Case study. Case study research is also is used primarily around descriptive and normative questions. While case studies can be quantitative, they are often used to gather qualitative information in support of findings obtained through survey research or other quantitative methods. See the separate section on case study research.

  • Field experiment Field experiments are used primarily around impact questions. It differs from other designs in that it requires a control or comparison group (though these are nice with other designs also). The three main types of field experiment are true experiments (subjects randomly assigned to treatment and control groups), nonequivalent comparison groups (subjects are not assigned randomly and so the control group is not equivalent), and before-and-after studies (sane subjects studied before and after treatment, no control group). See the separate section on research design..

  • Secondary data analysis Secondary data analysis is useful for all three types of evaluation question - descriptive, normative, and impact.

  
  
• Cost-benefit analysis may seem a straight-forward matter, but when measuring costs or benefits, a wide variety of questions arise. The problem of indirect benefits has its counterpart in the problem of intangible costs (ex., harm to agency reputation).
  • Questions Regarding Measurement of Benefits
    • Often there are different evaluative criteria for benefits (degree of consistency with the organization's mission, improvement in satisfaction with service, increased revenues, increased usage, and so on). What are the criteria to be? Are they to be weighted, such that some criteria are counted as more beneficial than others?
    • Should intangible benefits be "guess-timated" into dollar terms? Traditional financial models do not look at intangible benefits, but if they do, they convert to dollar terms.
    • Should dollar benefits be expressed in terms of net present value (NPV), such that the further in the future the a benefit of a given size, the less dollar value it has today? Traditional financial models use NPV.
    • How far into the future should the benefit stream be measured? Traditional financial models typically examine a short period such as 2 - 5 years.
    • Should "potential value" be included in benefits? Potential values are benefits which are contingent on future factors. If potential values are included, should they be discounted by multiplying by the probability of occurrence of the contingent factors? Traditional financial models do not look at potential values.
    • Should benefits be discounted by control factors representing other parallel causes of the benefit? If a benefit has multiple causes besides the IT initiative being evaluated, ideally a statistical path diagram linking causes to benefit effects could be constructed and the effect partitioned among paths leading from various causes. The path weight for the IT initiative as a cause could be used to discount the benefit (to establish the share of the benefit attributable to the IT initiative). In reality, causal models may be unknown and even where known, variables in the model may be intangible, unmeasured, or measured with error. Should discounting benefits for control factors be "guess-timated" when ideal statistical control is impossible to attain? Traditional financial models do not look at control factors.
    • Are benefits which are solely internal to the agency, such as employee satisfaction, "real" benefits which should be counted in return on investment? Traditional financial models do not look at internal benefits.
    • In view of the problem with objective measures of benefits, should subjective measures be used as, for example, whether managers, clients, experts, or some other group thinks the IT intervention has been beneficial, perhaps on a subjective scale from 1 to 10? Traditional financial models do not look at subjective measures.

  • Questions Regarding Measurement of Costs
    • Will measurement be in terms of total cost or marginal cost? For instance, an IT project might take no additional equipment and require no hiring of additional personnel, having a zero marginal cost, yet if machine cycles and personnel hours were costed proportionately, there might be considerable cost.
    • Will costs be discounted over time, as with applying net present value to benefits?
    • How much of agency infrastructure is to be allocated as cost to the IT project, and by what formula?
    • Are life-cycle maintenance costs included?
    • Are training costs included? Costs of upgrading personnel lines?
    • Is depreciation of facilities, equipment, or even software included?

    Life-cycle costing is an important aspect of cost evaluation. Ownership costs, operations and maintenance costs, and disposition costs can consume as much as 80% of a project's total life-cycle costs OMB, 1997: 79). Ownership and maintenance costs include such items as hardware upgrades, software upgrades, software modifications, and ongoing user training.

  
  

• Return on investment is perhaps the leading private sector approach to cost-benefit analysis.
  • ROI (return on investment) traditionally focuses on tangible benefits such as increased revenue, lower operational costs (ex., staff time savings), higher worker productivity, faster time to delivery, earlier problem detection, reduced postage or phone cost, reduced printing and paper cost, reduced travel costs, reduced consulting costs, reduction in redundant records, increased volume or traffic, faster and more reliable inventory, and increased external support. In the private-sector model, one begins by identifying market forces that may be opportunities for the firm, then proceeds to devise strategies to take advantage of these opportunities, managing investments and justifying them through quantitative performance indicators. Measures might involve such indicators as minutes per order fulfillment, order capacity per day per staff person, or orders fulfilled accurately the first time. As an example, return on investment might involve calculations such as Stoiber (1999) mentions: "..if the department currently processes 2,000 orders per day, the same department could process 2,600 orders using the new software. Assuming each order specialist earns roughly $40,000 annually...that translates into an annual cost savings of $240,000" for a staff of 20, not counting the savings of recruitment and training costs not incurred by not having to hire six additional staff to reach the higher capacity level. Using the net present value (NPV) function of any spreadsheet, figuring an initial investment cost of $500,000, a cost of money discount rate of 5%, and a three-year life for the software, the return on investment expressed as net present value would be $146,266. The software could cost as much as $653,590 and the IT project would still break even in net present value.

  • Intangible benefits. Private sector managers not infrequently choose not to measure intangible benefits when undertaking ROI analysis. Not only are intangible benefits difficult to measure, but even among tangible benefits there are problems because some, like faster time to delivery or increased traffic, do not necessarily directly affect revenue or costs. However, while it is plausible in many private sector settings to make decisions largely on the basis of direct tangible benefits such as the value of increased orders in the example above, doing so in the public sector can lead to poor choices. Intangible benefits can be as important as tangible ones, especially in the public sector. Examples of intangible benefits include client and citizen awareness, client and citizen satisfaction, policy consistency, upholding democratic accountability and transparency, staff morale, improving citizen choices regarding public health or the environment, improved information sharing, more non-hierarchical communication, higher organizational reputation, more effective targeting of services, enhancing organizational capacity for the future. Some intangible benefits can be measured, as with surveys of citizen satisfaction. But then how are survey satisfaction points to be compared to dollar costs to get return on investment? Moreover, the state of the art of social science is not adequate to model many intangible benefits. For instance, so many factors affect health choices of citizens besides public health agency educational projects that it is controversial at best to attribute such choices to specific agency investments.

  
  

• Performance Reference Model (PRM) is a public-sector framework for evaluation, developed by the Office of Management (OMB).
  PRM was released by the Office of Management and Budget in September, 2003, as a major framework for strategic planning. Acknowledging that performance metrics for the public sector cannot be directly borrowed from the private sector, the PRM called on agencies to use four types of evaluation measures:
  1. Mission and business results, particularly measures of conformity with Government Performance and Results Act (GPRA) strategic planning requirements.
  2. Customer results, particularly measures of accessibility and timeliness of services.
  3. Processes and activities, particularly measures dealing with finances, productivity, security, privacy, and innovation.
  4. Technology: Measures dealing with IT quality, efficiency, standardization, reliability, and effectiveness.
  Later PRM included human resources measures and fixed capital measures.

  For each type, the PRM provides a framework for categorizing operational evaluation measures. For instance, for the "Mission and business results" type, evaluation measures were broken into 31 subcategories, including Administrative Management, Community and Social Services, Controls and Oversight, Correctional Activities, Defense and National Security, Disaster Management, Economic Development, Financial Management, and more. Financial Management, in turn, was broken down into Accounting, Budget and Finance, Payments, Collections and Receivables, and Asset and Liability Management measurement categories. However, the measurement categories did not contain operational examples, only the injunction that "As agencies use the PRM for their specific IT initiatives they will create the inventory of operationalized indicators.

  
  

• Program Assessment Rating Tool (PART) is also a public-sector approach to evaluation developed by the Office of Management and Budget (OMB) to assess and improve program performance through a formal review process which was meant to identify a program's strengths and weaknesses.
  • PART reviews include assessment of program purpose and design, performance measurement, traditional evaluation, strategic planning documents, program management analyses, and reportage of program results over time. The intention is that by making PART reviews standardized, programs may be compared. Completing a PART review is considered by the OMB to constitute compliance with the Government Paperwork Reduction Act (GPRA). PART is a 25- to 30-part questionnaire, used by OMB starting in FY 2005. PART evaluation forms came in seven flavors, one for each type of federal program (direct federal programs like the National Weather Service, competitive grant programs, block/formula grant programs, regulatory based programs, capital asset and service acquisition programs, credit programs, and research and development programs). Most PART items appear in all seven forms. Several PART items dealt with the quality of a program's long-term and annual performance goals and measures, assessing whether they were meaningful, employed sound methodology, and could be verified with reliable data. OMB and agencies have to agree on appropriate measures early in PART reviews, involving relevant stakeholders, as required by the GPRA.

  
  

• Needs assessment serves to identify organizational problems (both internal and arising from the organization's environment), to investigate opportunities for change, and to document and defend projects which are proposed to address problems. Needs assessments assure that projects are directed to real problems faced by the organization rather than wasting scarce resources introducing innovations based on vendors' claims, "leading edge" novelties in the field, the practices of other jurisdictions which may differ in critical ways, or on other false premises. Properly implemented, a good needs assessment can re-energize an organization, providing hope that something fresh and constructive is being done about long-recognized shortcomings or problems faced by the organization. Needs assessments let participants specify the functions they want the solution to have and the needs the solution is supposed to meet. By creating awareness of needs, assessments help create the momentum for change. Needs assessments can be vehicles for involving stakeholders in the process of organizational transformation, particularly if implemented in a participatory way which gives members a feeling of ownership of what is being proposed.
  • Needs are not wants and conducting a needs assessment is not compiling a wish list. An agency may want to upgrade to wireless computing, for instance, but this is not a need unless it can be shown that wireless computing will allow the agency to achieve its objectives more effectively. It is not a priority need unless the gain in effectiveness is of greater value than gains from other proposed improvements. Even a priority need must be shown to be feasible in terms of the organizational resources identified in the needs assessment process. Where there is doubt about the capacity of the organization's financial, technical, and human resource ability to support a proposed solution, a feasibility study (discussed below) may need to be undertaken in parallel with the needs assessment process. An explicit financial analysis may need to accompany the needs assessment or feasibility study. The financial analysis is not an audit, though past audits may be consulted. Rather, a financial analysis is focused on gauging the organization's economic resources to undertake, implement, and maintain proposed changes associated with the needs assessment.

  • The mandate for needs assessment generally comes from the top management of the organization, who identify the general area of inquiry. Whether in-house or by consultants, the general needs assessment process is implemented in four stages:
    1. Collecting information
    2. Identifying and prioritizing problems
    3. Researching alternative possible solutions
    4. Seeking consensus on a proposed solution
    • Information collection. In the pre-assessment phase, a needs assessment committee and leader are established. Among the initial steps is to look at past needs assessments as well as the strategic plan. Past assessments help identify baseline data. Past assessments also help identify which data need to be updated. The strategic plan helps identify what information needs to be collected, if some data collected in the past is now unnecessary, and if new data is needed to throw light on new objectives. The information collection phase of needs assessment typically encompasses multiple methods, some of which are policy review of existing legislation, regulations, and resolutions; archival research on organization records (policy statements, job descriptions, operating manuals); analysis of records of organizational transactions; interviews of key staff and stakeholders about actual and desired practices; brainstorming by the needs assessment committee; Delphi method iterative interviewing of experts; organizational surveys of all employees; focus groups of staff and client groups; electronic discussion groups; participant observation by needs assessment staff; and in the case of certain agencies, forums or public hearings.

    • Problem identification and prioritization. Among the benefits of needs assessment is that the data gathered in the information collection stage forms a baseline measurement against which improvement may be measured. If needs assessment is conceived as an ongoing activity, the organization may benefit from time series data on performance. However, the needs assessment process does not stop with information gathering. As information is gathered, problems are identified and the process of prioritizing needs begins. The needs assessment analyst or facilitator functions to identify needs based on information revealing the gap between current practice and desired levels of performance; clustering needs into categories (ex., needs related to increasing client convenience of use); and assisting stakeholders to rank problems in terms of their importance in relation to the agency's mission. This is a sort of "triage" process, dividing needs into those the meeting of which is essential to the survival of the agency in the long run, those which are merely "wants" but do not appreciably increase organizational performance, and those in the middle which the agency would do well to implement but only after those in the top category are dealt with. The culmination of the needs identification phase is typically a statement of needs. This is not a wish list, nor is it a survey of leading edge technologies or other solutions which may be coming into use. Rather the needs statement is grounded in (1) documentation of needs articulated by stakeholders and by participants in the needs assessment process, and (2) documentation of how needs were prioritized in relation to the agency's mission and objectives. The needs statement is not a strategic plan, though it could be a supporting document for one. A good needs statement will present a convincing case that there are problems to be addressed, that some problems are more significant than others, and that problems might be ameliorated through introduction of new information systems with general specifications emerging from needs assessment information collection efforts. That is, the needs statement contains general parameters for solutions, but it is not as technical as a request for proposals (RFP) or a request for bids (RFB) will be later in the implementation process.

    • Researching solutions. When researching possible technological solutions as part of the needs assessment process, a number of routes are possible. First, there may be mandated information architectures (ex., from the OMB) which dictate to the agency the range of permitted solutions. Federal or state purchasing regulations may be in place which constrain the agency to select solutions from approved vendor lists. Professional associations may recommend "best practices" in a given area. Vendor conferences are a possible avenue for researching program solutions. The stage of researching solutions may have as its result a functional specifications document. This document spells out in greater detail than the needs statement what the capabilities and outcomes of a proposed information system should be (ex., system storage/retrieval capabilities, processing capabilities and scalability, reporting capabilities, networking capabilities). Various specific technologies may be capable of achieving the named outcomes. A good functional specification document is vendor-independent, does not use proprietary process models or other proprietary intellectual property, and is useful for comparing technical solutions across vendors. It should also be relatively free from jargon and accessible to the participants in the needs assessment process. Finally, to the extent that the agency's sunk investments require continuity with legacy information systems, compatibility with these systems is a common functional specification. It should be emphasized that the 'researching solutions' stage of needs assessment is purposely plural. It is rarely wise to research and document one technical solution for an organizational problem. Rather the needs assessment analysis is to support a broader decision-making process, such as strategic planning. In this context is it almost always more useful to decision-makers if the needs assessment effort results in alternatives for decision, not simply the case for "the one best way" as seen by the needs assessment analysts.

    • Seeking consensus. Needs assessment is a social marketing method, not just a decision support effort. That is, a good needs assessment is consumer-focused, even for public agencies. The information gathered, problems identified, and proposed solutions investigated and prioritized are all intended to help the agency more effectively meet the needs of its ultimate target audience - its consumers. The needs assessment process is not just meant to uncover data about problems, though some managers wrongly treat it as such. A good needs assessment process goes beyond this to build group consensus, a sense of participation and self-investment in change, and increased commitment to the organization's mission. That is, a good needs assessment is a form of organization development. Not the least of the concerns of the needs assessment team is selling the needs assessment recommendations to top management at the end of the process, which is really just the beginning of implementation. The more the needs assessment has focused on the nexus between the organization's mission and specific proposed changes, the easier it will be for the assessment team to convince top management that what has emerged from the needs assessment process deserves organizational priority. Management wants to know not just what needs are perceived, but rather how identified needs relate to the agency's mission and objectives. For this reason, the needs assessment report must be framed as a planning document, not as a technical evaluation.

    
    

  • Feasibility studies. What is needed may not be feasible, and what is feasible may not be needed. This logic would seem to be the basis for clearly differentiating needs assessments and feasibility studies into separate categories. In reality, however, needs assessments tend not only to assess needs but also to identify possible solutions and to evaluate their feasibility. When used in this way, feasibility studies are a component of comprehensive needs assessments. However, feasibility studies may be undertaken in their own right, to look more carefully into proposals emerging from strategic planning, legislative inquiry, executive mandate, or from whatever source. Either way, feasibility studies are more delimited than are needs assessments, concentrating on identifying obstacles to implementation and what is needed to overcome these obstacles. Feasibility studies normally are undertaken only after the organization has identified and defined a proposed project, and has a written requirements analysis and a general design statement for it.
    • Dimensions. In general, a feasibility study analyzes a business problem. It has three broad dimensions: operational (will the proposed solution, in fact, work?), economic (do benefits exceed costs?), and technical (can we integrate the proposed system with existing technology?). In commissioning an IT feasibility study, top management is looking for a distant early warning signal that can be used as a go/no-go decision support factor before the organization has committed its time, energy, and resources to the development of a proposed system. The higher the risk of the project, based on formal risk assessment procedures, the greater the need for a feasibility study.

    • Feasibility reports. What does a feasibility report look like? To take one example, the feasibility study format employed by the Department of Housing and Urban Development (HUD, 2004) requires the analyst to enter general, management, and system information. The general information section of the feasibility report includes a statement of purpose, an outline of the scope of the study, an overview of the system under study, project references, a list of acronyms and abbreviations used in the study, and a list of contacts that may be needed by the document user for informational and troubleshooting purposes, and a list of organizations that require coordination between the project and its specific functions. The management summary section describes the organizational environment, organizations involved, inputs and outputs of the organization, how the organization processes the flow of demands and services, how security is handled, interaction between the system and other systems, factors in the physical environment, current functional procedures, performance objectives, assumption and constraints, and evaluation criteria used to assess the system. The final section on the proposed system describes the proposed system, enumerates improvements the proposal makes over the present system, details time and resource costs, assesses system impacts, including equipment, software, organizational, operational, developmental, site and facility, security, and privacy impacts. There may be additional system sections comparing alternative proposed systems.

    • Difference from strategic plans. Feasibility studies can also be understood in terms of what they are not. Feasibility studies are different from strategic planning and do not assure that projects match agency mission, or if they do, that they do so more effectively than other projects competing for priority. Therefore, feasibility studies are not normally stand-alone efforts independent of agency strategic planning. Feasibility studies are different from needs assessments and do not establish need. Feasibility studies are not intended to support or supplement on-going projects. Feasibility studies are not a way of proving a preconceived project proposal is right. Feasibility studies are not final proven results, merely estimates. In fact, the track record for feasibility studies is that they are often not accurate when viewed with the benefit of hindsight, and they should be viewed not as static analyses on a particular date but rather they should be employed as a form of dynamic analysis illuminating costs, benefits, and risks for alternative proposed project options. In this way, feasibility studies bridge the distance between strategic planning and needs assessment on the one hand and the practical concerns of the project manager on the other.

  
  




Frequently Asked Questions

• What is a good web resource for evaluation research?
  Evaluation research is covered at a website maintained by Gene Shackman at http://gsociology.icaap.org/methods. Coverage includes survey research and some qualitative and quantitative methods as well as evaluation research directly. See also William Trochim's academic site at http://www.socialresearchmethods.net/kb/evaluation.php.




Bibliography

• Berk, Richard A. and Peter H. Rossi (1999). Thinking about program evaluation, Second edition. Thousand Oaks, CA: Sage Publications. Covers key concepts through examples of program evaluation.
• Boruch, R. F., et al. (1981). Reanalyzing program evaluations: Policies and practices for secondary analysis of social and educational programs. San Francisco: Jossey-Bass.
• Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
• Dunn, William N. (1994). Public policy analysis, second edition. Upper Saddle River, NJ: Prentice-Hall. See chapter on p=monitoring policy outcomes.
• Forehand, G. A., ed. (1982). Applications of time series analysis to evaluation. San Francisco: Jossey-Bass.
• Nagel, Stuart S. (1991). Public policy - goals, means, methods. University Press of America. Long section on evaluation.
• Orr, Larry L. (1999). Social experiments: Evaluating public programs with experimental methods. Thousand Oaks, CA: Sage Publications. Discusses how to interface random assignment with social program intake processes, sample design, the history of social experiments, and more.
• Owen, John and Patricia Rogers (1999). Program evaluation: Forms and approaches. Thousand Oaks, CA: Sage Publications. Standard introductory textbook. Covers introduction to evaluation processes and includes discussion of negotiating an evaluation plan, key players, proactive evaluations, and other aspects of the politics of implementing program evaluations.
• Patton, Michael Quinn (1980). Qualitative evaluation methods. Beverly Hills, Calif.: Sage Publications.
• Rossi, Peter H., Howard Freeman, and Mark W. Lipsey (1999). Evaluation, Sixth edition. Thousand Oaks, CA: Sage Publications. The classic program evaluation textbook. The new edition covers guidelines for selecting different types of evaluations, how to "tease out" theory implicit in program research, and strategies for impact assessment.
• Stoiber, John R. (1999). Maximizing IT investments. CIO. July 15. Retrieved 10/2/02 from http://www.cio.com/archive/enterprise.071599_checks_content.html.
• U. S. General Accounting Office (1990). Prospective evaluation methods: The prospective evaluation synthesis PEMD-10.1.10, 11/90 Available online in .pdf format.
• U. S. General Accounting Office (1991). Designing evaluations. PEMD-10.1.4, 5/91. Available online in .pdf format.

Copyright 1998, 2008 by G. David Garson.
Last update, 3/24/2008.