Members of the Improved Outcomes Forum have reviewed and generously contributed to this paper. Special thanks for their contributions go to Charles Bruner, Sid Gardner, Sharon Lynn Kagan, Karen Pittman, Harold Richman and Ralph Smith.

Because this paper is the product of many minds and diverse experiences, not all reviewers necessarily endorse its every particular.

The current ferment around using results as a way of assessing the success of efforts to improve the lives of children and families, and of shifting to a results-based accountability system has given rise to hope, fear, and confusion.

Hope that results-based accountability could be the key to

• freeing schools, health care, social agencies, and other human services from the rules that prevent them from operating flexibility in response to the needs of those they serve,

• restoring the public's faith that both public and private human service institutions can accomplish their intended purpose,

• encouraging communities to be more planful, more intentional, in how they support children and families.

Fear that results-based accountability will be misused and bring about

• the abandonment of attempts to better the conditions of disadvantaged children whose effects are difficult to measure or take a long time to occur

• the erosion of essential procedural protections and neglect of concerns for equity

• a smokescreen behind which further funding cutbacks will be made

• penalizing individual professional, institutions, and agencies who may not be achieving hoped-for results, but are trying hard and doing the best they can.

Confusion about how to devise and obtain agreement on a set of goals and outcomes, and reliable ways of measuring results that could justify the hopes and quell the fears of the many concerned constituencies.

This paper sets out some of the issues in the shift to results-based accountability, and identifies a start-up list of outcome measures with annotations on their use. The authors see results-based accountability as an essential part of a larger strategy to improve outcomes for children. Subsequent materials from the Improved Outcomes for Children Project will address how these outcomes can be translated into a program agenda; how that program agenda can then lead to a budget and a financial plan; and how, over time, results-based accountability can be combined with both rewards and penalties, based on performance. This paper sets the stage for those later discussions, but all readers, especially policy makers, should note that the real benefit of results-based accountability can be realized only through that full sequence of activities.

A. Results-based accountability can replace -- or at least diminish the need for -- centralized bureaucratic micromanagement and rigid rules. Effective services require a significant degree of both local variation and frontline discretion, which cannot be maintained in the face of detailed regulation of program inputs that tie the hands of front-line professionals. Management by results is the best alternative to top-down, centralized micromanagement, which holds programs responsible for adhering to rules that are so detailed that they interfere with a program's or institution's ability to respond to a wide range of urgent needs.

Policy makers can be encouraged to desist from regulating inputs and prescribing detailed procedures if they have the capacity to hold programs, institutions, and those who implement and run them accountable for results. The use of outcome indicators that reflect common sense and common understanding (indicators that show, for example, that the rates of low-weight births are being reduced, or that more students are demonstrating age-appropriate mastery of school skills) helps to focus attention on agency mission rather than rules, and permits the necessary flexibility and autonomy at the front-end. Auditors can spend less time reviewing records to see how many services were provided (e.g., how many families were seen) and whether the forms documenting eligibility for services were properly filled out. Instead, they spend more time on inquiring into the results achieved (such as the reduction of multiple or inappropriate out-of-home placements for children.) The question asked of professionals at the front-lines, be they teachers, social workers, or health professionals, would shift from "Did you do what they told you to do?" to "Did it work? What difference did it make in outcomes for children?" A changed organizational climate would be the result, in which well-trained professionals would be able to use their judgment and experience to respond flexibly to the needs of children and families, rather than being constrained by pressures of funders who see detailed input regulation as their only means of protecting the public interest.

One example of how this approach can work comes from Kentucky: under the Kentucky Education Reform Act, the state decides what, and local communities decide how. (The process of deciding what a state will hold communities accountable for will optimally involve extensive consultation, as we discuss in Section V.A. on page 15).

B. Outcome information can assure funders and the public that investments are producing results. Funders and the public are demanding information on which informed judgments can be made about whether institutions, programs and policies are in fact accomplishing their intended purpose. Polls show that a public that has lost confidence in government and other institutions is prepared to support new investments in schools and services when citizens are convinced that the investment is bringing promised results. Especially in a time of fiscal constraints, programs and agencies willing to be held accountable for achieving agreed-upon results will have a greater chance of obtaining needed funding and other support.

C. Agreement on desired results can facilitate cross-systems collaboration on behalf of children. As more individuals and agencies from different systems, disciplines, and backgrounds attempt to work together, they encounter barriers that are difficult to overcome in the absence of shared purposes that are explicitly defined and agreed upon. Agreement on a common set of goals and outcome measures not only makes collaboration easier, but also helps promote a community-wide "culture of responsibility" for children and families, and fuels the momentum for change.

D. Agreement on desired results helps to minimize investment in activities that don't contribute to improved results. Reflecting Alice in Wonderland's insight that if you don't know where you're going, any road will get you there, a focus on outcomes is likely to discourage expenditures of energy, political capital and funds on empty organizational changes and on ineffective services. Community-wide acceptance of shared outcomes thus helps to put service integration efforts into proper perspective. Fragmentation is such a big program in existing services, that a great deal of energy is currently going into efforts to link services together. But while collaboration is a necessary condition for providing effective services, it is not sufficient. Alone, it will not improve results. It is futile to put together services that are ineffective because they are of mediocre quality, are rendered grudgingly, are rendered by professionals who don't work respectfully with families, and are unable to respond to the unique characteristics of the community they serve. The shared commitment to improve outcomes for children is what can make efforts at collaboration fall into place -- not as an end, but as an essential means of working together toward improved results.

E. Information about results enhances community and agency capacity to judge the effectiveness of their efforts, and to modify activities in response to impact information. Because this is a time of rapid change in education, welfare, health and human services practices, programs, and systems, it is particularly important to improve the ability to make judgments about the effects of change: whether new or reformed services, new relationships among service providers and between providers and communities, and new connections among schools, services, and other community and economic development efforts, are working and worth the investment. When results are tracked on a timely basis and connected to service strategies, outcome information can provide guidance to program development as initiatives evolve.

F. A focus on results clarifies whether allocated resources are adequate to achieve the outcomes expected by funders and the public. An outcomes focus highlights how much investment is required to produce significant improvement in the lives of children and families. The new conversation about results is promising (or threatening) to end a conspiracy of silence between funders and program people. It exposes the fact that human service providers, educators, and community organizations are consistently expected to accomplish massive tasks with trivial resources and inadequate tools. It assures attention to the adequacy question: whether the resources being committed are adequate to achieve the intended purpose.

A parent education program may be expected to reduce the incidence of child abuse in a neighborhood although it consists of only a few didactic classes. An outreach program to get pregnant women into prenatal care may be expected to reduce the incidence of low birth weight in a community although the sources of prenatal care are overcrowded, impersonal, and have no capacity to deal with homelessness, drug abuse, or lack of social support. In the past, when a significant discrepancy occurred between aspirations and documentable (or even potential) accomplishments, the response has typically been to retreat from a commitment to changing outcomes. As program managers and funders face up to the difficulties of actually changing real-world outcomes, and recognize the relative weakness of single, underfunded, piecemeal interventions compared to powerful social, economic, and demographic forces that push outcomes in a negative direction, they often agree just to measure how many people were reached or how attitudes and knowledge have changed among program participants. Together, all parties agree that it is unfair or inappropriate to hold the program accountable for actually improving results when the program is doing the best it can. The reality that the investment is insufficient is thereby obscured.

In circumstances where it will take a critical mass of comprehensive, intensive, interactive interventions to change outcomes in a defined population, where interventions must be able to impact even widespread despair, hopelessness and social isolation in order to be effective, it is counterproductive to hide from ourselves the limitations of most current efforts. Recognition that the addition of a single new intervention, or even recognition that services alone may not be sufficient to change outcomes in the absence of changes in employment, economic security, physical safety and housing, is not an argument against results-based accountability. It is an argument for adequate funding of promising interventions, and an argument for human service agencies and schools to forge partnerships with efforts to create jobs and to improve housing and public safety, and to rec-create the infrastructure of communities. It is an argument for defining what the community considers a set of adequate outcomes and then making sure that the resources needed to reach those results are made available.

While the shift toward results-based accountability brings many clear advantages, it is not an unmixed blessing. In fact, it carries real risks, which much be recognized and minimized through careful strategizing and thoughtful implementation.

A. Funders, the public, and even program people, may underestimate how long it takes to achieve significant improvement in outcomes. The single most frequently cited lesson from major current reform efforts is that it takes much more time than expected -- both to get the initiative under way, and to get it to the point where it begins to show an impact on real-world outcomes.

B. Demands for documented results could drive programs away from achieving their mission of improving a broad range of results for a broad target population, when agencies

• engage in "creaming" and ducking hard cases

• distort their activities to emphasize those that will show measurable and rapid results while neglecting equally or more important efforts that are harder to quantify and whose results don't show up as quickly. (For example, if immunization rates are the outcome on which health services are judged, it will be important to guard against allowing an emphasis on immunization strategies to result in the neglect of other primary and preventive services.)

C. Funders could be led to confine their support to interventions whose effects are readily and quickly quantifiable, in preference to more effective or subtler interventions whose benefits are harder to document. (A church-sponsored children's choir or a recreation center for teenagers may add to a sense of community, and thereby contribute to improved outcomes, without being able to demonstrate a direct outcome impact.) Protection must be explicitly provided to interventions that are vulnerable because

• their effects show up years later

• the data to document their impact are not available, or the technology to measure their effects does not exist

• the intervention is one of many preventive efforts that interact to produce cumulative later effects.

D. Some forms of outcome measurement could lead to labeling and stigmatizing children. Efforts to assess school readiness, for example, must be designed to take into account that "readiness" does not inhere solely in the child, that children don't all develop at the same rate, and that "readiness" is not just a cognitive matter, but includes physical and emotional health and social competence. In addition, school readiness should be assessed on a sample rather than an individual basis, to avoid labeling of individual children.

E. Agency accountability could be weakened as attention shifts to community-wide accountability efforts. Achieving any of the core outcomes requires the activities of multiple agencies, as well as informal supports and community activities that are beyond a single agency's control. Since no one agency, acting alone, can achieve any of the agreed-upon results, the question arises, how should individual public agencies, or systems -- for example, the child welfare agency or the mental health system -- hold themselves or be held accountable.

The answer lies in a careful "unbundling" of each outcome into the detailed steps and strategies which, together, can produce the intended result. For example, to increase healthy births, a host of related actions are necessary, one of which is likely to be that the local health department must organize and conduct outreach to all high-risk mothers to ensure that they have adequate prenatal care. Once the health department's role is specified within the context of the overall strategy, measures can be applied to how well the health department is performing this role and the department can be held (and can hold itself) accountable for this result. So, to continue this example, if all parts of a strategy to increase health births are successfully accomplished except one (e.g., the health department's), then the health department can be held accountable by the community for better performance.

F. The shift to results-based accountability should not be seen as a panacea. An outcomes orientation does not guarantee the design of an effective community service and support system. Especially because current understanding of the precise connections between specific interventions and specific outcomes is limited, an outcomes orientation does not solve the problem of identifying needed and missing elements of a comprehensive approach to improving results. However, by informing and focusing the process, result-based thinking does seem to support the development of a culture that is less rule-bound and more mission-driven.

G. The shift to results-based accountability cannot be allowed to substitute for rock-bottom safeguards against fraud, abuse, poor services, and inequities or discrimination based on race, gender, disability or ethnic background. Supports of improved outcomes for children may in the past have overestimated the extent to which equity and quality could be safeguarded by regulation, and underestimated how much detailed mandating and strict rules could undermine responsiveness, flexibility, and the exercise of discretion. If the shift to results-based accountability is to attain its promise, it must be combined with better training of front-line workers and managers and careful monitoring, to make the achievement of less rigidity in the provision of services compatible with the protection of vulnerable children and families.

The new outcomes orientation should not lead to the abandonment of all input and process regulation. Procedural protections will have to be maintained to protect against fraud, poor services, and inequities or discrimination based on race, gender, disability, or ethnic background. Procedural protections will also have to be maintained where there is no other way to restrict the arbitrary exercise of front-line discretion by powerful institutions against the interests of powerless clients. At the same time, reliance on process measures should not be allowed to drive out or obscure a continuing focus on results-based accountability.

H. When a community adopts a results-oriented system, fundamental changes in the allocation of resources will result and must be prepared for. One key indicator of whether the outcomes orientation is influencing agency performance will be whether budget priorities change in response to the identification of shared outcome measures. The shift to results-based accountability will be of little use in a community if it does not translate into management and budget priorities. The adoption of community outcomes will be ineffective if agencies revert to tradition and divide the results into isolated subsets that would be pursued separately by schools, social service agencies, the health sector, mental health agencies, juvenile justice, and others. As agencies, communities and systems begin to manage their resources to achieve agreed-upon results, set priorities based on these outcomes, and are held accountable for making progress toward these outcomes, some agencies will actually lose resources in the process. That risk must be acknowledged and prepared for at the outset.

Communities, states, and collaboratives can best begin the results-based accountability process by asking the question, What do we want for our children? By beginning in this way, these deliberations are usually able to go beyond earlier project- or problem-bounded discussions. They are then more likely to produce a consensus around basic goals which together constitute a vision that can serve to:

• capture aspirations

• unify people who differ in many other ways around common aims for their children

• focus attention not only on a wide range of formal services, but also on informal supports and opportunities

• surface value issues in the context of what people want for their children, before they get into the question of what is to be measured.

Examples of the goals that communities might choose include

• higher rates of healthy births

• higher rates of children ready for school learning at the time of school entry

• higher rates of children succeeding at school

• higher rates of youth who achieve personal, social, and vocational competence (which might be further broken down into higher rates of youth who feel safe, who have a sense of self-worth, a sense of mastery, a sense of belonging, a sense of personal efficacy, who are socially, academically, and culturally competent, and who have the skills needed for productive employment.)

As a community embarks on the job of building an effective system of services and supports, these goals can become a framework that can be used for many purposes -- inspirational, guiding service delivery, and guiding the selection of outcome measures for accountability purposes.

Only some aspects of these goals can currently be measured with widely available data and with outcome measures around which it is possible to gain widespread agreement. Most communities will have aspirations for their children that exceed the results that are currently measurable. Goals and outcome measures serve different purposes. The former represent what the community is striving for. The latter represent what the community will be held accountable for -- by public and private funders and perhaps by higher levels of government. The goals can be general, but the outcome measures must be so specific, the public stake in their attainment so clear, and their validity and reliability so well established, that the community would be willing to see rewards\and penalties, as well as resource allocation decisions, attached to their achievement.

Thus the list of outcome measures that is attached represents minimum rather than maximum objectives, with the advantage of

• allowing measurement to begin

• allowing communities to reach agreement on results for which they are willing to be held accountable.

• allowing voters, professionals, and program participants to see the direct connections between the goals for children they are trying to achieve, and the outcome measures used for accountability purposes.

We believe that a commitment to more visionary goals is entirely compatible with a commitment to documenting progress toward the achievement of these goals by the use of the outcome measures we propose. Agencies and communities that align their activities with these measurable outcomes (i.e., "teaching to the test") would be moving dramatically in the right direction. Furthermore, and most important, while the outcome measures we propose stop short of a more ideal and ambitious set of goals, substantial progress toward these outcomes would signify a radical improvement in the prevailing conditions of children and families.

We believe that it is critical to start implementing systems of results-based accountability now, recognizing that the necessary procedures and technologies are still in the process of development. Too much is at stake not to begin. Not only is the overall well-being of children deteriorating in many ways, but the credibility of government capacity to help improve these outcomes is at its lowest point ever. If we are to maintain, let alone expand, investments in children's futures, we must be able to produce measurable results.

Although there are still many problems to solve in moving towards holding systems accountable on the basis of results achieved, we believe that the potential usefulness of results-based accountability now outweighs the risks. We believe that the shift to results-based accountability must be made carefully and thoughtfully, and that it must be led by those who care about both the process and the results, and not left to those who find it easy because they don't understand the issues.

Many communities and agencies around the country have come to similar conclusions, and have already been hard at work, defining the goals and outcomes they intend to pursue. We have tried to learn from these processes now under way, and have identified a list of Child and Youth Outcomes, around which it is relatively easy to obtain broad agreement, and about which data are readily available. We believe that the process of agreeing on set of outcome measures will be simplified and speeded up if every state, every local community and every new initiative didn't have to start from Square One. The list of outcome measures attached to this paper is intended to serve as a starting point for discussion and negotiations.

For each of the outcome measures on our list, we have also compiled the following information (to be found in separate volume, "A Start-up List of Outcome Measures With Annotations"):

• definition
• significance
• relevant facts
• national, state, and local data sources
• additional comments
• related measures

Our list is limited to child and youth outcomes, follows a developmental sequence (from birth through children to young adulthood), and is based on criteria specifically designed to produce a set of outcomes for immediate use. We present our criteria and our rationale for inclusion here so that they can stimulate the discussion that is essential if a results-based orientation is to be widely adopted.

A. A start-up list should consist of outcome measures that are considered important and meaningful by a wide range of policy makers, funders, and citizens. In our view, the results chosen for accountability purposes should be transparent measures of success that are persuasive to skeptics, not just to supporters of the programs and policies being assessed or held accountable. (For example, skeptics are more likely to be impressed by changes in the rate of unmarried teenage childbearing than by changes in self-esteem scales.)

B. A start-up list should consist of outcome measures about which data are relatively easy to obtain, primarily from existing official data sources, and to interpret. For example, outcomes such as the appropriate receipt of special education services are not included because the data are difficult to gather for large numbers of children and hard to interpret.

C. The distinction between outcome measures and process (or capacity) measures should be clearly maintained. Indicators that measure processes and capacities (rather than results) should be included as a significant part of a results-based accountability effort when they are clearly related by empirical evidence to significant outcomes. (For example, early receipt of high quality prenatal care has been shown to be clearly related to positive birth outcomes; on the other hand, participation in didactic parent education classes, even when it results in increased knowledge, has not been shown to improve parenting or decrease child abuse or neglect.) When process measures (e.g., school attendance, formation of a collaborative) or capacity measures (e.g., the availability of high quality child care) are used as part of results-based accountability, the rationale for using them instead of outcome measures should be clearly understood and stated.

D. The least ambiguous available measure of an outcome should always be used (e.g., confirmed child abuse reports are a better measure than child abuse reports, because an increase in child abuse reports could reflect a new hotline or greater community awareness, rather than an increase in the incidence of child abuse.)

E. The outcomes chosen should, to the fullest extent possible, not be subject to misuse. In the early childhood arena, for example, a community's achievements in raising the rates of children who enter school prepared for school success can and must be measured without stigmatizing or labeling any individual child. This can be done by collecting information only about groups and/or samples of children, rather than by assessing and reporting on individually identified children.

Which of the outcome measures on this or other start-up lists will actually be selected by programs, agencies, communities, and various political jurisdictions will depend on the nature and purpose of the efforts being measured, the nature and size of the target population, and who engages in the process of selecting outcomes. In recognition of the relationship among all of the major child and youth outcomes, we believe that communities beginning the process of shifting to results-based accountability should consider the full range of child and youth outcomes in their planning, although they may decide to undertake their efforts in pursuit of these outcomes sequentially.

We expect to modify our start-up list in the future, to reflect ongoing experience by those using these measures, progress in the development of more sophisticated technology, the availability of new kinds of data, and the identification of interim indicators that could measure short-term steps toward long-term outcomes and goals.

A. Who decides? Who selects the outcomes to be achieved for accountability purposes? When individuals representing diverse interests attempt to work together to identify outcomes for widespread use, the process may lead to a shared understanding of goals and the steps to reach them, but the process may also bring out differences that threaten to paralyze both the participants and the process. Tensions between "top-down" and "bottom-up" processes of selecting outcome measures are among the most difficult. On the one hand, many believe that society has so much at stake in the achievement of a core set of outcomes, that political bodies -- probably at the state level -- should be responsible for identifying a set of outcomes that should be achieved universally. Others believe that "outcome measures imposed from outside a community have no legitimacy in terms of a local consensus-building process...and cannot mobilize the resources needed to achieve the results sought...

There seems to be increasing agreement that the process of selecting outcomes for accountability purposes must have political legitimation, whether it is done by a state legislature or a local collaborative. Sid Gardner, of the Center for Collaboration for Children at the University of California, Fullerton, points out that the importance of going through a consensus building process, especially in the selection of over-arching goals, cannot be underestimated, because this is political, not primarily a technical process. Charles Bruner of the Iowa Child and Family Policy Center argues that those charged with achieving the outcomes must be involved in the outcomes selection process if it is to be regarded as fair, useful, legitimate, and if it is to reflect real-life experiences.

It is clear that all of those affected by results-based accountability -- as legislators representing tax payers, as providers, or as service beneficiaries or participants -- must have a role in the process. All concerned will be able to work more effectively toward common goals if they are able to engage in a consensus-building process, involving both providers and recipients of services, selected its education outcomes by holding meetings in towns throughout the state, eliciting extensive grass-roots participation in the process.

We also believe that if results are to be used for accountability purposes, the final decisions must be made by bodies at a higher or broader level of governance than those being held accountable. Many forms of interactive consultation are possible. For example, when an official state body selects the outcomes, localities may decide or negotiate the numerical value that will represent progress in the achievement of each outcome (e.g., the rate of low birthweight will be reduced by X percent each year, or racial discrepancies in low birthweight rates will be reduced by Y percent each year).

B. Who is responsible for achieving the selected results? It is difficult to envision an effective use of cross-sector outcomes in the absence of a governing body representing the major agencies, institutions, and interests within a community, because no one agency can achieve most of the core outcomes on its own. Governance structures that could take responsibility for results-based accountability across programs and systems and across political jurisdictions generally do not now exist. New arrangements will have to be made so that the community can (a) agree on a set of outcomes, (b) determine how best to achieve them, and (c) measure progress in meeting them.

Once a community decides on and disseminates a set of outcomes, public awareness of children's needs will increase, and it will be easier to track progress and to marshall public support for achieving the goals. For example, if there is broad public awareness that improved school achievement has been adopted as a community goal, then parents, neighbors, and informal community institutions as well as schools and human service agencies will be stimulated to be come more active and effective in contributing toward that goal.

C. What is the role of process measures in a system that places primary reliance for accountability on outcome measures? Because the present capacity to use outcome measures to judge program effectiveness is far from perfect, and because if often takes many years before outcomes improve in response to effective interventions, process measures will continue to play an important role in holding agencies, communities, and systems accountable. Existing process measures will be useful during the period of transition. Increasingly, however, new measures that are more closely and reliably related to outcomes should become available to measure initial progress toward ultimate goals, both through the measurement of interim outcomes, and by measuring the community's capacity to achieve identified long-term outcomes. In Section VI. C. on page 20, we discuss the sustained work that is now needed to produce reliable, short-term indicators.

It will be important to continually re-examine the balance between the use of process and outcome measures, so that communities and agencies can make sure they do not slip back into reliance on the input measures that results-based accountability was meant to replace.

D. Is there a role in results-based accountability for cost-savings outcomes? One outcome measure that has proven useful -- even though it is a departure from child and youth outcome measures -- is an indicator of cost-savings that occur as a result of the provision of improved services. Although many important savings occur in budgets and at times that are far removed from the new or improved service, in instances when the savings can be directly attributed to the intervention, the evidence of cost-effectiveness can be dramatic and persuasive. For example, high quality prenatal care and supports have been shown to save several times their cost in neo-natal intensive care. We advocate using cost-savings measures as a supplement to individual outcome measures rather than as a substitute.

E. What is the relationship between results-based accountability and evaluation research? The role of the evaluator and of evaluation research would undergo significant change in a world in which results-based accountability were the norm and not the exception and in which outcomes have become an important part of the everyday way that people think about programs, policies, and reform initiatives. This is especially true with respect to the evaluation of complex, multi-system, multi-disciplinary interventions that are expected to impact children, families, institutions, and whole communities.

First, when desired outcomes are specified by a community or state as part of its resource allocation or reform efforts, it is reasonable to base the impact evaluation of the program or community initiative on those outcomes. Then the evaluator would no longer assume the responsibility for selecting (or negotiating with program people to select) the outcomes on which the effort will be evaluated. In addition, since the outcome measures used for accountability purposes tend to be ones which are already being collected for other purposes, usually by official agencies (e.g., rates of low birth weight, immunization, arrests, school completion, employment, etc.), they are more likely to be widely recognized for their "real world" significance than are measures that show performance on scales constructed by researchers primarily for research purposes. The use of outcome measures that are already being collected for other (usually public) purposes has the effect of vastly reducing evaluation costs, where measured in money, time, intrusiveness, or required expertise. Outcome measurement would then no longer rely solely on an expert intermediary, on whom program managers, funders and communities are dependent for information as to whether the program is succeeding. The result could be a demystification and democratization of the outcomes evaluation process, encouragement of citizen monitoring of the welfare of children and families, greater access to outcome evaluation findings, and an increase in usable knowledge.

Second, when the impact of an initiative on selected results is measured for purposes of accountability, it is possible to draw a greater distinction between the two evaluation functions of assessing results (i.e., documenting the extent to which agreed-upon results were achieved), and of attributing causal connections (i.e., making judgments about the process by which those results were obtained). If evaluators were free to focus more on the "why" and "how" (as opposed to the "whether"), they could position themselves to better obtain an accurate and nuanced understanding of the nature of interventions, and to provide an accurate description of the interventions, because they would not have to maintain the traditional distance between evaluator and provider that has been required to provide the evaluator with an "outsider's" credibility.

Evaluators who are no longer responsible for answering whether the initiative worked, but can focus on how and why it is working, can move toward an explicit stance of helping practitioners to become more reflective, to extract theory from their daily experience, to learn contemporaneously from their experience, and thereby to improve their practice and their intervention. The new evaluator could provide feed-back to practitioners for mid-course corrections, enhancing their capacity to reflect and do on-the-spot experimentation. Evaluators could help practitioners to think more carefully about both the theory and practice of what they're doing. They could develop a new evaluation culture that would lead to a greater understanding of successful interventions and how results to interventions. Evaluators would become collaborators of reflective practitioners in the interest of program improvement, and of helping others to learn from proven and promising interventions.

A. Continuing work on making a start-up list more useful. We expect to continue work on helping communities and programs to implement results-based accountability. We expect that as communities and agencies being using results-based accountability tools, they will need additional information and advice on such matters as:

• how to get the necessary information,

• which measures are most appropriate with what size populations, in which specific circumstances, and over what period of time,

• improving the tools that will make it possible to measure outcomes in useful units (such as school catchment areas, neighborhoods, census tracts, zip codes, etc.),

• how changes in outcomes should be understood in relation to background factors (such as how to account for changes in outcomes that might be attributed to the closing of a factory or a sudden influx of new immigrants),

• how to select appropriate comparisons against which to measure outcomes, including the use of comparisons over time, comparisons with groups outside the community, and comparisons among various racial and income groups,

• how to avoid misuse of the chosen outcome indicators,

• how long it takes before it is reasonable to expect change in a given measure, and the magnitude of change that can be expected in relation to the size and nature of the problem and of the population,

• understanding options with regard to attaching rewards and penalties to the achievement of agreed-upon results, and

• identification of reliable short-term indicators that could measure small steps toward ultimate goals.

B. New work to increase the significance of what is being measured. Several of the outcome measures we recommend are not a perfect match for the outcomes we are recommending that communities get information about. For example, the indicator "Children living in their own family" is an imprecise measure of the outcome we are really after, which is "Children living safely in their own homes or in stable out-of-home care." We will continue to seek more reliable approximations, such as "Children who have experienced multiple out-of-home placements." The school-readiness outcome poses special measurement problems, which we expect to work on with communities.

Work should also proceed to identify new ways of achieving a closer match between outcome measures and long-term goals, by measuring additional widely acceptable indicators of positive well-being, such as responsible citizenship, healthy and nurturant family functioning, and healthy parent-child interactions. Consideration should be given to whether a major investment, on the order of the New Standards Project, is called for to achieve this purpose.

C. New work on identifying intermediate indicators. The many funders, practitioners, mangers, and systems reformers who are adopting an outcomes orientation increasingly recognize that they must obtain information about results during time periods that are meaningful to politicians, and that means relatively quickly -- often long before a program is "proud," and long before it has had a chance to make an impact on rates of school readiness, child abuse, school success, teenage pregnancy or violence. Sustained work is therefore now needed to produce reliable, short-term indicators that could measure initial progress toward ultimate goals.

There are two primary ways to assess progress toward the achievement of agreed-upon outcomes over the short-term: One is by measuring interim outcomes, and the other is by measuring the community's capacity to achieve the identified long-term outcomes.

New work on measuring interim outcomes. More work is needed to test hypothesized connections between short-term and long-term outcomes, although some connections seem to be fairly well established. For example, an improvement in school attendance rates is though to predict an improvement in school achievement rates. In a current evaluation of family preservation, such indicators as parent's sense of mastery, social support, and parental substance abuse are used as interim measures to predict such long-term outcomes as the recurrence of abuse or neglect. Such process measures as extent of participation, client satisfaction, and an increased sense of community may also come to be seen as reliable precursors of improved results.

New work on identifying indicators of community capacity. The biggest obstacle to identifying indicators of a community's capacity to reach long-term outcomes is that little is known about the precise relationships between the components of capacity and outcomes. The identification of indicators of community capacity that could guide communities in their efforts to reach long-term outcomes depends on having or developing reliable theories -- or at least sturdy hypotheses -- about the relationship between interventions and results, and about the constellation of conditions and interventions that will lead to good results. For example, when a community is developing strategies to reduce rates of low weight births, can a funder or governing entity say with confidence that the "enabling conditions" to reach that outcome are the capacity to provide high quality, responsive prenatal care, nutrition services, and family support to pregnant women, and family planning services to all persons of child bearing age? If the outcome in question were school readiness, is it possible to say that higher rates of school-readiness will be achieved if the community develops its capacity to provide high quality child health care, family support services, child care, Head Start, nutrition services, etc.?

There is little consensus on the constellation of services and supports that must exist in a community that can be reasonably assumed to constitute the preconditions for improved results, especially since many interventions rely on their interaction with other interventions for their effectiveness. For example, a high quality after-school program is probably effective only in combination with many other services and supports that could together make a difference. Furthermore, it is not enough to know of the simple existence of certain services, one would have to take their quality and other attributes into account. The distinction between service availability and the nature and quality of the service in accounting for improved results is absolutely crucial -- and requires much greater understanding than now exists. Ultimately, theoretical understanding might even allow one to take account of certain preconditions, such as an incentive structure to support desirable services, the availability of certain kinds of training and professional development activities, or the provisions made to respond in a culturally sensitive way to a variety of populations.

In some areas, existing empirical evidence permits clear connections to be made between inputs and results. For example, the work of Joyce Epstein and colleagues at Johns Hopkins University has shown that schools can enable more families to become and stay involved in their children's education, and that parent assistance at home has important consequences for children's achievement, attendance, school adaptability, and classroom behavior. In the main, however, much systematic empirical work is required before short-run measures of capacity can reasonably be related to long-term outcomes. For example, the availability of prenatal care and health insurance are surely related to improved birth outcomes, but whether the relationship is strong enough, and whether their effect on outcomes is actually a function of their availability (rather than of their quality), so that their availability can be used as an interim indicator, is an open question.

Little is also know, though much is hypothesized, about the connection between indicators of community-level change, and outcomes for children and youth. It is reasonable to assume that a neighborhood that is building an infrastructure of informal supports and economic opportunity is likely to lead to better outcomes for children, but there is little agreement on ways to measure community building and economic opportunity, and little understanding of the precise connections.

In primary and secondary education, the response to demands for interim measures has taken the form of what have come to be known as "Opportunity to Learn Standards," or "School Delivery Standards." The rationale for these standards is that it would be unreasonable to expect students to perform at world class levels in science if the school they go to has no chemistry or biology labs, and that the presence or absence of chemistry and biology labs would be something the community could look at to assess its progress toward desirable results, more simply and at an earlier time than measurable changes in achievement could be expected to occur.

The downside of the extensive application of capacity measures to assess progress toward long-term outcomes, of course, is how closely they could come to resemble the input regulation that results-based accountability was meant to replace.

This paper has discussed the case for results-based accountability, and has proposed a start-up list of outcome measures that could be used by communities as they begin to implement this approach. Agreement on outcome measures, however, is just the first step in installing a community-wide approach to service design, delivery, and accountability that is rooted in results. The key next step is for the community to use results to develop an effective program agenda. That step, in turn, must be complemented by attention to the collaborative governance strategy that maintains the results-based accountability; by the financial strategy for funding the program agenda; and by the professional development and training strategy that can help build frontline and administrative staff skills to support this new approach. Those topics are the subject of future materials from the Improved Outcomes for Children Project.

¹K. Pittman, S. Zeldin, “Evaluating Youth Development in Programs and Communities: The Need for an Integrated Framework and Collaborative Strategy.” Washington, DC: Center for Youth Development and Policy Research, 1992.

²Especially when it comes to adolescents, many communities and many programs will be working to achieve goals that go well beyond what they are able to measure. We believe that a programmatic commitment to achieving such goals as higher rates of youth who have a sense of self-worth, a sense of mastery, a sense of belonging, and a sense of personal efficacy, is not incompatible with accountability systems that rely on outcomes that are more easily measured and that may be more persuasive in a public policy context because their harm is widely agreed upon (e.g., reducing the rate of school drop out, teenage pregnancy, and juvenile crime.)

There are many who believe, with Albert Shanker, President of the American Federation of Teachers, that the opposition to Outcome Based Education is the result of over-reaching. Shanker cites the following examples of outcomes chosen by the States of Pennsylvania and Ohio which he believes were too ambitious and too ambiguous to command widespread acceptance: