Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance

ACKNOWLEDGEMENTS

Cynthia Helba and Kathy Langwell, Westat, and Jo Ann Kauffman, Kauffman and Associates, Inc., prepared this Final Report.  W. Sherman Edwards and Brian Colhoff, Westat, Victor Paternoster, Kauffman and Associates, Inc., Janet Sutton, Project HOPE Center for Health Affairs, and Frank Ryan, project consultant, made substantial contributions to the study.  We would like to thank the members of the Technical Working Group for their assistance and thorough reviews of draft reports throughout the study.  In addition, we are grateful to the Tribes/Tribal organizations that volunteered to participate in site visits that were conducted for the study, including:  Bois Forte Band of Chippewa, Choctaw Nation, Hopi, Little Traverse Bay Band of Odawa Indians, Port Gamble S’Klallam Tribe, and the Yukon-Kuskokwim Health Corporation.  Several individuals at the Indian Health Service also contributed generously of their time and provided assistance with data review activities.  Finally, we would like to thank Andy Rock and Tom Hertz, OASPE/DHHS Task Order Managers for the project, who provided valuable guidance and insights throughout the project.

DISCLAIMER

The conclusions and discussions presented in this report are those of the authors and do not necessarily reflect the views of the Office of the Assistant Secretary for Planning and Evaluation, the Indian Health Service, other agencies within the U.S. Department of Health and Human Services, or of the members of the Technical Working Group that provided advice and assistance to the study team.

Summary and Findings, Background, and Objectives

Summary and Findings

Purpose of Study

The Evaluation Feasibility Study was designed to provide information to the Office of the Assistant Secretary, U.S. Department of Health and Human Service (OASP) and the Tribes on several questions:

• Would an evaluation of Tribal Self-Governance be feasible?
• What would be the important evaluation issues and research questions?
• How would these issues and questions be addressed?
• Would there be sufficient data available to permit an evaluation to be conducted?
• What would be the costs of each alternative approach?

The Evaluation Feasibility Study was structured to obtain input and views of Tribal representatives, through a Technical Working Group, site visits, and discussion groups held at national American Indian and Alaska Native (AI/AN) conferences.  Technical Working Group members and other Tribal representatives provided input and comments on a range of issues, including feasibility considerations, the type of evaluation activities that would be desirable, and the potential benefits that evaluation could provide to Tribes. 

It is important to note that, while the purpose of this project has been to provide information helpful to the design of an evaluation, the project was not designed to produce a definitive evaluation design and methodology.  Results of this Study identify and describe feasible options for an evaluation and provide observations helpful to efforts on the part of DHHS and the Tribes regarding the range of issues that should be addressed.

Findings: Tribal Perspectives

The Technical Working Group and other representatives of self-governing Tribes who provided guidance and input to this project emphasized a number of points that were important to the development of the study findings:

1. Many Tribes that are involved in Self-Governance
2. Tribal Temporary Assistance for Needy Families
3. Low Income Home Energy Assistance
4. Community Services Block Grant
5. Child Care and Development Fund
6. Native Employment Works
7. Head Start
8. Child Welfare Services
9. Promoting Safe and Stable Families
10. Family Violence Prevention:  Grants for Battered Women’s Shelters

There are Tribes currently managing each of these DHHS programs that were determined feasible for inclusion in a Tribal Self-Governance Demonstration project, under contractual arrangements or grant awards.^[4]  A Self-Governance Demonstration program could, as detailed in the Final Report, permit a simpler, multiple-program application process and simpler and consolidated reporting requirements.  Importantly, the Demonstration program could provide “Tribes with the flexibility to change programs and reallocate funds among programs” to better address specific Tribal community priorities.

Congressional action is necessary to authorize a DHHS Tribal Self-Governance Demonstration, if such a demonstration were enacted.  The study reported here examines how an evaluation of outcomes and successes of DHHS programs managed by Tribes under Self-Governance might be conducted. While a number of assessments of Tribally-managed programs have been conducted, these have been primarily qualitative in nature. The Office of the Assistant Secretary for Planning and Evaluation (OASPE) was interested in determining the feasibility of conducting an evaluation that could include both qualitative and quantitative analyses of processes and outcomes associated with DHHS programs managed by Tribes under Self-Governance.

In September 2002, DHHS contracted with Westat, and its subcontractor, Kauffman and Associates, Inc., to conduct a study that would provide background information and assess the feasibility of conducting a rigorous and defensible evaluation of DHHS programs managed by Tribes under Self-Governance.

Objectives of the Study

The Evaluation Feasibility Study was designed to provide information to DHHS and to Tribes on several questions:

• Would an evaluation of DHHS programs operated under a new Tribal Self-Governance demonstration program be feasible?
• What evaluation issues and research questions would be important to address in an evaluation?
• What measures – both qualitative and quantitative – would be appropriate to use to address each of the selected evaluation issues and research questions?
• Would there be data available, within DHHS and from Tribes managing DHHS programs, which would permit an evaluation to be conducted, using a rigorous methodology that would be likely to produce reliable results?
• What would be the cost implications of alternative feasible evaluation strategies?

This project was intended to provide information helpful to the design of an evaluation and was not designed to produce a definitive evaluation design and methodology. Results of this Study were intended only to provide information on feasible options for an evaluation and considerations that can be used by DHHS in consultation with the Tribes about an evaluation that could be conducted and the range of issues that could be addressed.

Organization of this Draft Final Report

This Final Report on the Evaluation Feasibility Study provides background information and assesses the feasibility of conducting an evaluation of DHHS programs that may be managed by Tribes under a potential new Self-Governance demonstration.  Throughout the 15-month project, guidance and input were provided to the study team and to OASPE by the project’s Technical Working Group (TWG).^[5] In addition, preliminary findings of the site visits and data reviews were presented at discussion sessions held at three national conferences: National Indian Health Board, National DHHS/DOI Tribal Self-Governance meetings, and National Congress of American Indians.  The issues raised and perspectives of the TWG and participants in the discussion groups are reflected in the findings presented throughout this Final Report. 

Section 2 of this Report provides an overview of considerations for the feasibility of conducting an evaluation of DHHS programs managed by Tribes under Self-Governance and describes key feasibility issues that guide the study. The study methodology and background information activities conducted are described in Section 3.  Findings from the site visits and data reviews are presented in Section 4 and results of the discussion groups held at national conferences are presented in Section 5.  In Section 6, three illustrative evaluation models are described and issues and considerations for the feasibility of conducting these alternative evaluation models are discussed. 

Issues and Considerations for Evaluation Feasibility

Overview

The Evaluation Feasibility Study was initiated to provide information to the Department of Health and Human Services on the potential to evaluate DHHS programs that may be managed by Tribes under Self-Governance, if a demonstration program were to be authorized by Congress.^[6] Most demonstration programs within the Department frequently are developed with an evaluation component that is designed to assess the program’s operations, processes, and outcomes. 

The Department of Health and Human Services determined that examining the potential for evaluating DHHS programs that may be managed by Tribes under a potential new Self-Governance demonstration would be useful.  The focus of this planning study was to determine whether it would be feasible to conduct an evaluation that included quantitative measurement of process and outcomes, as well as qualitative information on program objectives, implementation, and operations.

Early in the project, it became clear that designing an evaluation of DHHS programs operated under Tribal Self-Governance was a more complex task than is the norm for other DHHS programs.  Tribal Self-Governance is intended to allow Tribes to manage their own programs with flexibility and with minimal requirements for reporting to Federal agencies.  The most common approach to evaluation requires substantial data collection that is consistent and comparable across organizations participating in a demonstration and a set of common objectives/outcome measures that may be the focus of the evaluation.  The goal of flexibility that is a principle of Self-Governance offers challenges in determining a consistent, comparable set of outcome measures and in defining an appropriate methodology for measuring outcomes across participating Tribes. The nature of the government-to-government relationship between the Federal government and individual Tribes also requires consultation and agreement on the type and the extent of any evaluation program. The Technical Working Group (TWG) stressed the importance of these issues at the initial meeting with the project team in February 2003.  In addition, the TWG members also stressed that most Self-Governance Tribes are convinced that Self-Governance is a successful and effective means for delivering services to Tribal members. 

Generally, many Tribes appear to be supportive of and willing to participate in any potential evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration.  Guidance provided by the TWG for assessing the feasibility of such an evaluation emphasized that there are a number of issues, however, that needed to be addressed in determining the feasibility of any particular evaluation model. 

First, the TWG stated it is important that the design of any evaluation be clear in communicating the objectives and benefits of the evaluation to Tribes and provide assurance that evaluation would not have detrimental effects on the future of Self-Governance.  The TWG also recommended that design of any evaluation should take into consideration that the underlying goal of Self-Governance is to offer Tribes flexibility to structure programs to better meet local priorities and, consequently, any evaluation that examines a set of outcomes solely Federally-determined would be inappropriate.  Instead, there should be an emphasis on flexibility in setting outcomes to be measured based on Tribal priorities and objectives.

The TWG also noted that no evaluation method should require Self-Governance Tribes to provide more extensive reporting than that required of Tribes carrying out programs under current legal authority.  Nor should any evaluation method impose extensive and burdensome data reporting on Tribes, rather than the minimal reporting that is one of the principles of Self-Governance.  If the burden on Tribes, and associated costs of data collection, are more than nominal, then an evaluation design should include funding to support the costs.  In addition, if a specific evaluation design includes extensive data collection, it is essential that there be clear and specific limitations on the time period for the evaluation and a specified date upon which data collection and reporting related to the evaluation would end.

Finally, members of the TWG noted that Self-Governance is an evolving process with new programs.  It may take two or three years for a Tribe managing a new program to identify priorities and develop more effective operational and management strategies.  Therefore, any potential evaluation should be designed to take into account the evolution of the program and the fact that Tribal goals and priorities may not be determined until the second or third year.

These issues and considerations guided the development of the evaluation feasibility study and the analysis of the feasibility of alternative evaluation approaches.

Perspectives on Evaluation Research:  Congress, DHHS, and Tribes

When Congress authorizes a demonstration program, it is generally with an understanding that it will be designed and implemented for a limited period of time to determine whether the new approach can effectively and efficiently meet specific goals.  Evaluation of demonstration programs provides information and evidence on the process through which the programs are implemented, operational issues, and impacts and outcomes of the demonstration programs, relative to the goals of the programs. Evaluation findings may also provide information that can be used to refine and improve the demonstration program as it transitions to permanent status.  Because evaluation is an accepted tool for assessing new programs, Congress often requires that an evaluation be conducted of new demonstration programs that it authorizes.

Agencies within the Department of Health and Human Services may have an interest in evaluating new strategies for delivering services to target populations. Management and program staff who have responsibility for specific programs have often worked in their fields for many years to develop program designs they believe are effective to provide services and meet defined needs of the population they serve.  When a new delivery strategy is undertaken, there may be concerns about whether the program objectives will continue to be achieved and whether the target population will be as well served as it was when DHHS dictates the strategy for program management.  Evaluations of how the newly structured delivery systems operate and meet the needs of the target population may allay concerns and provide increased support for the new delivery model. 

Tribes may perceive evaluation as having the potential to provide support for and evidence regarding the program improvements that Tribes can achieve through Self-Governance.  Evaluation may also provide useful information on innovative management approaches and on “best practices” that could be adopted by more Tribes and other recipients of Federal program funds. In addition, evaluation findings could provide evidence that suggests that additional funding of programs would be warranted in order for Tribes to achieve specific objectives and desirable outcomes.

Potential Benefits of an Evaluation

If an evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration were to be conducted, it may have several benefits both to DHHS and to Tribes.  DHHS program managers and staff which have had responsibility for administering programs that the Tribes would manage under a demonstration would perceive an evaluation as consistent with the normal approach to assessing the effectiveness and success of any new program arrangements undertaken within the Department.  Evaluation could provide evidence that the needs of clients are being met under the new management structure, even though some aspects of the program may be different than under direct Federal program management.  In addition, results of an evaluation could provide information that would increase understanding of Tribal issues and goals among DHHS program managers and staff and the benefits and successes of Tribal Self-Governance in better meeting the unique needs of Tribal members.

Evaluation of DHHS programs managed under Tribal Self-Governance could provide useful information to Tribes, as well.  Results could provide information on “best practices” and innovative programs that could be used by other Tribes to improve services and performance in program management.  There is also the potential for the findings to demonstrate that Tribal Self-Governance is an effective method for improving services and meeting needs of individual Tribes and Tribal members that would provide support for further expansion of Self-Governance to additional DHHS and other agencies’ programs.

Feasibility Considerations

The feasibility of conducting an evaluation would be dependent on a number of issues and considerations.  These include:

• Appropriate Comparison Groups.  Agreement by DHHS and Tribes on appropriate and acceptable – to DHHS and to Tribes – comparison groups are critical to the feasibility of any potential evaluation.  Evaluation research involves comparing a new program’s operations and impacts relative to what would have been observed in the absence of the new program.  Pre-post comparisons are generally accepted approaches, but do not take into account underlying trends and changes that may affect what is observed in the new program.  External comparison groups are usually defined and examined to adjust pre-post data for any outside trends that may affect programs. 
• Data Availability.  The availability of data for the pre-post demonstration period is a necessary condition for conducting an evaluation of DHHS programs operated by Tribes under a Self-Governance demonstration.  Similarly, if external comparison groups are to be used, comparable and consistent data must be available for the relevant time periods of the evaluation.
• Costs to DHHS and to Participating Tribes.  Costs of any evaluation approach considered are an important consideration in assessing evaluation feasibility.  Some evaluation alternatives may involve much higher costs than others that might be prohibitive.  Trade-offs may be considered between the comprehensiveness and rigor of evaluation alternatives and costs and the potential value of the findings that may be produced.
• Tribal Support for an Evaluation.  The likelihood that Tribes will support an evaluation will be affected by the goals of the evaluation, the nature and extent of consultation between the Tribes and DHHS on the goals and processes of the evaluation, and the costs and burden of participation to the Tribes. In addition, Tribal support may also depend on the total costs of an evaluation, even if these costs are fully borne by DHHS, relative to the potential value and benefits that may be obtained from the evaluation.

An additional important feasibility issue is the timing of an agreement between DHHS and the participating Tribes on the evaluation issues to be addressed and on the data that will be provided by participating Tribes for the evaluation.  This agreement would be essential prior to the implementation of the new demonstration program to assist participating Tribes to be aware of data requirements and to establish procedures to collect data from the initiation of the demonstration. In the absence of agreement prior to the demonstration and specification of necessary data, it would be difficult to ensure that appropriate and necessary data would be available for the evaluation

Study Methodology

Approach to Assessing Feasibility of an Evaluation

The Evaluation Feasibility Study involved the following activities:

• Establishment of a Technical Working Group that provided ongoing input and review of interim study products, as well as guidance on study objectives and processes;
• Broad communication about the project to all Tribal leaders, through mailings, a DHHS website (http://aspe.hhs.gov/SelfGovernance/Evaluation/actions.htm), and organization of discussion groups at national conferences:
• Extensive background information collection and summaries of findings;
• Review of DHHS program data and reporting requirements; and
• Conduct of site visits to six Tribes to collect background information and review data availability that might support an evaluation, if one were to occur.

Overview of Background Information Collection

During the initial months of this project, a substantial amount of information was assembled by the project team, as background for understanding and laying the groundwork for  the Evaluation Feasibility Study.  Each of these activities and the associated reports that were prepared are described in this section.^[7]

Review of Previous Research on Tribal Self-Governance/Tribal Management of Federal Programs

The Literature Review was conducted to provide a foundation for the development of the evaluation issues and related data requirements to guide the design of the feasibility study.  The objectives of the literature review included:

• Identification of existing studies and evaluations of Tribal Self-Governance and/or Tribal management of health and social service programs;
• Identification of data limitations and other factors that pose barriers to conducting comprehensive evaluations of Self-Governance and Tribal management of health and social service programs.
• Review of the methodologies and data sources used in previous studies, in order to assess both analytic rigor and the extent to which the findings are of general application;
• Synthesis of  available evidence and findings from existing studies; and
• Assessment of the implications of these findings for the Tribal Self-Governance Evaluation Feasibility Study.

Tribal Self-Governance of Indian Health Service programs have been in place for a decade and extensive detailed data are submitted by Self-Governance Tribes to IHS.  These data would make it possible to conduct a quantitative evaluation of Indian Health Service programs managed by Tribes under Self-Governance.  The evaluations of Self-Governance of IHS programs that have been conducted to date, however, have been primarily qualitative. There has been limited evaluation of other programs managed by Tribes under either Self-Governance or under contracts and grants. 

Most of the studies identified that have evaluated Self-Governance or Tribal management of health or social programs employ qualitative techniques, such as key informant interviews, which rely on stakeholders’ perceptions to reach conclusions about program effectiveness.  These qualitative studies provide insight into how various Tribes structure their health and social service programs, the characteristics of Tribal residents participating in these programs, and successes encountered in program implementation.  Further, these studies – particularly those that focused on the Tribal Temporary Assistance to Needy Families program – effectively highlight how social and economic conditions on Reservations, such as the high rate of poverty, high unemployment rates, and the lack of an economic base, may pose substantial barriers to achieving the intended goals of these programs.  These studies do not, however, provide definitive quantitative evidence on the extent to which and how Tribally-managed health and social service programs have operated to better meet the needs of their members. Most studies do not incorporate a comparison group in their design and, therefore, it is not possible to determine how persons participating in Tribal programs fare compared to how they would have fared if control over these programs were still vested with the Federal or State government.

Previous research on process, structure, and impacts of Tribal management of health and social services is limited in major ways:  1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not useful to other Tribes or other Federal grantees; and 3) adequate and comparable data across Tribally-managed programs and between Tribally-managed programs and Federal and State managed programs are not available.

Given these findings, a primary focus of the Evaluation Feasibility Study was to review and identify potential sources of data that would be adequate to permit a quantitative evaluation of relevant issues. 

Legislative History and Development of Tribal Self-Governance

In addition to the literature review, a summary of the legislative history and development of Tribal Self-Governance was prepared to provide background for understanding the context within which Self-Governance has evolved and the underlying principles on which the development of Tribal Self-Governance has been based.  The authority for Tribal management of Federally-funded programs was initially provided by Congress under the Indian Self-Determination and Education Assistance Act of 1975 (P.L. 93-638).^[8] Title I of that Act authorized Tribes to assume management of Bureau of Indian Affairs/DOI and Indian Health Service programs through contractual agreements.  From 1975 to the present, Congress has expanded the opportunities for Tribes to manage their own programs and has increased the degree of Tribal authority and discretion in management.  P.L. 100-472 amended P.L. 93-638 in 1988 to add Title III, which authorized the Tribal Self-Governance Demonstration Project that allowed Tribes to assume greater control over BIA programs that they managed, including consolidation and re-design of programs to better meet individual Tribal priorities and needs.

In 1992, as part of P.L. 102-477, Congress extended the Title III self-governance demonstration to provide for Tribal self-governance of Indian Health Service programs.  In that same Act, Congress created a program that permitted Tribes to consolidate employment and work-related programs from four Federal agencies.   Based on the perceived success of these demonstration projects, notwithstanding the lack of quantitative evaluations noted in this Report, Congress made Tribal self-governance authority permanent within BIA in 1994 (Title IV), and made permanent Tribal self-governance of IHS programs in 2000 (Title V).

Tribes manage other DHHS health and social service programs, under competitive contracts and formula grants.  These Tribally-managed programs do not generally offer the flexibility of program design and use of funds that self-governance provides to Tribes, and often require extensive program-specific application processes and detailed separate reporting requirements.

Tribal Matrix of Programs Managed Under Contracts and Compacts

As background for the Evaluation Feasibility Study, OASPE was interested in determining the extent to which Tribes are currently managing DHHS or other Federally-funded programs under compacts, contracts, and grants.  These other programs include programs of the Department of the Interior, Bureau of Indian Affairs carried out under the “477” program (P.L. 102-477).^[9]  The information on Tribes that are currently managing programs also provided background information for recruiting six Tribes to participate in the site visit component of the Evaluation Feasibility Study.

The construction of the Tribal Matrix and identification of programs that are managed by each Tribe required: 1) identification of each Federally-recognized Tribe (including those Tribes in Alaska and other States that have authorized a Tribal organization to carry out programs on their behalf); and 2) identification of data sources and individuals in the Federal government that could provide information on Tribal management of the specific DHHS and BIA programs of interest.

A complete list of all Federally-recognized Tribes was obtained from the Federal Register^[10].  This list was then cross-referenced with Indian Health Service information to match Tribes in Alaska and other States to the Tribal organizations they have authorized to carry out programs on their behalf.

Data were obtained from BIA staff on Tribal management of BIA programs under Title I self-determination contracts and under Title IV Self-Governance compacts, as well as Self-Governance under P.L. 102- 477 provisions. Project staff searched each DHHS program area web site, as an initial step, to determine whether the program maintained a list of Tribes and Tribal organizations that hold contracts, grants, or compacts to manage specific programs.  Then, direct telephone contacts were made with DHHS staff associated with each program area to verify the accuracy of information obtained from the web site or to request information on Tribal management of programs. For five programs, the information required was maintained on the Federal agencies’ web sites; information was provided by program staff for the remaining seven DHHS programs.

Tribal Population Characteristics and Related Data

To provide additional background information for the study, data were compiled from relevant sources that provided Tribal/Tribal organization-specific information on population size, age and gender distribution, socioeconomic characteristics, and any other variables that might be useful for describing and comparing Tribes.  This data compilation was intended to provide information for selecting and describing Tribes for the site visit component of the study, as well as information that could be useful to OASPE if an evaluation was conducted at some future time.

The Data Report provides information on a range of demographic and socioeconomic data that are useful for characterizing Tribes that manage both health and social service programs. Because of this broad interest, the Data Report is a compilation of information that may be relevant to Tribal management of many programs. 

Data sources that were used to develop the population and other characteristics, by Tribe, included the 2000 Census and documents from the Bureau of Indian Affairs and the Indian Health Service. There are serious limitations of these data sources and no good solutions are available to ensure that complete, accurate, and comparable data can be assembled for each Federally-recognized Tribe.  Despite these limitations, the Data Report provides some useful information on socio-economic and demographic characteristics of specific Tribes.

Review of Reporting Requirements for DHHS Programs That May Be Operated by Tribes Under a Self-Governance Demonstration

Current reporting requirements for each of the DHHS programs that have been identified as feasible to include in a potential new Self-Governance demonstration were also investigated by talking with several DHHS and Tribal program staff and reviewing the documents provided to us by the Tribes visited.  There is considerable variation in the level of detailed reporting that is required by different DHHS programs.  Some programs (e.g. Head Start) require lengthy and very detailed reporting on a large number of programmatic and outcomes measures.  Other programs do not have specific reporting requirements beyond minimal financial reports.  In addition to formal reporting requirements, some programs provide monitoring through ongoing contact between program staff and contracting Tribes. The current reporting requirements are described below.^[11]

Tribal Temporary Assistance for Needy Families (TTANF).     Currently, Tribes managing this program are required to submit (electronic submission is preferred) family-level and individual-level data elements for families receiving TTANF. For the family, these data elements include funding stream, number of family members, type of family for work participation, receiving subsidized housing, receiving medical assistance, receiving food stamps and amount, receiving subsidized child care and amount, child support, and family cash resources.  At the individual level, Tribes are required to submit characteristics such as adult and minor child head-of-household characteristic such as date for birth, ethnicity, gender, receipt of disability benefits, marital status, relationship to head of household, parent with minor child in the family, needs of pregnant women, educational level, citizenship, cooperation with child support, employment status, and work participation status.  The child characteristics submitted by TTANF grantees include family affiliation, race/ethnicity, gender, receiving disability benefits, relationship to head of household, educational level, amount of unearned income,

Low Income Home Energy Assistance Program.     Currently, Tribes managing this program are required to provide the Household Service Report—Short Format or a letter containing similar information.  This information includes number of households receiving the following types of assistance: heating, cooling, winter/year round crisis, summer crisis, or weatherization.

Community Services Block Grant.     Currently, there is no specific Federal required reporting form that documents services or activities conducted for this program.

Child Care and Development Fund.     The standard Child Care and Development Fund annual aggregate program report (ACF-700 form) requires the following information: number of families and children receiving services, age breakdown for children receiving services, reasons for needing childcare (e.g., working, in school), number of hours services provided, amount of CCDF subsidy, amount of parent co-payment, and poverty status of families receiving services.  A supplemental narrative report to the ACF-700 form requests general child care information, and is not restricted to CCDF-funded activities.  Financial reporting is required on the ACF-696-T form.

Native Employment Works.     Current Federal reporting requirements include a Program Report with a narrative section that compares achievements for the year to their plan for the year.  It also summarizes significant barriers to implementation, provides explanations for variances with the plan, and describes actions taken.  The Program Report also includes a statistical report that provides the following information: number of clients served characteristics of clients served (e.g., age, sex, TANF recipients), number of clients participating in types of NEW activities and services (e.g., classroom training, on-the-job training, counseling), and number of clients with selected outcomes (e.g., GED, unsubsidized employment).

Head Start.     The standard Head Start reports includes information in the broad categories of children enrolled by demographics, staff information by demographics, information on classes/ groups/ centers, volunteer information, and services provided. Head Start also currently has a requirement for extensive outcome measurement. 

Child Welfare Services.     As reported to us, there are no specific standardized reporting requirements; however each grantee must report how they are progressing toward their 5-year plan.

Promoting Safe and Stable Families.     Like other Child Welfare programs, we are aware of no specific reporting requirements beyond reporting concerning progress toward planned activities.

Family Violence Prevention: Grants for Battered Women’s Shelters.     Current Federal reporting requirements include narrative reports on number of clients served and on services provided.

Administration on Aging (AoA) Grants for Native Americans.     The standard report for AoA includes information on full-time/part-time staff; program resources and expenditures, including sources of income other than grants; unduplicated numbers of people that receive support services, congregate meals, home-delivered meals; total numbers of congregate and home-delivered meals; units of supportive services, legal services, at-home services, ombudsmen services, and others.

SAMHSA Targeted Capacity Expansion Grants.     Current Federal reporting requirements include a quarterly report and specified measures related to the Government Performance and Results Act (GPRA).  The quarterly and GPRA reports include the following information: grantee information; staffing information; data including number of new clients, services provided and individual-level information on the clients as required by SAMHSA to meet their GPRA reporting requirements^[12]; and narrative information about the project such as challenges and successes over the past quarter.

Methods for Obtaining Input from Technical Working Group and Others

The Evaluation Feasibility Study is just one component of the DHHS efforts to support effective Tribal Self-Governance.  DHHS consulted with Tribes on a variety of issues related to the Title VI Self-Governance demonstration feasibility study that was mandated by Congress to examine potential DHHS programs for Tribal Self-Governance and which has now been submitted to Congress.  A Title VI Advisory Group, comprised of Tribal Leaders with a commitment to Self-Governance and their designees, worked closely with DHHS throughout that earlier process.  In addition, DHHS made information regarding the Title VI Self-Governance demonstration feasibility study available to all interested persons through its website and through direct mailings to Tribes and other stakeholders.

The current Evaluation Feasibility Study has continued this practice of active consultation and information dissemination.  Specific communication activities that were undertaken included:

• A project description, explaining the project and its objectives, was sent to all Chairpersons or Presidents of Federally-recognized Tribes, with a background letter.  The packet sent included contact information for the DHHS Task Order Managers (TOMs) and for each of the Co-Principal Investigators, as well as the DHHS website address where ongoing information about the study and its progress is maintained.  Interested individuals were encouraged to provide comments and inquiries, through telephone or email to the OASPE Task Order Managers and/or the Co-Principal Investigators.
• The Task Order Managers and Co-Principal Investigators conducted presentations and question-and-answer sessions on the project at national or regional AI/AN meetings, including the National DHHS/DOI Tribal Self-Governance meetings in San Diego (November 2002) and Palm Springs (October 2003) and the Alaska Native Health Board meeting in August 2003.
• The Technical Working Group was established and met in February 2003 to review and comment on Draft Reports, the Project Work Plan, and to provide guidance to the project.  Monthly conference calls were conducted with the TWG to discuss interim reports and progress on the project, from March through August 2003.  The Technical Working Group met with the project team on December 9-10, 2003 to review and discuss the Draft Final Report on the project.
• Discussion sessions were arranged and held at the National Indian Health Board conference, National DHHS/DOI Tribal Self-Governance meetings, and NCAI conference to obtain broader Tribal comments and suggestions on the study objectives and preliminary findings.
• At the end of the project, the summary of the Final Report will be mailed to all Tribal Chairpersons with contact information for the OASPE Task Order Managers to encourage further comments and suggestions to DHHS.

The goals of all of these communication and information dissemination activities have been to ensure that the project and its development was conducted as an open and ‘transparent’ process and to glean the maximum useful advice and input from the Tribes in the conduct of the study, while seeking to assure a balanced inquiry and lead to valid and objective advice to DHHS. 

Methodology for Site Visits and Tribal Data Review

Site visits were made to five Tribes and one Tribal organization to assess the feasibility of conducting an evaluation of DHHS programs managed under Tribal Self-Governance.  The site visits focused on determining the extent to which there are historical documentation and knowledgeable individuals who are able to provide background and information on the development and goals of Tribal management of Federal programs, the Tribes’ management information systems capabilities, and the availability, sources, and completeness of data on each program managed by Tribes.  The two-day site visits were conducted by a two or three-person team during late June through August 2003. 

While selection of site visit participants depended on availability of willing volunteers, the Technical Working Group and DHHS developed several criteria for selecting Tribes for the study.  These criteria were:

• All Tribes selected should have three or more years of experience with compacting or contracting IHS health programs.
• Preference should be given to Tribes that have experience in managing, under contracts or grants, the DHHS programs determined to be feasible to include in a Self-Governance Demonstration of non-IHS DHHS programs.
• In addition, the sites selected, to the extent possible, were to reflect the following:
• Geographic diversity, reflecting the distribution of the AI/AN population;
• Variation in the size of the Tribe’s population; and
• Variation in economic conditions (i.e., average income, employment levels) of the Tribe.

These criteria were substantially met in the selection of the Tribes that participated in the site visits.

The process developed by OASPE, the Technical Working Group, and the project team for recruitment of Tribes to participate in site visits included:  1) a presentation on the project by the OASPE Project Officer at the Tribal Self-Governance meetings held in late April 2003 in Phoenix; 2) distribution of a letter of invitation to all Tribes to participate, and relevant background materials, at the Tribal Self-Governance meetings; and 3) mailing of the letter of invitation and background materials to all Tribal leaders during the last week in April.  Tribes interested in participating in the site visit were asked to contact the project staff by May 20, 2003 to indicate their interest and/or to obtain additional information.

The site visits were conducted in July and August 2003 to the six Tribes/Tribal organizations listed here:

• Bois Forte Band of Chippewa (Minnesota)
• The Choctaw Nation (Oklahoma)
• Port Gamble S’Klallam Tribe (Washington)
• Yukon-Kuskokwim Health Corporation (Alaska)
• Hopi Nation (Arizona)
• Little Traverse Bay Band of Odawa Indians (Michigan)

Methodology for Small Group Discussions

The Small Group Discussions, conducted by Kauffman and Associates, Inc. (KAI), brought together Tribal leaders, experienced Tribal program managers and technical experts in Self-Governance program management to provide feedback and response to preliminary findings and conclusions related to this study.  These discussions provided another means for review and analysis of draft findings and conclusions.  A qualitative analysis of these discussions was conducted to identify “major themes and issues” that emerged across the board.  These major themes informed the study team and the Technical Work Group prior to finalizing reports.

People recruited to participate in the Small Group Discussions included Tribal leaders, Tribal management and technical staff with direct experience in the administration of Self-Governance compacts, including financial managers, management information system (MIS) directors, legal or regulatory analysts, program administrators and related positions. Sign-up sheets were provided prior to each group to make sure we had the appropriate mix of expertise in each session and adequate space to conduct each session.

The Small Group Discussions occurred between September and November 2003.  Locations for these discussion groups included the annual consumer conference of the National Indian Health Board, September 29-October 2, 2003, in St. Paul, MN; the National DHHS/DOI Tribal Self-Governance meetings, October 6-10, 2003 in Palm Springs, CA; and. the annual convention of the National Congress of American Indians, November 16-21, 200, in Albuquerque, NM.  At both the NIHB conference and the DHHS/DOI Tribal Self-Governance meetings, a separate room or break-out session was provided to conduct these discussions.  Rich and substantial qualitative data was collected during these sessions.  Less effective was the session conducted at the NCAI gathering, where this topic was one of several on a busy agenda.  Comment sheets were distributed but few turned in from the NCAI event.  The majority of comments reflected in this report are from the NIHB and the Self-Governance meetings.

Each discussion group involved 10 to 25 individuals representing a mix of interests and experiences from Tribes and Tribal organizations, including both Self-Governance and non-Self-Governance Tribes.  Individuals were recruited through fliers, inserts in conference packets or by appearing on the conference agenda as a workshop option. 

Topic Areas and Prompt Questions:

KAI staff facilitated these discussions.  An overview was provided and a written summary of the Draft Findings and Conclusions distributed for review. The following discussion guide was generally followed, however the flow of conversation generally centered upon three main topics:  (1) reaction to the draft findings and conclusions; (2) omissions in the draft; and (3) best outcomes for this feasibility study.  The following are the questions in the formal Discussion Guide:

1. Facilitator describes what this feasibility study did and did not do:
   • Facilitator explains why this study was done. (purpose)
   • Facilitator describes the difference between an evaluation feasibility study and an actual program evaluation.
   • Q & A regarding this overall study
2. Identification of feasible alternatives for measuring success
   • What are participants’ reactions to the options proposed for measuring success?
   • What other means exist to evaluate success?
   • How would this differ between IHS and other HHS programs?
3. Facilitator reviews the preliminary findings and recommendations
   • Do these preliminary findings and recommendations reflect your experiences with SG?  Give examples of why or why not.
   • What’s missing?
4. What is the best outcome from your perspective for this feasibility study?
   • How do you see these preliminary findings or recommendations impacting future opportunities to expand SG compacting to other programs of HHS?
   • How can this study help local planning for SG?
   • Any other comments?

Site Visit Findings

Overview

These findings reflect the data collected at the six sites visited.  By agreement with the sites, this report does not present information about a single site by name nor does it compare one site to another or to all others.  Instead, the report provides overarching conclusions based on all six site visits and notes important exceptions to these conclusions. 

Specific findings about the availability of data on the history of Self-Governance and program management as well as management processes is presented below, followed by information on data availability for DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration. Following these discussions is a general summary of the findings as they relate to recommendations for the feasibility of a more quantitative evaluation to document the outcomes and successes of DHHS programs that may be included in a new demonstration.

Availability, Accessibility, and Quality of Data on History of Self-Governance and Program Management

When meeting with Tribal members at all six sites these issues were discussed:

• Individuals who have been involved in Self-Governance/Tribal management of Federal programs since these programs were first considered;
• Written documentation and reports that chronicle the initial steps that were taken when the Tribe first considered Self-Governance/management of Federal programs; and
• Individuals who have knowledge and information on the goals/objectives of the Tribe for the program, the extent to which those goals/objectives have been met, and how those goals/objectives have changed over time.

  Name

  Contact Information

  Email

  W. Ron Allen
  Tribal Council Chair

  Jamestown S’Klallam Tribe
  1033 Old Blyn Highway
  Sequim, WA 98382
  (360) 683-1109 Phone
  (360) 681-3405 Fax

  rallen@jamestowntribe.org;

  Julia Davis-Wheeler

  (alternates)
  Jim Roberts or Ed Fox at
  NPAIHB or
  J.T. Petherick at NIHB

  National Indian Health Board
  (208) 843-2965 Home
  (208) 843-2965 Home Fax

  Northwest Portland Area Indian Health Board
  627 SW Hall, Suite 300
  Portland, OR  97291

  jdavis@npaihb.org;

  JRoberts@npaihb.org;
  Edfox_phd@yahoo.com;
  jpetherick@nihb.org;

  Barbara Fabre
  Director of Tribal Child Care Services

  White Earth Band of Chippewa
  P.O. Box 418
  White Earth, MN 56591
  (218) 983-3285 ext. 407

  barbf@whiteearth.com;

  Dan Jordan
  Tribal Self Governance Coordinator

  Hoopa Valley Indian Reservation
  P.O. Box 1348
  Hoopa, CA 95546
  (530) 625-4211 Phone
  (530) 625-4594 Fax

  djordan@pcweb.net;

  Violet Mitchell-Enos
  Human Services Department Director

  Salt River Pima Maricopa Indian Community
  10005 Osborne Road
  Scottsdale, AZ 85256
  (480) 850-8410 Phone
  (480) 850-7373 Fax

  violet.mitchell-enos@srpmic-nsn.gov;

  Myra Munson
  Attorney

  Sonosky, Chambers, Sachse, Miller and Munson
  318 4^th Street
  Juneau, AK 99801
  (907) 586-5880 Phone
  (907) 586-5883 Fax

  myra@sonoskyjuneau.com

  Mickey Peercy
  Deputy Director of Health

  Choctaw Nation
  Drawer 1210
  Durant, OK 74702
  (580) 924-8280 Phone
  (580) 924-3393 Fax

  mpeercy@choctawnation.com;

  Anna Whiting Sorrell
  Director of Support Services

  Salish & Kootenai Confederated Tribes
  P.O. Box 278
  Pablo, MT 59855
  (406) 675-2700 Phone
  (406) 675-2806 Fax

  annas@cskt.org;

  Alvin Windy Boy
  Chairman

  (alternate)
  Tim Martin
  Executive Director

  Chippewa-Cree Tribe of Rocky Boy
  Rocky Boy Route, Box 544
  Box Elder, MT  59521
  (406) 395-4210 Work
  (406) 395-4497 Fax
  (406) 868-2134 Cell

  United South and Eastern Tribes (USET)
  711 Stewarts Ferry Pike, Suite 100
  Nashville, TN 37214
  (615) 872-7900 Phone
  (615) 872-7417 Fax

  alvin@cct.rockyboy.org;

  jtmartin@usetinc.org;

  Tribal Leader to be Named

  (alternates)
  Jackie Johnson
  or Lillian Sparks

  National Congress of American Indians
  1301 Connecticut Avenue NW
  Washington, DC 20036
  (202) 466-7767 phone
  (202) 466-7797 fax

  jjohnson@ncai.org

  lsparks@ncai.org

  
  

  Site Visit Data Collection Forms

  1. History of Tribal Self-Governance/Tribal Management of Federal Programs

     1. Are there individuals in the Tribe who have been involved in Self-Governance/Tribal management of Federal programs since these programs were first considered?  
                                       

        YES

        NO

                                                       
        • IF YES, who are they?
        • Are these individuals willing to and available to be interviewed, if an evaluation were to be conducted?                  

          YES

          NO

     2. Are there written documentation and reports that chronicle the initial steps that were taken when the Tribe first considered Self-Governance/management of Federal programs?                                

        YES

        NO

         

        YES

        NO

        • • IF YES, would these documents and reports be available to be reviewed, if an evaluation were to be conducted? 
        
        
        • For each Tribally-managed program, are there individuals who have knowledge and information on the goals/objectives of the Tribe for the program, the extent to which those goals/objectives have been met, and how those goals/objectives have changed over time?  

          TTANF	YES	NO	N/A	IF YES, who are they?
          HEAD START	YES	NO	N/A	IF YES, who are they?
          LOW-INCOME HOME ENERGY ASSISTANCE PROGRAM	YES	NO	N/A	IF YES, who are they?
          COMMUNITY SERVICE BLOCK GRANT	YES	NO	N/A	IF YES, who are they?
          NATIVE EMPLOYMENT WORKS	YES	NO	N/A	IF YES, who are they?
          CHILD CARE AND DEVELOPMENT FUND	YES	NO	N/A	IF YES, who are they?
          CHILD WELFARE PROGRAMS	YES	NO	N/A	IF YES, who are they?
          PROMOTING SAFE AND STABLE FAMILIES	YES	NO	N/A	IF YES, who are they?
          FAMILY VIOLENCE PREVENTION AND SERVICES GRANTS FOR BATTERED WOMEN’S SHELTERS	YES	NO	N/A	IF YES, who are they?
          ADMINISTRATION ON AGING:  GRANTS TO NATIVE AMERICANS	YES	NO	N/A	IF YES, who are they?
          SAMHSA TARGETED CAPACITY EXPANSION GRANTS	YES	NO	N/A	IF YES, who are they?
          HEALTH SERVICES	YES	NO	N/A	IF YES, who are they?

                  
        • For each Tribally-managed program, is there written documentation and reports that reflect the issues that were considered before the application for management of the program and the key factors that were considered? 
        • For each program, are there periodic written reports and documents that describe the implementation of the program, operational structure and changes over time, and services provided?

          TTANF
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          HEAD START
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          LOW-INCOME HOME ENERGY ASSISTANCE PROGRAM
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          COMMUNITY SERVICE BLOCK GRANT
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          NATIVE EMPLOYMENT WORKS
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          CHILD CARE AND DEVELOPMENT FUND
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          CHILD WELFARE PROGRAMS
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          PROMOTING SAFE AND STABLE FAMILIES
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          FAMILY VIOLENCE PREVENTION AND SERVICES GRANTS FOR BATTERED WOMEN’S SHELTERSFAMILY VIOLENCE PREVENTION AND SERVICES GRANTS FOR BATTERED WOMEN’S SHELTERS
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          ADMINISTRATION ON AGING:  GRANTS TO NATIVE AMERICANS
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          SAMHSA TARGETED CAPACITY EXPANSION GRANTS
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
          HEALTH SERVICES
          	Documentation of issues at application:	YES	NO	N/A
          	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A

  2. Effects of Tribal Self-Governance/Tribal Management of Federal Programs on Tribal Government and Management Processes
     1. Are there individuals in the Tribe who have been involved in Tribal governance and management for a sufficiently long period that they can describe the changes in Tribal governance and management that occurred as a result or in association with the Tribe’s undertaking management of Federal programs?                  

        YES

        NO

        
        IF YES, who are they?
     2. Are there written sources of information that describe the structure and operations of Tribal government prior to Tribal Self-Governance/management of Federal programs and the changes that have occurred over time?                                 

        YES

        NO

     3. What would you suggest as a strategy for assessing the effects of Tribal Self-Governance/management of Federal programs on Tribal government, management, and on community involvement?

  QUANTITATIVE DATA COLLECTION FORM

  1. What kinds of information are you collecting? 

     			COMMENTS	ATTACHMENT
     Do you collect detailed information on your clients/beneficiaries (number served, characteristics)?	YES NO			YES
     Do you collect information about the services provided to each client/beneficiary?	YES NO			YES
     Do you collect information about outcomes for each beneficiary?	YES NO			YES
     Do you have data on number of full-time and part-time personnel and on personnel ‘turnover’?	YES NO			YES
     Do you have cost information by cost component (administrative costs, personnel costs, other costs, by type)?	YES NO			YES
     Do you have records of funding allocations over time?	YES NO			YES
     Do you have records of funding allocations over time?	YES NO			YES

  2. How long have you been collecting this information? 
     
     Are these data available for each year since the Tribe began managing program?

     YES

     NO

  3. How and where is the information stored?
  4. If electronic storage (as opposed to paper files), how is it entered? 
     
     How frequently is it entered?
  5. If electronic, what kinds of formatting information and documentation do you have?
     Are file specifications available? 

     YES

     NO

  6. How could these data be provided for an evaluation (e.g., data file of XXX type, generated reports, hard copy for review)?
  7. What would be involved in getting permission to access these data?
  8. What kinds of problems exist in these data (e.g., missing data, miscoded data, long time lag between collection and entering)?

  Matrix of Issues and Concerns Raised at Discussion Groups  at NIHB and DHHS/DOI Self-Governance Conferences

  Discussion Guide Matrix
  Do the Draft Findings and Conclusions Ring True in Your Experience?

  Financial Issues	Political Issues	Employment		Data/Measures		Systematic Problems
  Opposition to looking at “total Tribal revenues”, look at program expenditures.	Comparing SG with non-SG programs may trigger other issues, i.e. direct services vs. Title I contract Tribes.		What does ‘turn-over’ measure? Why the focus?		Possible measures include: Tribal codes and levels of community participation.		Many Tribes are members of consortia and data/finances are mixed.
  Look at program revenues not Tribe total	Evaluation should provide forum for showing SG works		See if Tribal employment increased.		Look at ‘new services’ added to programs since SG.		Many Tribes lack infrastructure prior to take-over
  Gaming Tribes will resist gov’t examination of total revenues	Fear that pressure on data will move Tribes toward regionalization.		Look at ‘institutional history’ through interviews, not turnover		Pre/Post not fair, as Tribes not receiving same level of funds as States (TANF, Child Support) upon contract		Draft Findings document assumes there will be a standard for uniform reporting across sites. This is not realistic.
  	Tribal priorities drive SG, and can prioritize ‘quality’ over ‘quantity’.		Draft Findings document suggests Tribes do not have salary data by position, but most Tribes do have this in budgets.		Draft Findings document suggests a standard of an “integrated system” which Tribes lack.  Note that neither States nor most Federal agencies have this either.
  					The responsibility to secure baseline data from States/Federal agencies prior to SG must rest with DHHS.

  Discussion Matrix
  What Omissions Do You See?

  Stories	Purpose for Evaluation	Assumption of Trust	Cost Implications
  We want to see success stories	Better description of evaluation processes applied in this study	You need to convey trust is there.  Trust of Tribes will reduce skepticism.	The cost of moving toward an “integrated MIS” must be born by the feds, if it will be used as a standard for evaluation.  It is a Tribal decision to move toward integrated MIS.
  	Address issues of evaluating small populations, numbers.	Tribal Self-Governance already assumes a level of competence.	No across the board measures. The cost of doing these measures must be covered.
  	Important to communicate that this study looks at feasibility of evaluating Federal PROGRAMS operated under SG and not the concept of SG itself.	Tribes have to determine and define their own outcomes.	Tribes should have same access to resources as States to do this work.

  Discussion Matrix
  What Are Best Outcomes of This Feasibility Study?

  Success Stories/Lessons	Terminology	Tribal Base	Better Focus	Know Limits
  Document Tribal innovations, collaborations and creativity	Move away from terms like “failure” and use terms like “challenges”	Tribes starting from level of inadequate funding and lack of infrastructure	You cannot evaluate Self-Governance, but you can evaluate Federal programs operated under SG mechanism.	You cannot do an across the board evaluation design with SG, because SG allows for Tribal innovation and priorities.
  Communicate the service ‘values’ from the community perspective, i.e. increased control, increased participation, cultural appropriateness	Find another word for “evaluation”, it has negative connotation	Tribes already have program specific reporting requirements with 11 DHHS programs.	If Tribes perform poorly in an evaluation will the feds take-away funding? Tribes want to know.	Self-Governance was intended to allow Tribes flexibility to do the most with limited resources.
  	Program evaluations under SG, not evaluation of SG.	Statistical data is already available and should be used.	Begin with minimum standards not maximum standards as base.	Understand challenges to measuring small populations, small data.
  		Look at services provided not money spent	More complex than counting users, also look at intangibles like ‘ownership’ building a base, hiring Tribal members.

  Description of Illustrative Evaluation Models

  NOTE:   These Models are presented only to illustrate possible approaches to evaluation of DHHS programs that may be managed by Tribes under a new Self-Governance demonstration and to provide a framework for the discussion of evaluation feasibility.  If a future evaluation of DHHS programs operated under a new Self-Governance demonstration would be developed, there would be extensive consultation with Tribes to develop the specific evaluation approach.

  1. Comprehensive Evaluation Model:  DHHS Programs that May be Operated by Tribes Under a Self-Governance Demonstration
     1. Objectives
        1. Conduct a comprehensive evaluation of the implementation, process, and outcomes associated with a demonstration of DHHS programs that may be operated by Tribes under Self-Governance. 
     2. Assumptions
        1. Comprehensive Evaluation of Demonstration Programs
           1. 50 demonstration Tribes
           2. Demonstration Tribes may have contracted the programs they are managing under the demonstration prior to the demonstration, or they may elect to manage programs they have not previously contracted.
           3. An Annual Report format would be developed in consultation with Tribes participating in the demonstration and would be submitted by all participating Tribes.
           4. An Evaluation Data Set would be developed in consultation with Tribes. This Evaluation Data Set would include data on characteristics of individual clients/beneficiaries served by each program, services provided/received, and observed process and outcome measures at the individual client/beneficiary level.
           5. A subset of 15-25 Tribes would agree to voluntarily submit the Evaluation Data Set annually, as well as the Annual Report., for each program and for each year of the demonstration
           6. Two rounds of site visits would be conducted to 15-25 Tribes for in-depth evaluation, once during the initial six months of implementation and again approximately 18 months after initiation of the demonstration.
           7. DHHS program offices would provide baseline reports for demonstration programs managed by the Tribes and national benchmark data for all years required.
           8. All participating Tribes would be provided uniform financial reporting formats and Evaluation Data Set reporting formats and would be provided training and technical assistance to ensure comparable and consistent data.
           9. The evaluation would be conducted over a three-year timeframe.
     3. Research Questions to be Examined
        Note:  Specific research questions would be developed in consultation with the Tribes. Based on discussions conducted during the current study, the general research questions that are likely to be identified might include:
        1. Implementation Issues (first 6 months)
           1. What are the characteristics of Tribes that apply to participate in the demonstration?  That are selected to participate in the demonstration? Are these characteristics different from those of Tribes that do not apply?
           2. What factors are reported by demonstration Tribes as influential in their decision to participate?  What was the most important factor in their decision?  What concerns were identified during the decision process?
           3. Was the community involved in the decision to participate in the demonstration?  How was this accomplished?
           4. How was the planning for the demonstration program organized?  Where was responsibility for planning placed organizationally?  Who was involved?
           5. What changes in organization and staffing of each program occurred as a result of the demonstration planning?
           6. What goals/objectives were established for each program during planning?  Were these goals/objectives different from the goals/objectives that had been in place when the programs were contracted?  If so, what are the reasons?
           7. Was there community involvement in setting goals/objectives for each program?
           8. Were changes made in the funding available to each program under the demonstration?  If so, what were the reasons for the changes?
           9. Were changes made that resulted in cost-savings or more efficient use of resources?
           10. What problems were identified during implementation and how were they resolved?
           11. Was the implementation successful?  Why or why not?
        2. Process Questions (six months and throughout the demonstration period)
           1. What changes in programs, staffing, and organization occurred after the initial implementation period?  For each program, what were the reasons for these changes?
           2. Did the demonstration affect overall Tribal management structure and staffing?  Why or why not?
           3. Were there changes in the goals/objectives for each program after the initial implementation period?  If so, what were the reasons for the changes?
           4. How does the Tribe provide oversight and monitoring of each program?  What is the process for addressing problems or issues that are identified through monitoring?
           5. Was there ongoing community involvement in oversight and monitoring of each program?  If so, how was this achieved?
           6. Are goals/objectives for each program met, on a continuing basis?  What factors are important in achieving these goals?  If the goals/objectives are not met, what were the reasons?  What changes were made in response to identifying barriers to meeting goals/objectives?
           7. What are the perceptions of Tribal leaders and program managers of the benefits of Self-Governance, generally, and as a result of the demonstration?  Are there perceived disadvantages of Self-Governance, generally, and for this specific demonstration?
           8. What are the perceptions of Tribal members who receive services from the programs of the benefits and disadvantages of the changes in management and operation of each program?
           9. Were any program changes made to achieve cost-savings and increase efficiency?
           10. Were some program funds re-allocated to other priorities within the Tribe, after the initial implementation period?  How was the decision made?
        3. Quantitative Measures of Process and Outcomes Questions (to be addressed after two years of operation)
           1. Was maintenance of effort achieved?  That is, did each program serve as many people and provide at least the same quantity of services as were available prior to Self-Governance?  If not, what were the reasons?
           2. Did the mix of services provided change under the demonstration, for each program?
           3. For each program, was the Tribe able to achieve at least two quantifiable goals that were established at the initiation of the demonstration program?
           4. For each program, were any changes made in staff levels or types of staff employed? 
           5. Were there changes in the allocation of program funds to personnel, space, materials, administrative costs under the demonstration, compared to the previous contracted program (if the Tribe previously operated the program under contract)?
           6. Did program costs per person receiving services change under the demonstration programs?
           7. Are program users more/less satisfied with services provided under the demonstration program than they were before implementation? 
     4. Comparison Groups for Quantitative Measures Questions
        1. Demonstration
           1. Pre-Post comparisons
           2. Across-site comparisons
           3. National benchmark comparisons
     5. Data Necessary for the Evaluation
        Note:  Specific data needed would depend on the set of evaluation issues and research questions developed in consultation with Tribes.  Likely data needed would include:
        1. 1. DHHS Programs Under Demonstration
           1. Annual Report data for all Tribes participating in the demonstration.
           2. Evaluation Data Set on persons served, age-gender mix, services provided, outcome measures for each year of the demonstration, for each program, for 15-25 participating Tribes
           3. Detailed financial data for demonstration Tribes, baseline through evaluation period, for each program
           4. Detailed data on staffing, for each program
           5. Consumer satisfaction survey of Tribal members receiving services from each program, baseline and second year of demonstration
           6. Two rounds of site visits to 15-25 Tribes to collect qualitative data on implementation process during first six months and again at 18 months to collect information on operational experiences
        2. Other Data Needed
        1. Socio-economic and demographic data for each Tribe (2000 Census)
  2. Limited Evaluation Model
     1. Objectives
        1. To design and conduct an evaluation that addresses a limited set of evaluation issues that are identified by the Tribes and DHHS as high priority and valuable to understanding and assessing DHHS programs that may be operated by Tribes under a demonstration.
     2. Assumptions
        1. 50 Tribes participating in new demonstration.
        2. Demonstration Tribes may have contracted the DHHS programs they are managing under the demonstration prior to the demonstration or they may elect to manage programs they have not contracted.
        3. Site visits would be conducted to 15-25 Tribes for in-depth evaluation.
        4. A Minimum Data Set (MDS) would be developed in consultation with Tribes.
        5. All voluntarily participating Tribes would agree to submit this MDS for the baseline (pre-implementation) period and for each year of the demonstration.
        6. Additional data collection would be conducted only for the 15-25 Tribes selected for in-depth evaluation.
        7. DHHS program offices would provide baseline reports for demonstration programs managed by Tribes and national benchmark data for all years required.
        8. All participating Tribes would be provided uniform financial reporting formats and Minimum Data Set reporting formats and would be provided training and technical assistance to ensure comparable and consistent data.
        9. The evaluation would be conducted over a three-year timeframe.
     3. Research Questions
        Specific research questions would be developed in consultation with the Tribes. Based on discussions conducted during the current study, the general research questions that are likely to be identified would include:
        1. What are the overall benefits to Tribes of participating in Self-Governance of Federal programs?
        2. Do the Tribes use the flexibility of Self-Governance to make changes to programs?
        3. How are decisions made about goals of programs and changes that are made to achieve those goals?  To what extent is the community involved in those decisions?
        4. Do the Tribes meet the specific goals that are established for each program?
        5. Are there innovative approaches that are developed by the Tribes that contribute to effective and efficient management of programs and resources?
        6. What problems are encountered?  How are those problems resolved?
     4. Comparison Groups for Quantitative Measures
        1. Pre-post comparisons
        2. National benchmark comparisons
     5. Data Necessary for the Evaluation
        Note:  Specific data needed would depend on the set of evaluation issues and research questions developed through DHHS consultation with Tribes.  Likely data needed would include:
        1. Baseline data on persons served, age-gender mix, services provided, outcome measures, for each program.
        2. Minimum Data Set on persons served, age-gender mix, services provided, outcome measures for each year of the demonstration, for each program.
        3. Detailed financial data on programs operated by demonstration Tribes, baseline through evaluation period, for each program.
        4. Detailed data on staffing, for each program.
        5. Two rounds of site visits to 15-25 Tribes to collect qualitative data on implementation process during first six months and again at 18 months to collect information on operational experiences.
        6. Socio-economic and demographic data for each Tribe (2000 Census)
  3. Evaluation Model Using Only AGGREGATE  Reporting Data
     1. Objectives
        1. To conduct a limited evaluation that relies on aggregate periodic reports on programs managed by Tribes under the DHHS demonstration program.
     2. Assumptions
        1. 50 demonstration Tribes.
        2. A set of Annual Report Requirements, including Financial Reporting Requirements, would be developed, in consultation with Tribes.
        3. Participating Tribes would agree to submit these reports for each year of the demonstration.
        4. DHHS demonstration program officers would provide additional qualitative information to the evaluation team on implementation and process for demonstration Tribes, based on their ongoing interactions with the demonstration Tribes.
        5. Agencies responsible for programs included in the demonstration would provide national benchmark data for baseline and for all years of the demonstration.
        6. All participating Tribes would be provided uniform reporting formats and training and technical assistance to ensure comparable and consistent data.
        7. No individual-level analyses would be conducted of program clients/beneficiaries.  All evaluation analyses and program descriptions would be conducted at the aggregate level.
     3. Research Questions to be Examined
        Note:  Specific research questions would be developed in consultation with the Tribes.  Based on discussions conducted during the current study, the general research questions that are likely to be identified might include:
        1. Implementation Issues (first 6 months)
           1. What are the characteristics of Tribes that apply to participate in the demonstration?  Those that are selected to participate in the demonstration? Are these characteristics different from those of Tribes that do not apply?
           2. What changes in organization and staffing of each program occurred as a result of the demonstration planning?
           3. What goals/objectives were established for each program during planning? Were these goals/objectives different from the goals/objectives that had been in place when the programs were contracted?  If so, what are the reasons?
           4. Were changes made in the funding made available to each program under the demonstration?  If so, what were the reasons for the changes?
           5. Were changes made that resulted in cost-savings or more efficient use of resources?
           6. What problems were identified during implementation and how were they resolved?
           7. Was the implementation successful?  Why or why not?
        2. Process Questions (six months and throughout the demonstration period) 
        1. What changes in programs, staffing, and organization occurred after the initial implementation period?  For each program, what were the reasons for these changes?
        2. Were there changes in the goals/objectives for each program after the initial implementation period?  If so, what were the reasons for the changes?
        3. Are goals/objectives for each program met, on a continuing basis
        4. How does the Tribe provide oversight and monitoring of each program?  What is the process for addressing problems or issues that are identified through monitoring?
        5. Were any program changes made to achieve cost-savings and increase efficiency?
        6. Were some program funds re-allocated to other priorities within the Tribe, after the initial implementation period? 
        3. Quantitative Measures of Process and Outcome Issues
        1. Was maintenance of effort achieved?  That is, did each program serve as many people and provide at least the same quantity of services as were available prior to Self-Governance?
        2. Did the mix of services provided change under the demonstration, for each program?
        3. For each program, were any changes made in staff levels or types of staff employed?
        4. Were there changes in the allocation of program funds to personnel, space, materials, contracted services, administrative costs under the demonstration, compared to the previous contracted program and/or national benchmark data?
        5. Did program costs per person receiving services change under the demonstration?
     4. Comparison Groups for Quantitative Measures
        1. Pre-Post comparisons/Patterns over time
        2. National benchmark comparisons
     5. Data Necessary for the Evaluation
        Note:  Specific data needed would depend on the set of evaluation issues and research questions developed by the Tribal Working Group.  Likely data needed would include:
        1. Baseline data on persons served, age-gender mix, services provided, outcome measures, for each program.
        2. Annual Report data on persons served, services provided, outcome measures for each year of the demonstration, for each program.
        3. Annual Report financial data for programs managed by Tribes under the demonstration, baseline through evaluation period, for each program.
        4. Annual Report narrative information on goals/objectives, program changes, problems encountered, and how problems were resolved.
        5. Other Data Needed
           1. Socio-economic and demographic data for each Tribe (2000 Census).
  
  

  Cost Estimates for Each Illustrative Model

  Generic cost estimates were prepared for each of the illustrative models.  This Appendix provides information on the assumptions used for each cost estimate, cost components, and total cost estimates.  These cost estimates are provided to illustrate the potential range of costs and components of costs that might be reasonably expected, for each evaluation model.  Specific cost estimates would depend on the detailed evaluation approach that might be developed by DHHS and Tribes, if an evaluation were to actually be conducted.

  A. Comprehensive Evaluation Model

  Two sets of cost estimates are provided for the comprehensive evaluation model, one based on the in-depth evaluation involving either 25 or 15 Tribes.  The following assumptions were used to make general cost estimates:

  • Three meetings of representatives from Tribes involved in in-depth evaluation per year;
  • Data collection by Tribal members at all 50 Tribes participating in demonstration and in-depth data collection by Tribal members at either 25 or 15 Tribes;
  • Training and technical assistance for all Tribal data collectors;
  • Two site visits per evaluation Tribe during evaluation; and
  • Attendance by two evaluation staff at two conferences per year.

  The table below presents the cost estimates for the major components of the comprehensive evaluation and a total cost estimate.

  Estimated Costs for Comprehensive Evaluation Model

  	25 Tribes	15 Tribes
  Evaluation Team Labor	2,340,000	1,900,000
  Tribal Data Collectors’ Labor	590,000	460,000
  Tribal Travel to Work Group Meetings and Training	320,000	270,000
  Evaluation Team Travel	170,000	110,000
  Other Direct Costs	90,000	70,000
  Total	$3,510,000	$2,810,000

  B. Limited Evaluation Model

  Two sets of cost estimates also are provided for the limited evaluation model, also based on the in-depth evaluation involving either 25 or 15 Tribes.  The following assumptions were used to make general cost estimates:

  • One meeting of representatives from Tribes involved in in-depth evaluation per year;
  • Data collection by Tribal members at all 50 Tribes participating in demonstration and in-depth data collection by Tribal members at either 25 or 15 Tribes;
  • Training and technical assistance for all Tribal data collectors;
  • One site visit per evaluation Tribe during evaluation; and
  • Attendance by one evaluation staff member at two conferences per year.

  The table below presents the cost estimates for the major components of the limited evaluation and a total cost estimate.

  Estimated Costs for Limited Evaluation Model

  	25 Tribes	15 Tribes
  Evaluation Team Labor	950,000	800,000
  Tribal Data Collectors’ Labor	440,000	380,000
  Tribal Travel to Work Group Meetings and Training	150,000	120,000
  Evaluation Team Travel	100,000	80,000
  Other Direct Costs	40,000	30,000
  Total	$1,680,000	$1,410,000

  C. Evaluation Model Using Aggregate Reporting Data

  The following assumptions were used to estimate costs for the evaluation model suing aggregate reporting data:

  • Three meetings of representatives from Tribes during course of study;
  • Data collection by Tribal members at all 50 Tribes participating in demonstration;
  • Telephone training and technical assistance for all Tribal data collectors;
  • No site visits during evaluation; and
  • Attendance by one evaluation staff member at one conference per year.

  The table below presented the estimated costs for this model by component and a total estimated cost.

  Estimated Costs for Evaluation Model Using Aggregate Reporting Data

  	50 Tribes
  Evaluation Team Labor	420,000
  Tribal Data Collectors’ Labor	280,000
  Tribal Travel to Work Group Meetings	40,000
  Evaluation Team Travel	30,000
  Other Direct Costs	20,000
  Total	$790,000

  
  

  Feasibility of Evaluating Indian Health Service Programs Operated by Tribes Under Self-Governance

  The Scope of Work for the contract to conduct the Tribal Self-Governance Evaluation Feasibility Study included requirements to assess the feasibility of conducting an evaluation of Indian Health Service programs operated by Tribes under Self-Governance compacts, as well as the feasibility of an evaluation of DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration.  Early in the study, the Technical Working Group for the project indicated that an evaluation of IHS programs operated under Self-Governance was fundamentally different from an evaluation of a new demonstration program.  The TWG emphasized that Self-Governance of health programs is not a demonstration program and Tribes have been operating IHS programs under compacts for over a decade.  Any evaluation that would be conducted would be retrospective in nature and subject to more limitations than would evaluation of a new demonstration.

  In addition, the extensive experience with Self-Governance of IHS programs means that Tribes have had many years to develop management and operational systems and are confident that these programs are successful and effective.  The Indian Health Service also has monitored and worked with current compacted Tribes over this period and the program is well-established.

  Because nearly all Tribes report data to the Resource and Patient Management System (RPMS) maintained by the IHS, there are extensive data available to conduct a quantitative evaluation of processes and outcomes of Self-Governance. The Indian Health Service Resource and Patient Management System database is an IHS-wide system designed to provide detailed and comprehensive clinical information for assessment and management of IHS performance.  It has several components for reporting detailed information on patient characteristics, diagnoses, and specific services provided.  The data included on the Ambulatory Patient Care System component are collected and entered daily, one record per ambulatory medical visit, and includes information on diagnosis, treatments, and specific examinations and tests performed and their results.  In addition, results of special medical record audits are entered into the database.  The RPMS has the capability to produce special reports, by IHS Region, Reservation, Service Unit, and by individual physician.  A pre-post comparison strategy and/or an external comparison strategy based on direct service IHS facilities would be possible, assuming reliable and consistent RPMS data were available for the time frame that Self-Governance has been in place.  However, the Indian Health Service has not conducted such an evaluation.

  In this Appendix, we present the background information and evaluation feasibility findings for Indian Health Service programs operated under Self-Governance compacts.  Information on which the evaluation feasibility study was based includes a review of IHS RPMS data capabilities and site visit findings.  Results of these two activities are described below.  Then, illustrative models for the evaluation of IHS program Self-Governance are described and reviewed for feasibility.

  Indian Health Service Data Review

  Several IHS staff was interviewed about data available through the Resource and Patient Management System (RPMS).  There were two primary foci in these discussions:  patient-level data and administrative and personnel-related data.  The RPMS is an integrated software system for management of clinical and administrative data in IHS and Tribally operated healthcare facilities.  It is composed of several different data collection components.  The Patient Care Component (PCC) comprises data collected at the patient level regarding all care received through the service units and includes a number of client characteristics.  Among the data elements that were investigated and found to be available in some form from the RPMS, and particularly the PCC, from 1998 forward are those listed in the Table F1 below.

  Table F.1:  Selected data available through the Patient Care Component of the RPMS

  Unit of Measurement	Data Available
  At the Service Unit (SU) level	Number of patients provided services in SU, by age and gender
  	Number and type of Contract Health Services provided by quarter of the fiscal year
  	Percent of patients with Medicare coverage
  	Percent of patients with Medicaid coverage
  	Percent of patients with SCHIP coverage
  	Percent of patients with Private Health Insurance coverage
  	Number of hospital admissions
  	Number of hospital days
  	Number of primary care visits
  	Number of specialist physician visits
  	Number of dental visits
  	Number of prescriptions filled
  	Percent children under age 5 immunized
  	Percent of patients aged 50+ receiving influenza immunizations
  	Percent of women over 18 with annual Pap smears
  	Percent of pregnant women obtaining prenatal care in first trimester
  	Percent of adults screened for diabetes
  	Percent of patients diagnosed with breast cancer surviving 5 years
  	Percent of patients diagnosed with cervical cancer surviving 5 years
  	Percent of births that are low-weight or premature
  	Percent of births that are high-weight
  	Percent of deaths attributable to diabetes
  For each Service Unit, for all patients with diabetes, three years:	Percent of patients seeing physician at least once in 3 months
  	Percent of patients receiving HbA1c testing once in 3 months
  	Percent of patients receiving dilated eye exam annually
  	Percent of people with diabetes who have diabetic retinopathy
  	Percent of people with diabetes who have had amputation

  The availability of administrative and personnel-related data elements was also investigated.  These elements would include staffing information, information on staff credentials, staff turnover, pharmacy information, and payment information.  Some of this information are likely available through other components of the RPMS (e.g., accounts payable, contract health, staff credentials), if these components are in use by the Tribal entities of interest. 

  Site Visit Findings: Availability, Accessibility, and Quality of Data on Self-Governance of IHS Health Programs

  Four of the six Tribes visited have IHS compacts and two of the Tribes have 638 contracts.  The Tribes have managed components of health care for a minimum of four years. Each of the sites indicated that individuals knowledgeable about the Tribe’s experience in Self-Governance or management of health care are available and could provide historical background and other information for an evaluation.  

  Accounting and personnel data concerning health programs were available at all sites.  Detailed accounting data were available at all sites including cost information by cost component (e.g., administrative costs, personnel costs, and other) and funding allocations by source over time.  Personnel data including staff turnover information are available at all six sites but are not generally available electronically.  This information would, in most cases, have to be recreated by knowledgeable managers.

  All six Tribes use the RPMS system for collecting patient-care data. A few Tribes were also using RPMS data for third party billing or looking into using RPMS for third party billing activities.  Follow-up information gathering with the Indian Health Service revealed that most Tribes/Tribal organizations with Self-Governance compacts (78 of 81 compacts) do submit data to the RPMS.

  However, a few of those interviewed expressed concern over the quality of the data collected through the RPMS.  These Tribal interviewees suggested that, without intensive effort at the facility-level to enhance quality, the data were not extremely useful.  One site had enhanced the RPMS data by training staff members in its use, collecting additional outcome data, and conducting a separate patient satisfaction survey.

  Illustrative Evaluation Models for Indian Health Service Programs Operated Under Self-Governance Compacts

  Illustrative Comprehensive Evaluation Model – IHS Health Programs

  A comprehensive evaluation of Indian Health Service programs managed by Tribes under compacts would involve examination of implementation and operational experiences and analysis of the impact of Tribal management on process and outcomes.  As with the illustrative comprehensive evaluation of DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration, Tribes would identify unique health program objectives and quantitative evaluation of processes/outcomes would examine these unique objectives for each Tribe.

  The comprehensive evaluation of IHS health programs managed under compacts could be conducted as a separate evaluation, or could be conducted as part of a comprehensive evaluation of all DHHS programs managed by Tribes under Self-Governance.  In the latter case, a subset of Tribes that participate in the new demonstration program and that currently compact for health could be used to evaluate health programs managed under compacts.

  Tribes have been managing their health systems under compacts for a decade, so an evaluation of implementation of these programs and operational experiences would necessarily be retrospective in nature.  The retrospective nature of the evaluation might introduce some biases in the findings, but could provide useful information and insights for DHHS and Tribes.  Because nearly all Tribes and Tribal organizations that compact for health services report data to the IHS Resource and Patient Management System and these data are available for the pre-compact and post-compact period, little primary data collection would be necessary.

  With respect to the feasibility issues of interest:

  • Tribal Support for the Evaluation Approach.  Tribes that manage health programs under compacts may not be supportive of a comprehensive evaluation of these programs, because they have been managing them for a number of years and they view health program Self-Governance as established and successful. Even if the evaluation was limited only to implementation and operational experiences and an evaluation of the extent to which Tribes achieved specific Tribally-set objectives, it is uncertain whether most Tribes would support or consider such an evaluation useful.
  • Availability of Appropriate Comparisons.  Assuming that the IHS RPMS data are available for participating Tribes and for the past 10 years, pre-post comparisons would be possible.  External comparisons could be based on a selected sample of IHS direct service Tribes or on aggregate IHS data that includes both compact and direct service Tribes.  However, either of those alternative external comparison groups is likely to raise concerns about appropriateness of the comparisons and value of the potential results.
  • Data Availability.  The availability of the RPMS and other IHS data would make it feasible to conduct the comprehensive evaluation of DHHS health programs and little primary data collection would be required.  Some people with whom the project team discussed the study expressed concerns, however, about the quality and completeness of the RPMS data.  It is possible that substantial work would be required to create the evaluation data base to ensure that the quality of the data was sufficient to produce reliable evaluation results.
  • Costs.  

    Comprehensive Evaluation Model

    Limited Evaluation Model

    Evaluation Using Aggregate Monitoring and Reporting Data

    Tribal Support for Evaluation

    Low

    Possible

    Low

    Availability of Comparison Groups

    Possible but difficulties

    Possible

    Possible

    Data Availability

    RPMS and site visits

    RPMS and site visits

    Data are all currently being reported. No new data requirements

    Costs

    Highest

    Modest

    Low

    Trade-off between comprehensiveness and usefulness of results and costs

    Would produce reliable findings on a range of issues, but at high cost

    Useful findings for a limited set of issues at modest cost

    Limited findings at low cost

    Detailed Description Of Illustrative Comprehensive Evaluation Model: Indian Health Service Programs Operated By Tribes Under Self-Governance

    1. Objectives
       1. Conduct a comprehensive evaluation of the operations and outcomes of Indian Health Service health programs managed by Tribes under Self-Governance compacts.
    2. Assumptions
       1. Tribes to be included in the evaluation of health programs would be limited to those that have submitted data to RPMS.
       2. Participation of both compact and direct service Tribes in the evaluation would be voluntary.
       3. IHS would provide RPMS data for participating compact and direct service Tribes, for all years required.
       4. Site visits would be conducted to 15-25 compact and direct service Tribes that volunteer to participate in the in-depth evaluation.
    3. Research Questions to be Examined
       Note:  Specific research questions would be developed in consultation with the Tribes. Based on discussions conducted during the current study, the general research questions that are likely to be identified might include:
       1. Background Issues (retrospectively)—Health Programs
          1. What are the characteristics of Tribes that compact for health programs?  Are these characteristics different from those of Tribes that do not compact?
          2. What factors are reported by Tribes as influential in their decision to manage their health programs?  What was the most important factor in their decision?  What concerns were identified during the decision process?
          3. Was the community involved in the decision to manage health programs?  How was this accomplished?
       2. Process Issues (retrospectively)
       1. What changes in programs, staffing and organization have occurred since the Tribe began managing the health program?  What were the reasons for these changes?
       2. Were there changes in the goals/objectives for the health program/system after the initial implementation year?
       3. What oversight and monitoring of the health system does the Tribe conduct?  What is the process for addressing problems or issues that are identified?
       4. Is the community involved in decisions made about the health system?  How?
       5. Have the goals/objectives for the health system been met on an ongoing basis?
       6. Have program changes been made to achieve cost-savings and/or increase efficiencies?
       7. Have any health program funds been re-allocated to other priorities within the Tribe?  How was the decision made and who were the decision makers?
       8. What are the perceptions of Tribal leaders and others about the benefits of Tribal management of health programs?  Are there any perceived disadvantages?
       3. Quantitative Measures of Process and Outcome Issues – Health
       1. Do health programs managed by Tribes provide the same level and mix of services as would be available if the Tribes did not manage these programs?  If not, what are the reasons?
       2. Did the health program achieve at least two quantifiable goals, established by the Tribe, during the past three years?
       3. Does the Tribe employ directly all health providers that work at the health unit? If so, has this been the case since the Tribe began managing the health program?
       4. Has there been little or much staff turnover since the Tribe began managing the health program?
       5. Have there been changes in the allocation of program funds to personnel, space, materials, contract health services, administrative costs since the Tribe began managing the health program?
       6. Have program costs per person receiving services changed more/less than would be expected based on IHS funding levels?
       7. Are health program users more/less satisfied with services provided by the health program under Tribal management than are users who receive services through IHS direct service arrangements?
    4. Comparison Groups for Quantitative Measures
       1. Pre-post comparisons
       2. Self-Governance versus direct service comparisons
       3. National benchmark comparisons
    5. Data Necessary for the Evaluation
       Note:  Specific data needed would depend on the set of evaluation issues and research questions that would be developed through DHHS consultation with the Tribes. A set of possible data needs is described here.   
       1. RPMS data for pre-post compact period, for participating Tribes
       2. RPMS data for pre-post compact period, for participating direct service Tribes
       3. Detailed financial data on health programs managed by participating Tribes, baseline through evaluation year.
       4. Detailed data on staffing, staff-mix, salaries, credentials, and turnover of personnel for participating compact and direct service Tribes, since the beginning of health Self-Governance.
       5. Health care satisfaction survey of Tribal members (on/near Reservation), for participating Tribes managing health programs and for a sample of participating direct service Tribes.
       6. One round of site visits to participating Tribes managing health programs and participating direct service Tribes/IHS service units to collect qualitative data on operations, processes, goals/objectives, perceptions of advantages/disadvantages of Tribal management of health programs.
       7. Socioeconomic and demographic data for all health compact and direct service Tribes
    
    

    Notes

    http://aspe.hhs.gov/SelfGovernance/Evaluation/actions.htm.

    http://www.aspe.hhs.gov/selfgovernance/).