Core Performance Indicators for Homeless-Serving Programs Administered by the U.S. Department of Health and Human Services

Contents

• Characteristics of the Four Homeless-Serving Programs
• Analysis of Measures Derived from Homeless Administrative Data Systems (HADS)
• Potential Core Performance Measures For Homeless-Specific Service Programs
• Application of Suggested Core Performance Measures To DHHS Mainstream Programs Serving Homeless Individuals

Recent studies suggest that homelessness is a problem that afflicts many adults and children in the nation and can have a broad range of short- and long-term negative consequences. It is estimated that up to 600,000 people in the United States are homeless each night.⁽¹⁾ In developing programs to address the needs of the homeless, it is important to specify clearly the program goals and objectives to guide implementation of program activities, as well as a set of performance measures to facilitate documentation and analysis of the effectiveness of program interventions. This study explores the feasibility of developing a core set of performance measures for DHHS programs that focus on homelessness. It has two main objectives: (1) determine the feasibility of producing a core set of performance measures that describe accomplishments (as reflected in process and outcome measures) of the homeless-specific service programs of DHHS; and (2) determine if a core set of performance measures for homeless-specific programs in DHHS could be generated by other mainstream service programs supported by DHHS to assist low income or disabled persons.

A key focus of the study is on enhancing performance measurement across four homeless-serving programs administered by DHHS:

1. Programs for Runaway and Homeless Youth (RHY),
2. The Health Care for the Homeless (HCH) Program,
3. Projects for Assistance in Transition from Homelessness (PATH), and
4. The Treatment for Homeless Persons Program (formerly called the Addiction Treatment for Homeless Persons Program).

In addition, this project deals with an important government management requirement that has affected agencies and programs for the past several years: the Government Performance and Results Act of 1993 (GPRA), which requires government agencies to develop measures of performance, set standards for the measures, and track their accomplishments in meeting the standards.

This study mainly involved interviews with program officials knowledgeable about the four homeless-serving programs that were the main focus of this study, along with review of existing documentation. Interviews were conducted both by telephone and in-person. In addition, the research team conducted telephone interviews with program officials at four mainstream programs. Project staff also reviewed documents and interviewed program officials that operated homeless administrative data systems (HADS) or homeless registry systems in several localities across the country.

Characteristics of the Four Homeless-Serving Programs

Although the four homeless-serving programs shared the goal of providing services to the homeless, they also had significant differences. Some major findings from our interviews with program officials and review of documents are:

• Program funding, allocation, role of the federal/state governments, and number and types of agencies providing services vary substantially across programs. FY 2002 funding runs from $9 million (Treatment for Homeless Persons) to in excess of $100 million ($116 million for HCH). Three of the four programs allocate funds competitively; one of the programs allocates funds to states by formula (PATH). The federal government plays a significant role in all four of the programs  distributing funds to states (PATH) or competing grants and selecting grantees (in the case of the other three programs); providing oversight and collecting performance information; and providing technical assistance. In terms of state involvement, only under the PATH program among the four programs does the state play a significant role. The number of grantees ranges from 50 grantees selected under the Treatment for Homeless Persons Program to about 640 under the three RHY programs.
• While there is a similar focus on homeless individuals across the four programs, there are differences in terms of the number and types of individuals served, definitions of enrollment, and duration of involvement in services. The RHY programs target youth (both runaway and homeless), while the other three programs target services primarily on adult populations (though other family members are often also served). The HCH program funds initiatives that serve a broad range of homeless individuals (especially those unable to secure medical care by other means). The PATH and Treatment for Homeless Persons Programs serve a somewhat narrower subgroup of the homeless population than the other programs: the PATH program focuses on homeless individuals with serious mental illness; and the Treatment for Homeless Persons program targets homeless persons who have a substance abuse disorder, or both a diagnosable substance abuse disorder and co-occurring mental illness or emotional impairment. Enrollment practices also vary. In PATH  which is considered to be a funding stream at the local (operational) level  it is often difficult to identify a point at which someone is enrolled or terminates from PATH. In HCH, a homeless individual becomes a participant when he/she receives clinical services at an HCH site. Length of participation in HCH is highly variable  it could range from a single visit to years of involvement. For RHY  which is composed of three program components  there is considerable variation in what constitutes enrollment and duration of involvement. In RHYs Street Outreach Program (SOP), involvement is very brief (often a single contact) and presents little opportunity for collecting information about the individual. In contrast, RHYs Transitional Living Program (TLP) provides residential care for up to 18 months under the program and a broad range of other services to move homeless youth toward self-sufficiency and independent living. RHYs third program component  Basic Center Program (BCP)  offers up to 15 days of emergency residential care, help with family reunification, and other services. Of the four programs, enrollment in the Treatment for Homeless Persons Program appears to be most clearly defined. Homeless individuals are considered participants when the intake form (part of the Core Client Outcomes form) is completed on the individual. Involvement in the program is extended over a year or longer. Numbers served range from 7,700 over three years for the Treatment for Homeless Persons Program to about 500,000 annually for HCH.
• There is a wide range of program services offered through the four programs. All four programs try to improve prospects for long-term self-sufficiency, promote housing stability, and reduce the chances that participants will become chronically homeless. Each program has more specific goals that relate to the populations served and the original program intent  for example, RHYs BCP component has as one of its goals family reunification (when appropriate); HCH aims to improve health care status of homeless individuals; PATH aims to engage participants in mental health care services and improve mental health status; and the Treatment for Homeless Persons Program aims at engaging participants in substance abuse treatment and reducing/eliminating substance abuse dependency.
• The four homeless programs feature substantially different approaches to performance measurement, collection of data, and evaluation. With respect to GPRA measures, three of the four programs have explicit measures; there are no GPRA measures specific to HCH. GPRA measures apply to the BPHCs Health Centers Cluster of programs as a whole, of which HCH program is part.⁽²⁾ The measures used for the three other programs include both process and outcome measures. The Treatment for Homeless Persons Program has outcome-oriented GPRA measures, as well as a data collection methodology designed to provide participant-level data necessary to produce the outcome data needed to meet reporting requirements. For example, the GPRA measures for adults served by the Treatment for Homeless Persons Programs are the percent of service recipients who  (1) have no past month substance abuse; (2) have no or reduced alcohol or illegal drug consequences; (3) are permanently housed in the community; (4) are employed; (5) have no or reduced involvement with the criminal justice system; and (6) have good or improved health and mental health status. In contrast, the measures employed by PATH are process measures: (1) percentage of agencies funded providing outreach services; (2) number of persons contacted, (3) of those contacted, percent enrolled in PATH. Of the three main GPRA measures used in the RHY program, just the first one is outcome-oriented: (1) maintain the proportion of youth living in safe and appropriate settings after exiting ACF-funded services; (2) increase the proportion of BCP and TLP youth receiving peer counseling through program services; and (3) increase the proportion of ACF-supported youth programs that are using community networking and outreach activities to strengthen services. Methods of collecting performance data and the quality of the data collected also vary across the four programs. Three of the four programs have states (PATH) or grantees (HCH and RHY) submit aggregate data tables either annually or semi-annually. All four of the programs use (or are in the process of developing and implementing) some type of automated database for transmission of performance data to their federal administering agencies.

Differences among the four programs means that it will be a difficult and delicate task to develop a common set of performance measures for the four programs, and even more difficult for those measures to also be applicable to other DHHS programs serving homeless individuals. In addition, while federal agency officials are very willing to discuss their programs and share their knowledge of how they approach data collection and reporting, their willingness and ability to undertake change is uncertain. From our discussions, it appears that changes in how programs collect data and report on performance will require substantial efforts on the part of agency officials and programs. For example, with regard to RHY  which is currently involved in an effort to implement a streamlined data system  it would not only require change at the federal administering agency, but how over 600 grantee organizations collect and manage data.

Analysis of Measures Derived from Homeless Administrative Data Systems (HADS)

With input from DHHS, we selected five HADS (in New York City, Madison, Columbus, Kansas City, and Honolulu) for study. In the Summer 2002, we interviewed (by telephone) system administrators about the operations of each of the five HADS. We also conducted a site visit to New York Citys Department of Homeless Services to interview staff in greater depth and obtain additional background information on the operation of HADS. Major findings from the interviews are:

• The HADS system in New York has been operational since 1986, while the other four have been designed and implemented during the past decade; all five systems are either in the process of being upgraded to use the most recent technology or were recently developed using state-of-the-art technology.
• HADS tend to be system-wide  some cutting across a large number of partners  which avoids focusing narrowly on programs (e.g., silos).
• Some HADS have accumulated substantial numbers of records on homeless and other types of disadvantaged/low-income households.⁽³⁾
• HADS systems are not used principally for measuring program performance or outcomes  though have the capability to provide analyses of length of stay.
• A range of implementation challenges were reported  particularly with regard to training system users to make full use of system features.

While the HADS reviewed for this report provide some useful measures of program inputs and process, they do not provide a set of measures of program outcomes or performance (with the possible exception of length of stay) that are readily adaptable to the DHHS homeless-serving programs that are the focus of our overall study. There are, however, some interesting implications that can be drawn from HADS for developing performance measures for DHHS homeless-serving programs and the systems capable of maintaining data that might be collected as part of such systems. Several of the systems we reviewed do collect data on duration of episodes of receipt of homeless services (i.e., length of stay in emergency shelters and transitional facilities). Such a measure is particularly helpful in understanding frequency and total duration of homeless individuals receipt of assistance (e.g., duration of each spell of use of emergency shelters). Such data would be particularly helpful in understanding the extent of chronic homelessness and types of individuals most likely to have frequent and lengthy stays in emergency or transitional facilities. This points to the need to collect client-level data on service utilization, which includes dates that services begin and end so that it is possible to examine duration and intensity of services received, as well as multiple patterns of service use (i.e., multiple episodes of shelter use). The HADS also show that it is possible to collect detailed background characteristics on homeless individuals served, and especially in the case of Hawaiis HADS, to collect data at the time of entry and exit from homeless-serving programs to support pre/post analysis of participant outcomes.

Potential Core Performance Measures For Homeless-Specific Service Programs

In developing these measures, we took into consideration the following important factors:

• Extent currently collected. Items that are already collected by more programs have the advantages of already being highly regarded and contributing the least resistance for inclusion in a uniform system.
• Ease of collection. For items not universally collected, the ease at which an item can be collected is of interest. We are concerned with initial costs to establish the collection system as well as ongoing costs.
• Relationship to outcome and process measures of interest. In some instances, proxy measures for the measures of interest must be used because the proxies are preferable on criteria such as ease of collection and extent currently used.

Our earlier analysis of the four homeless-serving programs indicated that there are substantial cross-program differences that complicate efforts to develop similar performance measures and systems for collecting data. For example:

• Programs target different subpopulations of homeless individuals. For example, the RHY programs target youth (both runaway and homeless), while the other three programs target services primarily on adult populations (though other family members are often also served). While the HCH program funds initiatives that serve a broad range of homeless individuals (especially those unable to secure medical care by other means), the PATH program focuses on homeless individuals with serious mental illness; and the Treatment for Homeless Persons program targets homeless persons who have a substance abuse disorder, or both a diagnosable substance abuse disorder and co-occurring mental illness or emotional impairment.
• The definition of enrollment and termination in the programs and duration of involvement in services all vary considerably by program. For example, in PATH, it is often difficult to identify a point at which someone is enrolled or terminates from PATH. In HCH, a homeless individual becomes a participant when he/she receives clinical services at a HCH site. Enrollment in the Treatment and Homeless Persons Program appears to be most clearly defined  homeless individuals are considered participants when the intake form is completed on the individual.
• Numbers of homeless individuals served are quite different across the four programs. While actual numbers of individuals served or participating are difficult to compare because of varying definitions across programs, the sizes of programs appear quite different. For example, HCH reports that about 500,000 persons were seen in CY 2000. This compares with the RHY program estimates that it helps 80,000 runaway and homeless youth each year and estimates that PATH served (in FY 2000) about 64,000 homeless individuals with serious mental illness.
• Types of program services vary considerably across programs. Common themes across the programs include emphases on flexibility, providing community-based services, creating linkages across various types of homeless-serving agencies, tailoring services to individuals needs, and providing a continuum of care to help break the cycle of homelessness. However, the specific services provided are quite different. For example, the Treatment for Homeless Persons Program emphasizes linkages between substance abuse treatment, mental health, primary health, and housing assistance; HCH emphasizes a multidisciplinary approach to delivering care to homeless persons, combining aggressive street outreach, with integrated systems of primary care, mental health and substance abuse services, case management, and client advocacy. Of the four programs, the RHY program (in part, because it targets youth) provides perhaps the most unique mix of program services  and even within RHY, each program component provides a very distinctive blend of services (e.g., street outreach [the Street Outreach Program] versus emergency residential care [Basic Center Program] versus up to 18 months of residential living [Transitional Living Program]).

Our review of the performance measurement systems in existence across the four programs also indicates potential for both enhancement and movement toward more outcome-oriented measures. For example, the general approach to performance measurement used within the Treatment for Homeless Persons Program provides a potential approach that could be applicable to the other three programs. Of critical importance to our efforts to suggest core measures, all four of the programs are aimed at (1) improving prospects for long-term self-sufficiency, (2) promoting housing stability, and (3) reducing the chances that individuals will become chronically homeless. In addition, the four programs (some more than others) also stress addressing mental and physical health concerns, as well as potential substance abuse issues.

Based on the common objectives of these four programs, we suggest a core set of process and outcome measure that could potentially be adapted for use by the four homeless-service programs (see Exhibit ES-1). We suggest selection of the four process measures, which track numbers of homeless individuals (1) contacted/outreached, (2) enrolled, (3) comprehensively assessed, and (4) receiving one or more core services. We then suggest selection of several outcome measures from among those grouped into the following areas: (1) housing status, (2) employment and earnings status, and (3) health status. In addition, we have suggested a several additional outcome measures that could be applied to homeless youth.

With regard to housing outcomes, we have identified three potential outcome measures intended to track (1) changes in an individuals housing situation along a continuum (from living on the street and in emergency shelters to securing permanent housing), (2) whether the homeless individual secures permanent housing, and (3) days of homelessness during the preceding quarter (or month). Two earnings measures are identified  one that captures actual dollar amount of earnings during the past quarter (or month) and a second measure that captures whether an individuals earnings have improved. Two employment measures are also identified  one relating to whether the individual is engaged in work 30 or more hours per week and another that measures whether hours of work have increased. Three health-related measures are offered, focusing on use of drugs, improvement in physical health status, and improvement in mental health status. Finally, three measures are offered that are targeted exclusively on youth (though the other outcome measures would for the most part also be applicable to youth): (1) whether the youth is reunited with his/her family, (2) whether the homeless youth is attending school, and (3) whether the homeless youth graduates from high school or completes a GED.

A pre/post data collection approach is suggested with respect to obtaining needed performance data  for example, collecting data on housing, health, and substance abuse status of program participants at the time of intake/enrollment into a program and then periodically tracking status at different points during and after program services are provided (i.e., at termination/exit from the program and/or at 3, 6, or 12 months after enrollment). Collection of data on homeless individuals at the point of termination can be problematic because homeless individuals may abruptly stop coming for services. The transient nature of the homeless population can also present significant challenges to collecting data through follow-up surveys/interviews after homeless individuals have stopped participating in program services..

Given difficulties of tracking homeless individuals over extended periods, the extent to which existing administrative data can be utilized could increase the proportion of individuals for which it is possible to gather outcome data (at a relatively low cost). Probably the most useful source in this regard is quarterly unemployment insurance (UI) wage record data, which can be matched by Social Security number (though releases are required and it may also be necessary to pay for the data). A second potential source of administrative data that may have some potential utility for tracking housing outcomes are HADS system maintained by many states and/or localities. HADS systems are not used principally for measuring program performance or outcomes, but they have the capability to provide analyses of length of stay.

Finally, in terms of tracking self-sufficiency outcomes, data sharing agreements with state and local welfare agencies may provide possibilities for tracking dependence on TANF, food stamps, general assistance, emergency assistance, and other human services programs.

Application of Suggested Core Performance Measures To DHHS Mainstream Programs Serving Homeless Individuals

With input from the DHHS Project Officer, we selected four DHHS mainstream programs for analysis: (1) the Health Centers Cluster (administered by Health Resources and Services Administration [HRSA]), (2) the Substance Abuse Prevention and Treatment (SAPT) Block Grant (administered by Substance Abuse and Mental Health Services Administration [SAMHSA]), (3) Head Start (administered by Administration for Children and Families [ACF]), and (4) Medicaid (administered by the Centers for Medicaid and Medicare Services [CMS]). While these programs are not targeted specifically on homeless individuals, some homeless individuals are eligible for services provided under each program by virtue of low income, a disability, or other characteristics. In comparison to the four homeless-serving programs, the mainstream programs:

• Have much greater funding. The largest of the four homeless-serving programs in terms of budget is the Health Care for the Homeless (HCH) program, with an annual budget of slightly more than $100 million. The funding levels of HCH and the other homeless-serving programs pale in comparison to those of the four mainstream programs: Medicaid, with FY 2002 federal assistance to states of $147.3 billion; Head Start, with a FY 2002 budget of $6.5 billion; SAPT, with a FY 2002 budget of $1.7 billion, and the BPHCs Health Centers Cluster, with FY 2002 budget of $1.3 billion (which includes funding for HCH).
• Serve many more individuals. As might be expected given their greater funding levels and mandates to serve a broader range of disadvantaged individuals, the mainstream programs enroll and serve many more individuals  in 2002, Medicaid had nearly 40 million enrolled beneficiaries, far eclipsing the other mainstream and homeless-serving programs. In 2001, the Health Centers Cluster served an estimated 10.3 million individuals, while SAPT served an estimated 1.6 million individuals (in FY 2000) and Head Start enrolled nearly a million (912,345 in FY 2002) children.
• Serve a generally more broadly defined target population. While similarly targeted on low-income and needy individuals, the mainstream programs extend program services well beyond homeless individuals. Of the four mainstream programs, the two broadest programs are the Medicaid and Health Cluster Centers programs, both focusing on delivery of health care services to low-income and disadvantaged individuals. The Head Start program targets needy and low-income pre-schoolers ages 3 to 5; SAPT is primarily targeted on individuals who abuse alcohol and other drugs, but also extends preventive educational and counseling activities to a wider population of at-risk individuals (i.e., not less that 20 percent of block grant funds are to be spent to educate and counsel individuals who do not require treatment and provide activities to reduce risk of abuse).

Despite some differences, there are commonalities in terms of program goals and services offered by mainstream and homeless-serving programs. Three of the four mainstream programs (Medicaid, SAPT, the Health Centers Cluster) focus program services primarily on improving health care status of low-income individuals. Two of the programs  Medicaid and the Health Centers Cluster are aimed directly at delivery of health care services to improve health care status of low-income and needy individuals. Though more narrowly targeted on homeless individuals, HCH and PATH are similarly aimed at improving health care status of the disadvantaged individuals. The third mainstream program  SAPT  aims at improving substance abuse treatment and prevention services. Under SAPT, block grants funds are distributed to states, territories, and tribes aimed at the development and implementation of prevention, treatment, and rehabilitation activities directed to diseases of alcohol and drug abuse. In terms of program goals and services, Head Start is quite different from the three other mainstream programs and the four homeless-serving programs. The Head Start program is aimed principally at increasing school readiness and social competence of young children in low-income families. Our main findings from the review of mainstream programs are:

• Estimates of the number of homeless served are available for one of the four mainstream programs  Head Start.
• Three of the four mainstream programs, all except Medicaid, provide guidance on the definition of homeless.
• With the possible exception of counts of homeless individuals served, the mainstream programs do not collect sufficient information to address the suggested core performance measures.
• Mainstream program GPRA measures are combination of process- and outcome-oriented measures and are not closely aligned with suggested core performance measures for homeless-serving programs..
• Mainstream programs face substantial constraints to making changes to existing data systems to increase tracking of homeless individuals.

Recognizing the difficulties faced by the mainstream programs in making changes to their well-established data sets, it would be very useful to work with mainstream DHHS programs to: (1) add a single data element to data systems that would capture living arrangement or homeless status at the time of program enrollment in a consistent manner across programs; (2) provide the mainstream programs with a common definition of what constitutes homelessness and, if possible, the specific question(s) and close-ended response categories that programs should use in tracking homelessness; and (3) if mainstream programs conduct a follow-up interview or survey with participants, request that they include a follow-up question relating to homelessness or living arrangement.

For all four of the mainstream programs and the four homeless-serving programs, a step beyond collecting homeless status or living arrangement at the time of enrollment would be to collect such data at the time of exit from the program or at some follow-up point following enrollment or termination from the program. However, determining a convenient follow-up point to interact with the participant may be difficult or impossible in these programs. With regard to collecting homeless or living arrangement status at a follow-up point, it may be best to focus (at least initially) on implementing such follow-up measures in the homeless-serving programs, where long-term housing stability is a critical program objective.

Finally, where collection of information about homeless status either at the time of enrollment or some follow-up point prove either impossible to obtain or too costly, DHHS should consider potential opportunities for collecting data on homelessness as part of special studies or surveys. Several of the mainstream programs (as well as the homeless-serving programs) are periodically the subject of either special studies or survey efforts. For example, the Head Start program has implemented the FACES survey, which is conducted in 3-year waves on a sample of over 3,000 children and families served by 40 Head Start centers. Working with a sample, rather than in the universe in large programs such as Head Start (nearly 1 million children) and Medicaid (about 40 million beneficiaries) has great appeal from the standpoint of reducing burden and data collection costs.

Endnotes

(1) An estimate provided by the U.S. Department of Health and Human Services website (see http://aspe.hhs.gov/homeless/).

(2) HCH is clustered with several other programs, including Community Health Centers [CHCs], Migrant Health Centers, Health Services for Residents of Public Housing, and other community-based health programs.

(3) For example, New York City had over 800,000 records in its system, and Kansas City had 450,000 records.