Evaluability Assessment of Discharge Planning and the Prevention of Homelessness

Contents

• Introduction
• Key Research Questions
• Methods
• Key Findings
• Key Study Design and Measurement Issues
• Alternative Research Designs

Introduction

The purpose of this study was to determine the evaluability of discharge planning as a strategy to prevent homelessness. In 1994 the Federal Interagency Council on Homelessness identified inadequate discharge planning as a significant factor contributing to homelessness among persons with mental illnesses and/or substance use disorders. The prevention of homelessness is a key goal in the U.S. Department of Health and Human Services (DHHS) action plan to end chronic homelessness.⁽¹⁾ This study is related to a strategy in the plan that recommends identifying and promoting the use of effective, evidence-based homelessness prevention interventions.

Past research has indicated that many people with severe mental illnesses and substance abuse problems who experience homelessness travel in institutional circuits, or move repeatedly through systems and institutions such as state psychiatric hospitals, jails and prisons, homeless shelters, and drug treatment programs. However, although discharge planning is often recommended as a strategy to prevent homelessness among people released from institutions or youth aging out of foster care, very few studies have examined this strategy. The Assistant Secretary for Planning and Evaluation of DHHS sponsored this study in order to build knowledge for researchers and policy makers in the field of homelessness regarding the evaluability of discharge planning in institutional and custodial settings. The four institutional and custodial settings listed below were included in this study because previous research has indicated that many of those entering shelters have recently come from one of these settings.

• Adult inpatient psychiatric treatment units in state psychiatric, private psychiatric, or general hospitals;
• Residential treatment centers serving children and youth with serious emotional disturbances and/or substance use disorders;
• Residential treatment programs for adults with substance use disorders; and
• Foster care independent living programs.

Key Research Questions

Some of the key research questions of this study are listed below; a more exhaustive list can be found in this report's Methods section.

• Is a meaningful evaluation of discharge planning in relation to the goal of preventing homelessness feasible within and across settings?
• Can discharge planning be disentangled from other program activities in the various settings?
• What target population(s) in relation to homelessness could be evaluated, and how do these vary within and across settings?
• How available are the key community resources within and across settings, and what are the implications for evaluability?
• What are the relevant independent, dependent, and mediating variables that should be studied in each of these settings and how will these be defined and operationalized?
• What is the appropriate followup period after discharge to determine clients' housing status and other outcomes by setting, and what are the implications for evaluability?
• What alternative research designs could be considered for evaluating or studying discharge planning in relation to preventing homelessness in each of these settings, and what would their costs be?

[ Go to Contents ]

Methods

The study involved the following components:

1. A review of the literature and key analytic issues pertaining to discharge planning;
2. An Expert Panel process where members discussed key analytic issues and nominated exemplary discharge planning programs for use in a documentary analysis and site visits;
3. An analysis of discharge-planning-related documents (e.g., discharge planning policies, procedure manuals, job descriptions, forms, and screening instruments) from 19 programs;
4. Site visits to 8 of the 19 programs;
5. An Analytic Findings Report that synthesized findings from the documentary analysis and site visits in order to address key research questions and determine whether discharge planning is evaluable;
6. Development of evaluation design options outlining possible research studies on discharge planning and related homeless prevention issues; and
7. This Final Report, which summarizes the key findings from the study.

The primary data sources for this study included the documentary analysis materials; staff discussions at the primary and affiliate agencies of the site visit programs; and a review of procedures, forms, and a limited number of medical records (with client consent). The study has a number of methodological limitations. Most important, findings are based on qualitative examination of a modestly sized convenience sample of program sites selected because experts thought that their discharge planning processes were superior to other programs. The sites studied are examples from four extremely different and diverse settings. We did not interview clients, nor did we systematically examine quantitative data to confirm staff reports.

[ Go to Contents ]

Key Findings

In this Executive Summary we present only the key findings that hold across all four settings. Please refer to the Analytic Findings section of the report for key findings specific to each setting.

• A Summative Evaluation of Discharge Planning Is Not Justified at This Time

The study team concluded that a rigorous summative evaluation (i.e., an outcome or impact evaluation) of discharge planning as a strategy to prevent homelessness in institutional and/or custodial settings is not justified at this time. The recommendation against conducting a summative evaluation of discharge planning as a strategy to prevent subsequent homelessness is based on the findings that discharge planning is not readily separable from the broader program, that it is not well defined or consistently implemented, and that a summative study would be costly and is premature given the state of knowledge in this area. However, we found that alternative study designs to evaluate specific issues or activities related to discharge planning and homelessness prevention are feasible and justifiable, and these designs are described in this report.

• Discharge Planning Is Not Readily Separable From the Broader Program

A key evaluability question for each of the four settings is whether the discharge planning process is separable from the broader program in which it operates. Many discharge planning activities, such as client assessment, are also critical to treatment planning and are often performed by the same staff. While there are distinct, identifiable activities associated with discharge planning, they take place within the context of the broader treatment or service delivery process and cannot be clearly separated from that context. For example, the primary goal of a residential substance abuse treatment program is to reduce or eliminate a client's dependence on alcohol or other drugs.⁽²⁾ However, the intervention of interest for the evaluability assessment is not the entire treatment (which includes discharge planning) provided in the residential substance abuse treatment program. Rather, the evaluability assessment focuses on the discharge planning process alone; other activities that occur in the residential substance abuse treatment program would be mediating variables in an evaluation of the discharge planning process.

• The Discharge Planning Process Is Not Well Defined or Consistently Implemented

Few programs appear to have a well-designed and integrated model of the discharge planning process, nor have they implemented the process in a systematic manner likely to produce consistent results over time or across clients. Although most programs do have at least rudimentary discharge planning procedures and forms, few of the programs examined have a written protocol to ensure that staff members apply the interventions uniformly or document discharge planning processes well. None of the 19 programs studied used screening instruments to identify clients at risk of homelessness and in need of intensive discharge planning efforts.

In addition, most programs examined lack rigorous staff training and quality assurance activities in support of discharge planning. As a result, the discharge planning process is inconsistently applied within each program. Likewise, programs collect very little systematic postdischarge data that could create a feedback loop to improve the discharge planning process over time.

The discharge planning process consists of an imprecisely defined set of activities. While some governmental and professional organizations have developed consensus standards on what constitutes a model discharge planning process,⁽³⁾ we found little evidence that these models have been effectively disseminated or widely implemented. No studies have yet tested the effectiveness of these models in actual practice. Critics have suggested the models were not attuned to real world scarcity of housing and other resources, or to the tendency of organizations to pursue self-interest rather than collaborate effectively.

• Housing and Community Services Are Also Essential for Preventing Homelessness

The study team found that avoiding homelessness, the outcome of interest, is determined as much or more by the availability of suitable housing and support services in the community as by the discharge planning process. An example of this was found in the most well-structured and best implemented discharge planning process observed in this study. This model discharge planning process was implemented in a rural community so lacking in housing options that many clients were placed in large congregate semi-institutional conditions upon discharge. The best discharge planning process cannot overcome a lack of community housing and services.

• Practical Research Design Considerations Would Make a Summative Evaluation Challenging and Costly

The tremendous variability in the discharge planning process across clients, programs, settings, and communities dictates that a summative evaluation enroll thousands of clients across many programs. The discharge planning process is highly complex and tightly bound to programmatic, client, interorganizational, and community resource factors. Numerous mediating variables affect the discharge planning process and its outcomes; some of these variables lack well-formulated measures. A summative evaluation of the discharge planning process as a strategy to prevent homelessness would be complex, lengthy, costly, and could not be assured of producing clear and definitive findings. Further preliminary and exploratory research is called for before undertaking a study of such complexity and resource requirements.

• Eligibility and Funding Sources Dictate Intervention and Discharge Planning Tracks

Within a single residential or custodial program, there are several intervention and discharge planning tracks depending on who pays for or oversees a client's care and the community programs for which they are eligible. These tracks often result in differences in clinical interventions, lengths of stay, agencies involved in the discharge planning process, community housing alternatives, and available community services. For example, the treatment and discharge planning process a patient receives in a psychiatric hospital will be determined in part by whether the person meets eligibility criteria for Supplemental Security Income (SSI), Medicaid, a private insurer, the state mental health department, or is self-pay and of limited means. For youth in residential treatment centers, the discharge process is partially determined by who has custody of the youth  the family, child welfare, or juvenile justice. The availability of services in the patient's community of residence is another important determinant. Each funding source or community program may have its own complex eligibility standards and application process that is time-consuming to negotiate. An evaluator of the discharge planning process must address the complexity of multiple discharge planning tracks that results from the involvement of these diverse payers and community contexts.

• No Separate Payment Exists for Discharge Planning Activities

In most cases the discharge planning process is funded as incidental to the treatment and custodial care, often as part of a bundled per diem rate. Any study will face challenges in determining financial expenditures for discharge planning activities since they are not billed separately and are usually carried out by the same program staff who provide clinical care.

[ Go to Contents ]

Key Study Design and Measurement Issues

If a study of discharge planning is conducted, a number of key study design and measurement issues will need to be considered. The initial bullet points below address factors critical to designing a study, while later bullets discuss mediating variables that would need to be controlled for in a summative evaluation and other design considerations. In most cases, there are existing measures that could be used, although they will require some adaptation depending on the study context and setting. There are some key concepts, such as the availability of appropriate community housing, that will have to be carefully negotiated and for which no definitive measures are readily available. This section outlines some of these measurement, data collection, and other design issues.

• Sample Size Depends on Purpose of the Study

The sample size needed will depend on the purpose of the study. A sample of several thousand will likely be necessary to achieve sufficient statistical power if a summative evaluation is the goal. A more modest sample size of 100 or fewer might suffice if the purpose of the evaluation is formative or exploratory. Although obvious, it is important to state that precise calculations of sample size and statistical power will require a clear articulation of study goals and design.

• Recruit Clients Who Are at Risk of Homelessness

The study should recruit clients who are at significant risk of homelessness. Some programs serve only those who are homeless or at high risk of homelessness; however, most of the programs examined serve a significant proportion of clients who typically return to stable housing after the conclusion of their residential stays. Clients should be screened and included in an evaluation study only if they meet some risk threshold for homelessness.

• Develop Strategies To Track Early Terminators

Early terminators  those who leave programs after a brief stay, sometimes against professional advice  present particular challenges to any evaluation study of discharge planning. Some programs have high early termination rates (50 percent or higher) and followup data on these clients are often limited. These clients may be at the greatest risk of homelessness, yet are least likely to receive adequate treatment or discharge planning because of their early termination. They may also be more difficult to enroll and follow in a tracking study, but are critically important to include if the goal is to prevent homelessness.

• Use Followup Period of 1 Year or More for a Summative Evaluation

If a summative evaluation study of discharge planning in any of the four settings is undertaken, the study team recommends a followup period of 1 year or longer. The rationale for this position is based on two observations. First, the short-term base rate of homelessness following discharge is relatively modest in many settings, even given flawed, non-exemplary discharge planning processes. This rate can be highly variable across programs depending on the characteristics of the program and the population it serves, the availability of housing and services in the community, and other factors. Unless the followup period is extended to a year or more to allow a longer period for measuring homelessness (since the risk of homelessness often increases with time), it may not be possible to distinguish the impact of exemplary discharge planning in further lowering that already modest rate of homelessness. The second rationale relates to the ability of a study design to differentiate the effects of the discharge planning process from the progression of a disorder or the course of maturation. In the case of an individual in acute care treatment for mental illness and/or substance abuse, a short-term followup study of discharge planning runs the risk of confounding natural relapses with the outcomes of an inadequate discharge planning process. In parallel fashion, for studies of youth aging out of care, a longer followup period is necessary because of the possible confounding of developmental changes with factors attributable to discharge planning. A longitudinal design of a year or more allows for examining multiple transitions across settings and levels of care and better distinguishes between factors associated with the natural course of the individual's disorder or maturation and those factors attributable to the discharge planning and transition process.

• Meaningful Formative Evaluation Is Possible Without a Followup

Conversely, if the evaluation is formative or exploratory in nature an argument can be made for conducting a study that examines only what happens at the immediate point of discharge. In that case, the housing measure is not residential stability, but only what setting the client is placed in on the day of discharge. The measure of service linkage is not attendance at scheduled appointments, but only that appointments are made and the client informed. These types of measures are clearly insufficient to assure residential stability in the community but could, in conjunction with a variety of other measures, provide rough indicators of the quality of a program's discharge planning process.

• A Study Will Require Infrastructure for Data Collection and Followup

Any evaluation study examining homelessness outcomes will require a resource commitment to develop the infrastructure for data collection and client followup. Existing program data (hardcopy and electronic medical records) contain some but not all the information needed to conduct a discharge planning evaluation, and these data are of varying quality. Furthermore, most programs do not have the resources to follow up with clients after discharge or collect data on followup outcomes. The response rates for programs that do collect data on followup outcomes are inadequate for a rigorous evaluation. If a study is conducted it will be critical to tighten procedures and provide additional resources for program data collection and followup up; alternatively, the data collection activities could be contracted to external organizations with expertise in this area.

• Use Separate Studies for Different Program Settings

The four broad program settings addressed in this study actually represent many discrete types of programs. This observation suggests that care must be taken in determining which programs to compare in a discharge planning evaluation, and in generalizing evaluation findings from one program setting (or subtype within a setting) to another. For example, the psychiatric inpatient treatment settings category includes state psychiatric hospitals, psychiatric units of general hospitals, and free-standing private psychiatric hospitals. Yet these three types of psychiatric inpatient units differ in many important respects, including characteristics of clients served, length of stay, staffing patterns, risk of subsequent homelessness, and form and extent of linkage to community agencies, all of which bear upon the discharge planning process. Similar distinctions are apparent between subsets of programs within the other three settings.

Unlike the other three settings in the evaluability assessment, foster care independent living programs are not primarily treatment programs, but are fundamentally about assisting youth to make a transition to living independently in the community. Although these programs have processes that correspond to each element of exemplary discharge planning, they are unique in many respects and are subject to a range of particular influences and constraints. It would be particularly questionable to generalize findings from foster care independent living programs to other settings, or from the other settings to foster care.

• Identify Client Demographic and Clinical Characteristics To Be Measured

Differences in client characteristics affect the discharge planning process, options available to discharge planners, and postdischarge outcomes. Key client characteristics that should be measured in a discharge planning evaluation include presence of mental illnesses, substance use disorders, physical disabilities, developmental disabilities, co-occurring disorders, current and historical involvement with criminal or juvenile justice, and past success in mental health or substance abuse treatment. Critical factors for youth in foster care include the age of emancipation and educational attainment, as well as the presence of serious emotional disturbances, substance use disorders, and developmental disabilities. These individual history, demographic, and clinical characteristics also affect a client's eligibility for entitlements and services upon discharge, and influence discharge planning outcomes.

• Measure the Availability of Housing and Other Supports

Another critical variable is the availability of appropriate housing and supports in the communities the programs serve. The arrangement of stable housing and other needed services depends not only on the quality of the discharge planning, but also on the availability of appropriate resources in the community. Even in exemplary discharge planning programs, the outcomes achieved can be disappointing if the housing resources and services are not available.

• Measure the Policy Context

Each program is defined in part by the larger policy context in which the program operates (e.g., contractual obligations; accreditation standards and requirements; and state laws, rules, and regulations). The regulatory and accrediting bodies, like the payers, influence the conditions in the program and the discharge planning process. Their policies help determine which services are provided, how discharge planning activities are implemented, and who provides oversight.

• Measure the Program's Relationship to Other Organizations

The program's relationship to other organizations is also an important factor. If the treatment or custodial program is part of an umbrella agency that also provides outpatient care or housing, it may be easier to link clients to those intramural services. Similarly, if the program has invested in strong and trustful working relationships with community partner agencies, the housing and services provided by those agencies may be more easily accessible.

[ Go to Contents ]

Alternative Research Designs

We have identified at least four possible study designs, detailed in the full report, that would advance the field's understanding of discharge planning as an intervention to prevent subsequent homelessness. These studies are:

• Client Screening Protocols To Predict Risk of Homelessness. This study would examine the role of screening protocols in identifying people at risk of homelessness at discharge so that special efforts could be directed to securing appropriate placement. Such screening protocols have been developed, but their use does not appear to be common practice.
• Early Terminators/Foster Care Runaways and Methods To Engage Them. Foster care runaways and those who terminate prematurely from treatment programs are at high risk of homelessness. This study would aim to increase our knowledge of effective ways to engage this at-risk population and provide more effective discharge planning services.
• State Policies To Improve Discharge Planning and Prevent Homelessness. States have developed a range of policies intended to improve the discharge planning process in order to prevent homelessness. This study would catalogue those policies and their features for settings similar to those included in the evaluability assessment. It would also examine promising policies in greater detail, and identify common elements and themes associated with effectiveness; for example, use of performance measures, incentive provisions, penalties, and changes in the rates of homelessness over time.
• Discharge Planning Process and Outcomes. A quasi-experimental study targeting one of the four institutional or custodial settings in this evaluability assessment is the most rigorous alternative design proposed. It would examine the relationship between discharge planning practices and client outcomes over the 2-year period following discharge This research would be structured somewhat like the National Outcome Performance Assessment for the Collaborative Initiative to End Chronic Homelessness, but with comparison sites included in the original design, and could use some of the same instrumentation. The study could identify discharge planning practices that are effective in preventing homelessness.

[ Go to Contents ]

Endnotes

1. U.S. Department of Health and Human Services (DHHS). (2003). Ending chronic homelessness  Strategies for action:  A report from the Secretary's Workgroup on Ending Chronic Homelessness. Washington, DC: Author.

2. Center for Substance Abuse Treatment (CSAT). (1995). Detoxification from alcohol and other drugs, Treatment Improvement Protocol (TIP) #19. Department of Health and Human Services (DHHS Pub. No. BKD172). Substance Abuse and Mental Health Services Administration (SAMHSA).

3. Substance Abuse and Mental Health Services Administration (SAMHSA, 1997). Exemplary practices in discharge planning. Rockville, MD: Author.; American Association of Community Psychiatrists (AACP, 2001a). Continuity of care guidelines: Best practices for managing transitions between levels of care. Dallas: Author.; American Association of Community Psychiatrists (2001b). Continuity of care guidelines for addictions and co-occurring disorders. Dallas: Author.; Osher, F., Steadman, H. J., & Barr, H. (2002). A best practice approach to community re-entry from jails for inmates with co-occurring disorders: The APIC model. Delmar, NY: The National GAINS Center.