Evaluation of Family Preservation and Reunification Programs: Interim Report

Publication Date

Jan 7, 2001

Submitted to:
U.S. Department of Health and Human Services
Assistant Secretary for Planning and Evaluation
Room 450G, HHH Building
200 Independence Avenue, S.W.
Washington, D.C. 20201

Submitted by:
Westat, Inc.
1650 Research Boulevard
Rockville, MD 20850

Chapin Hall Center for Children
University of Chicago
1313 East Sixtieth Street
Chicago, IL 60637

James Bell Associates
2111 Wilson Boulevard, Suite 1120
Arlington, VA 22201

Executive Summary to the Interim Report

Background

This is a report of an evaluation of programs intended to prevent the placement of children in foster care when it can be avoided.¹ This report focuses on programs in three states, using a particular approach to family preservation, Homebuilders, thought by many to be the most promising approach.

Society has accepted a measure of responsibility for the well being of children, so government can intervene in family life when that well being is severely threatened by abuse or neglect, dependency due to death or disability of parents, or family conflict. Governmental intervention includes removing children from their homes when that is necessary. However, it has long been thought that children should remain in their parent's care whenever possible, consistent with their safety. The tension between assuring the safety of children and maintaining the integrity of families has been a perennial source of debate in the child welfare field and in our society more generally.

In 1980, Congress passed the Adoption Assistance and Child Welfare Act of 1980 (P.L. 96-272) which required states to make "reasonable efforts" to prevent children from entering foster care and to return children who are in foster care to their families. Part of the response of states to that Act was the development of family preservation programs. The emphasis on family preservation was further codified in the 1993 Omnibus Budget Reconciliation Act, which established a 5-year capped entitlement program to encourage the development of family preservation and family support programs. This program was revised and extended by the 1997 Adoption and Safe Families Act.

There have been a number of other evaluations of family preservation programs. Early evaluations suggested these programs had considerable promise but these studies were criticized for flaws in research design. Later, more rigorously designed studies began to cast doubt on the extensive claims of success. The largest of these studies were in California, New Jersey, and Illinois. No placement prevention effects were found in California and Illinois, while the study in New Jersey found short term effects that dissipated with time.² However, these studies were also criticized, most notably for not having examined programs thought to be most effective, those based on the Homebuilders approach.

The evaluation reported here was mandated by Congress in the 1993 legislation and was intended, in part, to provide information for deliberations on reauthorization of the funding. In using this report in discussions of reauthorization, it should be kept in mind that a relatively small portion of the Adoption and Safe Families Act funds is used for family preservation.³ It should also be noted that this report concerns only family preservation programs of a certain kind in three states. However, the three states were chosen because it was believed that they had implemented these programs effectively and because it was thought that they would represent the experience with the Homebuilders approach to service. It is hoped that the evaluation will also be useful to the states in making decisions about child welfare programs and to program planners and practitioners in developing responses to significant social problems.

The evaluation was designed to overcome shortcomings of previous studies of family preservation programs. It studied the Homebuilders model of service, the approach to family preservation that many observers believe to be the most effective. The evaluation also examined a number of outcomes. Placement prevention is a major goal of these programs, but family preservation is expected to achieve that goal while assuring the safety of children. A further important goal of these programs is improvement in functioning of parents, families, and children. Finally, it is expected that these programs will enable child welfare agencies to close cases more quickly, ending their involvement with families. Hence, besides placement prevention, the evaluation assessed the safety of children, changes in child and family functioning, and rates of case closing.

An additional issue raised in the earlier evaluations of family preservation concerned the targeting of these programs. It was found that the families served by these programs often were not those for whom they were intended: cases in which it was likely that at least one child would be placed in foster care without special intervention. The evaluation sought to throw light on this issue as well.

1. This is one of a series of reports from this evaluation. A previous report, The Evaluation of the New York City HomeRebuilders Demonstration reported on a program designed to facilitate the reunification of children in foster care with their families. A final report on the family preservation aspects of this project is forthcoming and will include data on a fourth site, Philadelphia, as well as further analysis of data from the three sites considered here.

2. J. Littell and J. Schuerman. (1995). A Synthesis of Research on Family Preservation and Family Reunification.http://aspe.hhs.gov/hsp/cyp/fplitrev.htm.

3. James Bell Associates. Family Preservation and Family Support (FP/FS) Services Implementation Study, Interim Report. March 1999. Prepared for U.S. Department of Health and Human Services, Administration for Children and Families. Contract # 105-94-8103.

The Homebuilders Model

Homebuilders, a foster care placement prevention program developed in 1974 in Tacoma, Washington, calls for short-term, time-limited services provided to the entire family in the home.⁴ The program is based, in part, on crisis intervention theory. This theory holds that families experiencing a crisis - that is, about to have a child placed in foster care - will be more amenable to receiving services and learning new behaviors. Social learning theory also plays a part in defining the Homebuilders model. Social learning theory rejects the belief that changes in thinking and feeling must precede changes in behavior. Instead, behavior, beliefs, and expectations influence each other in a reciprocal manner. Key program characteristics include: contact with the family within 24 hours of the crisis, caseload sizes of one or two families per worker, service duration of four to six weeks, provision of both concrete services and counseling, and up to 20 hours of service per family per week.

4. Jill Kinney, David Haapala, and Charlotte Booth. (1991). Keeping families together: The Homebuilders model. New York: Aldine de Gruyter.

Evaluation Design

The design for this evaluation was an experiment in which families were randomly assigned to either a Homebuilders family preservation program (the experimental group) or to other, "regular," services of the child welfare system (the control group). This report concerns programs in Louisville, Kentucky; seven counties in New Jersey; and Memphis, Tennessee. Information was collected through interviews with caseworkers and caretakers to examine caretakers' parenting practices, interaction with children, discipline, social networks, economic functioning, housing, abuse and neglect, psychological functioning, child well-being, and caseworker/caretaker interactions. These interviews were conducted with:

The investigating worker, caseworker, and caretaker of each family at the start of services;
The caseworker and the caretaker at the conclusion of family preservation services and at a comparable point in time for families in the control group; and
Caretakers one year after entry into the experiment.

After each in-person contact with families, experimental and control caseworkers completed a one-page form describing the services provided during the contact. Administrative data provided information on children's placements, reentries, and subsequent abuse and neglect allegations up to 18 months after entry into the experiment. Staff attitudes and characteristics were collected through a self-administered questionnaire. Throughout the project, discussions were held with personnel of the public agency and service provider agency to gather information about agency services, policies, staffing, training, and the context of services.

Site Descriptions

While data collection efforts were the same across sites, the sites varied in their approach to identifying families for services, the populations served, and the type of services provided (see Table 1).

Kentucky has a statewide program that uses the Homebuilders model. A state office coordinator is responsible for developing uniform selection criteria, training, contracting with family preservation providers, and overseeing the state program. The evaluation was conducted in Louisville, where there is a single family preservation provider agency, and child abuse and neglect cases are referred from intake or ongoing workers. A public agency screener reviewed all cases referred for family preservation services. Her role was to ensure that cases were appropriate for the service. There was no age limitation on the children included in the experiment. In Kentucky there were 174 cases in the experimental group and 175 in the control group.
New Jersey had a statewide program using the Homebuilders model at the time of the study. During the data collection, a state office coordinator was responsible for developing uniform selection criteria, training, contracting with providers, and program oversight. The study was conducted in seven counties: Bergen, Burlington, Camden, Essex, Monmouth, Ocean, and Passaic. Each county had a separate family preservation provider agency. The study population included Division of Youth and Family Service child abuse and neglect and family problem cases (primarily adolescent-parent conflict cases) referred from intake or ongoing workers. Each county had a screener to review cases referred for family preservation. Their major role was to review the appropriateness of the referrals and to make sure there were openings in the program. When the study began, the state was trying to refocus delivery of family preservation services to families with younger children. Not all counties conformed to this expectation, so all children under 18 were included in the experiment. In New Jersey there were 275 cases in the experimental group and 167 in the control group.
Tennessee had a statewide program using the Homebuilders model during the study period. It also had a state office coordinator responsible for developing uniform selection criteria, training, contracting with providers, and program oversight. The evaluation was conducted in Memphis and focused on families with children under 13 years old referred from the Department of Children's Services. Cases were referred only from intake workers. Prior to the study, workers referred cases directly to the family preservation program. For the study, cases were referred to a screener rather than directly to the program. In Tennessee there were 98 cases in the experimental group and 49 in the control group.

Table 1.
Study Site Descriptions
Program Description	Kentucky	New Jersey	Tennessee
Program Attributes
Location of evaluation	Jefferson County (Louisville) Fayette County (Lexington)	Bergen, Burlington, Camden, Essex, Monmouth, Ocean, and Passaic counties.	Shelby County (Memphis)
Program type	Statewide FP program	Statewide FP program	Statewide FP program
Program model	Homebuilders model	Homebuilders model	Homebuilders model
Responsibility for:
Selection criteria	State office coordinator	State office coordinator	State office coordinator
Training	State office coordinator	State office coordinator	State office coordinator
FP provider oversight	State office coordinator	State office coordinator	State office coordinator
Providers	Single FPS provider in study location.	Single FPS provider in each county location.	Single FPS provider in study.
Screener	Targeted cases were at high risk and should have entered care without FP. High risk family court cases where a petition was filed were reviewed for placement in the study. Public agency screener reviewed all cases referred to FPS for appropriateness.	Targeted cases were at high risk and should have entered care without FP. Each county had a screener to review cases referred for FP and make sure there were openings in the program.	Targeted cases were at high risk and would have entered care without FP. For the study, the screener referred cases to the FP program (prior to the study workers referred cases directly to program)
Population Attributes
Population criteria	FP cases referred from intake and ongoing units.	FP cases referred from intake and ongoing cases.	FP cases referred from intake only.
Child age limit	Children under 18 years of age. At time of study state the state was trying to refocus delivery of FP to younger children.	All children under 18. At the time of study the state was trying to refocus delivery of FP to younger children but not all counties modified targeting.	1 child in the family had to be under 13 years of age.

The Families

Most families in the study had birth mothers as the primary caretakers. About half of these women had not graduated from high school. Half of the households in Tennessee were headed by a single-birth mother, compared to 43 percent in Kentucky, and 34 percent in New Jersey (seeTable 2).

Table 2.
Description of the families at time of initial interviews
		Kentucky		New Jersey		Tennessee
		N	%	N	%	N	%
Gender of Caretaker/Respondent		311		328		117
	Male		7		12		7
	Female		93		88		93
Race of Caretaker/Respondent		310		327		116
	African American (not Hispanic)		43		42		83
	Caucasian (not Hispanic)		55		47		15
	Hispanic		1		9		1
	Other		1		2		0
Respondent's education level		311		325		116
	Elementary school or less		9		9		9
	Some high school		44		40		46
	High school graduate or obtained GED		32		26		18
	College		14		20		22
	Special education or vocational schooling		1		4		4
Respondent's marital status		310		328		117
	Married		24		30		17
	Divorced		19		23		13
	Separated		21		11		14
	Widowed		3		6		3
	Never Married		33		30		54
Respondent's relationship to youngest child		292		326		117
	Birth Mother		85		71		84
	Biological Father		7		10		6
	Grandmother		6		11		4
	Other Relative		2		8		5
Household composition		311		328		117
	Birth mother, no other adults		43		34		50
	Birth mother & 1 male adult		24		27		21
	Birth mother & extended family ^*		9		8		14
	Biological father ^*		6		9		6
	Other relative caretaker ^*		7		17		9
	Other ^**		10		4		1
Summary Statistics:		N	Mean	N	Mean	N	Mean
Age of respondent		306	33	324	39	116	33
Age of youngest		311	5	328	7	117	4
Age of oldest child		311	10	328	13	117	11
Number of kids		311	3	328	3	117	3
Number of adults		311	2	328	2	117	2

* These categories may also include other non-related adults in the home

** Includes: non-relative caretaker, adoptive or step-parent, birth mother & non-related females, or birth mother, and more than one non-related male.

At the time of referral to the Family Preservation program, families were experiencing a range of problems, some quite severe, others much less so (see Table 3). Examples included one case with children ages 10 and 12 who were not enrolled in school for nearly a month and were at risk of being removed from their home due to truancy and neglect. Another family was living in a home with no electricity, no heat, no food, no working appliances, a non-working toilet which was full of feces, and all 4 children slept in one bed. And yet another involved children who were sexually abused and who displayed extremely violent, uncontrollable and sexually inappropriate behavior at home and school. Although there was considerable diversity of problems, parental mental health and problematic child behavior were common issues.

At the time of the first interview, approximately half of the caretakers self-reported feelings of depression or stress. In Kentucky and New Jersey, approximately half of the caretakers answered affirmatively to each of three questions about emotional difficulties: "feeling blue or depressed," "feeling nervous or tense," and "feeling overwhelmed with work or family responsibility." Caretakers in Tennessee reported these difficulties at an even higher rate. Substantial proportions of caretakers reported behavioral problems in children. Between 60 and 74 percent said at least one of their children got upset easily, and two-thirds to four-fifths indicated that the children threw tantrums. Many said their children fight a lot with other kids (18% to 40%) and were very aggressive with their parents (18% to 43%). A number had problems in school, between 30 and 42 percent had children who had been suspended from school while 9 to 16 percent had children who had been expelled.

Table 3.
Selected Child and Family Problem Areas
(% responding yes)
Item	Kentucky	New Jersey	Tennessee
Caretaker Problems
Felt blue or depressed	55	58	62
Felt nervous or tense	56	52	53
Just wanted to give up	31	33	28
Overwhelmed with work or family responsibility	47	56	46
Not enough money for food, rent, or clothing	49	52	56
Participation in AFDC, food stamps, WIC, social security disability, or housing vouchers	82	68	80
Child Problems (%s of cases for which the question was relevant)
Child doesn't show much interest in what is going on	84	20	29
Child Get(s) upset easily	69	74	60
Throw(s) tantrums	83	79	67
Fight(s) a lot with other kids	33	40	18
Has/Have language problems	30	26	25
Is/Are very aggressive toward you	43	56	18
Hangs with friends you don't like	28	49	44
Been absent from school a lot	38	42	27
Run away from home overnight	10	26	21
Been temporarily suspended from school	30	32	42
Been expelled from school	11	9	16
Took something that didn't belong to him or her	34	42	27
Absent from school for no good reason	30	27	18
Failed any classes	27	41	38

Half or more of the respondents in all three states indicated that they did not have enough money for food, rent, or clothing. About two-thirds of the respondents in New Jersey and Tennessee reported they participated in at least 1 of the 5 income support programs: AFDC, food stamps, WIC, social security disability, and housing vouchers. In Kentucky, over 80 percent participated in one of these programs.

A number of families had previous involvement with the child welfare system. In Tennessee, 41 percent had previous substantiated allegations of abuse or neglect compared to 47 percent in Kentucky and 53 percent in New Jersey. In Kentucky and New Jersey, a fifth of the families had children who had previously been in foster care placement. In Tennessee, only 4 families had children who had previously been placed.

It might be noted that no mention is made here of substance abuse problems, thought by many to be a major issue in many families involved with the child welfare system. Very few caretakers admitted to alcohol or substance abuse in our initial interviews (fewer than five percent said they had either alcohol or drug problems except in Tennessee, where 8% said they "used drugs several times a week"). These are likely underestimates of the extent of substance misuse in the samples. However, states had policies regarding referrals to family preservation that may have limited the number of families with these problems. For example, New Jersey believed that family preservation should be used cautiously for substance abuse problems. Its FPS policy manual suggested that it is unlikely that a substance abuse problem can be resolved in a 5-6 week period. In Kentucky, families in which a drug dependent adult was not in active treatment were excluded from the program.

Service Provision

In all three states, the caretaker interview, the caseworker interview, and the contacts data generally confirmed the expectation that the experimental group would receive more services and more intensive services than the control group (see Table 4). In all three states, the number of experimental group caseworker activities reported by caretakers was greater than that reported by control group respondents, and this was also true of "helpful" caseworker activities. As for specific caseworker activities, experimental group workers in all three states were more likely to provide transportation, talk about discipline, and talk about how to handle anger.

The most common subject of counseling, interaction with children and in particular their discipline, reflects central problems in these families, problems of paramount concern to the child protective system. Experimental group caseworkers in all three states were more often reported to have talked about difficult issues, to have helped the caretaker to see her/his good qualities and problems, and to have "understood your situation."

Insofar as there are differences between groups, it can be assumed that the experimental conditions held since the experimental group received substantially more services than the control group. As is to be expected in real life implementations of models, the programs did not adhere completely to the Homebuilders approach as described above. In addition to other critical elements of family preservation, the Homebuilders model specifies that workers should provide an in-home contact within 72 hours of referral, and family preservation workers should be available 7 days per week. Substantial contact should take place within the first week; the model's developers suggest that the typical case receive 11 hours of service in that time. Concrete services are also an important component of service, particularly early in the case. Based on caseworker reports, families did not always receive contact within 72 hours, fewer than expected contacts occurred in the first week of the program, and few contacts occurred on weekends. There was relatively little provision of concrete services early on.

Table 4.
Summary of services, post-treatment interview
Caseworker Activities: (Proportion of affirmative answers by caretakers to yes/no questions)
	Kentucky			New Jersey			Tennessee
	C %	E %	p	C %	E %	p	C %	E %	p
Is Caseworker still working with family	79	64	0.006	75	31	0.001	57	34	0.02
Caseworker helped with money for rent, electricity, phone	3	17	0.001	5	4		5	10
Caseworker helped with money for other things	9	35	0.001	10	14		11	19
Caseworker provided transportation	16	42	0.001	12	25	0.003	19	34	0.10
Caseworker discussed proper feeding of child	14	20		5	11	0.06	16	28
Caseworker talked with you about discipline	35	55	0.001	39	60	0.001	46	70	0.01
Caseworker talked with you on relationship with spouse	16	18		8	14	0.09	11	34	0.01
Caseworker talked with you about how to handle anger	28	43	0.005	29	53	0.001	42	70	0.004
Caseworker told you about other agencies	38	43		42	56	0.01	19	33
Caseworker advised on job training programs	9	19	0.009	7	10		8	16
Caseworker talked about how to get paying job	6	17	0.004	5	8		11	18
Caseworker advised on how to continue school	9	18	0.04	5	8		14	23
Caseworker talked about uneasy issues	27	34		29	44	0.008	22	51	0.003
Caseworker helped you see good qualities	67	79	0.03	47	70	0.001	53	82	0.001
Caseworker helped you see your problem	66	76	0.10	52	72	0.001	50	82	0.001
Caseworker understood your situation	75	90	0.002	62	79	0.001	64	79	0.08
	C Mean	E Mean	p	C Mean	E Mean	p	C Mean	E Mean	p
CT report of # of Caseworker activities	2.18	3.90	0.0001	2.31	3.25	0.001	2.89	4.60	0.02
CT report of # of "helpful" Caseworker activities	1.04	1.68	0.0001	1.11	1.97	0.0001	0.83	1.33	0.04

NOTE: C = Control Group, E = Experimental Group

This table only includes items with a primary p-value less than .05 in at least one of the states; p-values greater than .10 are not reported.

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

Table 4.
Summary of services, post-treatment interview (continued)
Services Provided: (Proportion of affirmative answers by caretakers to yes/no questions)
	Kentucky			New Jersey			Tennessee
	C %	E %	p	C %	E %	p	C %	E %	p
Anyone been in job training program	3	8	0.09	2	3		3	4
Anyone been in WIC	32	45	0.02	22	20		51	41
Been in a marriage counseling program	0	7	0.006	2	2		0	1
Anyone receive daycare	5	19	0.001	10	7		26	26
Anyone receive transportation	7	16	0.02	14	12		17	19
Anyone receiving parent education/training classes	13	19		6	10		20	8	0.06
Anyone receive counseling	35	52	0.003	50	56		9	17
Anyone receive help finding a place to live	1	4		5	2		17	5	0.04
Anyone stay at an emergency shelter	1	1		2	1		6	0	0.03
Anyone receive medical or dental care	8	15	0.07	36	42		34	16	0.03
Anyone receive homemaker services	1	3		6	3		14	3	0.02
Were any needed services not gotten	27	19		56	42	0.01	39	24	0.10
	C Mean	E Mean	p	C Mean	E Mean	p	C Mean	E Mean	p
Caseworker report of # of services provided	3.16	4.99	0.001	2.31	3.17	0.001	1.58	3.19	0.0002

NOTE: C = Control Group, E = Experimental Group This table only includes items with a primary p-value less than .05 in at least one of the states; p-values greater than .10 are not reported. Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

Findings

This evaluation of family preservation programs was designed to assess the extent to which key goals of the programs are being met: the goals of reducing foster care placement, maintaining the safety of children, and improving family functioning.

The assessment of effects on placement and safety of children was based on administrative data which were available on families for at least one year after the beginning of service. Family functioning was assessed through interviews with caretakers at the beginning of service, one month later (at the end of service for the family preservation group), and a year after the beginning of service. Interviews with caseworkers were also conducted at the beginning and one month points.

No significant differences were found between the experimental and control groups on family level rates of placement, case closings, or subsequent maltreatment. There were a few child and family functioning items in which the experimental group displayed better outcomes than the control group in at least one of the states. However, these results did not occur in more than one state. It was found that family preservation programs in two states resulted in higher assessments by clients of the extent to which goals have been accomplished and of overall improvement in their families' lives.

Reducing foster care placement. In none of the three states were there statistically significant differences between the experimental and control groups on family level rates of placement or case closings (see Table 5). In Kentucky, placement rates at the end of one year were 23 and 24 percent for the experimental and control groups, respectively. In New Jersey and Tennessee, the percents were about 28 and 22 percent.

As to be expected with any program, some of the families assigned to family preservation programs did not receive the services or received a minimal dosage of the services. Also, a small number of the families in the control group were actually provided family preservation services. To address these issues, analyses were conducted in which these cases were dropped (secondary analysis). Results of the secondary analyses were quite similar to the primary analyses, also showing no significant differences between the groups in rates of placement.⁵

The ideal family preservation case is one in which there has been a recent significant crisis in the family, resulting in the maltreatment that triggers the possibility of removal of the child from the home. Subsamples of cases that approached this ideal were examined. Again in these analyses there were no statistically significant differences between the experimental and control groups in placement rates over time.

In addition to placement rates at various points in time, placement was examined in terms of proportion of time in substitute care subsequent to random assignment. No significant differences were found in care days for the families in any of the three states. In Kentucky both the experimental and control group children spent an average of 6 percent of the days subsequent to random assignment in care. In New Jersey, experimental group children spent an average of 6 percent of that time in placement compared to 4 percent for the control group children. In Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children.

Table 5.
Summary of Placement Data, Survival Analyses
Percents of families experiencing placement of at least one child within specified periods of time
Kentucky	One month (%)		6 months (%)		One year (%)
	E	C	E	C	E	C
Primary	6	5	18	18	23	24
Secondary	4	4	13	17	20	24
Refined Analyses:
Investigative cases, primary	8	5	16	14	26	15
Recent substantiated, primary	6	3	17	12	29	16
Petition cases, primary	7	10	14	26	18	33
Including relative placement, primary	8	9	21	25	27	32
Including relative placement, secondary	5	9	14	25	22	32

New Jersey	One month (%)		6 months (%)		One year (%)
	E	C	E	C	E	C
Primary	4	6	19	16	28	22
Secondary	3	6	18	16	26	22
Refined Analyses:
Investigative cases, primary	3	5	18	13	25	16
Recent substantiated, primary	8	5	20	12	27	15

Tennessee	One month (%)		6 months (%)		One year (%)
	E	C	E	C	E	C
CORS, primary	11	11	22	19	23	19
CORS, secondary	7	12	18	19	19	19
Including relative placement, primary	11	11	26	21	28	23
Including relative placement, secondary	7	12	20	19	23	21
Refined Analyses:
Recent investigation, CORS	7	12	15	15	17	15
Recent investigation, includes relative placement	7	12	18	18	22	21

NOTE: Primary analyses included all cases randomly assigned, except for cases that were determined to be inappropriate referrals. Secondary analyses dropped two categories of cases: families in the control group that were actually provided family preservation services ("violations") and families in the experimental group that received no or little service ("minimal service cases"). "Refined" analyses were limited to subgroups that were thought to represent better targeted cases. Most of the analyses above are of records of placements in administrative data. In Kentucky and Tennessee data were also available from case records on placements with relatives that were not recorded in the administrative data. Those data are included in the rows labeled "including relative placement."

Targeting. Since these programs were intended to prevent the placement of children, the target group for the services was families in which at least one child was "in imminent risk of placement." As in previous studies, it was found that most of the families served were not in that target group. This is shown by the placement rate within a short period of time in the control group, indicating the placement experience in the absence of family preservation services. In all three states, the placement rate in the control group within one month was quite low. It would, therefore, have been virtually impossible for the programs to be effective in preventing imminent placement, since very few families would have experienced placement within a month without family preservation services.

A number of subgroups that were thought to represent better targeting were examined. These included cases coming directly from the investigation of an allegation of abuse or neglect, cases with recent substantiated allegations of abuse or neglect, and, in Kentucky, a subgroup of cases in which workers had submitted petitions to the court for placement or some other court-ordered intervention. In none of these subgroups did placement rates in the control group within one month exceed 12 percent. Hence, even in these more refined (from the standpoint of targeting) subgroups, the intended target group was not in evidence.

It should be noted that the results found here occurred despite efforts in this project to improve targeting. In Kentucky and New Jersey a special screening form, developed by the evaluation team, was employed to rate the risk to children with the intent that cases with intermediate risk would be referred to the program. In Kentucky efforts were made to divert to family preservation cases that had been referred to the court. In Tennessee, special training efforts were instituted to address concerns about targeting.

Child Safety. Maltreatment subsequent to the beginning of service was generally not related to experimental group membership, except for one subgroup in Tennessee. Subsequent maltreatment was measured by the occurrence or nonoccurrence of a substantiated allegation of abuse or neglect following an investigation of such an allegation. The rate of subsequent maltreatment was relatively low, about 18 percent of the families in Kentucky had a substantiated allegation within one year of random assignment; in New Jersey the rate was 12 percent and in Tennessee, 25 percent. In Tennessee, in those families with an allegation within 30 days prior to random assignment, the experimental group children experienced fewer substantiated allegations than children in the control group did.

The findings of little difference between the experimental and control groups in subsequent maltreatment can be read in two ways. It indicates that families served by family preservation were no more likely than families not receiving the services to be subjects of allegations of harm. In this sense, children were, by and large, kept safely at home while receiving family preservation services. However, children in both groups were primarily in their homes, and family preservation did not result in lower incidence of maltreatment compared with children in the control group.

Subgroups. In an effort to identify groups of cases for which family preservation is effective, subgroups of Kentucky and New Jersey cases were examined.⁶ Subgroups were defined in terms of problems of the family (e.g., substance abuse, financial difficulties, and depression) and family structure. Within these subgroups, experimental and control groups were compared on placement and substantiated allegations subsequent to random assignment. Only one significant difference was found. Among single mothers in New Jersey, those in the experimental group were less likely to have a subsequent substantiated allegation than those in the control group. No subgroups were found in which there were effects on placement in either state.

Family functioning. In a few areas of family functioning, in one or the other of the states, families in the experimental group appeared to be doing better at the end of services (see Table 6). There were very few differences at the year follow-up and in changes over time. Those differences that did appear (primarily at the end of services) were not consistent across states and were not maintained. Family functioning was assessed through caretaker and caregiver interviews at three points in time, shortly after the beginning of services, four to six week later (at the end of services for the Homebuilders group), and again a year after services began. Areas assessed included life events, economic functioning, household condition, child care practices, caretaker depression, child behavior, and caretaker functioning. It can be said that family preservation services may have small, apparently short-term, effects on some areas of functioning. There was one item with some consistency across sites, the overall assessment of improvement by caretakers. At post treatment, a significantly larger proportion of experimental group caretakers in Kentucky and New Jersey generally thought there was "great improvement" in their lives. In Tennessee, although not significant, results tended in the same direction.

Table 6.
Summary of family and child functioning outcomes
Differences between experimental and control groups at post treatment, follow up, and change over time
Area	Post treatment	Follow up (one year after beginning of treatment)	Change over time
Life events
Positive life events	No significant differences	No significant differences	No significant differences
Negative life events	No significant differences	No significant differences	No significant differences
Depression	No significant differences	No significant differences	No significant differences
Family problems, individual items	KY: no significant differences NJ: fewer experimentals not enough money for food, rent, or clothing TN: fewer experimentals had few or no friends	No significant differences
Economic functioning
Individual items	KY: no significant differences NJ: fewer experimentals difficulty paying rent and buying clothes TN: no significant differences	KY: no significant differences NJ: no significant differences TN: fewer experimentals difficulty paying rent
Scale	KY: no significant difference NJ: experimental average lower (better) TN: no significant difference	No significant differences	No significant differences
Household condition
Individual items	KY: experimentals fewer broken windows or doors NJ: no significant differences TN: more experimentals in unsafe building because of illegal acts	No significant differences
Scale	No significant differences	No significant differences	No significant differences
Area	Post treatment	Follow up (one year after beginning of treatment)	Change over time
Child care practices
Individual items	KY: fewer experimentals used punishment for not finishing food NJ: experimentals less often got out of control when punishing child and more often encouraged child to read a book TN: more experimentals went to amusement park, pool, or picnic	No significant differences
Positive scale	No significant differences	No significant differences	No significant differences
Negative scale	KY: no significant difference NJ: experimentals lower (better) TN: no significant difference	No significant differences	No significant differences
Punishment	KY: no significant difference NJ: experimentals lower (better) TN: no significant difference	No significant differences	No significant differences
Caretaker depression	No significant differences	No significant differences	No significant differences
Child behavior
Aggression	No significant differences	No significant differences	No significant differences
School problems	No significant differences	No significant differences	No significant differences
Positive child behaviors	No significant differences	No significant differences	No significant differences
Negative child behaviors	KY: no significant differences NJ: experimental group lower (better) TN: no significant differences	No significant differences	No significant differences
Overall assessment of improvement	KY: experimentals, greater improvement NJ: experimentals, greater improvement TN: no significant difference	No significant differences
Caseworker report of caretaker functioning
Individual items	KY: no significant difference NJ: control group higher (better) in ability in giving affection and providing learning opportunities TN: experimental group higher (better) on five items		KY: respecting child's opinions: experimental group declined, control group increased NJ: control group had more positive change in respecting child's opinions TN: experimental group more positive change on setting firm and consistent limits
Scale	KY: no significant difference NJ: no significant difference TN: experimental group higher (better)		No significant differences
Caseworker report of household condition	KY: control group better NJ: control group better TN: no significant difference		No significant differences
Caseworker report of caretaker problems	KY: experimentals more problems NJ: no significant difference TN: no significant difference		KY: no significant difference NJ: no significant difference TN: experimentals declined more
Caseworker report of child problems	No significant differences		No significant differences

5. It should be noted that the most rigorous approach to analysis requires that cases be maintained in the groups to which they were randomly assigned. Random assignment is used to assure that the groups are as similar as possible at the outset of service. Removing cases from the groups or switching cases from one group to another threatens group equality and allows for the possibility that post-treatment differences could be explained by factors other than service. In particular, it is likely that violations and minimal service cases differ in systematic ways from other cases. Hence, the secondary analyses should be viewed with caution.

6. The number of cases in Tennessee was too small to allow subgroup analysis.

Implications

The findings of this study are not new. A number of previous evaluations with relatively rigorous designs have failed to produce evidence that family preservation programs with varying approaches to service have placement prevention effects or have more than minimal benefits in improved family or child functioning. The work reported here may be thought of as three independent evaluations, in three states, adding to the set of previous studies with similar results, this time focusing on Homebuilders programs. The accumulation of the findings from a number of studies in several states, with varying measures of outcome, is compelling.

The findings should not be taken as showing that these programs serve no useful purpose in the child welfare system. The results can be seen as a challenge to keep trying, to find new ways to deal with the problems of families in the child welfare system. The findings indicate the grave difficulties facing those who devise approaches to these problems, failure in such undertakings should not be surprising, and those who risk trying to find solutions should not be punished when evaluations such as this indicate they may have come up short.

The accumulation of findings suggests that the functions, target group, and characteristics of services in programs such as this need to be rethought. Obviously, function, target group, and services are closely intertwined. The foremost of these issues concerns the objectives of the programs. A number of observers have suggested that placement prevention be abandoned as the central objective in intensive family preservation services in favor of other objectives, notably the improvement of family and child functioning. Targeting these services on families at risk of placement is unlikely to be successful, so if these services are to continue, they will continue to serve "in-home" cases, families in which there has been a substantiated allegation of abuse or neglect or serious conflicts between parents and children but in which children remain in the home. Many, if not most, of these "intact" families need help. Relatively intensive and relatively short-term services such as those provided by family preservation programs are one source of such help. In this respect, family preservation programs can be thought of as an important part of the continuum of child welfare services.

There are some positives in the findings of this study. Services provided to Homebuilders clients were considerably more extensive and intensive than those provided to control group families. This translated into more positive assessments by caretakers of the relationships they had with workers (Homebuilders clients also rated their overall improvement during the service period as greater). Unfortunately, this apparently better relationship did not translate into observable effects on placement rates or changes in functioning. The challenge for programs is to make use of better relationships to bring about changes in functioning.

Another question that program designers must address is that of specialization. Subgroups for which the program was successful were not found, but these programs are quite generalist in character, and thus may sacrifice some of the benefits of specialization. Among those benefits are a clearer focus of services, tighter target group definition, specification of service characteristics such as length and intensity based on needs of the target group, and the development of more specific competencies on the part of workers. Specialization could be in terms of problems (e.g., substance abuse) or characteristics of clients (young, isolated mothers). There are clear drawbacks to specialization, including the tendency to define problems in terms of the service one offers. Furthermore, limiting target groups inherently limits the impact of programs. Nonetheless, it may be better to mount a series of small programs rather than putting all of one's resources into large, undifferentiated efforts.

Program planners must also address the issue of length and intensity. The extent to which the intensive-short-term-crisis approach of these services fits the needs of child welfare clients needs to be reexamined. The lives of these families are often full of difficulties — externally imposed and internally generated — such that their problems are better characterized as chronic, rather than crisis. Families with chronic difficulties can no doubt benefit from short-term, intensive services, but those services are unlikely to solve, or make much of a dent in the underlying problems. Of course, the hope is that family preservation programs will be able to connect families with on-gong services to treat more chronic problems, but that appears to happen far less than needed. The central point here is that we need a range of service lengths and service intensities to meet the needs of child welfare clients. It is essential that policy makers, planners, and program providers maintain realistic expectations of the effects of short-term family preservation programs.

Chapter 1: Study Overview

1.1 Background

In 1980, the Adoption Assistance and Child Welfare Act of 1980 (P.L. 96-272) required states to make “reasonable efforts” to prevent children from entering foster care and to reunify children who were placed out of the home with their families. A major focus of policy and planning in state child welfare systems was the development of family preservation programs. The emphasis on family preservation culminated in 1993 in the Family Preservation and Family Support provision of the Omnibus Budget Reconciliation Act (OBRA) (Title IV, subpart 2 of the Social Security Act), which encouraged states to institute or further develop family preservation and family support.

As part of the legislation, the Department of Health and Human Services (HHS) was authorized to set aside funds to evaluate State family preservation and family support programs. In support of this, HHS funded three separate studies in September 1994:

Family Preservation and Family Support Services Implementation Study. This study was awarded to James Bell Associates and is a process analysis of the implementation of the legislation, focusing on the types of programs developed and the barriers encountered. The interim report, “Family Preservation and Family Support (FP/FS) Services Implementation Study,” was released March 1999.
National Evaluation of Family Support Programs. This study was awarded to Abt Associates, Inc. and is an outcome evaluation of family support programs.
The Evaluation of Family Preservation and Reunification Services. This study was awarded to Westat, Chapin Hall Center for Children, and James Bell Associates and is the subject of this report. It is an outcome evaluation of family preservation and reunification programs.

The three projects are designed to be complementary. Although each focuses on a different aspect of the 1993 legislation, taken together they represent a comprehensive examination of the programs authorized.

More recently, the enactment of the Adoption and Safe Families Act of 1997 changed and clarified a number of policies established in the 1980 Act with a renewed emphasis on safety, permanency, and adoption. This legislation placed Federal family preservation initiatives under the rubric of “Promoting Safe and Stable Families” and extended the funding for three years, through FY 2001. The law made safety of children the paramount concern in service delivery. The law increased the need to understand how family preservation services strengthen families and prevent foster care placement and subsequent abuse and neglect allegations.

Concurrent with the development of legislation have been program initiatives in family preservation at the state and local levels. Since the 1970s, a number of programs have been developed to provide services to children and families who are experiencing serious problems that may eventually lead to the placement of children in foster care or otherwise result in the dissolution of the family unit. Although these programs share a common philosophy of family- centered services, they differ in their treatment theory, level of intensity of services, and length of service provision. Three models have emerged (Nelson, et al. 1990):

Crisis intervention model. This model, based on crisis theory and intervention stresses the situation of everyday people confronted with unstable and unsecure circumstances from precipitating events, and the belief that symptoms can be worked through in a brief amount of time (Barth, 1990). Crisis theory also holds that those experiencing a crisis — that is, families about to have a child placed in foster care — will be more amenable to receiving services and learning new behaviors (Nelson et al, 1990, citing Kinney et al, 1988). Homebuilders, a foster care placement prevention program developed in 1974 in Tacoma, Washington, is the prototype program for the crisis intervention model. The program calls for short-term, time-limited services provided to the entire family in the home. Services are provided to families with children who are at risk of an imminent placement into foster care. Social learning theory also plays a part in defining the Homebuilders program, providing the theoretical base for interventions employed (Nelson et al, 1990). Social learning theory stresses that behavior, beliefs, and expectations influence each other in a reciprocal manner, and rejects the belief that changes in thinking and feeling must precede changes in behavior (Barth, 1990). Concrete and supportive services are an important element of the Homebuilders program. Key program characteristics include: contact with the family within 24 hours of the crisis, caseload sizes of one or two families per worker, service duration of four to six weeks, provision of both concrete services and counseling, and up to 20 hours of service per family per week (Nelson et al, 1990).
Home-based model. This model focuses on the behavior of the family overall, how members interact with one another, and attempts to change the way in which the family functions as a whole and within the community. Programs using the home-base model stress longer term interventions based on family systems theory. The FAMILIES program which began in Iowa in 1974 is the original program using the home-base model. Under the original program in Iowa, teams of workers carried a caseload of 10 to 12 families whom they saw in the families’ homes for an average of four and one-half months. Both concrete and therapeutic services are provided (Nelson et al, 1990).
Family treatment model. This model focuses less on the provision of concrete and supportive services and more on family therapy (Nelson et al, 1990, citing Tavantzis et al, 1986). Services are provided in an office as well as in the home and are less intensive than those using the crisis intervention model. The Intensive Family Services (IFS) Program, which began in Oregon in 1980, is based on the family treatment model. The IFS program also uses family systems theory, which views individual behavioral problems as a reflection of other family problems. Therefore, treatment focuses on the family as a whole. Workers carry a caseload of approximately 11 families. Services are provided for 90 days with weekly follow-up services provided for three to five and one-half months (Nelson et al, 1990).

Over the years, various states have adopted these family preservation models, sometimes with variations. The growth in family preservation can be partly attributed to early evaluations that were “unequivocally positive and reported high placement prevention successes” (Bath, Howard, and Haapala, 1993). Primarily, these studies only measure family outcomes such as placement prevention for families who receive the treatment. No comparison was made to families who did not receive the services. It was assumed that nearly all children would be taken into foster care placement. However, it cannot be assumed that a high proportion of children receiving family preservation services were at imminent risk without observing the experiences of a comparison group that did not receive the intervention. More recent studies using experimental designs have shown that most of the cases referred were not at imminent risk of placement, as many children in the control groups did not become part of the foster care population.

Although many nonexperimental studies have suggested that high percentages of families remain intact after intensive family preservation services, the results of randomized experiments are mixed. Seven of eleven studies reviewed in A Synthesis of Research on Family Preservation and Family Reunification (Littell and Schuerman, 1995) found that the programs did not produce significant overall reductions in placement. In less than half of the control or comparison cases, placements did not occur within a short period of time after group assignment, which suggests that these programs were generally not delivered to families with children at risk of placement. When the risk of placement among family preservation clients is low, it is unlikely that a program will demonstrate significant reductions in placement.

Despite these findings, placement prevention remains a primary goal of family preservation programs. A review of family preservation programs was conducted in 1995 as part of the Evaluation of Family Preservation and Reunification Services. Information from that study was updated in 1997. As part of the update, 32 family preservation state coordinators were asked if placement prevention was the primary purpose of their program. The majority (78 percent) indicated that it was still the primary purpose, with the remaining coordinators identifying child safety (18 percent) and family functioning (4 percent) as the primary purpose. These goals broaden when county public agency and family preservation administrators were asked about the objectives of local family preservation progress. From the 32 states, 58 county public agency administrators and family preservation program administrators were asked to describe their family preservation objectives. Of the 58 administrators contacted, most offered multiple service objectives. The most frequently reported objective was placement prevention, followed by strengthening families and child and family safety. The purpose of the Evaluation of Family Preservation and Reunification is to test whether these service delivery objectives are attained.

1.2 Study Objectives

The Evaluation of Family Preservation and Reunification is intended to estimate the impact of family preservation and reunification services. The design of the evaluation was guided by the following objectives:

To identify and describe the range of existing placement prevention, family preservation, and reunification programs;
To determine the extent to which family preservation and reunification programs are effective in safely reducing unnecessary foster care placement;
To determine the extent to which family preservation programs are effective in meeting the basic needs of children and in promoting improved family functioning;
To explore the extent to which family preservation/reunification programs have varying degrees of success with different target populations;
To determine the extent to which program variables, child welfare system variables, and other factors in the service delivery environment affect the success of family preservation and reunification programs;
To identify the effects of each family preservation/reunification program on its related child welfare system; and
To compare the costs of family preservation/reunification services to those of control groups.

The evaluation is being conducted through randomized experiments in four family preservation sites: Kentucky, New Jersey, Tennessee, and Pennsylvania and the evaluation of an earlier implemented reunification program in New York City. The classic experimental design of this study is the best way to determine causal connections between interventions and outcomes. The control group received the “regular services” of the child welfare system; it was not a no-treatment control group. We studied the effects of the experimental services relative to ordinary services, services that would have been provided in the absence of family preservation services.

1.2.1 Site Selection and Recruitment

Site selection was based on a number of criteria, including selecting programs which were based on well-articulated theories, in place long enough to operate in the way expected by program managers, consistently implemented, and with sufficient numbers of families to provide adequate sample sizes. It was also important that programs have a primary focus on a population of children involved in abuse and neglect reports and that key policymakers, managers, and line staff were willing to allow evaluation. Initially, it was proposed that of the six sites to be evaluated, at least two would be placement prevention programs, two broader family preservation programs, and two reunification programs.

Emphasis was placed on selecting well-defined programs and those with characteristics useful for the development of knowledge (e.g., serving clientele with substance abuse problems). It was decided to evaluate three programs that use relatively “pure” versions of the Homebuilders model of service. These include Memphis, Tennessee; Louisville and Lexington,¹ Kentucky; and seven counties in New Jersey. The fourth family preservation/placement prevention site, Philadelphia, has a program in which the goal of family preservation services is defined more broadly than placement, compares family preservation services to less intensive in-home services, and has an explicit focus on substance abuse.

Our program review established that there were few reunification programs, and those that existed served small numbers of clients. Most reunification programs were part of family preservation programs and served families after discharge from foster care. We decided to examine the HomeRebuilders reunification program in New York City, by conducting the data collection for the experiment started by the New York State Department of Social Services. We were not able to identify a suitable site for a second experimental evaluation of reunification.

(1) Lexington, Kentucky, remained in the study only a short time. Further details on Lexington are presented in Chapter 3, Kentucky Overview.

1.2.2 Sample Size

Each site was evaluated separately. To detect a difference of 15 percentage points between the experimental and control groups in such characteristics as placement rates with a probability of .8, we set a goal of 500 cases in each site, about 250 in each group. Even with this sample size, our ability to detect small differences is limited. Initially we hoped to enroll 500 families in each site over a one-year period. However, the sample accumulation in sites in this report, Kentucky, New Jersey, and Tennessee was slower than expected. A 349-case sample size was achieved in Kentucky after enrolling families for two years. In New Jersey, 442 net cases were enrolled over an 18-month period and in Tennessee, 147 net cases were enrolled over a 21-month period.

1.3 Data Elements and Measures

Outcome measures relate to the goals of the programs and require multiple measures, including placement, subsequent maltreatment, family problems, and child and family functioning. Outcome measures are the heart of the experiment, but other types of measures were also needed in order to carry out the study and to more fully understand the observed overall impact in specific sites. Other measures include mediating and conditioning variables. Mediating variables reflect intervening factors that may be the underlying mechanism for achieving change in the more general outcomes, including parents’ coping skills, the family’s social isolation or embeddedness, and the general quality of interactions in the home environment. There is not always a clear dividing line between mediating and outcome measures. Moreover, an outcome in one realm may be a mediator in another. For instance, adequacy of the parent’s attention to a child’s health may be considered an outcome as itself, but it is also a key mediating variable in relation to other outcomes. Measures that may “condition” the effects of the treatment, such as demographic and household composition variables, were examined for their potential influence. For example, family preservation services may emerge as more effective for families with certain characteristics (e.g., single parent families, younger children). We also used check measures to ensure that the treatment that was intended actually occurred and to determine whether control group families received services that are supposed to be reserved for members of the experimental group. Finally, the study used service variables to identify at the program level those variables necessary for understanding the results at the family level.

1.4 Data Sources

To obtain these measures, we used multiple data sources, including administrative data, interviews with workers and caretakers, and qualitative data collection on program operation and context.

For family preservation/placement prevention sites, the study used a longitudinal design in which caretakers were interviewed at three points in time: when they entered the study, at the end of services, and at 1 year after entry to the study. Caseworkers were interviewed at two points in time, when the family entered the study and at the end of services. Investigating workers completed a self-administered form as quickly after assignment as possible. They were asked to provide a description of the allegation and the investigation findings. Caseworkers were asked to provide information on the actual services provided during in-person contacts with the family during treatment for the experimental cases and during a comparable time period for the control cases. Administrative data on placement and subsequent maltreatment were collected for 18 months after enrollment on each case. An analysis of costs of family preservation programs was also conducted.

This report includes analyses of data from Kentucky, New Jersey and Tennessee. A final report will provide further analyses on these three sites as well as on the Pennsylvania family preservation experiment.

References

Barth, Richard P. “Theories Guiding Home-Based Intensive Family Preservation Services.” In Reaching High-Risk Families, Eds., Whittaker et al. Aldine de Gruyter, New York: 1990.

James Bell Associates. (1997). “Interim Report: The Family Preservation and Family Support Services (FP/FS) Implementation Study Volume I,” submitted to DHHS, ACF, contract # 105-94-8103.

Nelson et al. “Three Models of Family-Centered Placement Prevention Services,” Child Welfare, January/February 1990.

Nelson K. (1994). Family-based services for families and children at risk of out-of-home placement. Child Welfare Research Review I, 83-108.

Westat, Chapin Hall Center for Children, and James Bell Associates. (1998). “A Review of Family Preservation and Reunification Programs” submitted to DHHS, ASPE, contract No HHS-100-94-0020.

Westat, Chapin Hall Center for Children, and James Bell Associates. (1998) “Evaluation of the New York City HomeRebuilders Demonstration,” submitted to DHHS, ASPE, contract No HHS-100-94-0020.

Chapter 2: Implemention

Designing a rigorous experimental study is only the first step; its implementation is a formidable task. Convincing administrators to subject their staff and programs to intense scrutiny is the first challenge, followed by implementing the evaluation in an ongoing service delivery environment. Negotiations required repeated meetings with administrative, supervisory, and front-line staff. We had to establish a dialogue to foster open communication in which fears, expectations, and study requirements could be discussed. Implementation required continual communication with site personnel. This communication included periodic site visits, monthly written reports to sites about the status of cases enrolled in the study, and a site coordinator stationed at each site to aid in the daily data collection effort. This chapter presents an overview of site selection, negotiations, and the data collection effort. Further description of site-specific implementation efforts in Kentucky, New Jersey, and Tennessee are presented in Chapters 3, 4, and 5.

2.1 Site Selection and Recruitment

The site selection process began with a discussion with personnel in potential sites of the issues and criteria surrounding site selection. The task of applying these criteria to real programs began a process of reconciling the differences between our hope of finding optimal sites and program and practice realities. We initially identified potential states and counties for the study through review of state plans, contacts with experts in the field, reviews of the literature, and previous studies conducted by the research team. Based on this review, we contacted 26 states and asked them about their family preservation and reunification programs with respect to our criteria for selection.

A list of programs and counties contacted is presented in Appendix A. Results of the telephone conversations with these sites were presented in the Review of Family Preservation and Reunification Programs. Based on responses to the telephone conversations and extensive discussion among research team members and the advisory panel, we eliminated a number of states or particular counties within states from consideration.

To obtain more detailed information about states, site visits were necessary. As we were unable to conduct site visits to all identified states, we established two levels of site visits. The first level targeted states that had some of the best and most mature programs in the country: Kentucky, Michigan, Missouri, New Jersey, and Washington. Project staff conducted 3 to 5 day site visits at the state level and in those local jurisdictions that might be included in the study. The visits included meetings with administrative, supervisory, and casework staff at the state and local levels of the public child welfare agency. We also conducted interviews with administrators and caseworkers of the local family preservation agencies. Through the interviews, we gathered information about family preservation services and the context in which the services were being delivered. States’ interest in the study and their ability to meet selection criteria were also explored. We then conducted further site visits in Tennessee, Oregon, California, Florida, New York, and Ohio.¹ Our emphasis was on selecting quality programs and those with characteristics useful for the development of knowledge (e.g., serving clientele with substance abuse problems). It was decided to evaluate three programs that reported using relatively “pure” versions of the Homebuilders model of service. The sites selected were Memphis, Tennessee; Louisville, Kentucky; and seven counties in New Jersey. These three sites met the original criteria set forth by contract requirements and also incorporated the other issues identified as important. All three sites identified a targeting problem and were interested in implementing targeting strategies, had a long and positive history of providing quality Homebuilders’s programs, had a limited number of providers, and had adequate support for the program in the responsible public agency. Also, all the sites identified a pool of families who were eligible for the services but not receiving them and had sufficient numbers to reach study sample size requirements (or agreed to continue the study for more than a year, if necessary).

For the fourth site, our efforts turned to identifying a non-Homebuilders-based family preservation program which was well defined and able to articulate its goals and objectives. While the study team visited Philadelphia to explore its reunification programs, its family preservation program was also presented as an option. The program has many interesting and policy relevant elements. The family preservation programs in Philadelphia are based on specialization, and the county has a strong focus on serving families with substance abuse problems. Philadelphia County represents a site in which the goal of family preservation services is defined more broadly than placement prevention, allowed comparison of family preservation services to less intensive in-home services, and has some agencies with an explicit focus on substance abuse. These criteria lent themselves to the selection of Philadelphia for the fourth site.

This report focuses on the initial findings from Kentucky, New Jersey, and Tennessee. The Philadelphia findings will be presented at a later date.

1 The Family Preservation and Family Support Implementation Study was selecting sites at the same time. It was decided that conducting both studies in the same site would be too burdensome for states: therefore, Alabama, Arizona, Texas, and Los Angeles. California were eliminated as candidates for the second round of site visits.

2.2 Negotiations

Negotiations began during the initial site visits. Discussions with staff focused on obtaining information on the state program and system while providing information to the state about the study. Site visitors needed to determine, as quickly as possible, if states were not interested in participating. Also, we had to establish site flexibility in working within the study guidelines and adhering to rigorous data collection methods early in the negotiation process. Negotiations always began at the state level to obtain permission from the child welfare commissioner or director. Although negotiations were tailored to individual sites, we followed general procedures which entailed numerous meetings with state and local personnel, written permission from the state director of child welfare services, and an agreed-upon detailed work plan delineating target populations, random assignment procedures, data collection plans, and targeting procedures. It was critical to go through a process with state and local agency personnel in which we explored their receptiveness to an experiment, including some alteration in referral procedures and a willingness to fill out our forms and partake in interviews. The most difficult process was working through workers’ concerns about withholding services from the control group. Extensive discussions were held about denying services to clients and having a computer make decisions about families’ lives. Although many caseworkers never felt totally comfortable with the idea of randomly assigning families to receive either family preservation services or other services, they eventually became resigned to the procedure. Many did come to accept that the experiment was set up to provide services to the same number of families served by family preservation prior to the study and understood that their present systems did not serve all families eligible for family preservation. It was more difficult for them to accept that particular families on their caseloads could not receive a service which they believed to be the best alternative for the families.

Targeting. A major problem that has plagued family preservation programs and their evaluations is targeting. To prevent placements effectively, these programs have been intended for cases in which there is an “imminent risk of placement.” Previous studies have indicated that family preservation services are often delivered to families in which placement is not likely. A goal of this evaluation was to address the targeting problem in at least some of the placement prevention programs to be studied so that the programs would have the best possible chance of success on the outcome measure of preventing foster care placement. We selected sites that realized that targeting was an issue and that were interested in developing strategies to improve targeting. We believed that targeting could be improved through removing from the referral pool some of the cases that would not experience placement in the absence of family preservation services or through diverting to the family preservation referral pool some cases that were placed. This might be called screening out the cases that are not at imminent risk of placement and screening in the cases that are going to be placed but can be safely maintained at home. To aid in this process the study team developed a screening tool for local agency personnel responsible for referring cases for family preservation services. The tool provides personnel the opportunity to review their decisions by using a risk index based on factual items such as previous substantiated complaints, more than one maltreated child, previous foster care placements, and the presence of substance abuse. The instrument yields a score, the midrange values of which were thought to suggest referral to funding preservation. A copy of the protocol is in Appendix B. A further discussion on the use of this screening protocol in Kentucky and New Jersey is included in each individual site report in Chapters 3 and 4.

Implementation plans for each site built upon already existing procedures. A written work plan was worked out with each site. A brief description of the plans for each site is presented below.

Kentucky has a statewide program using the Homebuilders model. A statewide coordinator is responsible for developing uniform selection criteria, training of and contracting with providers, and overseeing the program. The study was conducted in Louisville, where there is a single family preservation program provider, and child abuse and neglect cases are referred from intake or ongoing workers². There was no age limitation on the children included in the experiment. Because family preservation does not serve drug abuse cases unless the caretaker is in treatment, or sexual abuse cases in which the perpetrator is in the home, these cases were excluded from the experiment. Referral to family preservation begins with worker and supervisor approval. A screener reviews all cases referred for family preservation to determine appropriateness of the referral. Based on this process, we asked the screener to use the screening protocol developed for the study. The protocol aided the screener in reviewing the risk level of each case. In addition, all cases for which a court petition was filed were reviewed to determine whether they meet family preservation criteria. If they did, they were referred to the screener who decided whether to refer the case for family preservation services. We conducted this review to identify cases that might be diverted from potential placement. A full-time site coordinator in the Louisville office assisted the screener and workers with survey tasks.

New Jersey has a statewide program using the Homebuilders model. As in Kentucky, a state office coordinator is responsible for uniform selection criteria, training of and contracting with providers, and overseeing the program. The study was conducted in seven counties: Bergen, Burlington, Camden, Essex, Monmouth, Ocean, and Passaic. The study population included Division of Youth and Family Service child abuse and neglect and family problem cases referred from intake or ongoing workers. The state had been trying to refocus delivery of family preservation services to families with younger children. Not all counties made this change, so all children under 18 were included in the experiment. Each of the counties has a screener who reviews referrals to make sure necessary information is provided. The screener continued in this role during the experiment. In addition, we asked workers and their supervisors to apply the study screening protocol to all cases being referred to family preservation to review their referral decisions. In some counties, the screening protocol was also used on cases being referred for foster care placement Two site coordinators were assigned to help screeners and workers across the seven counties.

Tennessee. During the study period, Tennessee had a statewide program using the Homebuilders model. As with the other study sites, a state coordinator was responsible for developing uniform selection criteria, training and contracting with providers, and overseeing the program. The study was conducted in Shelby County. There was only one HomeBuilders agency in the county. However, Shelby County is a service rich county in which there were a number of other service options similar to HomeBuilders available to families in the control group. The study population included Division of Children Services child abuse and neglect cases referred from intake workers. Only families in which at least one of the referred children was under 13 were accepted into the study.

Prior to the study, caseworkers referred families directly to the Homebuilders program. For the study, two hotline workers served as study screeners. Referral to family preservation began with worker and supervisor approval. The worker then called the designated screener to find out if there was an opening in family preservation. If an opening was available, the screener would contact Westat to obtain a random assignment. A full-time site coordinator in the Shelby office assisted the screener and workers with data collection.

2 The study was also conducted in Lexington for a limited period of time. [Back to Text]

2.3 Random Assignment and Case Enrollment Status

Random assignment. Individual referral and random assignment procedures were developed for each site. These procedures built upon existing agency referral procedures to family preservation. In both Kentucky and New Jersey, the screener made random assignment referrals. Random assignment began in May 1996 in Kentucky and in November 1996 in New Jersey and Tennessee. In Kentucky and New Jersey random assignment ended in February 1998, and in Tennessee, random assignment ended in May 1998.

Cases were referred to the screener, who, depending upon the site, either determined if the case was appropriate for family preservation or merely made sure that space was available. The screener then called Westat for assignment of the case. The Westat assignment clerk asked for some basic information about the case. In most instances random assignment was done while the screener stayed on the telephone. The screener then mailed or faxed the family preservation referral form to provide more details about the case. This form was used to fill in the study’s random assignment form. (see Appendix C).

Westat personnel used a computer program to randomly assign the case to either the experimental or control group. For those cases randomly assigned to the experimental group, the family received family preservation services. For those cases assigned to the control group, the family received other services provided by the agency.

Case enrollment and status. Table 2-1 shows the number of cases enrolled in New Jersey, Kentucky, and Tennessee by county. A 50/50 experimental/control assignment was planned in Kentucky, and a 60-40 experimental/control assignment in New Jersey, Tennessee began with a 60/40 experimental/control assignment which changed to 70/30 about six months into the study. The actual proportions assigned to each group fell within the expected range.

Some eligible cases were not referred for random assignment and did not get into the study but did receive family preservation services. Exceptions were granted only with the approval of state officials who reviewed the case and determined whether to bypass the study. The state was asked to report exceptions, but sometimes these cases were only detected during review of agency logs and screener telephone calls. Over the course of the study, there were 5 exceptions in Kentucky, 33 exceptions in New Jersey, and none in Tenessee.

In Kentucky a total of 358 cases were randomly assigned by the Department of Social Services (DSS),³ 323 in Jefferson County (Louisville) and 35 in Fayette County (Lexington)⁴. Of these, 9 were determined to be inappropriate referrals and were excluded from the analyses (6 in the experimental group and 3 in the control group). The 9 inappropriate referrals included 3 reunification cases, 4 cases in which the children identified as at risk were out of the home, and 1 case where the custodial parent was incarcerated (in one case the reason for inappropriate referral was not identified). After removing the 9 inappropriate referrals, there were 174 net study cases in the experimental group and 175 net study cases in the control group.

Table 2-1.
Assignment of Cases by County
Kentucky
	Jefferson		Fayette		Total KY
	C	E	C	E	C	E
Randomly Assigned	165	158	13	22	178	180
Inappropriate Referrals	3	3	-	3	3	6
Net Study Cases	162	155	13	19	175	174

New Jersey
	Camden		Burlington		Ocean		Monmouth		Essex		Bergen		Passaic		Total NJ
	C	E	C	E	C	E	C	E	C	E	C	E	C	E	C	E
Randomly Assigned	20	40	23	51	29	42	24	27	49	66	24	29	13	33	182	288
Inappropriate Referrals	1	1	3	4	-	1	1	2	4	4	4	--	2	1	15	13
Net Study Cases	19	39	20	47	29	41	23	25	45	62	20	29	11	32	167	275

Tennessee
	Shelby
	C	E
Randomly Assigned	52	101
Inappropriate Referrals	3	3
Net Study Cases	49	98

Note: C = Control Group, E = Experimental Group

The New Jersey evaluation involved programs in seven counties. A total of 470 cases were randomly assigned from the Department of Youth and Family Services (DYFS), 288 in the experimental group and 182 in the control group. Of the 470 cases that were randomly assigned, 28 cases were determined to be inappropriate referrals (13 in the experimental group and 15 in the control group). Seventeen of these inappropriate referral cases were reunification cases. The remaining inappropriate referrals included foster care cases, cases with no child at risk in the home, or cases that had previously received family preservation services and were being re-referred for a “booster” session. After removing the 28 inappropriate referrals, there were 275 net study cases in the experimental group and 167 net study cases in the control group. The numbers of cases in each county in New Jersey are too small to allow for separate analyses of data by county, so we combine them in all analyses of this report.

The Tennessee evaluation in Shelby County included 153 cases randomly assigned by the Division of Children’s Services. Of these, six were determined to be inappropriate referrals and were excluded from the analyses (3 in each of the groups). The inappropriate referrals were due to no children under the age of 13 in the home (1 case), 3 reunification cases, and one case with children in foster care. The sixth inappropriate referral was screened out by DCS. After removing the six inappropriate referrals, there were 49 net study cases in the control group and 98 net study cases in the experimental group.

The basic analysis of differences between experimental and control groups concerned those cases labeled “Net Study Cases.” However, in a few cases the group assignment was violated, that is, the group to which a family was assigned was switched. Although cases that were deemed to absolutely require family preservation should have been designated as exceptions, we allowed each state 6 “approved violations,” that is, the state central office could switch the groups following random assignment, upon application from the local office. Despite the allowance of six violations, 9 Kentucky cases were switched from the control to the experimental group, 8 of these switches were approved and 1 additional violation was unapproved. New Jersey had 24 violations, 19 approved and 5 unapproved, 14 percent of the net study cases assigned to the control group. In Tennessee 3 cases were switched from the control to the experimental group. There were no recorded switches from the experimental group to the control group in any of the three states.

Some cases in the experimental group were provided minimal services because of refusal by the family to participate, failure of the family to comply with initial expectations of the program, or because the provider agency turned the case back. Turnbacks occurred when family preservation services workers were unable to contact the family or the family did not meet the criteria for service (in a few such cases, children were not considered to be at risk). There were 52 minimal service cases in Kentucky, 4 noncompliance, 18 refusals, and 30 turnbacks. In New Jersey, 44 cases assigned to the experimental group received minimal services because of refusal (14 cases), noncompliance by the caretaker (7 cases), or because the case was turned back by the family preservation agency (23 cases). Tennessee had eleven minimal service cases because of refusal (4 cases), the DCS worker never followed through (1 case), the family preservation agency turned back the case due to safety issues (3 cases), and children placed in foster care (3 cases). Seventeen of the 52 minimal service cases in Kentucky had at least one caseworker contact. One case had more than 5 contacts. In New Jersey, of the 44 minimal service cases, on 31 (70%) we had at least one contact. Seven of the 31 families had more than 5 contacts.

The distribution of violations and minimal service cases is shown in Table 2-2. In the analyses of Chapters 6, 7, and 8 we conducted analyses in which we dropped the violations and minimal service cases from the sample (the “secondary” analyses).

Table 2-2.
Violations and Minimal Service Cases by County
Kentucky
	Jefferson		Fayette		Total KY
	C	E	C	E	C	E
Net Study Cases	162	155	13	19	175	174
Violations	9	-	-	-	9	--
Minimal Service	-	47	-	5	-	52

New Jersey
	Camden		Burlington		Ocean		Monmouth		Essex		Bergen		Passaic		Total NJ
	C	E	C	E	C	E	C	E	C	E	C	E	C	E	C	E
Net Study Cases	19	39	20	47	29	41	23	25	45	62	20	29	11	32	167	275
Violations	1	-	-	-	6	-	3	-	6	-	6	-	2	--	24	--
Minimal Service	-	6	-	8	-	5	-	1	-	13	-	3	--	8	--	44

Tennessee
	Shelby
	C	E
Net Study Cases	49	98
Violations	3	--
Minimal Service	1	10

Note: C = Control Group, E = Experimental Group

3 Kentucky state social services have since been reorganized. DSS merged with the Department for Social Insurance to become the Department for Community Based Services.

4 In both Kentucky and New Jersey, two families were randomly assigned twice. The second of these assignments was considered an inappropriate referral and was dropped from this count.

2.4 Data Collection Activities

Data collection began with a baseline interview as soon as possible after families were randomly assigned to either group. At that time, we attempted to interview the investigating worker handling the case (if the case originated from an investigator), the caretaker, and the caseworker assigned to the case. The caseworker was also asked to report on all contacts with the family during the time services were provided. At the completion of family preservation services or at a comparable time for cases receiving regular services, we interviewed the caretaker and the caseworker again. One year after enrollment, we conducted a follow up interview with the caretaker. In addition to these interviews, we collected data from staff at the participating agencies. Administrative data were collected on individual cases up to eighteen months after random assignment.

Table 2-3 shows the data collection status of the study’s various questionnaires with agency staff.

The Staff Survey was a seven-page self-administered questionnaire designed to obtain a profile of staff at the participating agencies and information on their attitudes and opinions about family preservation services. The questionnaire was mailed to all staff who potentially could have a case in the study. A concerted effort was made to obtain questionnaires from investigating workers and workers in public and private agencies who had study cases. At most sites, this included all the workers at private agencies that provided family preservation services, any workers in family preservation units in public agencies, and workers in units of public agencies that provided in-home and foster care services. In addition to investigating workers and the workers to whom actual cases were assigned, those workers’ supervisors were also asked to complete the survey. The response rate for workers completing staff questionnaires for staff with cases in the study was 90 percent in Kentucky, 76 percent in New Jersey, and 79 percent in Tennessee.

Table 2-3.
Caseworker Response Rates

	Kentucky		New Jersey		Tennessee
	Number	Percent	Number	Percent	Number	Percent
Staff Questionnaires
Staff Questionnaires Mailed	215		344		81
Completed Staff Questionnaires	194	90	262	76	64	79
Investigating Worker Questionnaires
Investigating Questionnaires Mailed	212		223		140
Completed Investigating Workers Questionnaires	164	77	119	53	109	78
Cases with no Investigating Workers	138		219
Caseworker Interviewers
Initial Caseworker Interviews Fielded	349		442		147
Completed Initial Caseworker Interviews	280	80	388	88	112	76
Post-Treatment Caseworker Interviews Fielded	349		444		147
Completed Caseworker Post-Treatment Interviews	326	93	434	98	138	94
Contact Reports
Cases Expecting Contact Report Forms⁽¹⁾	324		28		140
Number of Cases with one or more completed Contact Report Forms	235	73	369	86	98	68

1. Staff indicated there were no contacts for 25 cases in Kentucky (18E and 7C); 14 cases in New Jersey (5E and 9C) and 7 cases in Tennessee (6C and 1E).

The Investigating Worker Questionnaire was a six-page self-administered questionnaire designed to capture information about the investigation of a complaint that led to a referral to family preservation services. Information collected included when and how the complaint was investigated, the nature of the allegation, a description of the home, and problems affecting the household.

As soon as a case referred by an investigating worker was randomly assigned, we mailed an Investigating Worker Questionnaire to the investigating worker reported on the Random Assignment Form. Investigating workers who did not respond to the initial request received reminder letters and second request mailings. If these requests failed, the site coordinator followed up with the worker in person. The response rate for investigating workers completing the questionnaire was 77 percent in Kentucky, 53 percent in New Jersey, and 78 percent in Tennessee. Not all cases were referred by investigating workers in Kentucky and New Jersey; ongoing workers referred 39 percent of the Kentucky cases and 50 percent of the New Jersey cases. All cases in Tennessee were to be referred by investigating workers. However, 5 percent of the cases (7 cases) did not have an investigating worker identified.

The Caseworker Interview was conducted by the Westat Telephone Research Center (TRC). The TRC attempted to conduct an initial and post-treatment interview with the caseworker for each case that was randomly assigned. The initial caseworker interview was to be completed within two weeks of random assignment. If the referring worker was an ongoing caseworker, telephone interviewers attempted to interview him or her as soon as possible. If the referring worker was an investigating worker and the case was a control case, Westat’s site coordinator tracked how quickly the investigating worker transferred the case to an on-going unit^.5 If the site coordinator did not get a response from the worker within 10 working days, the investigating worker was identified as the caseworker to be interviewed for the baseline interview, and TRC interviewers had an additional 5 days to obtain the initial interview. This procedure was instituted because some investigating workers did not immediately transfer their cases, which created difficulties in reaching caseworkers within the two-week time frame.

The telephone interviewers experienced some difficulty successfully reaching and interviewing caseworkers during the study’s time period, especially the initial caseworker interview period. The response rate for completed initial caseworker interviews was 80 percent in Kentucky, 88 percent in New Jersey, and 76 percent in Tennessee.

The post-treatment caseworker interview was scheduled to occur at the same time as the post-treatment caretaker interview, that is, at the end of family preservation services or at a comparable point for control group cases. In both the initial and post-treatment interviews, the caseworker was asked to describe the household, including all household members and their relationships to the children mentioned in the complaint; the condition of the home when visited by the caseworker; problems affecting the caretaker and other household members; and an assessment of the children’s well being. At the post-treatment interview, the caseworker was asked about services provided and was asked to assess whether the goals for the case were met. If the caseworker had not completed a staff survey questionnaire at the time of the post-treatment interview, the telephone interviewer attempted to ask the staff survey questionnaire questions at the conclusion of the post-treatment interview. The response rate for completed post-treatment caseworker interviews was 93 percent in Kentucky, 98 percent in New Jersey, and 94 percent in Tennessee. Data on completion of caseworker interviews by county are shown in Table 2-4.

Table 2-4.
Caseworker Interview Completion Rates by County
Kentucky
	Jefferson		Fayette		Total KY
	C	E	C	E	C	E
Net Study Cases	162	155	13	19	175	174
Initial Interviews	138	120	6	16	144	136
Post-Treatment Interviews	157	147	4	18	161	165
Both Interviews	136	119	3	16	139	135

New Jersey
	Camden		Burlington		Ocean		Monmouth		Essex		Bergen		Passaic		Total NJ
	C	E	C	E	C	E	C	E	C	E	C	E	C	E	C	E
Net Study Cases	19	39	20	47	29	41	23	25	45	62	20	29	11	32	167	275
Initial Interviews	16	35	16	45	21	40	17	24	39	55	19	23	9	29	137	251
Post-Treatment Interviews	19	39	20	47	28	41	23	25	42	60	19	28	11	32	162	272
Both Interviews	16	35	16	45	21	40	17	24	37	55	18	22	9	29	134	250

Tennessee
	Shelby
	C	E
Net Study Cases	49	98
Initial Interviews	46	66
Post-Treatment Interviews	48	90
Both Interviews	46	66

Note: C = Control Group, E = Experimental Group

Caseworker contact reports were to be completed by all caseworkers for each face-to-face contact with a family member during the time period designated for family preservation services. These forms were one-page checklists on which the workers indicated the services delivered at each contact. The forms capture information on concrete services and the content of counseling (e.g., parenting practices, anger management). For cases assigned to family preservation services, the caseworkers were expected to complete these forms from the time the case was first assigned to them through the end of services. Caseworkers with control cases were expected to complete forms for a comparable time period.

Each time a caseworker received another study case (after the first one), Westat mailed the caseworker a letter of notification. This letter identified the case and informed the caseworker that contact reports were to be completed for it, starting immediately. Caseworkers were instructed to complete the reports when a contact was made and to mail them to Westat at least once a week. Each participating caseworker was mailed a supply of contact report forms and postage-paid return envelopes. When it was time to stop completing reports for a case, Westat sent a letter notifying the caseworker. If no completed forms were received, the caseworker was asked to confirm that there were no in-person visits. Letters were sent to workers to obtain this confirmation. In addition, delinquency reports were sent to site coordinators who in turn contacted caseworkers to remind them to complete the form. Contact reports were received for 73 percent of Kentucky cases, 86 percent of New Jersey cases, and 69 percent of Tennessee cases. These response rates are based on only those cases for which we expected a contact report. Caseworkers returned letters indicating that no in-person visits were held for 7 percent of the Kentucky cases, 3 percent of the cases in New Jersey, and 5 percent of the Tennessee cases. All experimental cases where workers indicated there was no contact were minimal service cases.

Caretaker interviews were conducted at three points in time. Data collection began with a baseline interview soon after random assignment in order to get an accurate picture of the household just as services began. A Westat field interviewer attempted to interview the person designated as the caretaker on the random assignment form within two weeks of random assignment. During this interview, the caretaker was asked to enumerate and describe all members of the household and to answer questions about the functioning of the household and parenting philosophies and practices. A second or post-treatment interview was conducted at the time family preservation services ended, or a comparable time period for control cases. The post-treatment interview asked questions about the family’s makeup and functioning similar to those in the initial interview, as well as additional questions about the services received. A final followup interview with the caretaker was also attempted one year from the random assignment date. The final interview was designed to obtain information similar to that in the initial and post-treatment interviews to measure change over time.

As shown in Table 2-5, the response rate for completed initial caretaker interviews was 89 percent in Kentucky, 74 percent in New Jersey, and 80 percent in Tennessee. The response rate for completed Post-Treatment Caretaker Interviews was 84 percent in Kentucky, 78 percent in New Jersey, and 80 percent in Tennessee. For the follow-up interview, response rates showed a decrease to 71 percent in Kentucky, 62 percent in New Jersey, and 75 percent in Tennessee. Successfully completing the caretaker interviews was a data collection challenge for a variety of reasons. The main difficulties included the caretaker not having a telephone number and the mobility of the caretakers. Overall, refusals were rather low, 5 percent at initial and 3 percent at post-treatment and 4 percent in Kentucky; 6 percent at both initial and post-treatment, and 7 percent in New Jersey; and 5 percent at initial and 6 percent at both post-treatment and follow-up interviews. Another reason for noncompletion of interviews was that families could not be located. Table 2-6 shows caretaker interview completion rates by county.

Table 2-5.
Data Collection Status for Caretaker Interviews
	Kentucky		New Jersey		Tennessee
	Number	Percent	Number	Percent	Number	Percent
Initial Interviews
Number of Case Fielded	349		442		147
Total Completed	311	89	328	74	117	80
Refusals	16	5	29	6	8	5
Other reasons for closure	22		91	20	22	14
Post-treatment Interviews
Number of Cases Fielded	349		442		147
Total Completed	294	84	344	78	117	80
Refusals	11	3	26	6	9	6
Other reasons for closure	44	13	75	17	1	14
Follow-up Interviews
Number of Cases Fielded	349		442		147
Total Completed	49	71	274	62	110	75
Refusals	13	4	30	7	10	6
Other reasons for closure	87	25	138	31	27	19

Table 2-6.
Caretaker Interview Completion Rates by County
Kentucky
	Jefferson		Fayette		Total KY
	C	E	C	E	C	E
Net Study Cases	162	155	13	19	175	174
Initial Interviews	146	139	9	17	155	156
ost-Treatment Interviews	136	134	10	14	146	148
Follow-up Interviews	115	122	4	8	119	130
All Three Interviews	115	109	3	8	118	117

New Jersey
	Camden		Burlington		Ocean		Monmouth		Essex		Bergen		Passaic		Total NJ
	C	E	C	E	C	E	C	E	C	E	C	E	C	E	C	E
Net Study Cases	19	39	20	47	29	41	23	25	45	62	20	29	11	32	167	275
Initial Interviews	12	25	17	35	21	32	17	15	36	43	17	23	10	25	130	198
Post-Treatment Interviews	14	29	15	39	22	30	18	18	36	48	18	24	11	22	134	210
Follow-up Interviews	9	22	11	31	18	25	13	13	30	32	15	19	1	25	107	167
All Three Interviews	4	17	8	26	14	19	11	8	23	26	14	16	10	18	84	130

Tennessee
	Shelby
	C	E
Net Study Cases	49	98
Initial Interviews	37	80
Post-Treatment Interviews	37	80
Follow-up Interviews	36	74
All Three Interviews	28	61

Note: C = Control Group, E = Experimental Group

5 Transferring cases was not a problem for experimental cases as they went directly to a family preservation worker.

2.5 Lengths of Time from Random Assignment to the Interviews

Table 2-7 shows lengths of time between random assignment and each interview--initial, post-treatment, and follow-up--as well as the length of time between the initial and post-treatment interviews.

Kentucky. For the 311 initial interviews with caretakers, the length of time from random assignment to completion ranged from 1 to 50 days with an average of 12.6 days (s.d. = 8.1 days, 75% within 16 days, 90% within 23 days). For the 280 caseworker initial interviews, the length of time from random assignment to interview completion ranged from 3 days to 75 days with an average time of 17.6 days (s.d. = 9.36 days, 75% within 23 days, 90% within 28 days).

At post-treatment, 294 caretakers were interviewed, and the length of time from random assignment to interview completion ranged from 24 days to 111 days with an average of 44.8 days (s.d. = 10.5 days, 75% within 49 days, 90% within 58 days). Three hundred twenty-six caseworker interviews were completed in an average of 51 days (s.d. = 14.3 days) after random assignment, with a completion time ranging from 10 days to 142 days (75% within 55 days, 90% within 68 days). For 3 of the cases where services were terminated early (10, 14, and 17 days after random assignment), post-treatment interviews were completed at the time of termination, thus the minimum of 10 days.

With a goal of completing initial interviews within two weeks of the referral date, the intent was to capture each family’s situation at the inception of family preservation or regular services. However, initial interviews with caretakers took an average of over 12 days to complete and initial interviews with caseworkers took an average of over 2 weeks to complete. No significant differences were found between control and treatment groups with regard to the time from random assignment to completion of any of the interviews.

As already noted, the first interview was to be conducted within two weeks of the referral and the second interview was to be conducted at the end of service provision or a comparable time. Therefore, it is expected that for those cases where both interviews were completed, approximately four weeks should have passed between the dates of the first and second interviews. Two hundred and eighty-seven caretakers completed both the first and second interviews, and the average length of time between these interviews was 32.3 days (s.d. = 10.3 days, 75% within 37 days between interviews, 90% within 45 days between interviews). For the 274 caseworkers who completed both interviews, the average length of time between these interviews was 33.2 days (s.d. = 13.3 days, 75% within 38 days between interviews, 90% within 49 days between interviews).⁶

Table 2-7.
Timing To and Between Completion of Interviews
Number of days from random assignment to completion of initial interviews
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	Mean	N	Mean	p	N	Mean	N	Mean	p	N	Mean	N	Mean	p
Caretaker	155	13.3	156	11.9		128	15.5	197	15.2		37	17.0	80	14.8
Caseworker	144	17.6	136	17.6		137	16.0	251	15.4		46	11.8	66	16.5	.0004
Number of days from random assignment to completion of post-treatment interviews
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	Mean	N	Mean	p	N	Mean	N	Mean	p	N	Mean	N	Mean	p
Caretaker	146	5.0	148	44.6		134	53.7	210	51.2	.06	37	51.9	80	47.9	.05
Caseworker	161	50.9	165	51.3		162	58.7	72	54.4	.003	48	47.8	90	59.7	.0001
Number of days between initial and post-treatment interviews
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	Mean	N	Mean	p	N	Mean	N	Mean	p	N	Mean	N	Mean	p
Caretaker	142	31.6	145	33.1		117	38.4	175	36.5		33	35.4	75	32.8
Caseworker	139	32.4	135	34.0		134	1.6	250	38.9	.10	46	35.7	66	42.3	.03
Number of days from random assignment to completion of follow-up interviews
	Kentucky					Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	Mean	N	Mean	p	N	Mean	N	Mean	p	N	Mean	N	Mean	p
Caretaker	102	379.3	117	380.5		85	383.8	133	383.3		32	385.8	63	385.4

Follow-up interviews were completed by 219 caretakers an average of 379.9 days after random assignment (s.d. = 14.9, 75% within 387 days, 90% within 401 days). The difference between experimental and control groups with respect to the length of time between random assignment and follow-up interviews was not significant.

New Jersey. On average, the 325 initial interviews with caretakers were completed 15.3 days (s.d. = 8.5 days) following random assignment (for three cases we do not have the date of the interview). The range in time to completion was 1 to 50 days (75% were completed in 20 days, 90% in 27 days). As in Kentucky, it is not possible to consider the first interview as representing the situation at the inception of family preservation or regular services. In the case of the family preservation cases, these interviews were conducted, on average, two weeks into a four-week intervention. For the 388 caseworker initial interviews, the mean time to completion was 15.6 days (s.d. = 8.1) with a minimum of 2 and a maximum of 40 (75% within 21 days, 90% within 28 days).

For the 344 caretaker post-treatment interviews, the average length of time between random assignment and interview was 52.1 days (about 7 and a half weeks, s.d. = 11.8 days) with a minimum of 33 and a maximum of 116 days (75% within 58 days, 90% within 68 days). For 434 caseworker post-treatment interviews the average was 56.0 days (s.d. = 14.7) with a minimum of 13 and a maximum of 115 (75% within 63 days, 90% within 75 days, interviews on cases that terminated early were sometimes conducted before the end of the 28 day service period, hence the minimum of 13). There were no significant differences between the experimental and control groups in the average lengths of time to interview except for the caseworker post-treatment interview. For the control group, this interview was conducted an average of 58.7 days after random assignment while the average for the experimental group was 54.4 days (p = .003).⁷

Both caretaker interviews were completed in 292 cases, with an average of 37.3 days between interviews (s.d. = 11.2, 75% with not more than 43 days between interviews, 90% not more than 49 days). Three hundred eighty-four caseworkers completed both interviews with an average of 39.8 days between interviews (s.d. = 15.0).⁸

Follow-up interviews were completed by 218 caretakers an average of 383.5 days after random assignment (s.d. = 25.1, 75% within 389 days, 90% within 399 days). The difference between experimental and control groups with respect to the length of time between random assignment and the follow-up interviews was not significant.

Tennessee. On average, the 117 initial interviews with caretakers were completed approximately 15.4 days (s.d. = 7.0 days) after random assignment. The length of time to completion for these interviews ranged from 2 days to 36 days (75% were completed in 20 days, 90% in 26 days). Similar to both the Kentucky and New Jersey caretaker interviews, these interviews were conducted, on average, two weeks into a four-week intervention. Therefore, the first interview should not be considered representative of the family’s situation at the inception of family preservation or regular services. The 112 initial caseworker interviews were completed an average of 14.5 days (s.d. 7.2 days) after random assignment, with a range of 3 to 34 days (75% completed within 18 days, 90% completed within 23 days). The length of time from random assignment to completion of the initial interview with caseworkers was significantly shorter for the control group than for the experimental group (11.8 days vs. 16.5 days, p = .0004).

The length of time between random assignment and the post-treatment interview was significantly different for experimental and control groups on both the caretaker and the caseworker interviews. Therefore, these timeframes are reported separately for each group. The 80 post-treatment interviews with caretakers in the experimental group were completed an average of 47.9 days (s.d. = 10.3 days) after random assignment, while the 37 post-treatment interviews with control group caretakers were completed an average of 51.9 days (s.d. = 10.6 days) after random assignment (p = .05). This time period ranged from 32 days to 80 days for the experimental group (75% in 54 days, 90% in 64 days) and from 29 days to 70 days for the control group (75% in 61 days, 90% in 67 days). For the caseworker interviews, the 90 post-treatment interviews in the experimental group were completed an average of 59.7 days (s.d. = 18.5 days) after random assignment, whereas the 48 post-treatment interviews in the control group were completed an average of 47.8 days (s.d. = 9.7 days) after random assignment (p = .0001). The length of time from random assignment to the initial caseworker interview ranged from 37 to 135 days (75% within 68 days, 90% within 83 days) for the experimental group, and from 36 days to 91 days (75% within 48 days and 90% within 61 days) for the control group.

One hundred and eight caretakers completed both the initial and post-treatment interviews, with an average of 33.6 days between the interviews (s.d. = 10.1 days, 75% with not more than 41 days between interviews, 90% with not more than 47 days). For the 112 caseworkers completing both the initial and post-treatment interviews, there was a significant difference between the experimental and control groups in length of time between interviews (p = .03). Forty-six caseworkers in the control group completed both interviews with an average of 35.7 days between the interviews (s.d. = 12.3, 75% with no more than 38 days between, 90% with no more than 55 days). Sixty-six caseworkers in the experimental group completed both interviews with an average of 42.3 days between the interviews (s.d. = 17.5, 75% with not more than 49 days between, 90% with not more than 60 days between).

Follow-up interviews were completed by 95 caretakers an average of 385.5 days after random assignment (s.d. = 25.9, 75% within 394 days, 90% within 404 days). The difference between experimental and control groups with respect to the length of time between random assignment and the follow-up interviews was not significant.

6 One case in the experimental group was a turnback where the second interview was conducted with the public agency worker and the first interview was conducted with the FPS worker 20 days after the second interview had already been conducted. For this case and four others where there were less than 10 days between the two caseworker interviews, computed scores measuring the change between initial and post-treatment interviews were dropped from the caseworker data.

7 The difference in times to interview for the caretaker post-treatment interviews was nearly significant: experimental group, 51.2 days vs. 53.7 days for the control group, p = 0.056.

8 Two cases in the experimental group were closed by the time the worker was contacted for the initial interview, so both caseworker interviews were conducted on the same day. In all, 9 sets of initial and post-treatment interviews (3 caretaker and 6 caseworker) were conducted with less than 10 days between completion dates. For these cases, computed scores measuring the change between initial and post-treatment were dropped from the caseworker data.

2.6 Administrative Data

We attempted to gather administrative data on foster care placements and reports of maltreatment both before and after assignment into the study on all of the net study cases. This administrative data also contained other information such as case opening dates, types of maltreatment, and some demographic data. In Kentucky, of the 358 randomly assigned cases, no administrative data were obtained from DSS on 22 cases, an additional 20 cases had no recent activity in the administrative data,⁹ and as already noted, 9 cases were inappropriate referrals. These 51 cases were excluded from the administrative data analyses.¹⁰ In New Jersey, we obtained administrative data on 462 cases, 28 of which were inappropriate referrals, so that we had administrative data on 434 of the 442 net study cases (98%), 269 in the experimental group and 165 in the control group (see Table 2-8).

Table 2-8.
Numbers of Cases on which Administrative Data are Available by County
Kentucky
	Jefferson		Fayette		Total KY
	C	E	C	E	C	E
Net Study Cases	162	155	13	19	175	174
Cases with administrative data	140	141	12	14	2	155

Note: Administrative data on one KY case in the experimental group contained only opening and closing data on an adult family member. No data on placements or reports of maltreatment were available for this case.

New Jersey
	Camden		Burlington		Ocean		Monmouth		Essex		Bergen		Passaic		Total NJ
	C	E	C	E	C	E	C	E	C	E	C	E	C	E	C	E
Net Study Cases	19	39	20	47	29	41	23	25	45	62	20	29	11	32	167	275
Cases with administrative data	19	38	20	46	29	41	23	25	44	60	20	29	10	30	165	269

Tennessee
	Shelby
	C	E
Net study cases	49	98
Cases with administrative data	48	96

New Jersey administrative data included some information on services other than placement. In Tennessee, we obtained information on placement and reports of maltreatment from administrative data and case records. Placement data were available for 140 (95%) of the cases, 47 in the control group and 93 in the experimental group. Allegation data were available for 144 (98%) cases, 48 in the control group and 96 in the experimental group.

9 For all cases in Kentucky, we calculated the length of time between the last activity recorded in the administrative data before referral to family preservation services and the date of referral to family preservation services. For each of these 20 cases, there was no recorded activity within 3 years prior to the referral date. It appears that for these cases, recent administrative data were not obtained from the DSS system.

10 In the course of the evaluation, Kentucky changed administrative data systems, which resulted in some difficulties in the retrieval of administrative data.

2.7 Maintaining Study Integrity

It was through the site coordinator activities that many aspects of the study integrity were controlled. This was accomplished in a variety of ways. The site coordinators served as the points of contact between the home office and agency liaisons. They monitored performance by the participating agencies, alerted the home office to problems, and became actively involved in resolving problems as they arose.

The site coordinator was responsible for tracking down needed information to complete interviews (e.g., addresses, caseworker names). Additionally, the SC monitored the status of individual cases to report changes in service end dates, or to identify and seek explanations for cases in which the assignment to regular or experimental services appeared to have been violated. These included cases that should have been but were not referred to random assignment, cases that were randomly assigned but did not get referred to the appropriate service provider, and cases that were not eligible for the study, but were receiving family preservation services. This was accomplished by comparing results of random assignment to agency logs on a monthly basis. State and local personnel were provided monthly reports delineating the cases assigned, their status, and problem areas.

The site coordinator also had a weekly meeting with the public agency screeners and private agency liaisons to review concerns and problems. By keeping in touch with caseworkers and persons in critical positions to the project, the SC was able to gather information about changes in policies, procedures, and staff so that necessary changes could be made. In both Kentucky and Tennessee, there was one site coordinator for one site, while in New Jersey, two site coordinators traveled across seven counties. Having only one site allowed the Kentucky/Tennessee site coordinators to provide better oversight of study activities.

A detailed description of study procedures, including forms and materials, will be available upon completion of the study. The remaining chapters of this report present an overview of the implementation of the study in Kentucky (Chapter 3), New Jersey (Chapter 4), and Tennessee (Chapter 5); a description of the families served in all three sites (Chapter 6); the services received (Chapter 7); preliminary outcomes (Chapter 8); and conclusions (Chapter 9).

Chapter 3: Kentucky

3.1 Introduction

In Kentucky the Family Preservation Program (FPP) is a resource within the state's Department of Community Based Services (DCBS), a division of the Kentucky Cabinet for Families and Children.¹The 120 Kentucky counties are grouped into 16 regions for purposes of FPP administration. There is a family services specialist in Frankfort who has responsibility for statewide coordination of family preservation services including program oversight of contracts, providing training and meeting program reporting requirements. Direct services are delivered by private providers under contract to the state.

Kentucky counties participating in the evaluation originally included Jefferson County (Louisville) and Fayette County (Lexington). Fayette County only participated in the data collection effort for eight months and referred 32 of the 349 net study cases. Therefore, this chapter highlights service delivery, family preservation services, and the implementation of the evaluation in Jefferson County. Study enrollment began in May 1996 and concluded in February 1998.

The sources of material for this chapter are reports and documents produced by the state and interviews with personnel at the DCBS and FPP programs. This information is presented to help understand the context in which services were provided, and to identify any changes that occurred during the implementation of the evaluation. The observations only reflect the perceptions of the individuals we interviewed.

This chapter begins with an overview of the characteristics of Kentucky's children and families. Details of the Kentucky family preservation program, service delivery in Jefferson County, implementation of the evaluation, and other organizational initiatives are then provided.

(1) At study inception the Department was known as the Department of Social Services (DSS).

3.2 Characteristics of Kentucky's Children and Families

This section provides demographic statistics on Kentucky's children and families. Child welfare statistics are presented for Jefferson County (Louisville), which was the focus of the family preservation study in Kentucky.

There are approximately 1,000,000 children under age 18 in Kentucky, with the majority being white (89 percent), and nearly two-thirds under twelve years old (Table 3-1).

Table 3-1.
Age and race distribution of children in Kentucky
Total number of children under age 18 in 1997	961,200
Age	Percent (%)
0-5 years old	32
6-11 years old	32
12-14 years old	18
15-17 years old	18
Race/Ethnicity 1997
White	89
African American	9
Hispanic	1
Other	1

Indicators of child health, education, and social and economic welfare in Kentucky as compared to the nation are presented in Table 3-2. Data have been abstracted from the Kids Count Data Book, published by Annie E. Casey Foundation. With respect to most indicators, Kentucky's families and children are similar to the national average. The Casey Foundation developed a family risk index based on the following indicators: 1) number of children who are not living with two parents; 2) households in which the head of household did not have a high school degree; 3) family income below poverty level; 4) parents did not have steady employment; 5) the family was receiving welfare; and 6) no health insurance for the children. Using the Casey risk calculation, in Kentucky, 17 percent of the children are considered at risk as compared to 14 percent of children in the nation.

Table 3-2.
Indicators of children and family health, education, social and economic welfare in Kentucky as compared to Nation
	Kentucky	Nation
Health:
Percent low birth weight babies (1996)	7.9%	7.4%
Infant mortality rate (deaths per 1,000 live births, 1996)	7.5	7.3
Percent of 2 year olds immunized (1997)	81.0%	78.0%
Percent of children without health insurance (1996)	14.0%	14.0%
Percent of children covered by Medicaid or other public-sector health insurance (1996)	31.0%	25.0%
Child death rate (deaths per 100,000 ages 1-14 in 1996)	27.0	26.0
Teen violent death rates (deaths per 100,000 ages 15-19 in 1996)	73.0	62.0
Teen birth rate (Births per 1,000 15-17 females in 1996)	37.0	34.0
Education:
Percent of teens who are high school dropouts (1998)	14.0%	10.0%
Percent of 4^th grade student scoring below basic reading level (1998)	37.0%	39.0%
Percent of 8^th grade students scoring below basic math reading level (1998)	26.0%	28.0%
Welfare, Social, and Economic:
Median income of families with children (1996)	$33,900	$39,700
Percent of children in poverty (1996)	25.0%	20.0%
Percent of children in poverty (1996)	12.0%	9.0%
Percent of children living with parents who do not have full time employment (1996)	33.0%	30.0%
Percent of families with children headed by a single parent (1996)	25.0%	27.0%
Source: Kids Count Data Book, Published by Annie E. Casey Foundation, 1999.

Table 3-3.
Number of children with child abuse and neglect reports, and percent substantiated by age and race,
Jefferson County, Kentucky
	1992		1993		1994		1995
	Number Reported	Percentage Substantiated (%)	Number Reported	Percentage Substantiated (%)	Number Reported	Percentage Substantiated (%)	Number Reported	Percentage Substantiated (%)
Age:
Total all ages	10,170	48	9,940	51	10,660	50	12,621	48
0-5 Years Old	4,857	42	4,526	47	4,697	46	5,810	44
6-10 Years Old	2,807	50	2,623	56	2,912	47	3,606	51
11-15 Years Old	2,287	52	2,286	56	2,477	54	2,576	54
16-17 Years Old	536	50	505	58	574	49	629	52
Race:
White	6,216	45	5,944	54	6,127	47	7,213	53
Hispanic	17	47	31	46	44	59	26	73
African American	3,534	53	3,556	66	4,082	56	4,785	61
Asian	44	45	51	55	34	47	51	31
American Indian	0	--	2	100	4	54	5	0
Bi-racial	342	48	347	65	349	25	508	61

Child welfare statistics for Louisville. To provide background for the findings from the evaluation, an overview of the number of child abuse and neglect reports and percent of substantiations for four years prior to the study and the first year of the study are presented (Table 3-3). For the calendar years 1992-1994, the number of children for whom there were abuse and neglect reports remained fairly stable, around 10,000. An increase of about 2,000 was seen in 1995. In 1996 there were 12,118 children reported and 49% substantiated, similar to the number of children reported in 1995.

In 1998, the year the study ended, there were 11,797 children reported and 44% of those children substantiated.

Across all five years, approximately 50 percent of the children reported were substantiated victims. The percentage of cases substantiated by age remained fairly constant over the years, with children over ten having a slightly higher rate of substantiation than children under 5 years old. African American children had a consistently higher rate of substantiation than white children.

Substitute care placements in Louisville, Kentucky for the year prior to the study (1995) and the first full year of data collection (1997) are presented below (Table 3- 4). There were a greater number of children in care at the beginning of 1997 than 1995. This may reflect the increase in rise in abuse and neglect cases in 1995 and 1996. However, there is a definite decrease in the number of new entrants and an increase in the number of discharges in 1997.

Table 3-4.
Children served in substitute care in FY 1995 and FY 1997 In Louisville, Kentucky
	1995	1997
Children in care at beginning of year	1534	1774
New entrants	943	591
Discharge	885	1458
Total served	2477	2365

Note: Includes children in foster homes, group homes, treatment facilities, and with relative foster parents.

3.3 History of Family Preservation in Kentucky

Family preservation programs began in Kentucky in 1985 with pilot projects funded by the Edna McConnell Clark foundation. These pilot studies, initiated through local efforts, were the impetus for three 1989 state grants to pilot family preservation programs in Louisville, Lexington, and western Kentucky. The pilot projects were replications of the Homebuilders Model. In 1990 the Kentucky Family Preservation Act established the Family Preservation Program (FPP), "a short-term intensive, crisis-intervention resource intended to prevent the unnecessary placement of children at imminent risk of placement." According to legislation, family preservation programs were to "follow intensive, home-based service models with demonstrated effectiveness in reducing or avoiding the need for out-of-home placement."²

Initially the 1990 Kentucky Family Preservation Act provided for grants to 47 counties to establish family preservation programs. By 1992 the program expanded to 90 counties, and in April 1996 services were available in all 120 Kentucky counties.

By law, family preservation services can be provided by the Department of Community Based Services or through contracts with private, nonprofit social service agencies. Currently all services are purchased through contracts with private agencies.

Until 1994 family preservation programs were entirely state funded when the decision was made to use federal Title IV-A Emergency Assistance Funds (EAF). The purpose of this was to maximize available state and federal dollars by applying for Title IV-A emergency funds for families eligible for family preservation services. In 1997, with the inception of Temporary Assistance for Needy Families (TANF) and the block granting of IV-A funds, the state implemented an eight percent decrease in family preservation contracts. At that time, there was a short-term effort to draw down Medicaid Rehabilitation Funds for FPP. Presently TANF funds are being used to supplement state funding. In 1998 there was an increase in the budget due to an increase in funding for reunification services. The 1998 budget reflected a blending of funds for the two programs, family preservation and reunification.

The family preservation funds available through the 1993 Omnibus Budget Reconciliation Act (OBRA), Title IV subpart 2 of the Social Security Act were mainly used in the planning year to develop regional planning for service provision. A small proportion of the funds were used to pilot family reunification services in five sites for six months. In subsequent years, the funding was divided between family support programs and family preservation. The proportion of funds devoted to family preservation were used for the development of reunification programs rather than further expansion of family preservation programs. All regions were given funds to develop reunification programs. These funds could be used to expand reunification services provided through the pilot studies or new programs could be developed. New initiatives were developed to provide reunification services at the time a child entered foster care as well as targeting those children who were in foster care for extended periods of time.

(2) Kentucky Family Preservation Services Legislation, 200.75 Family preservation services programs.

3.3.1 Description of State Family Preservation Program Model

According to Kentucky policy, "Family Preservation and Program Responsibilities,"³ the Family Preservation Program (FPP) is a short-term, intensive, crisis-intervention resource intended to prevent the unnecessary placement of children at imminent risk of placement. The program serves children and their families who are at risk of commitment as dependent, abused, or neglected; who are identified as needing juvenile services because families are unable to exercise reasonable control of the child; who are identified as having mental health problems; or who are receiving services through the Kentucky Impact program. The purpose of the program is to make reasonable efforts by the Department to prevent the removal of children from their homes.

Programs are to:

Assess the situation and FPP's ability to maximize safety of family members;
Stabilize the family in time of crisis;
Develop goals with the family for family preservation services;
Teach skills to family members; and
Empower the family to make changes that may alleviate the need for out-of-home placement during the crisis.

Families referred to the FPP are expected to meet the following criteria:

At least one parent willing to work with the FPP
The family is in crisis
At least one child is at imminent risk of out of home placement. Both the public agency caseworker and family members shall believe that without immediate intensive intervention, out-of-home placement is imminent.
The family may not be served effectively by using other existing, or less intensive services.
In cases where there has been an emergency removal, it can not have exceeded seven working days and the Department must be willing to return the child home upon FPP acceptance.

Families not eligible for family preservation services include families in which there has been sexual abuse of a child and the perpetrator is still in the home or the child is at risk from recurring sexual abuse and families in which an adult is drug dependent and he or she is not in active treatment.

Direct services are provided by private providers under contract to the state. State policy dictates that caseload size, intensity and duration of services and accessibility of services are based on the Homebuilder's model and are outlined in policy as summarized below.

Provide 20 hours of direct and indirect services according to the needs of each family each week for an average of 4 to 6 weeks;
Provide at least half of the services in the family's home or other natural community setting;
Each worker carries a maximum of two cases at one time;
The worker shall be available to provide services to the family 24 hours a day, seven days a week;
FPP will make referrals as needed to other available community resources, including but not limited to, housing, child care, education and job training, local, state, and federally funded public assistance, and other basic support needs;
Aid in the solution of practical problems that contribute to family stress so as to effect improved parental performance and enhanced functioning of the family unit;
Have available monies (flex dollars) to help the success of the intervention;
Provide services beyond six weeks, if necessary. But no longer than eight weeks.

Policy also specifies that the family preservation provider is to conduct a home visit within 24 hours of referral and make a determination of service provision within 72 hours of the referral.

To aid in the implementation of family preservation services in each region, policy outlines the development of a Family Preservation Program Management Team. The team consists of the contract agency Executive Director, the Department's District Manager, a Department staff person who assumes responsibility for reviewing all referrals to the FPP, the central office family preservation program coordinator, and the FPP supervisor.

(3) Department for Social Services Program Manual, Family Preservation Section.

3.3.2 Family Preservation Services in Jefferson County

Jefferson County (Louisville) is the largest district of the Department. Jefferson County did not become part of the Department until 1989. Prior to that time the Department contracted with Jefferson County to provide child protective services. In Jefferson County reports of child abuse and neglect are made to a state hot line. These reports are then investigated by the Intake and Investigation unit. After investigation, families needing further service are referred to CPS ongoing treatment units. Transfers are to occur within 10 days of conducting the investigation.

During the evaluation, there were nine intake and investigation teams and nine ongoing treatment units. There were also special teams to serve the medically fragile, adolescents, adoption, recruitment, domestic violence cases, and provide court support. During the study period, approximately half way through data collection, the District Manager moved to a state office position, and a new District Manager was appointed.

Prior to beginning data collection for the study, interviews with public and private agency staff were conducted to understand how family preservation services were delivered and the relationship between FPP and DSS.⁴ Comments from these interviews are included in the following description.

Presently, family preservation services are provided in Jefferson County by the private provider, Seven Counties Services, Inc. However, this was not always the case. Originally, family preservation services were provided through a unit within the county public child welfare agency and Seven Counties Services. Public agency staff who experienced both the internal family preservation program and the program provided by Seven Counties preferred the services provided by the public agency program. They felt that the public agency program was more successful, more accessible, there was better collaboration, and services were provided for a longer time period, (12 weeks as compared to 4-6 weeks). The family preservation unit had a screener who reviewed all cases referred for services. When the decision was made to contract for family preservation services, the screener position remained within the public agency.

Referral procedures. Referrals to family preservation come from the intake, ongoing, and adolescent child welfare agency units. Workers are required to discuss all referrals with the team supervisor and then present the case to the family preservation screener. A family preservation referral form is completed (see Appendix D), and the worker must discuss with the family its interest in the service prior to referral. The screener is responsible for making sure the referral is appropriate and also acts as the liaison with the family preservation program.

The screener maintains a log of cases needing family preservation services that have not been referred because of unavailability of slots. In the year prior to the study (1995), 195 child welfare cases were referred to family preservation services. Of these cases, 58 percent were from ongoing units and 42 percent were from intake and investigation units.⁵

As discussed earlier, state regulation stated that referrals should only be cases in which there was imminent risk of placement. In fact, during early negotiations with Louisville, the screener said she estimated about 80 percent of the referred cases were at "imminent risk" of placement. However, in subsequent conversations with workers, they indicated they referred cases that they felt really needed services, but were not necessarily facing imminent placement.

Referrals from ongoing units did not always involve a specific incident of maltreatment. It was reported that if a case appeared to be in crisis or involve chronic problems that were getting worse, it would be referred. An example was a child who had behavior problems and the parent had no idea how to parent or set boundaries. In such a case, family preservation was used to prevent possible abuse. One worker indicated that she referred a lot of neglect cases where parents had poor parenting skills, problems with depression, possibly substance abuse, and problems keeping the home organized, but these cases were not necessarily at imminent risk of placement. Also, some workers said that family preservation was used as a respite for parents as well as caseworkers.

When intake and investigation (I&I) workers were asked specifically about the types of cases referred for family preservation services, they responded:

Low functioning parents with no parenting skills
Young mothers who are overwhelmed and need help getting supportive services
Dirty house cases, something very concrete that family preservation can work on and can see improvement if it is not a chronic problem.
Domestic violence cases, family preservation provides ongoing support to the mother, who needs to repeatedly hear that she is worthy in order to make the decision to move out.
Psychiatric cases--parent is schizophrenic and won't take medication.

The I&I workers believed that family preservation helped families get organized and taught daily living and parenting skills.

FPP Program. Seven Counties, Inc. is the agency that provides family preservation services for Jefferson County. The FPP program is referred to as the "HELP" program and the workers are referred to as "therapists." Seven Counties is a community mental health agency with a staff of almost 1000. The agency has a variety of programs for seriously mentally ill adults, including outpatient treatment, case management, day treatment, and medication management. Programs for children and families include services for violence problems—outreach, office based, and in-home services for perpetrators and victims. Other programs include school outreach, a parent aide program, and Kentucky Impact, a program providing long term wraparound services for severely emotionally disturbed children. The family preservation program also has a reunification component. Reunification services are provided to families just prior to returning the children home.

FPP cases are referred from Kentucky Impact as well as the courts and DCBS. Referrals from the courts and Kentucky Impact account for approximately one-third of all family preservation cases served each year by Seven Counties. However, family preservation service cases referred from Kentucky Impact focus on prevention of psychiatric hospitalization and lasts for six to eight weeks rather than the four weeks of the general family preservation program.

Each FPP therapist handles two cases at a time, and must complete 15 cases per year. The agency is budgeted at 124 cases per year (1000 face to face hours per worker/year). Since its inception in 1990-1991, the program has doubled in size. In 1992 the intervention was shortened from six to four weeks to meet the goal of serving 124 cases.

To provide family preservation services Seven Counties has one supervisor, ten therapists, 1 reunification therapist, and one therapist who works solely with cases serving severely emotionally distributed children. Almost all therapists have Masters degrees (either MSW or M.Ed.). All therapists receive Homebuilders training, with some specific training provided on substance abuse. Twenty percent of the therapists in the program are African American and the remaining are white. Therapists were very adamant in their belief in the Homebuilder philosophy, particularly its emphasis on respecting clients, self-determination, and advocating for clients.

State policy required that the therapist contact the family within 24 hours of referral. If they were unable to reach the family within 48 hours they were to contact the public agency for assistance. By 72 hours a complete initial family contact was to occur, with a determination of whether the family would be active with FPP. If a family could not be contacted or was not willing to work with FPP, then DCBS was to be immediately notified. Therapists indicated they had an unwritten "3 strikes" rule. A family was given three attempts to contact or visit, and if a therapist could not reach the family, the family was "out," and the referral was turned back to DCBS.

FPP provides an acceptance letter on each case, but CPS investigators rarely have direct contact with FPP therapists. Some FPP therapists said they liked to meet with ongoing workers while others did not. In some instances, case conferences were held. If there was no conference, at the closure of FPP case, the therapist would call the worker.

Conversations with therapists revealed some tensions between the public agency ongoing workers and Seven Counties therapists. Therapists felt that some ongoing workers referred cases because they wanted a break from overwhelming cases so they could work on other cases. Seven Counties therapists felt that the workers should stay involved with the family while the case was receiving FPP services. In contrast to the therapists' reports, supervisors of ongoing workers indicated that their workers do keep visiting families during FPP, dealing with the child protection issues.

Seven Counties therapists worried that cases would be closed immediately after FPP was done, although this did not often actually happen. They felt that this was not appropriate as many families needed extended services.

There were generally positive views from the intake and investigation workers who wanted many more FPP slots. However, they were concerned about the short-term intervention because they felt that positive family changes were just beginning to happen at the end of service. They believed some of the therapists do good work, while others were not as good. When asked, workers described an inadequate therapist as one who was not flexible and did not really connect with the family. Supervisor comments stressed the positive value of FPP, but suggested several changes: They felt the program should have more slots, change the substance abuse policy, and have a longer period of intervention.

Overall, workers and supervisors indicated they had a mostly positive experience with the program. They believed FPP was timely in responding, took difficult cases, and shared information. One worker said, "even when placement occurs we still find out a lot about a family, and good joint decisions are made." Supervisors stressed that referrals were made based on crisis, immediate need, and risk of placement, not to assess a family. While assessment is not the "reason" for referrals, they noted that FPP may find out more about family problems such as drug abuse.

There were differing opinions about the rule that families with substance abuse problems can only be referred to family preservation if they are in or about to enter treatment. Supervisors felt that 70 percent of the cases involve some kind of substance abuse. They indicated that FPP can help to get parents into treatment and that FPP should change its focus in order to deal with these cases. The rule does not prohibit the referral of a family with an adolescent with substance abuse problems. Others felt that FPP is too short an intervention for dealing with substance abuse problems and parents need to admit to their problems first in order to make use of FPP.

Court System. The court system in Jefferson County is very supportive of family preservation programs. There is a strong commitment to families. At about the time family preservation programs were being piloted, a family court pilot project was implemented. Beginning with a Family Court Feasibility Task Force in 1988, the Kentucky General Assembly adopted Resolution Number 30. The resolution recognized that the courts were routinely required to make judicial determinations about families, the jurisdiction of the various courts overlapped, and the establishment of a court devoted to and specializing in family law might promote continuity of judicial decision-making. The Family Court Pilot Project was established. The jurisdiction of the family court includes divorce cases, adoptions and terminations of parental rights, dependency, neglect, and abuse cases, paternity status, and emergency protective order cases. The court also conducts the reviews of children in substitute care placement. A 1993 poll conducted by the Survey Research Center at the University of Kentucky found the concept of the court was strongly favored by attorneys and litigants. The majority of the people interviewed believed that family legal disputes should be adjudicated in a single court system, that it was an improvement for families, that the court's rulings met family needs, and that it created additional support mechanisms available to the judge. The family court concept is still functioning in Jefferson County, and the court continues to play an integral role in service delivery and in particular is a proponent of family preservation services.

The policy on involvement of the court was revised and strengthened in 1995 to aid in the protection of children. In substantiated cases of intrafamilial child abuse or neglect in which the alleged perpetrator has continued access to the victim, a juvenile abuse, neglect or dependency petition shall be filed on cases meeting the following guidelines.

Substantiated physical abuse of any child under five years old;
Any child with injuries to critical areas of the body (head, neck, face, abdomen, genitals, lower back) as a result of physical abuse or any unexplained or abuse-related serious physical injury;
Neglect resulting in significant risk of injury or harm;
Sexual abuse; or
Any case in which staff determine that the family will not cooperate with services or action by the court which is necessary for the protection of the child.

Due to this policy there was a substantial rise in the number of petitions filed on cases. Workers indicated they were pleased to have the clout of the court when working with families. However, due to the increase in petitions, a deferred court process was also instituted, court proceedings could be deferred 90 days.

We met with judges and the court administrator prior to starting the study. Initially there was support for the evaluation and a strong belief that family preservation services were a good service. As discussed in Section 3.4, judges did become perturbed with the random assignment process, especially if it affected a case in which they wanted family preservation to be provided. The public agency administrator played a major role in working with the judges throughout the study. She talked with judges about their concerns, and while sympathetic to their concerns, helped maintain study procedures.

(4) At the beginning of the study, the Department was DSS.

(5) The ongoing case total includes adolescent service units.

(6) Fiscal years go from July to June.

3.4.3 Workers' major concerns about the study

Throughout the course of data collection, meetings were held with supervisory and casework staff at the public agency and at the family preservation program. Initially group meetings were held to explain study procedures and identify staff concerns. Subsequent meetings were held to try to allay workers' worries and keep communication open. The study site coordinator was housed at the Louisville public agency so that she was available to have individual meetings with workers as concerns about the study and its effect on operations and services to families came up. Concerns fell into two major categories: 1) the ethics of random assignment and denying services to clients; and 2) the disruption of service delivery caused by study procedures.

Random assignment. Implementing random assignment procedures met with much resistance by caseworkers and family preservation therapists. They were concerned about the ethics of random assignment and what they perceived as denying services to families who needed them. The study design was based on the assumption that each participating county had a higher demand for service than the existing slots permitted. This was true in Louisville where nearly twice as many cases were referred to the screener as were entered into the study. However, as workers noted, in the past when services were not available, a worker might patch services together for a family until there was an available FPP slot. This made it particularly difficult for workers to accept a case being assigned to the control group, because control cases could not be referred again at a later date.

Workers' beliefs that random assignment denied services to families were fueled by the fact that random assignment removed worker control over decisions about their cases. This was complicated by the fact that workers who were good advocates for their families could not get the system to provide the service they believed the family should have. A good example of this tension was one worker's experience with the study. Apparently there had been some confusion between the screener and worker about being able to re-refer a case that went control. Initially the worker told the family that they could be re-referred after going control. The worker was very concerned that this was a child with Attention Deficit Hyperactive Disorder and the mother needed immediate help in controlling the child's behavior. The worker found other services for the child that the family had to pay for. The worker blamed random assignment for denying a service to a family and forcing them to pay for a service that they were entitled to receive for free. Workers were particularly galled by not being able to have control over their decisions and giving that decision up to a computer.

Another worker reported that random assignment denied services to families in his opinion, because appropriate cases did not necessarily come in on a regular basis. There were times when there were more cases referred than slots available. However, there were also times when there were fewer appropriate cases than openings in FPP.

It was often difficult to differentiate workers' angst about the ethics of random assignment and their discomfort over changing service delivery procedures. Both were realities for them and they identified a number of service delivery issues they felt would be affected by the experiment.

Service Delivery. Workers' concerns about how study procedures and random assignment affected service delivery to families were varied. While some of these concerns did materialize, other anticipated concerns did not come to fruition.

One of the more problematic issues concerned the procedures for obtaining IV-A eligibility for families. To obtain IV-A funding for family preservation services, workers were required to have families sign a form. This form also served the purpose of obtaining a family's commitment to FPP, if services were available. Policy required workers to tell families that services might not be available, and the form only showed their interest in receiving services. Signing the form did not mean that the family would get FPP. However, in practice, the workers did not use the form in this way. Workers expected that eventually, families would get FPP, even if there was not a current opening, there would be a future opening. Therefore, they did not tell families they would get the service only if it was available. Instead, they used the form as the family's acknowledgement of accepting services. This procedure became a major hurdle in the implementation of random assignment. Caseworkers could not refer a case to family preservation without the signed eligibility Title IV-A form. Because of random assignment, workers could not promise a family that they would get services. This often required the worker to make two trips to a family's home.⁷

Workers also feared that the study would create a higher turnover of workers. They believed that FPP helped relieve workers of difficult cases so they could concentrate on other cases. By having to keep difficult cases they would not spend time on other cases, become frustrated and quit. Interviews with caseworkers and supervisors indicated that this did not happen.

Caseworkers indicated that family preservation was often used to show "reasonable efforts" prior to placing a child in foster care to meet the P.L.96-272 requirements. Workers were concerned that if a family became a control case, and could not receive family preservation services, it could be interpreted by the family's attorney that workers had not tried every possible option to keep a child from going into foster care. In turn, they thought this would prolong termination of parental rights as a case could not be made that everything had been done to prevent termination. While these were legitimate concerns, there were no reports of this actually occurring.

Caseworkers were upset with the study's disruption of court procedures. There were instances in which a judge would order a family to have FPP. It was incumbent upon a worker to remind the judge that a study was in process, and families were being randomly assigned to the program. Workers were very upset about having to tell judges that a case could not get family preservation services because of the study. In Kentucky, the judge can order services, but not a particular service. But as a number of workers said, "tell that to the judge."

Many workers indicated that there were simply no alternatives to FPP and "when a case is appropriate for the HELP team, that is what is needed, not something else." Investigation and Intake workers complained that random assignment forced them to patch services together when a case went control, further evidence that FPP was not always used to deter foster care placement, but as an alternative to other services.

Caseworkers were also interviewed after the study was over. Not surprisingly, many of the issues they raised during earlier conversations remained as concerns. They were never comfortable with random assignment. When queried about how the families they referred during the study differed from families that would usually be referred, they indicated that they felt there was a difference. The study caused them to refer families they wouldn't ordinarily refer because more referrals were necessary to meet study demands. Supervisors did not agree with this assessment. They felt that workers often had to be encouraged to refer a case, especially if many of their cases went control. However, there was not a difference in the type of family that was referred. Both Seven Counties and CPS workers stated that it was hard to find foster care placements and the motive to provide alternative services lessened the risk of placement of the control group cases.

Violations and Exceptions. As part of our negotiations with DCBS, it was agreed that a limited number of eligible cases could be excluded from the study. DCBS administrators felt it was important that they not deny services to families that local agency staff identified as having an unacceptably high level of risk. It was agreed that eight cases could be considered exceptions prior to random assignment. There were six post random assignment exceptions (called violations in this report) to be used for cases that were assigned into the control group, but later, due to new circumstances, were determined to require FPP. All exclusions and violations had to be approved by the District Manager. In Kentucky a total of five exceptions and nine violations occurred. Interviews with caretakers and caseworkers were conducted on all cases randomly assigned, except for inappropriate referrals.

Inappropriate Referrals The family preservation program has a reunification component for families whose children are in foster care but are to be returned home within two weeks. These cases were not supposed to be enrolled in the study. However, circumstances of cases were not always clearly understood at the time of referral. Therefore, some cases that were referred to the study were identified as having children in foster care for over seven days at the time of referral. These cases were later removed from the study. There were a total of nine inappropriate referrals in Kentucky.

7 We considered changing the study procedure, but local management wanted workers to follow the policy as it was written. It was believed that by shortcutting the policy, workers were not necessarily using family preservation for imminent risk cases. Study procedures did allow workers to call for an assignment from a family's home, but they never used this procedure.

3.4 Implementation of the Evaluation of Family Preservation and Reunification

Having a well-established statewide program, Kentucky was one of the original sites considered for study participation. This site was selected as it met all study selection criteria-a well-defined, mature program using a "relatively" pure version of the Homebuilders model that had more families to serve than slots available.

Site visits were conducted and state office administrators were very interested in participating in the study, with some trepidation about a randomized experiment. Additional meetings were held with state and local personnel to address concerns and to explain the dimensions of the study. To meet required study sample sizes, it was decided that the study would take place in Jefferson county (Louisville), Fayette county (Lexington), and ten Bluegrass counties. A two-day meeting was held with personnel from all these sites to work out the details of the study. At that time it was decided to drop the Bluegrass counties and concentrate efforts on Louisville and Lexington, even if it meant extending data collection past one year to achieve the necessary sample size. Meetings were set up with all of Louisville and Lexington supervisors and workers that would be affected by the study. As was expected, these meetings focused on staff concerns about random assignment. Their concerns are discussed in further detail in Section 3.4.3.

State and local personnel indicated that families currently being referred for family preservation were not necessarily those at imminent risk of placement and that there were many eligible families not being referred for services. Referral problems were particularly salient in Lexington. To address these concerns procedures were implemented to help tighten the screening and referral of families to family preservation. As described below, different procedures were designed for Louisville and Lexington in conjunction with central office DCBS staff.

3.4.1 Louisville study procedures

Preparation and training for the experiment were conducted in the winter and early spring of 1996. Training sessions were held with both DCBS screeners and FPP program coordinators. During the one-day training sessions study procedures were reviewed including use of study forms, the screening protocol, random assignment procedures, and the role of the study site coordinator.

In addition, initial group meetings were conducted with caseworkers and supervisors from each unit. Workers were very resistant to random assignment and concerned that it would deny services to families, cause extra tasks, and delay referrals for their caseloads.

A site coordinator was hired locally and provided by the study to assist DCBS and FPP staff with case information needed for random assignment and the conduct of interviews.

Referral to FPP. Prior to implementing study procedures, workers identified families they felt were appropriate to receive family preservation services, got supervisor approval for the referral, and then made the referral to the public agency screener through a referral form. The screener was then responsible for determining whether or not the referral was appropriate and contacting the family preservation agency to see if there were any openings. If there were time periods when referrals were low, the screener was also responsible for working with caseworkers to identify appropriate new referrals. The plans for implementing the evaluation built upon these procedures. As almost equal numbers of referrals came from the intake and ongoing units, it was decided to maintain this practice. Procedures for the two types of units only varied in the definition of an eligible case. Procedures established to refer cases were:

When a worker decided to refer a case for family preservation services, he or she determined the family's willingness to participate in the program;
The worker discussed the referral with his or her supervisor;
The worker called the screener to see if there was an opening;
If there was an opening, the worker referred the case and the DCBS screener determined eligibility for family preservation;
Eligibility was determined using Kentucky's current review procedures. The screener was also asked to complete a screening protocol that contained a scoring procedure for determining risk. If the screener's decision differed from the recommendation indicated by the score, the circumstances of the case that supported the decision that was made were to be described (see section on targeting);
Once the case was determined eligible, the screener notified Westat that an eligible case was ready for assignment. A computerized program was used to randomly assign a case as experimental or control;
The screener notified the worker that the family had been accepted into the experimental group or assigned to the regular service group;
If the case was assigned to the experimental group, it was referred for family preservation services.

It was expected that cases referred from intake were cases with recent abuse/neglect reports. Cases referred from ongoing were defined as: The family unit was maintained at home without family preservation service and a new situation emerged which indicated that without family preservation services, the children would be placed in out-of-home care.

Targeting. As discussed earlier, DCBS administrators were concerned that FPP was not always targeted at families in which placement for a child was imminent. The study implemented two new procedures to help improve targeting and identify those cases that were at "imminent risk of placement." The first procedure was to identify potentially "high risk" eligible family preservation cases that were not being referred for services. In Louisville, there was a Court Liaison who reviewed every case in which a petition to the court was being filed because there was concern for the child's safety or the case was being referred for foster care placement. Efforts were made to include these cases in the study as more "severe" cases that might not otherwise be referred for family preservation services. For each case in which a petition was filed, we asked that a Worker Safety Checklist be completed by the worker (see Appendix E). For all recently investigated cases in which a petition was being filed, the workers were asked to complete a checklist which covered such issues as: whether or not they were considering foster care, whether the location of the primary caretaker was known, whether the caretaker refused to care for the children, whether the caretaker was chemically dependent without a plan for treatment, whether there was potential for recurring risk of sexual abuse, whether the caretaker was willing to work with an agency, and whether the caretaker was the perpetrator of harm to the child. These questions were developed in conjunction with family preservation and public agency staff to identify potential foster care cases, yet screen out those cases that would not be eligible for family preservation.

The study Site Coordinator reviewed the checklists using established criteria to ascertain whether or not the case should be sent to the screener for family preservation eligibility determination. These criteria excluded cases where workers indicated they were not considering foster care placement, the caretaker could not be located, the caretaker refused to care for the children, the caretaker was chemically dependent without a current treatment plan, or there was potential for recurring risk of sexual abuse. To help ensure that previously referred cases were not re-referred for family preservation, the procedures also included a question about previous referrals.

Throughout the 22 months of data collection, the Site Coordinator reviewed 2103 petitions. Of those petitions, 177 (8 percent) were identified as potentially eligible for family preservation. Of those referred to the screener, 42 percent (74) were randomly assigned. For 51 percent (53) of the cases not enrolled, the reason was that there was no space available in family preservation. Insufficient information accounted for another 23 cases (Table 3-6).

Table 3-6.
DResults of petition review
	N
Number of petitions reviewed	2103
Total petitions sent to FP screener	177
- Cases enrolled in experiment	74
- Cases not enrolled	103
Reasons cases not reviewed
- No space	53
- Insufficient information	23
- Worker plans to enroll	11
- Other	16

The second procedure to help tighten targeting had the screener use a protocol to review cases referred to her. Screening protocols were developed by the study team to aid the screener. The protocol provided the screener the opportunity to review his or her decision by using a risk index based on factual items such as previous substantiated complaints, more than one maltreated child in the family, previous foster care placements, and the presence of substance abuse. The instrument yielded a score, the midrange values of which were thought to suggest referral to family preservation. Guidelines provided to the screener stated that cases receiving a score greater than 2 and less than 5 fell within reasonable risk, and should be referred. Cases with a score of less than 2 might not be considered at risk. Cases with a score greater than 5 might have too high risk. Although cases outside the 2-5 range could be referred, the screener was asked to provide the reason she believed the case should receive family preservation services. The screener was asked to explain why she was still referring cases that fell below or above the midrange. The screener was told that the tool was not to be used to determine referral, but as a review of her decision. As indicated earlier, copy of the protocol is presented in Appendix B.

Although there were many cases referred to the screener that were not sent to family preservation, screening tools were only completed on those cases referred to family preservation. Table 3-7 presents a breakdown of item responses for each of the screening questions. A total of 327 protocols were completed, approximately 91 percent of the 358 cases randomly assigned.

The majority (77%) of the screener's scores for the cases referred to family preservation fell in the midrange between 3 and 5, with an average score of 4.2. The screener's comments about why she referred cases with scores below 3 and above 5 focused on the above 5 scores. She did not provide comments when a case with a score below 3 was referred. However, for the cases with scores above 5, the comments indicated that family preservation was necessary to prevent placement and procedures were in place to ensure safety.

The screening protocol depicted the majority of children (85%) having previous abuse and neglect allegations, with 85 percent of the allegations within the last six months. About one-third of the families had a child previously placed in substitute care. The reader is reminded that these findings are based on the screener's knowledge of the case at the time of referral to FPP.

Table 3-7.
Screening protocol responses
Screening protocol questions	(%)
1. Number of children in family at risk of placement
One	35
Two	31
Three	16
Four	9
Five or more	8
Unknown Organization	1
2. Number of previous substantiated abuse and neglect reports:
Two or more	60
One	25
None	14
Unknown	1
3. Substantiated or confirmed allegation in last six months:
Yes	85
No	12
Unknown	3
4. Has a child in the family previously been removed and placed in substitute care because of maltreatment
Yes	33
No	49
Unknown	18
5. Perpetrator currently living in the home made threats of physical harm to the family in the last two weeks?
Yes	12
No	51
Unknown	37
6. Perpetrator currently living in the family ever been convicted of a crime against a person
Yes	3
No	37
Unknown	60
7. Perpetrator currently living in the family abuses drugs
Yes	3
No	54
Unknown	43
8. At least one of the victims 3 years old or less
Yes	43
No	57
9. Single-female-headed household
Yes	48
No	52
10. Any income from employment
Yes	27
No	37
Unknown	36
11. Protocol score
0	<1
1	1
2	11
3	22
4	29
5	26
6	7
7	3
8	<1
Average Score	4.2

Throughout the 22 months of data collection, approximately 683 cases were referred to the Louisville screener for family preservation. Of these cases, 323 were randomly assigned to FPP or the control group. A monthly breakdown of the number of cases referred to the screener and then referred for randomization for Louisville is provided in Table 3-8. Only DCBS referrals to family preservation were considered for random assignment. Excluded from the study were cases referred by non-DCBS sources, and family reunification cases.

The number of cases referred for random assignment stayed constant for the first year of the study. A slight decrease in referrals was experienced in the second year. There was not an immediate rise in referrals as the study ended. Referrals in March and April 1998 maintained at 28 and 29 per month respectively. Conversations with the screener indicated that it became more difficult to continue to get workers to refer cases for family preservation, as they experienced having more of their cases go control.

3.4.2 Lexington procedures

Based on Family Preservation yearly reports from Seven Counties, the number of referrals received from DCBS increased from 1996 – 1998. The number of referrals was 185 in 1996, 244 in FY 1997, and 294 in FY 1998. There was also a slight increase in the number of referrals accepted, 109 in FY 1996 as compared to 135 in FY 1998. For FY 1997 the referrals accepted were similar to FY 1996, 110.⁶

Although negotiations for Louisville and Lexington started at the same time, the study did not begin in Lexington until the end of August 1996, almost four months after start-up in Louisville. Setting up the experiment in Lexington met with much resistance by the local agency staff. Prior to the study beginning, staff morale was low and referrals to family preservation were waning. State office and family preservation personnel (both local and state) were committed to implementing the study. They hoped that the study would increase referrals to the FPP program. It was decided that the study would go ahead, and the state family preservation coordinator became the family preservation screener. Prior to this, screening of cases was done by the supervisor of the family preservation program, Bluegrass Comp Care. Thirty-two net study cases were enrolled in Lexington over eight months. Resistance of local public agency staff was not overcome, and it was decided that it was best to discontinue the experiment in Lexington.

Table 3-8.
Number of cases referred to the screener and enrolled in study in Louisville
Month	Number of Cases Referred to Screener	Number of Cases Enrolled
Month	Number of Cases Referred to Screener	E	C	TOTAL
May-96	43	6	9	15
June-96	37	10	4	14
July-96	37	8	8	16
August-96	Incomplete	3	7	10
September-96	46	5	2	7
October-96	40	7	9	16
November-96	52	9	7	16
December-96	41	4	1	5
January-97	42	9	3	12
February-97	31	8	10	18
March-97	34	3	7	10
April-97	40	11	14	25
May-97	24	8	6	14
June-97	20	4	6	10
July-97	37	9	9	18
August-97	21	11	4	15
September-97	22	11	4	15
October-97	38	10	22	32
November-97	26	8	15	23
December-97	Incomplete	5	5	10
January-98	29	9	11	20
February-98	23	1	3	4
March-98	29	Random Assignment Over
April-98	28	Random Assignment Over
Total	740	159	166	325

317 Net study cases, 6 inappropriate referrals and 2 cases referred twice

6 Fiscal years go from July to June.

3.5 Other initiatives

Near the end of the study period, the Department underwent a reorganization and welfare reform activities were implemented.

3.5.1 Restructuring

In 1998 the Cabinet for Families and Children began a major restructuring. Two Departments, the Department for Social Insurance and the Department for Social Services were combined to form the Department for Community –Based Services. The separate regions of these two Departments were combined into common regions. According to Cabinet Secretary Viola Miller, "Welfare reform, the need for a more community-minded approach to human services, and the demands of our own employees have been the driving forces behind this decision." The goals of the restructuring are to:

Improve quality of service to families – less crisis intervention, more prevention/stabilization;
More emphasis on positive outcomes for families;
Comprehensive service delivery, instead of fragmentation;
Community orientation;
Let families feel like part of the solution, rather than be lost in rules and red tape;
Maximize federal resources;
Blend fiscal and human resources, reducing duplication and increasing productivity;
Provide a team approach; and
Create greater flexibility and autonomy at the local level

The seeds of the restructuring began in 1996 with the EMPOWER Kentucky initiative. The goal of the initiative was to save taxpayer money, improve the efficiency of services, and better equip state employees with the tools they need to perform their jobs. With the advent of welfare reform it was felt that the Department for Social Insurance had to expand its operations from merely getting benefits out in a timely and efficient manner. There had to be programs in place to help recipients get back to work. The state believed that the best way to accomplish this was to collaborate with the community and other agencies. The Cabinet also wanted to use a community-oriented approach to more effectively fight child abuse by building partnerships with churches, neighborhood groups, and other individuals and groups using preventive techniques. It is intended that increased decision making authority will be given to the regions, with the central office in Frankfort providing training, technology, and technical assistance.

3.5.2 Welfare Reform Initiative

With the passage of the 1996 Welfare Reform Act, the Cabinet for Families and Children was concerned about the impact that the new time limits and work requirements would have on the Kentucky welfare population. Questions to be answered included: Would there be adequate resources to train people within the five year limit and would homelessness and poverty become endemic? Could the hard-core unemployable go to work and would clients keep jobs and become self-sufficient? The Cabinet contracted with the University of Louisville's Urban Studies Institute (USI) to conduct a longitudinal, outcome-based evaluation of the effects of Welfare Reform. The evaluation has two components. The first is to work with Cabinet Departments to develop a database to track the trends and impact of reform on individual clients, and enable the Cabinet to meet the research and evaluation mandates accompanying welfare reform. USI is also conducting a panel study of current and former clients to measure their quality of life for up to five years, with additional cohorts added each consecutive year. Recipients prior to welfare reform will also be included in the study. Administrative data files summarizing client activity in 1994, 1995, and 1996 will be included. The data from years prior to the establishment of TANF will be used as a source of baseline data.

3.6 Summary

Kentucky has offered family preservation services since 1985, when it served as one of the original pilot projects funded by the Edna McConnell Clark Foundation. The state funded three sites in 1989, and by 1996 family preservation programs were available in all 120 Kentucky counties. Over the years state policy and procedures have remained consistent, based on the HomeBuilders Model. The program has remained focused on identifying children at imminent risk of foster care placement and preventing that placement from occurring. Although policy has been consistent, caseworkers acknowledged that their definition of imminent risk was varied. Often caseworkers perceived family preservation services as an alternative service, which might aid in preventing future placement, but not necessarily targeting children at imminent risk of placement.

State and local administrators recognized the targeting problem and worked with the study staff to implement more stringent procedures. A screening protocol was used by the local screener to review all cases referred for FPP. Also, a procedure was implemented to review all cases in which a worker was filing a court petition for foster care placement or for the court's involvement in protecting the safety of the child.

In Louisville, the main study site, the family preservation program was well regarded by both caseworkers and the courts. There were some suggestions for improvement in communication between family preservation therapists and caseworkers. Some caseworkers believed that therapists needed more flexibility when working with families, while some therapists felt that caseworkers needed to stay more involved with families once they were referred for family preservation services. A major concern of both the public and private agencies was services for families in which drug abuse was a problem. While all staff agreed that this was a prevalent problem, there was not consensus as to whether FPP was the appropriate resource to address the issue. Overall, the courts, therapists, and caseworkers believed that family preservation services were a needed resource for families.

Kentucky random assignment for the evaluation was conducted from May 1996 through February 1998. The study mainly took place in Jefferson County (Louisville), with Fayette County (Lexington) participating for eight months. A net sample of 349 cases were assigned, 317 cases from Louisville and 32 cases from Lexington. Interviews with caretakers and caseworkers were conducted. Administrative data were also collected. The analyses of these interview and administrative data are presented in Chapters 6, 7, and 8.

While Kentucky staff were frustrated with study procedures and could not wait for random assignment to end, all levels of staff -- administrators, screeners, supervisors, caseworkers, and therapists put forth a tremendous effort and helped to maintain study integrity.

Chapter 4: New Jersey

4.1 Introduction

The New Jersey Division of Youth and Family Services (DYFS) is a state administered child welfare system with four regions, 21 counties, and a total of 35 field offices. There is a statewide family preservation program which during the study time frames was using the Homebuilders model,¹ with the service provided by contract with not-for-profit agencies in each county. A state office coordinator is responsible for developing uniform selection criteria, training, contracting procedures, and oversight of the contracts for family preservation service (FPS). While all FPS workers are trained using a consistent program model, the county DYFS offices maintain some autonomy in determining how the program is used. Family preservation was originally funded in New Jersey to serve adolescents and prevent placement in residential care. A shift in state policy to change the emphasis to serving families with younger children was made just prior to the implementation of the evaluation.

The sources of material for this section are reports and documents produced by the state and interviews with personnel at DYFS and FPS programs. The comments from staff offer insight into individual practice in the counties or offices in which they work. This helps provide an understanding of the context in which services are provided. However, these observations only reflect the perceptions of the individuals we interviewed.

A summary of how New Jersey compares to national child indicators is reflected in Table 4-1. Data has been abstracted from the Kids Count Data Book, published by the Annie E. Casey Foundation. New Jersey has 16 percent of children covered by Medicaid or other public-sector health insurance compared to 25 percent of all children in the nation. With respect to most indicators, New Jersey's children and families are similar to the national average. As described in Section 3.2, the Casey Foundation has developed a family risk index. Using the Casey risk calculation, in New Jersey, 11 percent of the children are consider at risk as compared to 14 percent of children in the nation.

Table 4-1.
Indicators of child health, education and welfare in New Jersey as compared to nation
	New Jersey	Nation
Percent low birth weight babies	7.7%	7.4%
Infant mortality rate (deaths per 1,000 live births)	6.9	7.3
Percent of 2 year olds immunized (1994)	78.0%	78.0%
Percent of children without health insurance	14.0%	14.0%
Percent of children covered by Medicaid or other public-sector health insurance (1996)	16.0%	25.0%
Child death rate (deaths per 100,000 ages 1-14) (1996)	22	26
Teen violent death rates (deaths per 100,000 ages 15-19)	36	62
Teen birth rate (Birth per 1,000 15-17 females)	37	34
Percent teens who are high school dropouts	6.0%	10.0%
Percent of 4^th grade student scoring below basic reading level (1998)	N/A	39.0%
Percent of 8^th grade students scoring below basic math reading level (1998)	N/A	28.0%
Median income of families with children in 1996	$54,200	$39,700
Percent of children in poverty in 1996	14%	20%
Percent of children in extreme poverty	7%	9%
Percent of children living with parents who do not have full time year-round employment	26%	30%
Percent of families with children headed by a single parent	22%	27%

Source: Kids Count Data Book, Published by Annie E. Casey Foundation, 1999.

1 DYFS discontinued the contract for HomeBuilders training in March 1998. The new model is called New Jersey FPS.

4.2 History of FPS Service in New Jersey

New Jersey has provided FPS services since 1987 using the Homebuilders Model. A Project Director was hired in September 1986 and program design and contracting enabled four programs to begin operations in June 1987. These initial four programs were in Cape May, Cumberland, Essex, and Hudson Counties. By the end of the following year, four additional programs were initiated. FPS services were available in 14 of its 21 counties by 1990. Following the passage of the federal legislation, New Jersey passed a Family Preservation Act in 1993.

The new legislation resulted in the extension of FPS programs to all 21 counties by October 1995. In addition, the bill established the requirement for a statewide coordinating unit, the Family Preservation Technical Support Unit, TSU, to implement the FPS philosophy consistently statewide and to monitor FPS contracts for service. The bill also required the development of a manual of standards for all districts and monitoring by the state legislature including a yearly report. The report must include, at a minimum, the number of families served; the number of children placed in foster care, group homes, and residential settings; the average cost of providing services to a family; the number of children who remain with their families for one year after receiving services; and recommendations for improving the delivery of FPS services in the state.

The state used Title IV-A emergency assistance funds (EAF) to support the expansion of FPS to all 21 counties in 1995. The annual budget for FY1995, prior to the use of EAF was $3.4 million. Current administrators report that the recent block granting of IV-A funds has not affected the funding of FPS.

Description of FPS Model. DYFS chose to utilize the Homebuilders model for family preservation services, considered “a gatekeeper” to out-of-home care in the last community-based effort to prevent out-of-home placement for a child. It was initially established to reduce the number of congregate and institutional placements of adolescents in the state. It is now described as playing an important role in the continuum of care available within the state's children's services.

Caseload size, intensity and duration, and accessibility of the family preservation service are defined in state legislation. These requirements are summarized as follows:

Each worker carries a maximum caseload of 2 families at a time. He or she is allowed to add a third family when one of the two cases enters the last week of service. The worker may serve a total of 18 families within a 12 month period;²
An eligible family shall receive an initial visit within 24 hours of the referral to family preservation;
The worker shall be available to provide services to the family for 24 hours a day, seven days a week;
The program shall provide services to a family for four to eight weeks as appropriate; and
The worker shall provide for no less than five hours of direct service each week.

The state standards for FPS workers stress flexibility of schedule. As to the intensity of service, there is a five-hour per week minimum for contact with families. This is interpreted in the standards as an average of ten face-to-face hours per week with a minimum of three face-to-face contacts per week. More intense services are provided during the initial weeks and in cases with extensive safety issues or other severe needs. Workers are required to keep a phone beeper active or maintain a backup beeper for another worker at all times.

Each program is budgeted to provide limited financial assistance to families. Since the inception of the program in 1987, an average of $75 per family has been budgeted. The money is available to help families with concrete needs such as unpaid utility bills or household appliances or to be used as a token reinforcement to facilitate progress in goal achievement. FPS programs also can apply for Protective Services Emergency Funds (PRS) through the referring DYFS office. This additional funding is available to ameliorate a situation of abuse and neglect where there is an immediate threat to the child's well being or inability of the parent to continue caring for the child. Allowable expenditures include household equipment, food, and payment for shelter.

Each FPS program is required to establish a county based FPS Advisory Council. The Advisory Council provides input to the FPS program and DYFS from the local perspective. The council is chaired by the FPS director and co-chaired by the DYFS worker responsible for screening cases to the FPS program in each county (DYFS screener). The body includes at least one representative from each of the referring agencies in the county as well as key agencies involved in follow-up services for families. Issues for discussion include eligibility criteria, case management, follow-up service, case closure, defining imminent risk, and how to use the program for substance abusing parents. The councils have been most successful in counties where referrals come from many sources, but are inactive in counties that focus only on DYFS cases (e.g., Bergen, Ocean).

2 In FY '2000, the contract changed to 14 families.

4.2.1 Referral Process

Although decisions to refer families are made by the worker and supervisor, workers do not directly refer cases to family preservation programs. Each county has an appointed DYFS screener. All referrals to FPS must be made through the DYFS screener in the designated county. The screener makes referrals to the FPS program when slots are available and maintains a log of unmet need, when no slots are available at time of referral. The screener monitors the referral process, making sure that the referring worker has completed all necessary forms and processes. When a vacancy becomes available in the FPS program, the screener makes a referral on a first-come first-serve basis. Consistent with the criterion of imminent risk, DYFS policy precludes keeping a waiting list for service.

The DYFS referral process allows a DYFS worker to respond immediately to the service needs of a family with a child at risk of placement. Before referring for FPS, the worker must:

Determine that the family has children at imminent risk of placement;
Conduct an assessment in a face-to-face interview within 3-5 days prior to the referral;
Discuss the availability of the FPS services with the family to assess their likely interest and willingness to participate;
Determine that the children's safety is not at risk, if left in the home;
Determine that other less intense services have been used, have not reduced the imminent risk or are not appropriate or not available.³

The worker first presents the recommendations to her or his supervisor and then, if approved by the supervisor, to the screener for referral.

DYFS has decided that FPS should be used cautiously for three populations that require specialized resources: homeless families, out-of-county residents, and families with identified substance abuse problems. These special circumstances must be considered during this assessment by the caseworker. Families that are homeless and living in a shelter can be considered on a case-by-case basis. Families that move across county lines are eligible for FPS in their county of residence. Referral can be made by a caseworker in the former county to the screener in the new county. Similarly, there are limits on services to families with substance abuse problems. The policy suggests that it is unlikely that a substance problem can be resolved in a 5-6 week period. FPS can be used in these cases to help with parenting skills and to provide coordination with the treatment program.⁴

Targeting of Referrals. FPS in New Jersey was initially intended to enhance the continuum of services available for adolescents. In the last few years, the state has encouraged a shift in the focus of their targeting to families with young children. FPS is designed to work with families with children at imminent risk of placement in order to prevent unnecessary placement. As stated in the FPS manual, given that each county's caseloads and placement options vary, discussion should outline the types of families considered to be at imminent risk of placement.⁵ Final decisions concerning policy related to FPS are made jointly by the FPS providers and DYFS.

According to state legislation, FPS is targeted at families with substantiated abuse or neglect, where the children are at risk of harm from maltreatment. Referrals from non-DYFS sources must have risk of placement, but do not require confirmed abuse or neglect. The state defines three levels of eligibility for targeting purposes:

Level one includes families with at least one child at imminent risk of placement, unless changes in family coping or behavior patterns occur, placement will occur or there is one child in temporary placement less than thirty days.
Level two includes families where at least one child is in a temporary placement and was in a placement for less than ninety days in the past or at least one child who is living at home and who was previously in placement for no more than six months or at least one child who is living at home and who has been in a previous shelter, detention, or foster home placement of any kind for any duration.
Level three includes families preparing for reunification where a child is currently in placement and is expected to reunite within seven days regardless of the length of time the child has been in placement.

These broad and overlapping criteria for targeting allow individual counties the flexibility to look very different from the state legislative vision.

County practices certainly varied from this model. Workers interviewed from our seven study counties presented several alternatives. In most counties the major types of referrals are ongoing cases, cases in which workers have worked with the family for many months or years. A worker has to demonstrate that many alternative services have been offered. This so-called three-service rule, in practice, often discourages workers from making a referral to FPS until very late into the life of the case. For many families, workers seemed to consider FPS because it was the only option of service left to offer a family in long-term cases. For example, in Bergen County, the screener reported that traditionally only a small percentage of cases originated from intake. She estimated that only 30% of cases result from recent incidents of maltreatment. Across the seven study counties, 50 percent of the cases were from investigating workers. A statewide referral form was used in all counties, (see Appendix F).

According to Statewide guidelines, counties cannot maintain waiting lists for FPS service. However, DYFS screeners are permitted to maintain a list of “chronic families” who might benefit from FPS service if a vacancy occurs. In practice, the distinction between this list and a waiting list is trivial and was difficult to distinguish in interviews with county staff. One county clearly reported the use of a waiting list, particularly for families with adolescents with behavior problems. The children were temporarily maintained in their homes, often using homemaker services, until a FPS vacancy occurred. Workers indicated that being on the waiting list provided relief to the stressed caretaker, knowing that intensive FPS service would eventually become available.

Sources of Referrals. DYFS is the primary funder of FPS in New Jersey. DYFS screeners are the only authorized individuals who can make referrals to the contracted FPS provider in each county. While DYFS is the primary referral source, in some counties, referrals can also be made by other sources. In 1996 the breakdown of referral was 73 percent from DYFS and the remaining referrals from the following sources.

Family Court: About eight percent of the referrals came from Family Court and were comprised of voluntary requests for family preservation services from families ordered directly by a judge;
County Crisis Intervention Units (CIU's): This is a delinquency diversion program which works primarily with unadjudicated teens and comprised about 12 percent of the referrals;
Children's Crisis Intervention Services (CCIS): This is a diversion service provided by the Division of Mental Health and was responsible for about 4 percent of the referrals; and
Other: The remaining three percent of the referrals were made by other sources. The main source was The Case Assessment Resource Team (CART). The CART is an interdisciplinary team including DYFS and other state agencies to prevent teens from being placed in out-of-state residential treatment facilities and works towards returning those teens that are placed out-of-state.

The non-DYFS referrals must be made through the screener. DYFS cases and families known to DYFS get priority, if a vacancy is available in the program.

The table below (Table 4-2) shows the number of referrals and percentage of referrals that come from DYFS as a referral source. “N/A” means the program was not yet in operation. The new programs that began in FY1996 (Hunterdon, Middlesex, Ocean, Somerset and Warren) serve only DYFS cases. Bergen County, one of the earlier programs, also serves only DYFS cases.

Table 4-2
Number and percentage of DYFS referrals to FPS by County for FY 1994-1996
	FY1997		FY1996		FY1995		FY1994
County	N	%DYFS	N	%DYFS	N	% DYFS	N	%DYFS
Atlantic	45	68.9	64	46.9	66	33.3	67	29.9
Bergen*	103	100	109	100	110	100	100	100
Burlington*	105	73.3	88	80.7	78	55.1	94	56.4
Camden*	83	66.3	83	61.4	159	58.5	226	52.7
Cape May	54	74.1	51	74.5	60	56.7	59	55.9
Cumberland	145	93.1	150	74.7	119	68.1	122	57.4
Essex*	214	71.0	166	73.5	229	52.4	161	67.1
Gloucester	38	50.0	47	36.2	70	22.9	94	28.7
Hudson	194	73.2	151	67.5	179	57.0	129	76.7
Hunterdon	40	100	21	100	N/A	N/A	N/A	N/A
Mercer	93	89.2	95	72.6	108	50.0	125	48.8
Middlesex	107	100	85	100	N/A	N/A	N/A	N/A
Monmouth*	97	74.2	88	71.6	110	57.3	100	79.0
Morris	95	67.4	84	73.8	N/A	N/A	N/A	N/A
Ocean*	56	100	53	100	N/A	N/A	N/A	N/A
Passaic*	82	86.6	103	59.2	113	56.6	99	61.6
Salem	68	94.1	71	88.7	76	64.5	81	59.3
Somerset	40	100	23	100	N/A	N/A	N/A	N/A
Sussex	60	48.3	37	70.3	N/A	N/A	N/A	N/A
Union	79	86.1	126	47.6	140	26.4	167	49.7
Warren	43	100.0	36	100	N/A	N/A	N/A	N/A
TOTAL	1841	81.0	1731	73.6	1617	55.0	1624	59.2
* Evaluation Sites

As can be seen from the above table, there is considerable variation in the proportion of cases coming from DYFS. Looking at the evaluation sites (marked with *), Passaic County increased from 59.2% in FY 1994 to 86.6% in FY 1997 of cases documented as DYFS referrals. In actual numbers, this is reflected by a major reduction in non-DYFS referrals in Passaic County from 42 in FY1996 to 10 in FY97.

Counties also developed separately funded FPS programs for targeted populations. Essex had a second FPS program specifically funded for boarder baby referrals. Monmouth and Burlington Counties also reported the funding of additional slots specifically for the Crisis Intervention Unit (CIU) used primarily by the court.

3 DYFS Referral Handout for Casework Staff, 1996.

4 NJ FPS Standards Manual, Chapter 3, page 2.

5 NJ FPS Standards Manual, Chapter 6, page 13.

4.2.2 Statewide FPS Case Characteristics

FPS programs are required by contract to collect and report social and demographic information on the families that they serve. Statistics are available through Fiscal Year 1997, as reported in the Family Preservation Annual Report. DYFS has placed an emphasis on having the families served by FPS programs reflect the composition of the DYFS general population and the communities they serve.

In New Jersey, the FPS programs have historically served more one-parent families. In FY96, 57% of families served were one-parent families (single, divorced, separated, and widowed). Additionally, from FY 1992-95, approximately 39% of families served were reported as having AFDC as their primary source of income. FY96 showed an increase in this percentage to 48%.

The largest racial group served statewide is white families, including around 40% of families served in FPS each year, as shown in Table 4-3below.⁶ The percentage of white families increased slightly from 41.5% in FY 1994 to 44.8% in FY 97 with a corresponding 4.3 percent decrease in the percentage of African American families served.

Table 4-3.
Race/ethnicity of families served in FPS
Race of Families Served	FY1994	FY1995	FY1996	FY1997
White	41.9%	39.7%	42.7%	44.8%
African-American	37.1%	37.7%	36.5%	32.8%
Hispanic	14.3%	15.7%	14.4%	16.5%
Other	6.7%	6.9%	6.4%	5.9%

In FY1995, there was discussion during state budget planning that FPS programs were not adequately serving the appropriate population.⁷The DYFS Program Report for Fiscal Years 1995 and 1996 suggests that the FPS caseload should ideally reflect the active DYFS and foster care caseloads. Statewide, about two-thirds of the children in foster care are African-American, while 37% of FPS population served during the year was African-American. There is some county variation. The African-American populations of certain counties' FPS and total active caseloads (e.g., Essex, 84% FPS and 86% active caseload) more closely match (see Table 4-4). While there is some possible bias in comparing FPS full-year statistics to point-in time DYFS caseload demographics, DYFS suggests that caseload demographics have not changed much over the past several years. Most programs have not served African-American families in the same proportion as the foster care caseload.

Table 4-4.
Percent of FPS, DYFS total, and foster care caseload that is African-American by County
County	Percent of Caseload^b
County	FPS^a	DYFS Total Caseload	DYFS Foster Care Caseload
Bergen Burlington Camden Essex Monmouth Ocean Passaic Statewide	15% 23% 33% 84% 26% 12% 39% 37%	25% 32% 54% 86% 39% 19% 45% 50%	33% 44% 67% 92% 59% 37% 60% 67%

^a The report uses cumulative yearly percentages as reported by FPS programs. The DYFS Active Caseload counts for Total and Foster Care are point in time. Active caseload can reflect duration bias for those children remaining longer in the DYFS actual or foster care caseloads.

^b Family Preservation Services. Program Report for Fiscal Years 1995 and 1996, July 1997, pg. 43.

During the three-year period FY1994-FY1996, older children continued to be targeted by family preservations programs. However, the percentage of older children, age 13-17, decreased from 56% in FY95 to 37% in FY 97. DYFS credits this shift to an increase in service to children in reunification cases, where the distribution of ages of children tends to be younger. In addition, several of the newer programs, including Huntingdon, Ocean, and Somerset served a majority of younger children in FY1996.

6 Family Preservation Services, Annual Program Report for Fiscal Year 1997 (Draft), April 1999, Pg. 47, DYFS Office of Policy, Planning and Support.

7 Excerpted from Background paper, DHS budget 1995-96 , New Jersey State Auditor.

4.2.3 Reunification Component of the Program

In New Jersey, referrals can be made to family preservation services for both placement prevention and family reunification. As described in the DYFS family preservation standards manual, placement prevention applies to families where one or more children are at imminent risk being placed into foster care. Children in short term emergency placements at risk of longer placements are also eligible for placement prevention services. Families with children already in placement for any period of time are eligible for referral to FPS as reunification cases. Workers can refer families when they are preparing to reunite with a child currently in placement within 7 days. This is regardless of the length of time the child was in placement.

Family reunification cases in family preservation are eligible for the same services as placement prevention cases. By definition, the criterion for imminent risk of placement does not apply for reunification cases. For reunification cases, workers and FPS screeners appear to have broader latitude in determining when a family will benefit from FPS service. In addition, FPS programs are monitored for contractual compliance in preventing children from avoiding placement. Reunification cases are excluded from this monitoring, so these cases are perceived as under less scrutiny by the programs.

4.2.4 Training

In FY1996 all Child Protective Service (CPS) workers, around 2,000 field staff statewide, participated in a full day of training on the philosophy and practice of family preservation services. The intent of the training was to encourage a conformity of type of cases referred to FPS around the state and to train new workers. The training emphasized that child safety is paramount. CPS and FPS workers should only consider or continue family preservation services if there is minimal safety risk to the children in leaving them in their own homes. Separating children from families and creating new temporary or permanent families was emphasized as good practice in some situations. The training reviewed the basics of family preservation assessment, interventions and referrals, and the techniques that are used with families.

In New Jersey, there has been consistency in the content and philosophy of training of FPS workers. Since the inception of FPS in New Jersey, Behavioral Science Institute (BSI) conducted the training sessions for new workers at the 13 programs in the state. In March 1998, the state ended their contract with BSI. According to the FPS administrator, it was felt that the BSI program was too generic and that a New Jersey-specific program was needed. The Family Preservation Institute, a joint program with Rutgers University, began training in September 1998.

4.3 Implementation of the Evaluation of Family Preservation and Reunification Services

DYFS, as reflected in interviews with FPS administrators and in their manual materials, is interested in integrating ongoing evaluation into the development of its FPS program statewide. The FPS state legislation in 1993 also required monitoring of outcomes of the program. Before discussing the implementation of the evaluation in New Jersey, Section 4.2.1 will review previous studies in the state and the lessons learned.

4.3.1 State's Interest in FPS Evaluation

Changes in the FPS programs have resulted from the previous studies conducted by the state. DYFS administrators described them as important and necessary catalysts for some of the changes in the FPS service and delivery models that have occurred over its ten-year history. The following sections briefly describe DYFS's evaluation and the Targeting Referrals Project.

4.3.2 DYFS Evaluation

A DYFS evaluation of the New Jersey family preservation model was conducted in four counties (Feldman, 1991). Cases were randomly assigned and followed for one year after service. Data are available on 117 experimental and 97 control cases. Thirty-three families that were “turned back” from the experimental group were excluded from the analysis.

Findings. Analyses were conducted on both placement prevention and improvement in family functioning. Measurement occurred at several points in time and comparisons were made between the treatment and control groups. Both the treatment and control groups made gains on the Moos Family Environment Scale, Interpersonal Support Evaluation List, and Child Well-being Scales. However, there were few significant differences between groups in the amount of change.

The differences in placement rates between the treatment and control groups were also examined (see Table 4-5). During the intervention period, approximately 6 weeks, 6% of families in the experimental group and 17% of families in the control group experienced placement of at least one target child. At 6 months post-termination, 27% of families in the experimental group and 50% of control group families had experienced at least one placement. At one year post-termination 43% of those in the experimental group and 57% of families in the control group had experienced placement.

Table 4-5.
Placement Data by Months Since Termination
Months Since Termination	% of Families with Child Placed
Months Since Termination	FPS treatment	Control
3 Months	22	37
6 Months	27	50
12 months	43	57

The state concluded⁸ that FPS services can be effective in preventing placement for the short term. If used as a short-term "front-end" it can be useful as part of the continuum of services needed by a family. However, more is needed to be known about the targeting of families and outcomes. In particular, staff wanted to know which families are likely to get the best outcome from the short-term service.

8 FPS Manual, Section 900, pg. 44.

4.3.3 The New Jersey Family Preservation Services Targeting Referrals Project

Following the DYFS study, it was felt that the decision-making process involved in making a referral to FPS needed to be evaluated. In 1992, DYFS, with funding from the Tri-State Network of HomeBuilders, conducted a study to examine the caseworker decision making process to assess the targeting issue. DYFS was concerned that targeting was not solely directed at children at imminent risk of placement. It conducted a series of case record reviews, caseworker interviews, caseworker focus groups, and a survey in four district offices. The project sought to examine why workers refer, how the referral process functioned, and what factors influenced the selection of families for referral.

Findings. The study included findings about the referral process, reasons for selecting families, and the perception of FPS. In regard to the referral process, workers appeared to understand the process including forms, screener's role, and procedures. Some workers admitted to making referrals only when they knew that a slot in FPS was available. Most workers considered the acceptance process random, since it required referring a case that met the criteria for referral at the time a slot was available.

The consideration of a family for referral appears to be related to availability of resources for families, especially adolescents. Many workers expressed frustration regarding the availability of community resources. FPS was often used as crisis intervention, in response to a parent's request to remove a troubled teen from the home. During that time, some workers recast the definition of imminent risk because of the availability of voluntary placements. Children, especially adolescents, while not at risk of harm, could be at imminent risk of placement by parent request. Parental cooperation and desire for placement of their children were considered to be an important factor in making a referral to FPS.

The Project defined eight policy issues and implications:

The required timing of family in crisis and at imminent risk when a FPS slot is available is unrealistic. The project recommended increasing the number of slots in each county to a saturation level.
FPS is not being used as placement prevention as intended. The existence of voluntary placements allows families at low risk of harm into placement. The recommendations include more policy education. They particularly recommend establishing routine referrals of children who are headed for placement.
FPS is being used as an adolescent crisis intervention and treatment program. The recommendation is that DYFS should develop more services in the community for adolescents.
There must be more emphasis on follow-up services for clients after FPS. The recommendation is for service delivery standards and broader funding for continuum of care services.
Some families spiral into crisis after a previous period of FPS services. DYFS procedures allow those families to be referred for an additional period of FPS service, called a "booster."; Workers do not adequately use FPS booster services.
DYFS families referred to FPS are not being tracked routinely by SIS, the state child welfare tracking system. It was recommended that training is required to ensure workers record the FPS activity into SIS. In addition, it was recommended that FPS agencies get linked up to the DYFS computers to enhance tracking of families. This will allow DYFS to track referrals and service data, but will not provide access to the SIS for FPS agencies.
The DYFS System is a reactive one. The recommendation calls for DYFS to develop intensive services for families prior to imminent risk.
Local FPS issues are not being resolved at the local level. The recommendation calls for a better use of conflict resolution.

While some of these recommendations (#1 and #2) were reflected by the legislative implementation of FPS in 1993, the need for expanded services mentioned in #3 and #7 is still under consideration and embodied in new strategies outlined by DYFS administrators as new directions for the FPS program. Some of the problems identified as issues in this early project are still obstacles today.

4.3.4 Evaluation of Family Preservation and Reunification Services

Executive staff in New Jersey expressed early interest in participation in this evaluation to obtain a thorough assessment of their family preservation services. FPS services in New Jersey had been operational for almost ten years. They were recently expanded to all counties. The emphasis, while originally focused on adolescents with family problems, was undergoing a shift to maltreatment cases involving young children.

In addition to such things as maturity of the program and the use of the Homebuilders model, New Jersey also met the study's criterion that there was not saturation of FPS services. To avoid the ethical concern of denying services to families, sites were considered where service demand exceeded the number of slots available. Ten counties were identified as possible sites. They were Warren, Ocean, Bergen, Cape May, Monmouth, Salem, Cumberland, Essex, Hudson and Middlesex. DYFS administrators decided on the final sites to be included. They wanted a balance of northern and southern counties as well as urban and suburban ones. In addition, the DYFS random assignment study had been conducted in four counties. DYFS wanted to limit the research burden on these counties. DYFS selected seven counties that agreed to participate: Bergen, Burlington, Camden, Essex, Monmouth, Ocean and Passaic Counties. As seen in Figure 4-1, this resulted in a cluster of three counties in the Northern part of the state and four in central New Jersey. A target of 500 cases was set for New Jersey to allow for adequate subsample analysis.

DYFS requested a 60-40 split of cases in experimental and control groups. Having a better than 50% chance of obtaining family preservation services was thought to encourage caseworkers to make referrals.

Figure 4.1. New Jersey counties participating as evaluation sites (shown in color)

The procedures for targeting and screening were determined with Central Office DYFS staff, then individualized with counties to fit their service delivery procedures. DYFS administrators, while interested in participating in the evaluation, had concern about the random assignment. They wanted to work out all ethical and procedural concerns before allowing the evaluators to talk with county staff. As a result, state FPS administrators did not include county DYFS staff or FPS administrators in discussions with evaluators until workplan, procedures, and protocols were completed. This delayed and possibly lost some of the "buy-in" by the local administrators and workers.

The screening protocol, developed for the evaluation and discussed in Chapter 2, was offered to counties to assist with targeting cases for family preservation. It was asked that the tool be completed for all cases considered for FPS, as well as children being referred for placement into foster care.⁹ For counties that had a formal pre-placement conference, the screening protocol would be completed during that meeting. For other cases, a referring worker would complete a screening tool with her or his supervisor prior to submitting a referral to the screener. All counties, except Passaic County, agreed to use the screening protocols for cases referred to FPS and randomly assigned to the evaluation. For placement cases, DYFS staff from all counties felt they could not commit to using the protocol, because it would be considered a paperwork burden to staff.

We received screening protocols on 56 percent of the 442 net study cases.¹⁰ In addition, workers completed protocols on 15 cases that were not referred for random assignment. Our intent had been for workers to use the screening tool for all cases considered for family preservation services. However, this did not occur. Of the screening protocols for cases that were randomly assigned, 60 percent (147) were experimental and 40 percent (99) were control. Table 4-6 presents a breakdown of item responses for each of the screening questions.

The screening protocol asked nine questions to establish a risk score. The worker and his or her supervisor were to complete the form at the time the case was reviewed for referral to FPS. The purpose of the form was to have workers reassess certain conditions of the case to make sure it was appropriate for family preservation services. The form was not intended to replace worker judgement, but to give them an opportunity to review their decisions about the appropriateness of the case for FPS.

Table 4-6.
Item Response For Screening Protocol
Screening Protocol Question	Cases Randomly Assigned (%) (N=245)	Cases Not Randomly Assigned (%) (N=15)
1. Number of previous substantiated abuse and neglect reports:
None	29	15
One	43	54
Two	24	23
Unknown	4	8
Total %	100%	100%
2. Substantiated report of abuse and neglect within the last six months:
No	59	59
Yes	32	33
Unknown	8	8
Total %	100%	100%
3. Has a child been previously removed and placed in substitute care because of maltreatment?
No	65	50
Yes	27	42
Unknown	9	8
Total %	100%	100%
4. Has a perpetrator currently living in the family made threats of physical harm to the family in the last two weeks?
No	71	69
Yes	18	23
Unknown	11	8
Total %	100%	100%
5. Perpetrator in family ever convicted of a crime against a person:
No	68	92
Yes	4	--
Unknown	26	8
Total	100%	100%
6. Perpetrator in family abuses drugs:
No	60	69
Yes	19	8
Unknown	21	23
Total	100%	100%
7. At least one of the victims 3 years old or less:
No	76	77
Yes	23	23
Unknown	1	--
Total %	100%	100%
8. Single-female-headed household:
No	49	47
Yes	45	53
Unknown	6	--
Total %	100%	100%
9. Any income from employment:
No	59	47
Yes	33	53
Unknown	8	--
Total%	100%	100%
10. Total Score
0	8	7
1	12	13
2	20	13
3	22	27
4	24	20
5	9	13
6	3	--
7	1	7
8	1	--
Total %	100%	100%
Average	2.9	3.1

Guidelines provided to the workers said that cases receiving a score greater than 2 and less than 5 fell within reasonable risk, and should be referred. Cases with a score of less than 2 might not be considered at risk and cases with a score greater than 5 might have too high risk. Although workers could refer cases outside the 2-5 range, they were asked to provide the reason they believed the case should receive family preservation services. Examples of reasons that were offered for scores below 2 are acting–out teenagers and teenagers with suicidal tendencies. The majority of cases received a score between 2 and 5 (75 percent). Only 5 percent of the cases had a score greater than 5 and 20 percent had a score less than 2. In New Jersey, caseworkers indicated they did not believe the risk scale sufficiently addressed the problems of teenagers, and therefore there were cases that did not score as high as they should have.

The workers completing the screening protocols depicted the majority of the children having previous abuse and neglect allegations (67 percent), but less than one-third of them within the last six months. Workers reported that 23 percent of cases had a child age three or less. The reader is reminded that these findings are based on screening protocol data completed by workers at the time of referral to family preservation. Overall scores on experimental and control cases were similar and are not presented here.

The second column of the table provides a breakdown of the responses to the screening protocol for the cases not submitted for random assignment. As there are so few of these cases, comparisons with randomly assigned cases are problematic. The average score for the two groups is similar, 3.1 for the non-study cases, and 2.9 for the study cases.

9 Workers mainly completed screening forms only for families that they actually referred.

10 Workers in Passaic County did not complete screening protocols for the study. Excluding Passaic County from the total, we received protocols for 246 or 62% of the 399 net study cases.

4.3.5 Initiation of Project

Preparation and training for the experiment were conducted in the summer and early fall of 1996. Training sessions were held with both DYFS screeners and FPS program coordinators. During one-day training sessions, study procedures were reviewed including use of study forms, the screening protocol, random assignment procedures, and the role of the study site coordinator. There is some variation in the number of screeners, depending on the number of field offices in each county (see Table 4-7).

Table 4-7.
Number of DYFS screeners and local offices by County
County	Number of DYFS Screeners	Number of DYFS Local Offices
Bergen Burlington Camden Essex Monmouth Ocean Passaic	1 1 1 5^a 1^b 1 2	1 1 2 5 2 1 2

a. The five screeners alternated as screeners for Essex County on particular days each week. There was also a supervisor assigned to the unit.

b. The Monmouth screener was stationed in the southern district office. The northern district office had a worker assigned as "gatekeeper" to screen cases and relay each referral to the screener.

In addition, meetings groups were conducted with self-selected groups of caseworkers and supervisors in each county prior to the start of random assignment. Study staff traveled to each county and met with public agency caseworkers, supervisors, and agency administrators as well as FPS workers and the administrator at each contracted private agency. Workers were very resistant to random assignment and concerned that it would deny service to families, cause extra tasks, and delay referrals for their caseloads.

Two site coordinators were hired and assigned to assist DYFS and FPS staff with the collection of information needed to complete random assignment and the conducting of interviews. One site coordinator worked with the three northern counties (Essex, Bergen and Passaic) and the other worked with the four southern and central counties (Monmouth, Ocean, Camden and Burlington). Random assignment was initiated in late November 1996.

4.3.6 The Random Assignment Process

Only DYFS referrals to family preservation in the selected sites were considered for random assignment. Excluded from the study were cases referred by non-DYFS sources, cases served in family preservation prior to the study that were returning for a second "booster" service, and reunification cases.

DYFS uses a broad definition of reunification as a referral criterion. The policy does not put time limits on how long a child was in placement. Caseworker, supervisor, and screener jointly decide whether a child is considered being reunified from a placement or whether the case is considered a placement prevention case, that is, the goal was preventing the child from entering a long-term placement. For the study we used a guideline of seven days. If a child had been in placement more than 7 days, we excluded him or her from the study. Although this process was monitored closely, nine such cases entered the study, but were removed from the analyses because children were in care for longer than seven days at the time of random assignment.

DYFS workers could also re-refer a case for FPS anytime after the completion of the program. This "booster" or "booster shot" still must meet the criteria of "imminent risk" and has a maximum service of four weeks. There is no maximum waiting time between the first period of intervention and the booster; however, after a year, a booster would be counted as a full case with a new case number. Cases being referred for booster service where the original FPS service was conducted prior to the study were excluded from random assignment.

4.3.7 Concerns of DYFS and FPS Staff

Initial and subsequent meetings were held with supervisory and casework staff at both the county DYFS offices and the family preservation programs. Many concerns about the study and its impact on operations and service to families were discussed. The most unanimous concern in every DYFS office was the requirement for additional paperwork. Supervisors were concerned that requiring caseworkers to perform additional paperwork would be a barrier to participation. Study personnel agreed to modify procedures to minimize the burden on workers and to assure staff that paperwork would not be duplicative. Some counties agreed to consider using study forms as substitutes for existing forms.

Another common concern to both DYFS and FPS staff in all counties was the possibility of denial of service to families. The study design is based on the assumption that each participating county had a higher demand for service than the existing slots permitted. When asked by a DYFS administrator if they could fill an additional slot with families, every county screener said, "yes." The issue was most pressing in counties where a waiting list was kept. In Bergen, for example, workers spoke of promising families FPS when a slot was available. It was felt that the promise of future availability of service to a family with a troubled adolescent was an important incentive to a parent not to insist on placement of the child. DYFS staff acknowledged that a waiting list was not consistent with the imminent risk criterion of the FPS service.

Staff in many of the counties stated other concerns. DYFS workers, DYFS screeners, and FPS staff were concerned that the random assignment process would disrupt the relationship between DYFS and FPS staff. This was voiced for both counties with good and bad working relations. For counties with good working relationships, it was believed that the random assignment mechanism would interrupt the good communication between DYFS and FPS in regard to vacancies, case characteristics, and relaying of information. For counties where communication was already poor between DYFS and FPS personnel, it was felt that the study mechanisms would cause things to get worse.

In addition, DYFS supervisors were concerned that the random assignment process would interfere with the Title IV-A eligibility process. DYFS claimed a portion of FPS spending toward Emergency Assistance funding (EAF), under Title IV-A of the Social Security Act. Workers were required to have the family sign a IV-A eligibility form prior to referral to FPS. By getting the signature, workers begin the engagement process of getting a family ready to agree to participate in an intensive family service. Since workers could not know the results of random assignment until they returned to the office to make the referral, the workers felt they could be less forthright with families regarding the availability of the service. This appeared to be more an issue in counties such as Bergen and Monmouth, where the screeners prioritized cases for referral and did not seem to fully adhere to a first-come first service rule for cases.¹¹

Several ongoing concerns were discussed during the meetings and continued to surface in discussions with staff during the course of the evaluation. These issues include:

DYFS supervisors were concerned about whether the confidentiality of sensitive information about families would be maintained by the interviewers.
The proposed screening tool was criticized for being too focused on child abuse and neglect issues. Many of the workers still considered the FPS service most appropriate for family problem cases, especially with adolescent issues. The screening tool was used in six of the experimental counties. The protocol was used during pre-placement conferences for only one county on a limited basis.
DYFS workers and screeners were concerned that the referral process would cause delays and a reduction in referrals.

11 The eligibility form was phased out during the course of the evaluation, as a result of the federal Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) legislation signed by President Clinton in August 1996.

4.3.8 Violations and Exceptions

As part of our negotiations with DYFS, it was agreed that a limited number of eligible cases could be excluded from the study. DYFS administrators felt it was important that they not deny services to families that district office staff felt were at an unacceptably high level of risk. It was agreed that eight cases prior to random assignment and six cases post- random assignment could be identified as exceptions in the study. The number of exceptions was calculated to minimize the impact of the exclusions on the outcome analysis. The post-random assignment exclusions (called violations in this report) would be used for cases that were assigned into the control group and became higher risk after a few weeks due to the return of an abusive spouse or other critical changes in the case. Interviews with caretakers and caseworkers were conducted on cases excluded from the study after random assignment and they are included in the analysis.

A district office manager or screener made requests for exclusion to Central office staff. Only the Coordinator of the Technical Support Unit (TSU) or the Administrator of the Office of Case Planning, Screening and Emergency Response (OCPSER) could approve exceptions. The TSU took responsibility for developing criteria that could be used statewide for approving the exclusions. Many more violations (24) and exceptions (33) occurred than originally planned.

4.3.9 Inappropriate Referrals

Shortly after random assignment began in November 1996, study staff observed that some county screeners misinterpreted criteria for the evaluation and made inappropriate referrals. This was a particular problem for counties with multiple screeners (Essex and Passaic), as well as during periods when screeners were on vacation and substitute screeners were used. Sometimes, screeners would make a referral and subsequently additional information became available indicating the case was an inappropriate referral. Often these cases were identified by evaluation staff upon review of the DYFS referral materials or in conversation with screeners. These could occur either in the treatment or control group.

Many of the inappropriate referrals were reunification cases, not eligible for the study. Since our criteria excluded cases after placement of seven days, often screeners were not informed by caseworkers that the children were already out of home for an extended period prior to the referral. DYFS issued a memorandum to DYFS screeners clarifying that reunification cases where the children at risk were in care less than seven days should be included in the study. This appeared to help reduce inappropriate referrals in some counties. There was some concern that reunification as a reason for an inappropriate referral was being used to game the system by removing a case from the study that went "control." However, overall, 28 cases were identified as inappropriate referrals, 15 in the control group, and 13 in the experimental group.

4.4 Child Welfare Issues in New Jersey

To provide further understanding of the context in which the study was conducted, the following gives a brief overview of issues in child welfare in New Jersey. Child welfare services in New Jersey are administered centrally by the Division of Youth and Family Services (DYFS), a branch of the Department of Human Services (DHS).

Organization of Child Welfare Services in New Jersey. The state is divided into four service regions: Northern: Sussex, Warren, Morris, Passaic, Bergen and Hudson Counties, Metro: Essex, Union and Middlesex Counties; Central: Hunterdon, Somerset, Mercer, Monmouth and Ocean Counties; and Southern Atlantic: Burlington, Camden, Gloucester, Salem, Cumberland, and Cape May. Counties have one or more district offices.

Reports of child abuse can be made to either the local DYFS district offices or to a centralized Office of Child Abuse Control (OCAC), which handles all calls during evenings, weekends, and holidays. OCAC transfers all calls during regular business hours to the appropriate district offices. OCAC refers calls requiring an immediate response to an on call special response unit (SPRU) worker when the district offices are closed. Non-emergency cases are forwarded to the district offices for response. Emergency placement cases investigated by OCAC are transferred to the District office for follow-up. District offices have both intake and ongoing units. Some counties (e.g., Bergen) have converted to generic units, in which caseworkers perform both intake and ongoing case intervention.

The FPS Technical Support Unit (TSU) through a contract with DYFS, coordinates family preservation policy and programs on a state-wide level. The contract is supervised by the DYFS Office of Case Practice, Screening and Emergency Response, recently renamed the Program Support and Permanency Office. Because they were not DYFS staff, TSU staff reported that their authority was limited with most DYFS staff and district officials. The contractor during the entire study period was the Family Service Association of New Jersey. The TSU staff are responsible for the monitoring of all FPS providers.

Three tiered screening of child abuse/neglect reporting. DYFS utilizes a three tiered response system for inquiries for service through its hotline or individual district offices. An initial screening is conducted upon receipt of the call. Screening can result in one of three recommendations or tracks:

Investigation to determine if a child is abused or neglected [Child Protective Services(CPS)]
A request for service resulting in a child welfare services assessment to determine if DYFS can provide or refer for services [Family Problems]
Information or referral to another resource with no direct involvement by DYFS [Information and referral (I&R)].

In the screening process, cases with less risk will be referred to the two latter tracks: family problems or I&R. Family problems can include both child-related problems and family problems. Child-related problems include child substance abuse, medical and psychiatric issues, and pregnant or teen parent issues. Family problems include domestic violence, homelessness, lack of supervision, parenting issues, and parental substance abuse.

Family preservation referrals come from both CPS and family problem cases. A substantiated maltreatment report is not required to meet the criteria for referral.

Number of Child Welfare Reports and Indicated Cases. In New Jersey, the count of official reports of abuse and neglect to the state is very broadly defined and uses a two-tiered definition. It includes both abuse and neglect, as well as requests for family services. The latter is defined in the state data as "family problems." According to 1995 NCCAN Report, New Jersey had a rate of 32.43 children reported per 1000 children in the population.¹² This was based on a duplicated count of 63,684 child-based reports in 1995, including 28,924 reports of child abuse and neglect and 34,760 reports relating to family problems. Thirty-two percent or 9,279 child-based reports of abuse or neglect were indicated, compared to a national average of 34%. Of reports substantiated, 608 or 7% of children named in reports were removed from the home during or as a result of the investigation. This is in comparison to a national average of 15% for 1994.

New Case Handling Standards. In 1996, DYFS revised its case handling standards to ensure that the risk of harm to children was given emphasis by workers during an investigation. A two-day training was provided to all case managers and supervisors. One key component of the new standards is the priority that is given to evidence about parental substance abuse. Up to this point in time, a report identifying a drug-exposed newborn was identified as a family problem case. The new state policy now requires that a report of a drug-exposed newborn to also be classified as a neglect allegation.¹³ The change reflects the state's heightened concern about the effects of substance abuse.

DYFS attributes an increase in reports being classified as abuse and neglect to the change in standards. In January 1996, 44% of cases were classified abuse or neglect, compared to the total that includes cases classified as family problems. In December 1996, 58% of new reports and referrals were classified as child abuse or neglect.¹⁴

Table 4-8 compares total reports and referrals and counts of family problems and abuse or neglect reports and referrals from 1995-1996. There is an increase in total reports and referrals statewide. In particular, this increase occurred in six of our seven participant counties. However, for Bergen County, the total number of reports and referrals decreased from the previous year, from 3,564 to 3,323.

Table 4-8.
Total referrals by county by reason for referral
	Family Problems		Child Abuse/Neglect		Total
	1995	1996	1995	1996	1995	1996
Bergen	2,323	2,052	1,241	1,271	3,564	3,323
Burlington	1,234	1,234	1,348	1,634	2,582	2,868
Camden	2,550	2,294	3,996	4,418	6,516	6,712
Essex Total	5,528	4,356	3,796	5,994	9,324	10,350
--Newark City	3,636	2,680	2,332	3,911	5,968	6,591
--Other Essex localities	1,892	1,676	1,464	2,083	3,356	3,759
Monmouth	3,033	2,496	2,030	2,607	5,063	5,103
Ocean	868	671	1,829	2,318	2,697	2,989
Passaic	2,550	2,506	2,151	2,623	4,701	5,129
TOTAL	34,760	30,638	28,924	37,179	63,684	67,817

Boarder Babies. Concern about infants in the care and custody of the state remaining in hospitals beyond medical necessity has been a significant policy and political issue for many states, including New Jersey. In 1996, DYFS responded to the issue with the development of a Boarder Baby Project Team and recommendations for several initiatives that were implemented the same year. The initiatives included a statewide program for the recruitment and training of foster parents, in order to maintain a standby pool of foster homes for boarder babies. In addition, a pilot program was initiated for the recruitment of foster parents interested in adoption, but willing to care temporarily for children. This program would allow concurrent planning for children, encouraging reunification, while preparing an alternative placement, in case the child stayed in care beyond a year.

A federal class-action suit was filed against DYFS and DHS by the Association to Benefit Children on behalf of foster children who remain in hospitals beyond medical necessity. A Final Order of Settlement was entered in December 1996 with several requirements. With the receipt of a Federal Abandoned Infants Assistance Grant, several program elements were added or modified. Since Essex County accounted for 80% of the boarder baby population, a Boarder Baby Unit was established in the Metropolitan Regional Office. Case managers, on call, provided expedited care management with a goal to ensure permanency within 30-60 days of initial placement. The family preservation provider in Essex County, The Bridge, also received additional family preservation slots to provide support to birth parents upon discharge from the hospital.

(12) Child Maltreatment 1995: Report From the States to the National Child Abuse and Neglect Data System.

(13) Children at Risk DYFS July 1995, page 33.

(14) Excerpt of draft of DYFS 1995-1996 Child Neglect report.

4.5 Current Status in New Jersey

4.5.1 Feedback From Counties Post-random Assignment.

Interviews were conducted with DYFS district office staff and FPS staff in three participating counties in the Spring of 1998. The perceptions of staff regarding random assignment for the evaluation, changes in referrals to FPS during and after the evaluation, effects of the study, and the outcome of FPS were discussed.

DYFS and FPS staff believed that referrals to FPS increased after random assignment ended. Camden reported that new referrals increased from four cases per month in the fall of 1997 to seven cases per month in the spring of 1998 after random assignment ended. In fact, they began keeping a waiting list. Similarly, DYFS staff from Ocean claimed they had used a waiting list prior to the study and following the completion of random assignment they returned to using a waiting list and a triage procedure instead of first-come, first-serve.

Most DYFS and FPS staff attributed a drop in referrals to the evaluation. Camden staff indicated that individual workers became frustrated if one of their referrals became a control case. In Camden, some units, as a whole, did not refer cases at all. Several FPS providers spoke of low contractual utilization during the year. In addition, both FPS and DYFS staff described some changes in the types of referrals. According to staff, reunification cases increased during the study. Many felt this was a response to the study once workers learned that reunification cases were excluded from random assignment. For example, one worker asked a screener to consider a case of reunification because the birth father had left the home and a goal of services was to reunite him with his spouse and children. In Camden, staff spoke of a new Juvenile Court Judge who was ordering an FPS referral for reunification cases. Although staff voiced much concern about the number of cases served, there was little fluctuation in the number served in FY'96 through FY'98 (see Table 4-9).

Table 4-9.
Number of Families Served by FPS, 'FY 96 - FY 98' by County
County	FY '96	FY '97*	FY '98
Bergen	53	59	58
Burlington	48	54.5	47
Camden	57	69	51
Essex	82	73.5	84
Monmouth	54	56	53
Ocean	52	44	53
Passaic	52	50	51

* FY '97 totals include booster cases, counted as 0.5 case.

Despite the state's emphasis on serving more young children in FPS cases, targeting of teen children was still frequent in every county. Two reasons were cited. First, placement resources are often limited or expensive for this group. Therefore, FPS is considered while a resource is located. Second, ongoing cases with teenagers often exhaust all community resources and FPS is considered as a last resort to help the family.

One FPS director described a change in referral type due to the lower utilization of services during the study period. The DYFS screener could refer cases of lower risk when vacancies remained open. The screener felt that the study caused a delay in the referral process and some workers were concerned about referring high-risk cases. (Random assignments were made at the time of the initial phone call by the screener.) Camden FPS staff also reported that they relaxed their turnback policy, keeping low risk cases to avoid extensive vacancies in their caseload.

4.5.2 FY1998 Case Characteristics of Participating Counties

Aggregate data describing the service in FPS programs are available from state reports. County-specific annual monitoring is presented for FY 1998. The random assignment period in New Jersey, November 1996-February, 1998 overlapped partially with this aggregate data. The data are based on information self-reported by each FPS program as part of their contract obligations and oversight. It provides a snapshot of the caseload of families served during the fiscal year closest to the end of the random assignment period. It includes only those DYFS families served by the program and excludes families "turnbacked" from service. The following table (Table 4-10), lists the number of families served by each county, the total number of children in each family and the number of children identified by the referring DYFS worker as at risk. All programs operated at similar service levels, except for Essex, which served 84 families.

Table 4-10.
Number of families, children served, and children at risk in Family Preservation by County, FY 1998
County	No. Families	No. Children	No. Children at Risk	Percentage of Children at Risk(%)
Bergen	58	118	81	69%
Burlington	47	75	75	100%
Camden	51	124	91	73%
Essex	84	228	130	57%
Monmouth	53	172	104	61%
Ocean	53	130	104	80%
Passaic	51	118	67	57%

The ages of children at risk in the seven selected counties are presented in the following Table 4-11. Over 40% of the children at risk in five of the programs were 13-17 years in age. In Passaic County, 75% were in that age range. The state's policy of serving younger children at risk of child abuse and neglect was not being followed during this time period. In Essex County, there is a FPS program for boarder babies which are not included in these data or in our study. Many of the infants at risk in that county would have been referred to its "Boarder Baby" program possibly affecting the number of young children reported in Essex's service.

Table 4-11.
Age category of children at risk (percentage) by County
County	No. Children at Risk	Ages 0-5 (%)	Ages 6-9 (%)	Ages 10-12 (%)	Ages 13-17 (%)
Bergen	81	23	16	20	41
Burlington	75	12	17	21	47
Camden	91	21	19	20	40
Essex	130	18	24	14	45
Monmouth	104	28	20	19	33
Ocean	104	31	25	22	22
Passaic	67	6	4	15	75

During FY1998, the emphasis on referring cases with abuse/neglect or risk of abuse/neglect was not apparent in the seven participating counties. According to the Annual Monitoring data as shown in Table 4-12 below, the majority of cases in most counties were referred for reasons related to the behavior or activity of a child. The only exception was Ocean County that reported 51% of cases had abuse/neglect or risk of abuse/neglect as reason for referral. Over a third (36%) of Ocean's cases were referred as reunification cases.

Table 4-12.
Reason for referral (percentage) by County
County	Abuse/Neglect (%)	Risk of Abuse /Neglecta (%)	Child-Relatedb (%)	Otherc (%)
Bergen	18	21	57	4
Burlington	14	7	59	12
Camden	15	12	60	13
Essex	4	5	87	5
Monmouth	21	14	54	12
Ocean	25	26	13	36
Passaic	6	4	81	9

a. Risk of abuse and neglect includes cases referred for unknown injury cause.

b. Child-related reasons include runaway, behavior out of control, parent/child relationship, juvenile delinquency, and child is suicidal.

c. "Other" is primarily reunification in Ocean County.

Substance abuse continues to be a key problem in the FPS service of these counties. The data in Table 4-13, reported by the FPS programs, identifies the number of families served in which substance abuse was identified. It was identified as a problem at any point during the intervention and was not necessarily known at the time of referral to FPS. This is an important distinction since New Jersey now specifies that substance abuse by caregiver is grounds for reporting of child neglect. In addition, counties (DYFS and FPS programs) have local discretion on determining whether a family with substance abuse problems would benefit from FPS service. In Bergen County, almost half of the families served had substance abuse problems during FY 1998. In Essex, Monmouth, and Passaic approximately a third (39%, 36%, 32%) of the families were identified with substance abuse problems. In Burlington, Camden, and Passaic Counties, child substance abuse problems were more prevalent than parent/guardian substance abuse problems.

Table 4-13.
Percentage of families with substance abuse problems by County
County	Families with Substance Abuse Problems	Parent/ Guardian Only^a	Child Only	Parent/ Guardian and Child	Other Member of Household Only
Bergen	46	33	9	2	2
Burlington	24	9	13	2	0
Camden	14	6	8	0	0
Essex	39	25	10	2	2
Monmouth	36	28	2	4	2
Ocean	27	21	2	2	2
Passaic	32	8	14	6	4

a. Households where parent /guardian and another member of the household were identified with substance abuse problems are included in this category.

4.5.3 Current Policy Context

Several statewide changes have occurred since random assignment began in November 1996. These were the new FPS computer system, changes in administration, the Governor's Blue Ribbon Panel on Child Protection Services, statewide DYFS strategy planning, the federal adoption initiative, and welfare reform.

New computer system connecting DYFS and FPS programs. Problems in communication between public and private agencies can limit effectiveness of child welfare services. One of the major initiatives mentioned by state FPS administrators is the linkage of the 13 FPS agencies (serving 21 counties) with each other and the DYFS District and Regional offices. The prototype was scheduled to be in place in June 1997. The system will allow electronic exchange of referral and case information and more intensive program monitoring.

The FPS administrator reports that the system, now called the Electronic Case File System, was actually implemented in 1998 with most components activated. Staff at FPS programs were trained in spring 1998, and all programs began using electronic versions of forms at that time. There has been a delay in the communication component between the FPS and DYFS offices, while Internet security issues are resolved.

Change in Administration. Several leadership changes occurred during the implementation of the experiment. Several months after random assignment began, the Director of DYFS left office. In June 1997, the Director of the Technical Support Unit, changed. In September 1997, the Administrator of the DYFS Office of Case Practice, Screening and Emergency Response (OCPSER), changed positions.

The full impact of the change in leadership on the experiment is not discernible, however two effects can be identified. First, the approval of exceptions and violations were case-by-case determinations made by the TSU Director or the Administrator of OCPSER. Their threshold for approving an exception or violation was based on case specifics, but also reflected interpretation of county specific practice and policy, as well as state policy and politics. One would expect that different individuals have different thresholds for what is extremely high risk. At a briefing with several counties, one screener requested that the exception criteria be clarified, claiming it had changed as a result of the personnel changes.

Secondly, a new agreement with the study was made in regard to the length of the random assignment period. It was hoped that the original target of 500 cases would be reached in a one-year period of random assignment. The target was not reached in that time and shortly after the transition, a meeting was requested by DYFS administrators and FPS contractors to discuss the conclusion of the random assignment period. The new administrators requested that random assignment end by February 28, 1998, instead of continuing random assignment until a specified sample size was reached. A net sample size of 442 cases was achieved by the designated end date.

Governor's Blue Ribbon Panel on Child Protection. State administrators emphasized the importance of the Governor in defining the direction and priority for DYFS. In January 1997, Governor Whitman created the Governor's Blue Ribbon Panel on Children Services (BRP) to review the status of the child welfare system in general and the performance of DYFS, in particular. A final report was issued in February 1998, highlighting strengths and weaknesses of the child welfare system and recommendations for every component of DYFS and other components of the broader statewide system of services for families and children. The report was very critical of DYFS, stating that resources had fallen behind need, that staff morale was low, and that the system was in a state of crisis.

The values included in the Panel Report include emphasis on child protection. As stated in a discussion of "Child Protection vs. Family Preservation"

Child protection is of paramount value. When there is a conflict between the safety of a child and a family's right privacy and autonomy, the child's safety overrides all other considerations. Any ambiguity regarding the safety of a child will be resolved in favor of eliminating the source of harm or separating the child from it. This may include the removal of a child from his/her family. While it asserts that child safety is the paramount value, the Panel affirms the need to support families.¹⁵

In regard to family preservation targeting, the report suggested that for children at imminent risk, DYFS should err on the side of placement over preservation. The Panel emphasized that workers do not have clear guidelines when to remove children and when to leave them in their homes and provide in-home services.

Additional observations made by the panel are relevant to FPS targeting and effectiveness. The Panel found that standards for placement were inconsistent across districts. It observed that availability of resources to serve families were often used in deciding whether to place a child. In particular it was concerned that availability of foster homes in sufficient numbers were influencing workers' decisions to place or use family services. This observation was noted also in our briefing sessions with workers in the seven experimental sites.

The Panel was critical of the state's continuum of family support and preservation services. It observed that the state uses most of its in-home dollars in the state-run FPS Program. It found the FPS model lacking in flexibility stating:

Unfortunately, the program contains explicit limitations, offering very intensive services over a very short time period of four to eight weeks. This program has never received sufficient resources to meet the demand for services. But even more critically, it is too limited in terms of the minimum and maximum amount of time a worker can devote to a family. Most families have multiple long-term problems that cannot be addressed within one or two months. In addition, some families are unable to use such an intense approach and find it too intrusive.¹⁶

The Panel recommended that the FPS program be evaluated to determine what kind of cases it serves best and that existing slots be targeted to that type of case. In addition, the resources of the program should be expanded to fit the full continuum of preservation needs. This issue remains. The evaluation team heard comments from workers and administrators in several counties reiterating the dilemma that a very specific HomeBuilders model for placement prevention as the only DYFS funded resource was being stretched by workers and courts to fill the whole continuum of need.

Statewide DYFS Strategic Planning. In response to the Panel Report and need to plan for compliance with new Federal ASFA legislation, DYFS implemented a strategic planning process with DYFS staff and its community of service providers. A report in response was produced in June 1998. The report was organized according to six strategic goals: reform New Jersey's foster care system; improve safety and expedite permanency for children; improve the quality and accountability of DYFS direct services and administrative operations; enhance the professionalism of the child welfare workforce; improve case assessment and planning for children and families; and strengthen New Jersey's system of prevention services for at-risk children and families.

The plan mentions FPS services specifically only in the section on foster care reform. In that section, the plan recommends the expansion of FPS to include more reunification services as an approach to reduce the length of stay and to increase the number of children who reach successful permanency. In prevention services the plan does call for the coordination of all prevention services, to identify gaps and develop recommendations to improve the continuum of services.

DYFS is considering more specific changes to the FPS program statewide. According to the administrator for family preservation services, many changes are expected, stemming from a philosophical shift from preventing placement to a broader emphasis on family functioning and child and family stability. While placement prevention and attention to cases involving imminent risk will still have priority, county workers will be able to refer cases at a lower standard of substantial risk. Assessment cases and reunification cases will be eligible, as well as adoptive families and family foster homes where there is a risk of replacement for a child to another foster home.

A contractual change in service units is also being considered.¹⁷ Presently, an FPS program is expected to serve a contracted number of families with duration of intervention from four to eight weeks (an average of 4 and one-half weeks per family). The standard for duration will be made more flexible to allow programs to serve families requiring shorter or longer periods. This will allow the flexibility to serve families in the broader eligibility categories described above.

Counties and local FPS programs will be given discretion to expand eligibility and standards for case practice. This will result in some movement away from the HomeBuilders model that has guided the New Jersey program model since 1987. Planning for these changes and a new service manual continues to be in development.

Federal Adoption Project. In October 1996, New Jersey began an Adoption Opportunities Grant to implement concurrent planning with the expressed goal of expediting permanency outcomes for children in three counties: Union, Middlesex, and Essex Counties. As part of the state's permanency reform, the initiative developed a new program model known as fost-adopt. Fost-Adopt parents provide foster care, but also offer an adoption commitment if this becomes the child's long-term goal. In return, agencies provide intensive reunification services w/the birth family. Timnely decision-making for the child and adoption planning for those children who remain in care for more than a year.

Welfare Reform. "WorkFirst New Jersey" is New Jersey's response to the federal welfare reform bill and the implementation of TANF (Temporary Assistance to Needy Families). New Jersey passed the WorkFirst New Jersey Act effective March 1997. It is not yet certain how TANF will affect the child welfare system and the population it serves. There are several areas that might affect families. Persons seeking assistance are expected to engage in employment or work activity. It is not clear how this will affect families with children in regard to day care and the supervision of children. Secondly, there is a cumulative 60-month lifetime limit for the receipt of TANF for an individual. Next, of concern because of the high incidence of substance abuse among the child welfare population, individuals convicted of a felony involving the distribution, possession, or use of a controlled substance shall not be eligible for TANF. A person convicted of possession or use can be determined to be eligible only if they successfully complete a drug treatment program and remain drug free for a period of sixty days after completion of the program. Non-citizens who entered the country after August 22,1996 will be ineligible for TANF benefits.

One procedural change, which affects the FPS operation specifically, occurred in June 1997. Because TANF funding was converted into a federal block grant, the state no longer had to demonstrate eligibility for IV-A funding for FPS service. Workers previously had to have families sign an eligibility form prior to referral. The change simplified the referral process, requiring one fewer form. The state still required a visit within 72 hours of referral, but a signature was no longer needed to pursue the referral.¹⁸ This eliminated a service barrier which DYFS workers had described during our interviews in participating counties. The full impact of WorkFirst on families must be monitored closely.

(15) Governor's Blue Ribbon Panel on Child Protection Services, Final Report, February 20, 1998, Part Two, page 3.

(16) Panel Report, Part Three, page 5.

(17) Implemented in FY '2000.

(18) In 2000, the process was reinstituted, using the authorization form.

4.6 Summary

New Jersey has offered family preservation services since 1987, using the Homebuilders Model. Since its inception, referrals have been targeted at adolescents. Since 1995, the state has tried to redirect targeting to families with young children at risk of placement. There has been little success to date in this retargeting. While DYFS used a statewide training model and procedures, there was much variation in access to FPS in the seven participating counties. Differences were observed in screening practices, use of waiting lists, targeting, the use and definition of FPS for reunification, and the availability of other intensive services in each county. All counties continue to serve predominately adolescent at-risk populations.

In New Jersey random assignment for the evaluation was conducted from November 1996 through February 1998 in seven selected counties. A net sample of 442 cases were assigned. Interviews with caretakers and caseworkers were conducted. Administrative data were also collected. The analyses of these interviews and administrative data are presented in the chapters 5, 6, and 7.

Chapter 5: Tennessee

5.1 Introduction

In Tennessee the Family Preservation Program (HomeTies) is a resource within the state’s Department of Children’s Services (DCS).¹ The 95 Tennessee counties are grouped into 12 regions for purposes of service delivery. During the study period, there was a family preservation coordinator who was responsible for overseeing the administration of the family preservation programs, including setting standards, contracting with private providers throughout the state, and providing training and technical assistance. Direct services were delivered by private providers under contract to the state.²

Shelby County (Memphis) participated in the evaluation. Study enrollment began in November 1996 and concluded in May 1998. Frayser Family Counseling provides the HomeTies Program in Shelby County.

The sources of material for this chapter are reports and documents produced by the state and interviews with personnel at the DCS and HomeTies program.³ This information is presented to help understand the context in which services were provided, and to identify any changes that occurred during the implementation of the evaluation. The observations only reflect the perceptions of the individuals we interviewed.

This chapter begins with an overview of the characteristics of Tennessee’s children and families. Details of the Tennessee family preservation program, service delivery in Shelby County, implementation of the evaluation, and other organizational initiatives are then provided.

(1) Formerly the Department of Human Services.

(2) As discussed later, implementation of managed care for non-custodial services has changed this structure.

(3) Sources of data for this report include Tennessee's Family Preservation/Family Support Five Year Plan (1994); Family Preservation in Tennessee, The Home Ties Interventions: Selected Findings from the Program's Operation from 1989 to 1995 (Homer, K.S. Cunningham, M.L., Bass, A.S., Collette, S., and Evans, M.S., 1996); the State of Tennessee's Assessment of Children and Youth Committed to State Care (1989); Tennessee Home Ties History, and interviews with public and private agency staff.

5.2 Characteristics of Tennessee's Children and Families

This section provides demographic statistics on Tennessee's children and families. Child welfare statistics are presented for Shelby County, which was the focus of the family preservation study in Tennessee.

There are approximately 1,300,000 children under age 18 in Tennessee, with the majority being white (76 percent), and two-thirds under twelve years old (Table 5-1).

Table 5-1.
Age and race distribution of children in Tennessee
Total number of children under age 18 in 1997	1,324,800
Age	%
0-5 years old	32
6-11 years old	32
12-14 years old	18
15-17 years old	18
Race/Ethnicity 1997
White	76
African American	21
Hispanic	2
Other	1

Indicators of child health, education, and social and economic welfare in Tennessee as compared to the nation are presented in Table 5-2. Data have been abstracted from Kids Count Data Book, published by the Annie E. Casey Foundation. With respect to most indicators, Tennessee's families and children are similar to the national averages. The Casey Foundation developed a family risk index based on the following indicators: 1) number of children who are not living with two parents; 2) households in which the head of household did not have a high school degree; 3) family income below poverty level; 4) parents did not have steady employment; 5) the family was receiving welfare; and 6) no health insurance for the children. Using the Casey risk calculation, the percentage of children in Tennessee considered at risk is the same as in the nation as a whole, 14 percent.

Table 5-2.
Indicators of children and family health, education, social and economic welfare in Tennessee as compared to Nation
	Tennessee	Nation
Health:
Percent low birth weight babies (1996)	8.8%	7.4%
Infant mortality rate (deaths per 1,000 live births, 1996)	8.5	7.3
Percent of 2 year olds immunized (1997)	78%	78%
Percent of children without health insurance (1996)	13%	14%
Percent of children covered by Medicaid or other public-sector health insurance (1996)	35%	25%
Child death rate (deaths per 100,000 ages 1-14 in 1996)	30	26
Teen violent death rates (deaths per 100,000 ages 15-19 in 1996)	81	62
Teen birth rate (Births per 1,000 15-17 females in 1996)	40	34
Education:
Percent of teens who are high school dropouts (1998)	13%	10%
Percent of 4^th grade students scoring below basic reading level (1998)	42%	39%
Percent of 8^th grade students scoring below basic math reading level (1998)	29%	28%
Welfare, Social, and Economic:
Median income of families with children (1996)	$33,500	$39,700
Percent of children in poverty (1996)	22%	20%
Percent of children in extreme poverty (1996)*	11%	9%
Percent of children living with parents who do not have full time employment (1996)	29%	30%
Percent of families with children headed by a single parent (1996)	29%	27%

Source: Kids Count Data Book, Published by Annie E. Casey Foundation, 1999.

* Extreme poverty is defined as income below 50 percent of poverty level.

Child Welfare Statistics for Shelby County. To provide background for the evaluation findings, an overview of the number of child abuse and neglect investigations and percent of indicated reports for fiscal years 1995–1998 is presented in Table 5-3. The number of children for whom there were abuse and neglect investigations shows a slight decrease in FY's 97 and 98. However, agency staff reported that lower abuse and neglect investigations may be due to administrative undercount rather than a decline in the number of children investigated. During those two years, administrative systems were being updated and the staff shortages in Shelby County resulted in data entry being a low priority. The percentage of cases substantiated remained fairly constant over the study years: FY 95, 36 percent; FY 96, 35 percent; FY 97, 41 percent; and FY98, 38 percent.

For all 4 years, children under one year of age had a slightly higher rate of substantiation than older children. Other than FY 96, males and females had similar rates of substantiation. Substantiation rates fluctuated by types of maltreatment within each year with failure to thrive, abandonment, educational neglect, physical abuse, substantial risk of physical injury, and substance affected infants being substantiated at higher rates.

Children in substitute care also remained fairly constant throughout the study period. In FY 95, the year prior to random assignment, 1,772 children were served. The number of children in care on the last day of each fiscal year rose slightly over the study years: 1,880 children in FY 96; 1,963 children in FY 97; and 1,943 children in FY 98.

5.3 History of Family Preservation in Tennessee

5.3.1 Background

The family preservation program in Tennessee, HomeTies, began in October of 1989. State funding ($1.71 million in FY 90) for the program was provided through a joint legislative resolution signed by the Governor directing the Departments of Human Services, Mental Health, and Youth Corrections to proceed with an inter-departmental family preservation program. Eight teams, serving 24 counties, including Shelby County, were funded in 1989 (FY90) as pilot projects. The program initially served families with children diagnosed as seriously emotionally disturbed and adolescents charged with delinquent acts or status offenses who were at imminent risk of placement in substitute (i.e., out-of-home) care. Referrals were made by staff in the three state agencies who could place or cause the placement of children. HomeTies was, and is, based on the Homebuilders model of family preservation services--serving multi-problem families for 4-6 weeks using behavioral and cognitive therapeutic interventions and concrete services in order to prevent placement. Services are delivered through contracts with private and public agencies, most often community mental health agencies. This structure changed in 1998 when the state moved to a managed care model of purchasing and delivering services in one-half of the state.

Table 5-3.
Number of children with child abuse and neglect investigations and percent indicated by type of maltreatment, age, and gender in Shelby County
	1995		1996		1997		1998
	Number Investigated	Percentage Indicated	Number Investigated	Percentage Indicated	Number Investigated	Percentage Indicated	Number Investigated	Percentage Indicated
Total	6,606	36%	6,642	35%	5,029	41%	4,578	38%
Types of Maltreatment
Minor physical abuse	1,415	26	1,438	26	1,101	30	1,057	30
Severe physical abuse	95	41	120	53	88	67	67	70
Failure to thrive	27	74	27	85	26	80	21	76
Malnutrition	4	50	6	50	4	50	3	33
Physical neglect	2,252	33	2,281	32	1,683	40	1,494	35
Medical neglect	306	37	254	40	197	38	180	36
Lack of supervision	630	48	541	42	409	47	480	46
Abandonment	185	66	203	62	144	65	118	69
Sexual Abuse/ Exploitation	954	40	1,063	35	956	41	755	37
Moral abuse	12	58	9	22	4	50	5	60
Emotional abuse	95	38	94	40	65	38	30	33
Emotional neglect	56	63	34	44	19	94	11	45
Educational neglect	12	83	26	69	20	75	9	88
Other	451	20	360	23	181	25	178	17
Substantial Risk of physical injury	43	72	93	65	68	78	115	64
Substance affected infant	48	96	78	90	61	97	45	89
Age:
<1 year	609	46	558	47	434	55	394	54
1-2 years	951	35	905	33	639	38	596	38
3-5 years	1,385	36	1,425	31	1,056	38	888	37
6-11 years	2,078	35	2,058	34	1,612	41	1,606	37
12 years and older	1,577	33	1,657	36	1,283	40	1,090	35
Gender:
Male	3,132	35	3,099	35	2,318	41	2,141	38
Female	3,471	36	3,542	25	2,711	40	2,436	38
Unknown	3	--	1	--	--	--	1	--

The FPS and family reunification programs offered by HomeTies constitute one of the state's four programs designed to preserve families. Wraparound services (i.e., individualized services purchased to prevent placement, reunify families, or support community/family based placements) are also available statewide.⁴ Community intervention and intensive aftercare programs are also available in selected counties for families with youth in the correctional system.

4 Wraparound services are not available to families receiving services in the Home Ties program, but they are available for use following intensive family preservation services as aftercare services.

5.3.2 Initial Planning, Program Development, and Training

The impetus for HomeTies began at a conference in Nashville of southern state service providers and legislators in October 1987. Various models of intensive family preservation services were presented. Members of the Tennessee Select Committee on Children and Youth and others in Tennessee attended this conference and became strong advocates of FPS. There were initial differences of opinion about which FPS model or models should be chosen and, ultimately, the Homebuilders model was recommended to legislators in Tennessee. Family preservation advocates from the Behavioral Sciences Institute (BSI, the developers of the Homebuilders model of family preservation services in Washington State), the National Conference of State Legislators, the Center for the Study of Social Policy, and the Edna McConnell Clark Foundation made presentations to the state's Select Committee on Children and Youth about the value of and need for FPS in Tennessee. Legislators responded quite positively, and there was little controversy about starting the program. Significantly, there was a new Democratic governor in Tennessee at the time, and FPS fit well with his emphasis on shaking up the status quo and developing creative government programs that could make a difference.

The development and implementation of HomeTies involved collaboration among multiple state agencies, initially including the Departments of Human Services (DHS), Mental Health, Youth Development,⁵ and Finance and Administration. Representatives from these agencies met in 1988 to examine financing options, interdepartmental service coordination, and existing FPS models. This committee completed a policy-procedures manual, developed forms, and, with researchers from the University of Tennessee, designed the evaluation of HomeTies. The request for proposals was generated from this work, and required that agencies replicate the Homebuilders model.

The $1.71 million in initial funding for HomeTies in FY90 came from redirected foster care funds, block grants, and state dollars. No additional dollars were added to the state budget to fund FPS. The table below shows the source and types of funds used to provide initial program funding.

Table 5-4.
Source and type of funds used to provide initial program funding.
Department	FY90 Funding	Source
Human Service	$850,000	Redirected foster care funds
Mental Health	$647,500	Block grant funds and state dollars
Youth Development	$212,500	State dollars

Start-up training included: (a) inter-departmental training for all referring staff on FPS policies and procedures; (b) a Homebuilders orientation by BSI for all referring staff; and (c) training by BSI on the Homebuilders model for all HomeTies workers in the contract agencies.

(5) Until 1996, the Department of Youth Development provided all youth correctional services in Tennessee. In 1996, these three agencies, along with others were combined to form the Department of Children's Services.

5.3.3 Program Expansion

The HomeTies program was expanded several times between 1990 and 1994.

In 1990, the program was expanded by three additional teams as a result of additional Title IV-E allocations. The Edna McConnell Clark Foundation also awarded the state $104,000 for coordination, training, and research, resulting in the hiring of a State Coordinator for HomeTies.
The program expanded much more dramatically in 1991. The 1991 Tennessee Family Preservation Act mandated that the program serve all eligible families in the state. Seventeen new teams were funded. Expansion began in the fall of 1991 and was completed in May 1992.
In 1992, child abuse and neglect cases were made eligible for the program. This policy change was preceded by extensive discussion and distribution of policies and guidelines for serving this new population.
In 1993, program regulations added juvenile court judges and their staff to the list of professionals from whom the program could receive referrals.
In 1994, HomeTies added another worker to each team and began accepting family reunification cases.
Between FY90 and FY94, there was a 650 percent increase in the number of families served by placement prevention services, from 400 cases in FY90 to 2,976 in FY94. In FY95, there was a slight (4%) decrease in the total number of families served.
Despite the slight decline in total families served in FY95, family reunification cases increased statewide by 18 percent from FY94 to FY95. (See the table below for the number of families served, percent change in the number of families served, and the number of FPS teams since the inception of the program through FY95.)

In its five-year plan for family preservation and family support services, Tennessee chose to put all new federal funds into family support rather than family preservation services. In FY95, the state planned to expand Healthy Start--an early intervention program for parents with newborns at risk of child maltreatment. In FY96, the state planned to add 31 Family Resource Centers -- networks of state and community based services designed to help families solve problems before crises occur.

HomeTies contracts for service providers were originally based on a $2000 per unit cost. In FY93, the state began reimbursing the agencies for cases served rather than a preset number of cases. This may, in part, explain the decreasing time frame of interventions and the increased numbers of families being served. Due to rising costs ($2028 per family in FY92 and $2624 in FY93), the state capped the contracts in FY95, resulting in lowered total expenditures ($7.8 million in FY96). HomeTies is a Medicaid reimbursable service and rates are set by the state's TennCare system. As of November, 1997, the Tennessee Director of Budget reported that 58 percent of HomeTies cases were eligible for full Medicaid reimbursement for services.

5.3.4 Significant Events Affecting HomeTies

In 1991, the Department of Finance and Administration established the Children's Plan by creating a single funding pool to finance children's services. Looking for an independent agency that did not have a vested interest in maintaining the status quo, the Department of Health was selected to administer the Plan. Assessment, Care, and Coordination Teams (ACCT) were formed to provide comprehensive assessments of children entering care in order to make better initial placement decisions for children and youth. The ACCT would also monitor the child's progress through the placement system, manage the expenditures of flex funds, and function as a single portal of entry for children needing state services (pre-custodial, custodial, post-custodial). ACCT was housed in Community Health Agencies (CHAs) which were originally created in statute to advocate for community based medical care for the poor across the state. The CHAs were administered by the Department of Health and were located in 12 regional offices. The ACCT was suppose to review all referrals to HomeTies and to provide an assessment of whether children were at imminent risk of placement. The degree to which this actually occurred varied widely across the state and, in the fall of 1993, ACCT was dropped as the gatekeeper of HomeTies referrals. One example of the difficulty in implementing this referral strategy was that some juvenile court judges refused to send referrals through ACCT. In FY 95, ACCT staff continued to make referrals to HomeTies, accounting for slightly less than one-third of referrals.

Table 5-5.
Number of families served in fiscal year 1990-1995.
	FY90	FY91	FY92	FY93	FY94	FY95^b
TENNESSEE HomeTies Program
Placement prevention, Number of families served	400	788	1282	2781	2976	2777
Reunification, Number of families served	--	--	--	--	332	391
Total number families served, prevention and reunification	400	788	1282	2781	3308	3168
HomeTies funding (in millions of dollars)^a	--	1.8	2.6	7.3	8.8	8.2

a. Funding for FY96 was $7.8 million, and was budgeted at $8.5 million for FY97.

b. There are no caseload data available for FY96, 97, 98, or 99. FY96 is lost.

In April, 1996, the Tennessee General Assembly passed legislation to remove child welfare services from the Department of Human Services and create the Tennessee Department of Children's Services. The new Department consolidates family and children services from several Departments and includes: child welfare, child development, day care licensing, pregnancy and parenting services, youth corrections, and the children's fiscal division. While the Department of Human Services continues to administer the Social Services Block Grant and Title IV-A funds, the Department of Children's Services (DCS) administers all Title IV-E and Title IV-B funds.

DCS has recently been operating under tight fiscal constraints. During site visits in 1997, DCS was altering the structure of service delivery in an effort to increase service provision without increasing personnel expenditures, strengthen follow-up services, and decrease duplication and problems associated with case transfers. The conversion process affected workers both in- and outside of DCS. The Assessment Care and Coordination Teams were dissolved and ACCT staff no longer reviewed any referrals to FPS. The community health agencies called Community Service Agencies (CSAs) became contract agencies that provided services directly to families (one person described this as quasi-privatization). CSA child welfare staff began to carry their own cases and be part of teams (along with high risk CPS staff, outreach, crisis intervention, youth development, and foster care staff). The Community Service Agencies were the fiscal monitors of flexible funds to prevent or reduce time spent in state custody.

The conversion process affected investigative staff. The emphasis changed from service provision to investigations which required strict adherence to the policy of completing investigation in 60 days or referring cases to ongoing service units. Staff were required to close or transfer cases within 60 days of case opening. Also if investigator caseloads were greater than 30 families at a time, they had to justify the number in writing. The conversion process had a strong negative effect on the morale of DCS workers due to a high level of uncertainty about how their job status would be affected.

Since 1997, Tennessee has been moving toward a managed care model of service delivery for noncustodial cases. Fifty percent of the state is currently using the new managed care system, but Shelby County, among others, is not expected to be converted until July 2000.⁶

The Department of Children's Services (DCS) reported that most non-custodial service contracts had been based on a fee-for-service basis without regard to level of service or quality of performance. Little evaluation had taken place and services were not distributed evenly across the state. As a consequence, the state asked all 12 Community Service Agencies to conduct local needs assessments for their regions defining service priorities and gaps in their current service continuum. The assessments were completed in late January 1998. The assessments focussed on three levels of service: prevention (community education and early prevention), intervention (treatment), and diversion (just prior to commitment services).

As a continued move towards managed care, the Department of Children's Services then issued a Request For Proposals for each region based on the local needs assessments. As part of the proposals, networks of agencies bid a case rate for families. Once in place, the network will decide the amount and kind of services families require and the length of service delivery to prevent placement.⁷ The state plans to have 12 networks across the state each with a lead agency which will subcontract with other agencies for services or with a coalition of service providers. The 12 networks will replace approximately 70 existing contracts for service. The networks are to be outcome focussed and will be financed by Social Service Block Grant (SSBG), some of the state's Family Preservation /Family Support funds, and all state HomeTies dollars.8

The state planned to have the networks in place by July 1, 1998. However, the state only approved proposals from six regions and rejected the remainder largely due to service cost estimates, particularly the capitaled rate amount for families receiving in-home services, with a specific annual cap of $1,550 per family. Following withdrawal of the RFP from the six regions not funded, Shelby's CSA submitted a plan which proposed a five year pilot program using “an integrated fee-for-service and risk- adjusted model” for children at risk of state custody

In essence, the plan is to have the CSA convene community members, including service providers, the courts and DCS, who will develop both a risk adjustment scale to classify children into moderate, high, and imminent risk of placement categories and a service delivery model to address each level of service need. Both the University of Tennessee and the University of Memphis will be part of this group to help review data and design the service model. Case rates will be established looking at historic expenditures of flexible funds, SSBG, and IV-E dollars. Once the model and fee structure are established, the group will prepare a program evaluation, funded by local resources and conducted by the two local universities. Finally, an RFP will be written, and after approval from the state, will be released into the community. Network provided services are expected to begin on July 1, 2000. The state's move toward managed care will also eliminate the state's Homebuilders family preservation program. While the new service networks will be required to offer some form of intensive family preservation, they will not be required to offer a Homebuilders model and the state will no longer provide uniform training and oversight.

At this time, there is no consensus about the role of Shelby County DCS in case oversight once the network is involved with the family. In other parts of the state, DCS acts only as a gatekeeper (accepting calls, conducting investigations, and making referrals) and the CSA monitors families' progress. It has also not been determined whether the Frayser HomeTies program will continue under the network; if it does not, a less intensive service model will be used.

(6) At the same time that DCS is preparing to shift to a managed care model for noncustodial cases, the state is experiencing a significant budget shortfall which threatens to eliminate large amounts of DCS prevention services. If the state is unable to raise additional funds through tax increases, the shift to managed care will probably not occur.

(7) The Director of State DCS Finance reported that because networks will not bill for individual services, state finance will no longer track the exact service families receive.

(8) The state's 98/99 APSR reported that funding is also coming from savings generated by the continuum of residential care.

5.3.5 Description of Tennessee Family Preservation Model

HomeTies follows the Homebuilders model and utilizes a behavioral cognitive approach to work with multi-problem families. Workers try to engage the entire family and teach skills that will increase their ability to function more effectively. Workers carry two families for four to six weeks, and are available 24 hours a day, seven days a week. Through a wide range of services and the ability to access $250 per family in flexible funding, workers address crises, monitor family stability, assist families, create linkages, and obtain services in the community.

State guidelines rule out referring the following case types for HomeTies Services.

Physical Abuse
The physical abuse is considered life threatening, necessitating the children be immediately placed to ensure safety (for example, the parent threatens homicide of the child).
Sexual Abuse
The perpetrator of the sexual abuse resides in the same home as the victim.
Substance Abuse
The adults in the home are found incoherent all of the time due to substance abuse and all of their resources are used to support their addiction.
Family members, including parents, fear being murdered by the drug community and move constantly to avoid harm.
A parent wants the child(ren) to be placed and refuses to consider services that might enable the child(ren) to remain in the home.
Neglect
Neglect cases are not ruled out unless the family refuses services.

CPS intake workers complete a risk assessment form to identify high, intermediate, low, or no risk. High risk cases are identified as cases where “the child or children in the home are at imminent risk of serious harm if there is no intervention in the situation.”

A typical high risk case might involve such factors as: 1) a vulnerable child; 2) a history of previous maltreatment; 3) an active perpetrator who has continued access to the child, and; 4) no available support or family strengths to offset the stated risks.

5.3.6 Family Preservation Services in Shelby County

Since Tennessee does not operate a central state hotline, all CAN calls for Shelby County come directly to two screeners within the county. The screeners determine risk levels and using a manual intake system, assign calls to the appropriate investigative unit.

During the study period, there were ten Child Protective Services units in Shelby County with approximately 65 staff. In addition to the Intake Unit, there were four Emergency Response Units (investigation within 48 hours), two Non-Emergency Units (investigation within seven days), and a single High Risk/CPS Ongoing Unit. There was also a High Risk, Multi-Victim/Multi-Perpetrator Unit, and a Court Unit which was primarily responsible for conducting home studies and visits for relative care and custody change cases.

Some investigative caseloads were as high as 150 cases/families per worker, causing great strain on and concern among staff at all levels. Caseloads within the Ongoing Unit averaged about 20. In May 1997, the service delivery plan for the new Department of Children's Services was implemented in Shelby County. Child Protective Services was divided into six work units, with a supervisor (team leader) and eight case managers, plus case manager positions responsible for CPS intake. The new service model called for CPS case managers to only do the investigative piece, referring any families who needed services beyond the investigation to child and family teams. Existing CPS policy requiring that investigations be completed in a 60 day time frame was strictly enforced. In Shelby County, the CSA provides all follow-up services and case management for these CPS cases. CPS case managers continue to refer to HomeTies because of the crisis nature of the service and its use to prevent placement.

In FY95, Shelby County served 12 percent of the state's accepted HomeTies cases (an increase from 8.3% in FY90), making it the second largest HomeTies program in Tennessee. The Shelby County HomeTies Program grew from 317 in FY93 to 391 in FY95, a 23.3 percent increase. This overall increase was due primarily to the inclusion of reunification cases (14 in FY94 and 58 in FY95). During the study years, the number of families served slightly decreased. (See Table 5-6.)

Table 5-6.
Families served by HomeTies in Shelby County from FY 93-98
	FY93	FY94	FY95	FY96	FY97	FY98
Placement prevention, Number of families served	317	351	333	336	292	331
Reunification, Number of families served	--	14	58	38	27	16
Total number families served, prevention and reunification	317	365	391	374	319	347

Shelby County DCS workers' views and use of HomeTies. Investigative and ongoing staff reported referring equal numbers of cases to HomeTies and cited several reasons that they use the program. Investigative workers reported that HomeTies was used as their first resort for families at imminent risk of removal because program staff could be in the home monitoring and assessing families around the clock. Ongoing workers reported that they used HomeTies as a last resort, after they had tried less intensive services because of the intensity of the intervention and the availability of concrete resources (flexible funding, transportation). Both investigative and ongoing workers said that HomeTies staff could be relied upon to provide thorough and frequent feedback about families, both during the course of treatment and at the end of treatment. Feedback was particularly useful because it included information on both family strengths and weaknesses.

Unlike many child welfare jurisdictions, Shelby County has a variety of in-home and office-based therapeutic programs to which workers can refer (these are described in Exhibit A, provided at the end of the chapter). However, supervisors and workers noted that, prior to the study, HomeTies generally had a waiting list and was a preferred option for many workers for a number of reasons.⁹ HomeTies could be relied upon to monitor and assess new cases in crisis and provide intensive support to ongoing cases which were perceived to be on the brink of placement. To a large extent, public agency workers had previously been able to make referrals directly to a specific HomeTies worker and they could contact this individual directly to set up and coordinate the intervention. Also, there was no paperwork or external review of referrals associated with referral to HomeTies. Some people stressed how important it was that HomeTies had been accessible to emergency staff around the clock and would engage the family within 24 hours of the referral--day, night, or weekend, which helped to stabilize families. This was especially important to investigative staff who have historically referred the cases, when they perceived families to be in crisis. Some workers thought that some of these advantages were reduced or eliminated by the initiation of the study (see discussion of the impact of the study on referrals below).

In general, most administrators and workers viewed HomeTies as successful in working with a wide range of families. DCS workers said that the best candidates for HomeTies were families who needed assistance with communication skills or anger management. Public agency supervisors said that HomeTies staff are often perceived by clients as allies whereas DCS staff are perceived as the enemy. The supervisors also said that HomeTies has been particularly successful with acting-out teenagers, and with families where parents do not want to work with DCS. HomeTies is willing to try a number of workers to create a “good fit” with a family.

DCS staff also had some negative comments and concerns about the program.¹⁰ These included:

Uneven staff - some staff are too “gullible” - they believe “stories” families tell;
Some workers are intimidated by families or refuse to go to some homes (this appeared to be particularly frustrating to public agency staff because they do not have the option to refuse a home visit);
Some staff are reluctant to work with drug using families (DCS staff believe this is because of personal risk issues of HomeTies staff and not a clinical decision);
HomeTies recommends removal more frequently than agency staff.

HomeTies has very few “turnbacks,” DCS staff estimate 2-3 percent of all referrals are turned back to the agency, almost all within the first seven days. The majority of turnbacks are the result of a family's unwillingness to cooperate with the program. The other two reasons cited for turnbacks are: a) a family has too many problems (generally a violent adolescent) and the worker is at risk; or b) the children are not at imminent risk of state custody.

Once HomeTies has completed its four to six weeks of intervention, the worker reports to DCS staff about the continuing level of risk in the family and makes recommendations about the family's continuing service needs. DCS staff report that they almost always accept the program's recommendations about the family. According to Emergency Response workers interviewed (those that investigate within 48 hours), 90 percent of their cases are closed directly after HomeTies intervention. The remaining 10 percent are transferred to ongoing services for continued supervision. Ongoing/High Risk Workers estimated that 60 percent of their cases are closed directly after HomeTies intervention; the remaining 40 percent remain open.

Frayser Family Counseling's (FFC) HomeTies Program. In Shelby County, HomeTies is offered by Frayser Family Counseling, a private, non-profit community mental health center. The center has 95 employees including psychologists, psychiatrists, nurses, and other mental health personnel. The center provides voluntary outpatient services to individuals of all ages. Among its many services are individual and group therapy, in-home family preservation and support services, alcohol and drug therapy, victim assistance, and child and adolescent evaluations.

In May 1997, HomeTies had three supervisors and fifteen counselors,¹¹ with 5-6 workers per supervisor. In 1997 and 1998, HomeTies was funded for 21 counselor positions and three supervisors. Community mental health started losing dollars because of TennCare, and quickly learned that if they worked outside the model and saw more of the same numbers of families with fewer staff, they increased their revenue. The program director serves as one of the supervisors. Another HomeTies supervisor is responsible for the Life Coach program. Nine of the HomeTies workers also take Life Coach cases (see discussion below).

Two of the workers had over fifteen years of experience in the field, five workers had 5-10 years of experience in the field, and the other eight workers had 2-5 years experience. All staff are required to have two years of experience when they are hired. Twelve of the workers were female and twelve were African American. Sixty percent of the workers have master's degrees (the state requires at least 50%), six of which are in counseling, two have MSWs, and one has a masters degree in criminal justice. One of the staff previously worked at DCS.

Workers are supposed to serve 1.5 cases per month (21 case workers x 18 cases per year), for a total of 378 cases per year. The program director estimates that 60 percent of the cases are referred to HomeTies by DCS, 30 percent by Community Service Agencies, and 10 percent by the juvenile court, with less than 1 percent from mental health centers.

HomeTies cases can be extended for up to two weeks, but this occurs in less than 5 percent of cases. One possible reason for the rarity of extensions is the availability of other services in the agency (i.e., Life Coach, see below). The agency also provides a six-month check-in with families when the child is still in the home.

HomeTies and Life Coach. Because Life Coach serves some control group cases, it is important to describe the relationship of HomeTies to Life Coach. In addition to sharing staff, HomeTies and Life Coach (LC) are intermingled in several other ways. First, workers reported that approximately 35 percent of HomeTies cases go to Life Coach for follow-up services, usually with the same worker providing the services. These services ($60 per day, about 70 percent of the HomeTies rate) are usually provided for 30 days, but can last as long as needed. Second, control group cases were being referred to LC. The Life Coach supervisor said that there is no difference between LC and HomeTies. The program director basically agreed, but said that LC workers spent slightly less time with families (4-7 hours per week).

One difference between HomeTies and Life Coach is that referrals to LC must be reviewed by the prevention team (at the time, DCS and ACCT). Also, LC cases did not have access to flexible funds (i.e., $250 in cash). HomeTies workers often work overtime on LC cases. If a worker has two HomeTies cases, he or she can only have one LC case.

Other information about referrals. Many of the referrals involve parent-child conflicts in which the parent wants the worker to fix the child. According to therapists, approximately 65-70 percent of families have substance abuse problems and 95 percent include one person (usually the mother or the child) who takes psychotropic medication. Other prominent problems of children and families include school behavior and attendance, child behavior at home (e.g., not doing chores, not following rules), housing problems, parents' relationships, domestic violence (relatively few cases, some with past incidents), failure-to-thrive infants, and drug-exposed infants. Referrals of drug-exposed infants were more frequent earlier, and staff were unclear why these cases are not being referred.

Workers and supervisors were generally satisfied with the types of referrals they receive, though workers stressed that DCS should screen parents who are mentally ill for appropriateness. Turnbacks of referrals to DCS occur if there are seven days without contact with a child because of parents refusing services, parents wanting the child placed, the child running away, or failure to comply with safety plans.

When asked which cases were most appropriate or inappropriate, supervisors contested the idea of a typology of cases based on problems (such as drug abuse or mental illness) or even problem severity. They stressed instead that the issue of motivation was more important in determining the difficulty of a case, and they stressed techniques for building motivation (see below). This is consistent with some of the issues that have been raised previously in discussions of the difficulty of targeting families for referral to FPS--that one cannot know before referral the extent of family problems or the family's responses to intervention except within the context of the helping relationship.

Cases are assigned to specific workers based on openings, except for a small number of cases, for example, a sex abuse case might require a female therapist.

Training and supervision. All staff, called therapists, are trained by BSI in the Homebuilders model. While this basic training was viewed positively by supervisors, it was not considered sufficient preparation for actual work in the field, especially for younger, non-Masters level staff. Newer therapists receive individual supervision for 3-6 months, and they shadow other therapists for at least one full case, present cases at weekly staffings, and are shadowed by another therapist when they take on cases.

Supervisors provide general professional support to workers and personalized coaching on clinical skills. In addition, they described supervision as a process of helping workers learn to: a) focus their efforts with families by picking workable issues (i.e., ones that could be addressed in four weeks) and reducing DCS goals to core issues and goals; b) communicate to the family and DCS that the therapist is working with the family's agenda (knowing also that the family's goals can change as they become more aware of opportunities); and c) continually assess the family strengths, needs, and goals, and the situation, and to be flexible in their approaches to helping families based on assessments.

Practice approach. Supervisors and therapists identified important purposes and strategies of working with families (in addition to those mentioned above related to supervision), and some of the benefits of in-home services. The descriptions here are intended to be illustrative of how staff approach practice at FFC, not a comprehensive description of practice.

Staff noted the importance of identifying family strengths by looking at the situation and family members' motivation. Staff emphasized the importance of building motivation in the family to change and of building a sense of empowerment. These appeared to be interrelated goals that are particularly important for families who are referred to HomeTies --who wouldn't ordinarily seek help. These goals are accomplished through a variety of means, including:

Spending time with families in their world and at times that are convenient for them;
Assuring families that HomeTies staff are not from DCS and the families can ask them to leave;
Listening to family members' perspectives in a non-blaming, respectful manner--this is often the first time families have experienced this;
Determine what the family's goals are and examine how they can relate to the goals of the public agency;
Showing them that they have power to change some things by identifying small steps that can be made to improve the situation--showing parents they can be different by breaking down big foggy clouds of problems into small parts; and
Focusing on solutions.

Therapists note that the first things that they do is to assess and address safety issues and concrete needs. Safety issues include running away (e.g., you don't tell them what to do, but you talk with them about what they do to stay in the home), suicide assessment (e.g., ask about attempts, weapons, and pills; lock up pills), and physical abuse (agree to a no-hit policy while HomeTies is in the home).

Staff also noted that using flexible funds ($250 per family) generously and creatively (e.g., refrigerator, rent, car, utilities, moving, food, meals out) to meet a family's initial concrete needs is a very helpful strategy in HomeTies. Use of flexible funds must be approved by supervisors, and workers consider or try other means of addressing concrete needs first.

When working with parent/child conflict cases, therapists suggest that parents have generally lost their power, have their own issues with conflict, or inappropriately want the child to be their friend. Therapists often work with parents separately, and try to show parents that they can be powerful and help parents see the good in their children and respect the perspectives of the children. Therapists also noted that behavioral charts with agreed upon goals and reinforcers are very helpful in promoting specific changes in roles and behaviors.

Therapists refer to other social services in 50-60 percent of cases. They try to identify needs as early as possible so that referrals can be made. Sometimes families are able to start other services during HomeTies, other times they are placed on waiting lists. Services used include day care, homemaker services, and parenting groups, as well as other state and federally funded rape crisis services, HIV support groups, vocational rehabilitation for the mentally retarded, mentoring, respite, drug treatment, psychiatric treatment, housing advocacy, counseling, telephone hook-up, and free concrete services provided by churches.

While noting that in-home services are more difficult and stressful than traditional therapy and that they involve a shorter engagement period, the therapists believe that in-home services are better, “one month of in-home is worth 6-12 months of outpatient.” Therapists noted the following benefits of HomeTies:

Better assessment: they know much more about families because they see conflict, caring, housekeeping, and parenting in the natural environment;
Parents can see that they have power, that children have positives, and that children will change;
Parents like the program;
Families are empowered, and gain improved communication, relationship, and anger management skills;
There are more teachable moments with in-home services; and
Workers can be real with families.

Relationship with DCS. Supervisors expressed concerns about the low proportion of DCS workers who refer to HomeTies, the high turnover of DCS staff (many new DCS workers don't know about HomeTies), and the poor training and supervision provided to DCS workers. They viewed the DCS workers as pleasant, but noted that they frequently need to educate them about HomeTies. Sometimes, though this happens infrequently, DCS staff expect HomeTies staff to act as investigators rather than therapists. There was some concern among supervisors and therapists that DCS workers are hard to reach by phone, but therapists said that communication with DCS occurs during services and is generally good, and that DCS really tries to be available for meetings.

(9) It is important to note that both DCS and HomeTies staff had been concerned that many DCS workers didn't refer to HomeTies. One person estimated that 50% of DCS workers did not make any referrals to HomeTies prior to the study, suggesting a large degree of indifference to or ignorance of the program among some workers. Based on our interviews, antipathy toward the program appears to be an unlikely explanation for non-referral for most workers.

(10) One worker was no longer using HomeTies because of these issues, while other staff appeared to be merely pointing out the program's shortcomings and will continue to use the program.

(11) The Shelby County director explained that while rates for HomeTies had not increased since 1992, the costs of providing services have increased substantially. Consequently, he was only able to support 18 workers. Because of lower than average caseloads, he has been forced to keep the number of staff below 18.

5.4 Implementing the Evaluation

Having a well-established statewide program, Tennessee was one of the original sites considered for study participation. This site was selected as it met all study selection criteria – a well-defined, mature program using a relatively pure version of the Homebuilders model.

Site visits were conducted and state office administrators were very interested in participating in the study, with trepidation about a randomized experiment, the impending reorganization of state services for children, and whether or not the state would be in compliance with the “reasonable effort” requirement of Title IVB. Additional meetings were held with state and local personnel to address concerns and to explain the dimensions of the study.

Usual referral procedures in Tennessee included referring workers learning of an opening in family preservation or waiting to refer a case until an opening existed. If a worker learned a program was full, he or she might ask when an opening was expected, leading to cases being held until an opening occured. To address concerns about random assignment, it was suggested that since not all cases could be served and since it was largely a chance matter whether or not a case received services, random assignment might be just as ethical as the current procedure.

Random assignment was eventually agreed to, but not without major objections. One concern was whether or not the state was in compliance with “reasonable efforts” requirements to provide services necessary to prevent foster care placement. It was believed by agency staff that family preservation was the best way to prevent foster care placement. After conversations with the federal government, it was determined that random assignment did not prohibit efforts to keep children out of foster care and the state would not be out of compliance with “reasonable effort” requirements.

State and local personnel indicated that targeting was a concern, families currently being referred for family preservation were not necessarily those at imminent risk of placement and that there were many eligible families not being referred for services. To address these concerns, training was conducted to help tighten the screening and referral of families to family preservation. The state family preservation coordinator developed training materials to review appropriate cases for referrals to the HomeTies program. Prior to the study beginning a one day training was held with the entire CPS and HomeTies staff in Shelby County. Study procedures were presented at the same training. There were plans to have training “tune-ups” throughout the study but these did not occur.

5.4.1 Study Procedures

Preparation and training for the experiment were conducted in the summer of 1996. Training sessions were held with both DCS screeners and family preservation program coordinators. During the training sessions study procedures were reviewed including use of study forms, the screening protocol, random assignment procedures, and the role of the study site coordinator. A meeting was also held with all HomeTies staff and DCS staff. The purpose of the meeting was to train staff on Shelby procedures and targeting appropriate cases for family preservation. In addition, periodic group meetings were conducted with caseworkers and supervisors to reinforce study procedures and solicit their feedback on the study. Workers were very resistant to random assignment and concerned that it would deny services to families, cause extra tasks, and delay referrals for their caseloads.

A site coordinator was hired locally to assist DCS and HomeTies staff in providing case information needed for random assignment and the conduct of interviews.

Referral to Family Preservation. Prior to implementing study procedures, workers identified families they felt were appropriate to receive family preservation services, got supervisor approval for the referral, and then made the referral directly to the HomeTies program. The HomeTies worker would determine whether or not the referral was appropriate and if there were any openings. The evaluation slightly altered these procedures. The major change was that workers no longer directly referred cases to HomeTies. Instead, if a worker saw the need for in-home services, he or she asked the family to sign a release form to participate in the study. If the worker selected HomeTies, the worker then completed a random assignment form and a DCS screener checked that the primary child was under 13, not a juvenile court case, and not already in foster care. Screeners assessed whether the child was at imminent risk; it was up to the worker's supervisor to assess level of need.

The screener's role was to call HomeTies to see if there was an opening and if there was, contact Westat to randomly assign the case to the experimental or control group.

Cases eligible for the evaluation were limited to those served by child welfare, even though HomeTies also served cases referred from juvenile justice and mental health. Also, referrals were limited to those cases that were in the course of an investigation of abuse or neglect or shortly thereafter, and had at least one child under the age of 13 years old. A 60/40 (60% treatment) random assignment ratio was used at the beginning. This was changed to 70/30 when the evaluation was under way.

Impact of the study on DCS workers. Both CPS workers and supervisors expressed frustration about the impact of the study. Supervisors reported that there was no change in the characteristics of families referred to HomeTies after the study began. The most commonly cited problems resulting from the study included:

“Emergencies happen around the clock”; front line staff could no longer make referrals directly to HomeTies after 4:30PM weekdays or during the weekends due to the office hours of DCS screeners. (Frayser staff estimate this accounted for 5-10 percent of their referrals.)
With caseloads higher than usual due to the hiring freeze, workers did not have time “to scramble” to identify other services when the screener told them their cases were in the control group.
Additional time related to the random assignment process. For families without telephones, workers had to make one trip to investigate and get consent to participate in HomeTies and then a second trip to tell families about the services they were to receive. To compound this, workers were still required to complete all of the paper work associated with a control group case.
A worker could no longer choose a particular HomeTies worker to help with a case. This was troubling to some CPS workers who had built relationships and felt particularly confident in the abilities of certain workers.
The “transition” was already requiring workers to review all of their open cases and close as many as possible. Workers and supervisors reported that they already had a lot of paperwork to complete and that the evaluation contact sheets were a very low priority.
Workers and supervisors also reported getting calls from Westat about cases two months after they had closed the cases. By this time, workers did not remember all of the specifics and did not have time to go back through their records. If a worker left or was transferred, the supervisor had to answer the questions and it was even more difficult for them to know the specifics of a case.

In response to these issues, many workers reported that they were referring their families to other services instead of HomeTies to avoid “the hassle” of possible control group selection. They identified nine such programs. Some workers even asked staff within the Juvenile Court system to make referrals directly to HomeTies to ensure that cases got HomeTies services. Screeners estimated that only 20 percent of front line staff made referrals to HomeTies during the evaluation, whereas in the past, closer to 50 percent made referrals.

In response to the staff's “rebellion” against the study, administrators and CPS supervisors actively encouraged front line staff to use HomeTies. Due to the strain of the uncertainty of random assignment, supervisors reported that front line staff perceived the evaluation to be more cumbersome than it really was. The screeners reported that the local Westat site coordinator/data collector was “very motivating” to staff. Using information from case records, the data collector filled in gaps in the initial referral forms and completed contact forms for workers. Workers also had the option of filling out the contact forms instead of completing the case narrative in the case record.

Reduced referrals and financial issues. At the end of April, 1997, HomeTies was down 56 cases in comparison to budget projections (at approximately $2,500 per case, this is over $125,000), the program's worst financial year to date. Referrals were low before random assignment, and were reduced further after random assignment. The HomeTies program director was working with DCS to increase referrals. There was considerable frustration and hostility among some of the staff regarding random assignment and the reduction in referrals. While acknowledging that random assignment was not the only problem, one supervisor believed that promises had not been kept, stating that Westat had said that referrals would increase and the state had said that they would not allow HomeTies to suffer financially--neither of these things was happening. This person noted that people's livelihoods were in jeopardy and this had a big impact on worker's attitudes and on data collection.

During study interviews with staff, the program director of HomeTies said that low referrals were having negative financial implications on the program. The state was considering reimbursing the Frayser Family Counseling at a higher rate for HomeTies referrals for the rest of the year to make up for the shortfall. (Because of the lower number of staff, it was not clear the extent to which Frayser Family Counseling was actually losing money.) The program director stressed that the agency was not accepting different cases just to meet the budget, that is, clinical decisions were not to be affected by the present shortfall.

With regard to the assumption made prior to the decision to use random assignment in Tennessee-that more families needed services than were actually referred, one of the supervisors noted that the number of families in need of HomeTies had little to do with referrals to HomeTies. He noted that relatively few DCS workers actually referred to HomeTies and that there never had been enough referrals--it was very rare that HomeTies was not able to see a family within seven days.

To allay some of staff concerns, the random assignment was changed to 70 percent treatment and 30 percent control.

Other issues related to the research and its effects on practitioners. HomeTies supervisors identified a number of other concerns related to the research:

Supervisors had to do much more hand holding with staff in supervision because of the stress of fewer referrals/lowered caseloads and the increased paperwork related to the research cases.
Workers said that Westat forms asked them to be judgmental and blaming toward families.
The yes-no questions were often impossible to answer, and didn't fit complex situations.
Westat interviewers were often not available.
Therapists didn't like doing the contact form for each visit and thought a weekly form could be used.

Therapists noted that clients said that they liked the gift certificate from McDonalds that they get for participating in interviews and that the Westat interviewers were nice. Only one family had an issue with the consent form.

5.4.2 Other Evaluations

Two studies are useful for understanding the development and implementation of the HomeTies Program: a study by the Tennessee Department of Finance and Administration, Division of Budget, of children in state care in Tennessee in 1989; and the evaluation of the HomeTies program by the University of Tennessee Social Work Office of Research and Public Service (SWORPS).

In 1989, during the pilot phase of the HomeTies program, the State of Tennessee conducted the Assessment of Children and Youth Committed to State Care. This report was compiled with the hope of locating inefficiencies in the current placement, tracking, and management process in child welfare. It explored the kinds of children committed to state care and the types of placements and services provided and needed by those children. Teams of professionals reviewed a random sample of 247 children (out of a total of 3,018 children) who were in various types of substitute care through the Department of Human Services as of May 1, 1989. Based on reviewers' judgments, the researchers found that 59 percent of children committed to the state's care were appropriately placed, 31 percent needed less intensive placement (including the option of not being in substitute care), and 10 percent needed more intensive placement. The study also found that too many children were placed in foster care. These and other findings from the study were used extensively in discussions with legislators to support the need to expand the HomeTies program; resulting ultimately in the dramatic increases in the program in the early 1990s.

One important limitation of this study was that only those cases in which children were already in substitute care were examined. This sample of cases skews the findings in the direction of concluding that more children need less intensive placements by: a) not examining non-placement cases, some proportion of which would likely to have been rated as needing more intensive services, including placement; and b) selecting cases only at the high end of the continuum of case severity, setting a ceiling for many of the cases on the possibility of recommending more intensive placements.

The University of Tennessee's statewide evaluation of the HomeTies program “was designed in response to both a legislative mandate and an interest in generating management information for ongoing program planning” (Homer, Cunningham, Bass, Collette, and Evans, 5/15/96). This research provides helpful descriptive information about referral sources, characteristics and problems of the population served, presenting problems in the family, prior placements, length of service termination status, and trends over time in these areas between FY93 and FY95. Some key information and findings are described below.

Demographic characteristics of children. Table 5-7 shows the age, race, and gender of children targeted as being at risk of placement for the state. For children at risk in FY95, 27 percent were under 10 years of age while 60 percent were teenagers (aged 13-18). There was no substantial change in the age of children at risk between FY93 and FY95. A large majority of the children served in The HomeTies program were white (67%), with African American children comprising 31 percent in FY95. This represented a slight increase in African American children, from 27 percent in FY93. The percentage of males grew from 52 percent in FY93 to 55 percent in FY95.

The relatively small proportion of cases referred for child maltreatment shows that, although CPS cases became eligible in 1991, the HomeTies program continued to serve a large majority of families with older children and families that were not referred because of child abuse or neglect.

Table 5-7.
Demographic Characteristics of Children at Risk Presenting Problems of Children and Parents Demographic Characteristics of Parents and Families at Time of Referral
	Percent of All Families or Children (FY95 Prevention Cases) N = 2,777 families N= 3,591 children
Age of child^a
under 10 10-12 13-15 16-18	27 14 39 21
Race of child
African American White Other	31 67 2
Gender of child
Female Male	45 55
Child behavioral difficulties
Child behavior problems School problems of child Running away—child Juvenile delinquency	85 64 29 23
Maltreatment-child problems
Physical child abuse Neglect Sexual abuse	11 9 9
Maltreatment-parent problems
Physical child abuse Neglect Sexual abuse	11 12 2
Parent problems
Criminal/police involvement Physical violence Alcohol/drug abuse Mental illness Parenting problems	5 17 17 13 91
Poverty related parental needs
Concrete service needs Home management needs Severe financial hardship	21 27 16
Prior out of home placement of children at risk at the time of referral	28
Age of mother figures (percentage of the 93.5% of families in which mother figures were reported as present and data on age were provided)
19 or younger 20-29 30-39 40-49 50-59 60-69	1 12 52 25 7 3
Marital status (percent of families in which mother or father were present and data were provided)^a
mothers who are single fathers who are single mothers separated/divorced fathers separated/divorced mothers who are married fathers who are married mothers widowed fathers widowed mothers cohabitating fathers cohabitating	15 3 30 11 43 73 4 1 7 11
Family composition (percent of families in which mother or father were present and data were provided)
Birth or adoptive mother only Birth or adoptive parents Birth mother/stepfather or adoptive father Birth mother and other adults Birth father and stepmother or adoptive mother Birth or adoptive father only Other	32 16 13 15 4 4 16
Employment status (percent of non-missing data where mother or father figures were present)^a
mother employed full time father employed full time mother employed part time father employed part time mother homemaker father homemaker mother unemployed father unemployed mother disabled father disabled mother student/working father student/not working	44 72 9 5 12 <1 26 11 7 11 1 <1
Gross Family Income (percent of non-missing data)
Less than $5,000 $5,000-9,999 $10,000-14,999 $15,000-19,999 $20,000-24,999 $25,000-29,999 $30,000-34,999 $35,000 and over	14 23 22 14 9 6 4 8

a. Percentages that should add up to 100 but do not because of rounding errors.

(Note: missing data make up no more than 4.3 percent of the total of children or families for the characteristics listed here)

Presenting problems of parents and children. The most common presenting problems of families entering the placement prevention program in FY95 were parenting issues (91% of parents), child behavior problems (85% of children at risk), family conflict (78% of parents and of children at risk), and school problems (64% of the children at risk). Running away (29%) and juvenile delinquency (23%) were other frequent problems associated with children. These items are also indicative of the types of problems of families with older children and adolescents.

Home management needs (27% of parents), concrete service needs (21%), child and parental violence (19% and 17%), parental and child alcohol/drug abuse (17% for each), and severe financial hardship (16%) were also common problems of families. Mental illness of parents was listed as a presenting problem in 13 percent of families. The three types of maltreatment — physical abuse, neglect, and sexual abuse — were each listed as presenting problems in less than twelve percent of children at risk and in a separate listing of the problems of parents. There were few changes in presenting problems or demographic characteristics over time, although severe financial hardship declined by 6 percent from FY93 to FY95 — paralleling a 7 percent decline (from 20% to 13%) in families with gross family incomes of less than $5,000 and a 4 percent decline in families with concrete service needs.

Prior out-of-home placements. For children at risk at the time of referral to the placement prevention program, 28 percent had experienced at least one prior out-of-home placement. The mean number of prior placements was 1.6 for this population. Emergency/runaway shelters (43% of all prior placements) and juvenile court (37%) placements were the most common types of prior placements — no other placement types constituted over 10 percent. It is not clear how many children were in placement at the time of referral. Given the types of prior placements experienced by children, it is possible that many children were in short-term placements immediately prior to referral.

Demographic information about parents and families. Consistent with the paucity of infants served, only 13 percent of mother figures whose age was known were younger than thirty. Fifteen percent of the mothers being served by HomeTies were single, 30 percent were separated or divorced, and 43 percent were married. Only 3 percent of fathers being served were single, 11 percent were separated or divorced, and 73 percent were married. With regard to family composition, single parent families headed by birth or adoptive mothers (with no other adults) were the most common type of family — 32 percent of all families; followed by birth or adoptive parents (16%), birth mother and other adults (15%), and birth mother with stepfather or adoptive father (13%).

Forty-four percent of mothers served were employed full time, compared with 72 percent of fathers. Twenty-six percent of mothers were unemployed, compared with 11 percent of fathers. Seventy-three percent of families had gross incomes of less than $20,000 in FY95, with 37 percent of families earning less than $10,000, and 14 percent earning less than $5,000.

Findings: Out-of-home placement. The Homer et al report examined placement status of children at termination of HomeTies and six and twelve months later. “Placement data were obtained from the Client Operation and Review System database (CORS) by matching the information about children to HomeTies information” (Homer et al., 1995, p. 79). Two limitations of the data should be noted: only first placements were counted and data on the type of placement are available only for placements at termination of services. Data on identifying information (3.0%) or placement (.6%) were missing on 3.6 percent of cases. For children who received placement prevention services in FY95:

85.0 percent had no out-of-home placements for one year, conversely 15 percent (n = 523) of the children were placed;
5.3 percent were placed at termination of services; of these 186 children, most were placed in psychiatric hospitals (28.5% of the 186 children), foster homes (23.1%), or correctional institutions (14.0%);
In addition to the 5.3 percent of children placed at termination, another 5.2 percent were living with friends or relatives and .9 percent were classified as runaways;
8.1 percent were placed between termination and six months after termination; and
1.6 percent were placed between six and 12 months after termination.

The figure of 15 percent of children placed within one year in FY95 is substantially lower than FY94 (20.4% of children placed within a year) and FY93 (24.7% of children placed within a year). Thus, there was s 40 percent decrease in the one year placement rate from FY93 to FY95. It is not clear whether differences are due to larger numbers of records missing in previous years (704 in FY93, and 216 in FY94), a trend toward less risky referrals, or improved program targeting and outcomes.

Cost analysis. The University of Tennessee report initially recognized the limitations of studying outcomes without a comparison group. Despite this, a detailed analysis of costs concluded that over $74 million was saved by the HomeTies placement prevention program as a result of preventing various types of placements. Like other optimistic estimates of cost savings, this estimate incorrectly assumes that all children at risk would have been placed in the absence of the program.

Exhibit A. Other services available to referring workers in Shelby County

In addition to HomeTies, Shelby County has a large number of both in-home and office-based programs that provide counseling and some that provide concrete services. Most of these services are free to families, and few have waiting lists. They are either DCS funded programs or community based programs funded through other agencies, such as the schools. Some require TennCare (Tennessee Medicaid) eligibility, some require private insurance. The programs that front line CPS workers are using in place of, or in addition to HomeTies are:

Life Coach, also provided by Frayser Family Counseling, is an in-home, case management program with case staffing comprised of one therapist and one supervisor. Service intensity varies depending on family needs. At minimum, there are three face-to-face visits a week. Therapists provide counseling, teaching, and concrete services including transportation. Services are funded through TennCare, the Community Mental Health Agency, (formerly ACCT), and DCS, and self pay on a fee-for-service basis. (While Life Coach is viewed as one of the most viable options for some control group cases, workers note that making a referral to Life Coach also requires considerable paperwork and involves uncertainty about whether cases will be accepted into the program.)
Homeworks, also provided by Frayser Family Counseling, is an in-home, case management program staffed by one therapist and one supervisor. Less intensive, Homeworks therapists provide counseling and teaching services on average once a week. Services are funded through TennCare, the Community Mental Health Agency (formerly ACCT), DCS, and self pay on a fee-for service basis.
Frayser Family Counseling Outpatient Therapy is weekly individual, couple, or family therapy. Services are funded through TennCare, Social Service Block Grant, and private insurance.
The Exchange Club provides office based counseling in parenting and anger management. It is funded by DCS or TennCare.
Intercept, like HomeTies, provides intensive, home-based services (24 hour availability, four to six cases a worker, service duration of three to six months, minimum of three visits a week) and is offered by Youth Village. The program requires an eight page application form. Intercept is “very expensive” and requires ACCT approval and funding but the program also accepts TennCare and some private insurance. For nearly half of the workers we interviewed, Intercept has replaced HomeTies as the intensive model of choice since the study began. This is partly because in the words of both front line staff and supervisors, Intercept has been coming to DCS to “drum up business.”
Memphis City School Mental Health offers services for children who have been sexually abused, including child-on-child sexual abuse. While the services are free to families, there is generally a waiting list. Services funded by DCS Social Services Block Grant.
Child Advocacy Center offers counseling to children and their parents. Services are funded by DCS, private funds, federal grants, and the city and county government.
The Center for Children in Crises provides comprehensive evaluation (social, medical, psychological, and psychiatric) of all family members in abuse/neglect cases and makes placement recommendations. Services are funded by DCS and TennCare.
Homemaking Services in-home services provided by the Memphis City Schools. Services are funded by DCS.
Lakeside Hospitals, in-home service using BSI-trained staff.
Monitor Prime, in-home services, largely case management, that are sometimes tried before HomeTies.

According to the DCS front line staff interviewed, approximately 50 to 60 percent of substantiated CPS cases are encouraged to accept some services. Jean Taylor, the CPS Program Supervisor, estimated that for control cases, over 50 percent currently go to Community Service Agencies to access services not otherwise funded by DCS. For families in treatment, most of the requests for flexible funds are to support concrete needs like home repair or specialized psychiatric services not otherwise covered by TennCare.

Chapter 6: The Families

6.1 State Policies on Referral

Before describing the family characteristics, services provided, and outcomes of the study, we review the state policies and practices that guided the types of families referred for family preservation services.

In all three study states there were policies specifying the types of families eligible for family preservation services. These criteria emphasized the imminent risk of placing children in foster care if the services were not provided. All three states used the Homebuilders family preservation model and reported they followed the guidelines set forth by the Behavioral Sciences Institute (BSI), where Homebuilders began.¹ According to BSI, the family characteristics that are key to an appropriate referral are:

Child is at imminent risk of placement. Placement has already been initiated or will be initiated at once without family preservation services;
The family is in severe crisis;
One parent is willing to meet with the family preservation worker at least once;
There are some family strengths, resources, or social supports available that can be utilized to increase safety;
There are no options for long-term placement with relatives;
The family has been told that placement is imminent; and
Other services have been tried and failed, or other less intense services would not be sufficient to resolve the problems that will cause placement.

Kentucky law defines FPS as “a short-term intensive, crisis-intervention resource intended to prevent the unnecessary placement of children at imminent risk of placement.” Kentucky policy specifies that imminent risk includes children who are at risk of commitment as dependent, abused, or neglected; who are identified through the Regional Interagency Council as severely emotionally disturbed; or whose families are in conflict such that they are unable to exercise reasonable control of the child.

In New Jersey, family preservation is considered to be a “gatekeeper” to prevent out-of-home care for a child. According to state legislation, family preservation services are targeted at families with substantiated abuse or neglect, with the children at risk of harm from maltreatment. The state defines three levels of eligibility for targeting purposes. Only the first level applies to families with children at risk of foster care placement. It includes families with at least one child at imminent risk of placement, unless changes in family coping or behavior patterns are made, placement will occur. Cases in which there is one child in temporary placement less than thirty days are also eligible. The referring worker must base the assessment of imminent risk on a face-to-face interview with the family no more than 5 days prior to the referral.²Although the term “imminent risk” is used as the litmus test for referring families, definitions of this term are left to the counties and ultimately the individual caseworker and his or her supervisor.

In Tennessee, criteria outlined in policy are also based on the criteria established by the BSI. CPS intake workers complete a risk assessment form to identify high, intermediate, low, or no risk situations. High risk cases are identified as cases where “the child or children in the home are at imminent risk of serious harm if there is no intervention in the situation.” A typical high risk case might involve such factors as: (1) a vulnerable child; (2) a history of previous maltreatment; (3) a perpetrator who has continued access to the child; and (4) no available support or family strengths to offset the risks.

In Kentucky and New Jersey workers were being encouraged to focus family preservation referrals on younger children. Although not a written policy, managers were emphatic that families with younger children should be a priority for family preservation referral. Conversations with workers revealed that this was not necessarily being adhered to. In addition, when workers were queried about the types of families they actually referred to family preservation their responses varied.

Divergence of Practice from Policy. As expected, policy and practice were not always synchronized. In New Jersey, county practices on referral varied. Workers interviewed in the seven study counties presented several alternatives. In most counties the workers indicated they mainly referred ongoing cases, cases in which they had worked with families for an extended period of time. Workers had to demonstrate that they offered many alternative services and workers said that they used family preservation because it was the only service option left to offer a long-term case. Workers also indicated that they considered family preservation services most appropriate for family problem cases, rather than child abuse and neglect cases, especially those with adolescent issues.

In Kentucky, criteria outlined in policy mirror the criteria established by BSI. However, in practice workers said they referred cases that they felt really needed services, and were not necessarily facing imminent placement. Workers who referred cases from ongoing units as opposed to intake and investigation units said that ongoing referrals did not involve a specific incident of maltreatment. Instead, referrals of ongoing cases were more likely to involve chronic problems that were getting worse. When asked specifically about the types of cases referred for family preservation services, some workers identified:

Low functioning parents with no parenting skills;
Young mothers who are overwhelmed and need help getting supportive services;
Dirty house cases, something very concrete that family preservation services could work on and see improvement in;
Domestic violence cases; and
Psychiatric cases where a parent might be schizophrenic and would not take medication.

When queried, supervisors stressed that referrals are made based on families in crisis who have an immediate need because of risk of placement.

Investigative workers in Tennessee reported that Home Ties was used as their first resort for families at imminent risk of removal because program staff could be in the home monitoring and assessing families. Ongoing workers reported that they used Home Ties as a last resort, after they had tried less intensive services because of the intensity of the intervention and the availability of concrete resources (flexible funding, transportation) that could be used. Department of Children’s Services workers also said that the best candidates for Home Ties were families who needed assistance with communication skills and anger management

Both Kentucky and New Jersey policies excluded families in which there was a substance abuse problem and a current plan for treatment was not being pursued. Kentucky excluded families in which there was sexual abuse and the perpetrator was still in the home.

We turn now to a description of the families in the evaluation. Descriptive information about the families was gathered from the initial interviews with caretakers. Those interviews included information on the families involvement with social programs prior to referral to family preservation. Questions on family problems and social program participation were also asked in the post-treatment and follow-up interviews. Data from those interviews are presented in Chapter 8. In addition, administrative data were used to describe prior involvement of families with the child welfare system. Because families were randomly assigned, we would expect the families in the experimental and control groups to be similar at the time of random assignment, and for that reason, the sample is described as a whole. However, by chance it is expected that the groups would differ in statistically significant ways on a few variables. We identify below those characteristics on which the groups differed significantly.

(1) Behavioral Sciences Institute, "Key elements of an appropriate referral," Behavioral Sciences Institute, Federal Way, Washington, 1992.

(2) As discussed in Chapter 4 the remaining two levels of eligibility focus on reunifying children with their families after they have already been in placement less than 90 days or are about to return home within the next two weeks.

6.2 The Kentucky Families

Table 6-1 summarizes certain characteristics of 311 Kentucky caretakers and families for which we have initial caretaker interviews (89% of the 349 net study cases). The respondents were primarily women (93%). Most (85%) of the respondents were birth mothers, 7 percent were biological fathers, 6 percent grandmothers, and the rest were other relatives, including one adoptive mother (for 6% the relationship to the child was not ascertained). The racial composition of the respondent group was mostly white (55%) and African American (not Hispanic) (43%), along with 1 percent Hispanic and 1 percent other. The average age of the respondents was 32 (n = 306, s.d. = 9.49). Nine percent of the respondents had less than a high school level education, 44 percent had some high school, 32 percent had graduated from high school or obtained a GED, 14 percent had at least some college education, and 1 percent had special education or vocational schooling. Approximately 24 percent of the respondents indicated they were married, 19 percent divorced, 21 percent separated, 3 percent widowed, and 33 percent never married.³ Thirty-five percent reported that they were living with a spouse or partner. At the time of the first interview, 38 percent of the respondents indicated they were employed, 29 percent were unemployed and looking for work, and 33 percent were unemployed and not looking for work.⁴ Overall, 83 percent of the respondents rented their homes. Respondents in the experimental group were more likely to rent their homes than those in the control group (89% vs. 77%, p = .005). Provided with a list of income categories, respondents were asked to approximate their household incomes. Of the 300 respondents who answered the question, 15 percent reported an income less than $5,000, 23 percent between $5,000 and $10,000, 43 percent between $10,000 and $20,000, 16 percent between $20,000 and $40,000, and 3 percent reported an income of $40,000 or more. There were no significant differences between experimental and control group respondents in reported household income.

Table 6-1.
Description of the Kentucky families at time of initial interviews
	N	%
Gender of Caretaker/Respondent	311
Male		6.8
Female		93.2
Race of Caretaker/Respondent	310
African American (not Hispanic)		43
Caucasian (not Hispanic)		55
Hispanic		1
Other		1
Respondent's education level	311
Elementary school or less		9
Some high school		44
High school graduate or obtained GED		32
College		14
Special education or vocational schooling		1
Respondent's marital status	310
Married		24
Divorced		19
Separated		21
Widowed		3
Never Married		33
Respondent's Relationship to youngest child	292
Birth mother		85
Biological father		6.5
Grandmother		5.8
Other relative		2.4
Household composition	311
Birth mother, no other adults		43
Birth mother& 1 male adult		24
Birth mother& extended family^*		9.3
Biological father^*		6.1
Other relative caretaker^*		7.4
Other^**		10
	N	Mean
Age of respondent	306	32.2
Age of youngest child	311	4.6
Age of oldest child	311	9.9
Number of Children	311	3.0
Number of adults	311	1.6

^* These categories may also include other non-related adults in the home

^**Includes: nonrelative caretaker, adoptive or step-parent, birth mother& non-related females, or birth mother, and more than one non-related male

On average, these families were comprised of 1.6 adults and 3 children for an overall average family size of 4.6 persons. The average age of the youngest child in the family was 4.6 years (n = 311, s.d. = 4.35), and the average age of the oldest child in the family was 9.9 years (n = 311, s.d. = 5.00). The distribution of the age of the youngest child was 19 percent under 1 year, 42 percent between 1 and 4, 33 percent between 5 and 12, and 6 percent 13 and over. The distribution of the age of the oldest child was 3 percent under 1 year, 16 percent 1 to 4, 42 percent between 5 and 12, and 39 percent 13 and over.

While there were no significant differences between families in the experimental and control groups with regard to total number of persons, number of children in the home, or ages of youngest and oldest child in the home,⁵ there was a statistically significant difference in the number of adults in the home. The control group averaged 1.7 adults per household (n = 155) whereas the experimental group averaged 1.5 adults per household (n = 156; p = .012). Respondents were also asked to provide information regarding the relationship of other adults in the home relative to the youngest child in the home. This information was then used to determine household composition for these families. Forty-three percent of households were headed by a single birth mother, 24 percent had a birth mother residing with one male adult, 9 percent had a birth mother and extended family, 6 percent were headed by a biological father, and 17 percent were headed by another relative caretaker.

Family Problems. We can get some sense of the difficulties families faced from the first interviews with caretakers, in which we asked whether they had experienced certain problems in the last month (see Table 6-2). In Kentucky there were few significant differences on these items between the experimental and control groups at the initial interview. With regard to emotional problems, 55 percent of the respondents reported feeling "blue or depressed,"56 percent reported feeling nervous or tense, 47 percent were overwhelmed by work or family responsibilities, 31 percent said they had just wanted to give up at some point in the last month, and 30 percent felt they had few or no friends. With regard to financial difficulties, 49 percent responded that in the past month they did not feel they had enough money for food, rent, or clothing. In response to more specific questions about difficulties paying bills in the past 3 months, 24 percent reported difficulty paying rent, 32 percent reported difficulty paying electric or heating bills, 23 percent difficulty buying food for the family, and 31 percent difficulty buying clothes for their children.⁶

Table 6-2.
Caretaker problems and strengths, caretaker initial interview, Kentucky
Problems	Percent responding yes
Felt blue or depressed	55
Felt nervous or tense	56
Just wanted to give up	31
Overwhelmed with work or family responsibility	47
Felt you had few or no friends	30
Not enough money for food, rent, or clothing	49
Gotten in trouble with the law	7
Had too much to drink in a week	3
Used drugs several times a week	1
Economic Items
Had difficulty paying rent	24
Had difficulty paying electric/heat	32
Had difficulty buying enough food	23
Had difficulty buying clothes	31
Positive Items
Have you felt happy	82
Gotten together with anyone to have fun/relax	53
Doing a pretty good job raising kids	90

Three percent of respondents acknowledged having too much to drink several times a week, and 1 percent reported using drugs several times a week. Seven percent of respondents indicated they had gotten in trouble with the law in the past month.⁷

Most (90%) respondents felt they were "doing a pretty good job raising [their] kids"(94% of the experimental group, compared to 86 percent of the control group, a difference significant at p = .02).

Table 6-3 shows problems of children identified by caretakers. Over four-fifths of caretakers said at least one child in the family threw tantrums and about the same proportion said a child "didn't show much interest in what is going on." Over two-thirds said a child "gets upset easily." Items identifying difficulties in school were endorsed by a quarter to a third of respondents (frequent absences, suspension, failed classes). Aggressive behavior was a fairly common problem, a third of the caretakers said a child fights a lot with other kids and 43 percent said a child was very aggressive toward them.

Caretaker Abuse or Neglect as a Child. When asked two separate questions about whether they had been abused or neglected as a child, 31 percent of the 311 initial interview respondents reported having been abused and 20 percent neglected. Sixteen percent responded affirmatively to both questions, and overall, 35 percent of the caretakers reported having either been abused, neglected, or both as a child. Eighteen percent of caretakers had been in a foster home or institution. Experimental and control groups did not differ significantly with respect to these previous experiences.

Previous Allegations and Placement. Historical reports of maltreatment and of placement in substitute care were available from the administrative data files. Two hundred and ninety-five (96%) of the Kentucky families had been investigated for maltreatment prior to random assignment. Two hundred and thirty-six (77%) of the families had experienced at least one substantiated⁸ allegation prior to random assignment. The administrative files reported five types of allegations; dependency, emotional, neglect, physical abuse, and sexual maltreatment. The allegation just prior to random assignment was of primary interest. This particular allegation provides some indication of reason for referral to family preservation. The distribution of last allegation prior to random assignment is: 34 percent dependency, 5 percent emotional, 32 percent neglect, 44 percent physical abuse, and 24 percent sexual maltreatment. The distribution of last substantiated allegation prior to random assignment is as follows: 34 percent dependency, 3 percent emotional, 34 percent neglect, 41 percent physical, and 19 percent sexual maltreatment. As individual families can have multiple allegations on any given day, percentages add to more than 100 percent. In 68 cases (29% of the 236), the only substantiated allegation just prior to random assignment was dependency. Hence, there were a substantial number of cases referred for family preservation services in which it appears that abuse or neglect were not major issues.

Table 6-3.
Concerns and problems regarding children,
Caretaker Initial Interview, Kentucky
(% responding yes regarding any child that the respondent cares for)
	Kentucky
Item	N	%
Asked about all children...
Child went through alcohol withdrawal at birth	309	2
Child went through drug withdrawal when born	309	2
Child doesn't show much interest in what is going on	308	84
Child is Smaller/Lighter than other children	308	29
Child Get(s) upset easily	303	69
Asked for children over 3 months old...
Is/Are Funny and makes you laugh	303	95
Like(s) to share things with others	296	70
Throw(s) tantrums	302	83
Is/Are shy and withdrawn	302	24
Is/Are outgoing and friendly	298	85
Is/Are good looking	297	99
Fight(s) a lot with other kids	289	33
Has/Have language problems	286	30
Asked for children over 4 years old...
Is/Are very aggressive toward you	247	43
Has/Have a special talent in music	232	32
Like(s) animals	248	95
Is/Are good at sports	204	51
Usually does the right thing	241	74
Hangs with friends you don't like	243	28
In the past 3 months has any child you care for...
Gone to church regularly	247	34
Been absent from school a lot	240	38
Run away from home overnight	240	10
Been temporarily suspended from school	240	30
Been expelled from school	239	11
Taken care of younger children	220	40
Took something that didn't belong	245	34
Absent from school/no good reason	238	30
Received special education at school	241	40
Failed any classes	237	27
Received counseling	245	61
Asked for any child over age 7...
In the last 3 months, has any child been arrested	197	13
Asked only for children over age 10...
Has child age 11 or older had alcohol problems	141	4
Has child age 11 or older had a drug problem	138	7
Has any girl age 12 to 18 been pregnant	82	12
Has any boy age 14 to 18 fathered a child	53	6

The above data describe the allegations that may be considered to be associated with the current involvement of the family with the child welfare system. The administrative data can also be used to explore the extent of prior involvement with the system. Of the 295 Kentucky families with at least one allegation prior to random assignment, 139 (47%) had a substantiated report of maltreatment prior to the allegation just before referral to family preservation.

Regarding substitute care placement, 124 children in 53 (17%) families had experienced placement prior to random assignment.⁹ The administrative files contained placement dates for 123 of these 124 children. On average, 20.2 months elapsed between the last day of care and random assignment. In the placement spell just prior to random assignment the average length of time in substitute care was 5.9 months.¹⁰

Length of Time from Case Opening to Referral to Family Preservation Services. The Kentucky administrative data also contained information about case opening and closing dates. In Kentucky, opening and closing data are recorded at the individual rather than family level, and the dates of opening and closing for various members of a family may differ. Our analyses, however, were conducted at the family level. We considered a case open from the date of the first open record for any person in the family to the time that the last record for any person in the family had been closed out. In other words, the opening and closing data described here refer to periods of time during which DSS was involved with at least one person in the family. It should also be noted that in Kentucky a family does not necessarily need an open record in order to receive services, as services or referrals for services may be provided by the investigating worker prior to opening the case. Presumably, such cases should be opened shortly after referrals for services. With this in mind, cases were examined for the date of case opening or the date of the last maltreatment report, both of which may indicate DSS involvement in that case.

Of the 307 cases for which administrative data were available, 183 (60%) were open at the time of the referral to family preservation services. An additional 89 cases were not open at the time of referral to FPS, but had had a prior maltreatment report (15 of these cases had been open previously). In 59 of the 272 cases open at the time of referral or with prior maltreatment reports, the most recent case opening or maltreatment report occurred over six months prior to referral, in 34 cases, over a year prior. Appendix G provides a more detailed breakdown of case openings and maltreatment reports as well as the timing of these events in relation to the referral to family preservation services.

Social Program Participation. In the initial interview, respondents were asked whether they or anyone else in the household had participated in various social programs within the past 3 months. The overall rates of participation by Kentucky families are provided in Table 6-4. Over two-thirds indicated that they received food stamps, just under half received AFDC, more than a third received WIC, about a third received social security disability,¹¹ and just under a fifth received a housing voucher. Overall, respondents indicated that they participated in an average of 2.1 of the 5 income support programs listed (s.d. = 1.36) and 82 percent of the sample participated in at least one of the five programs.¹² Differences in the rates of program participation were found for WIC and community mental health programs, with both programs showing higher rates of participation among the experimental group. Forty-eight percent of the experimental group reported WIC participation within the last 3 months compared to 34 percent of the control group (p = .01) and 16 percent of respondents in the experimental group reported participation in community mental health programs compared to 9 percent of respondents in the control group (p = .04). Reports of participation in alcoholism, drug treatment, marriage counseling, and job training programs were less than 10 percent for each. Slightly less than a third of the sample reported participation in Head Start or another pre-school program.

Table 6-4.
Participation in social programs prior to initial interview,
Kentucky
Program	Percent %
Food Stamps	67
Job Training	7
WIC	41
AFDC	48
Housing Vouchers	18
Social Security Disability	36
Alcoholism Program	6
Drug Treatment Program	1
Marriage Counseling	5
Community Mental Health program	12
Head Start/Pre-school	30

(3) When married, divorced, and separated categories are collapsed and compared to never married, a larger percentage of respondents in the experimental group were never married, 40% vs. 28%, p = .04 (8 widowed respondents and 1 not ascertained respondent are not included in these collapsed analyses).

(4) When the 2 unemployed categories are collapsed and compared to the employed category, a larger percentage of respondents in the control group were employed at the time of the first interview, 43% vs. 33%, p = .12.

(5) Though not a statistically significant difference, the average age of the oldest child was greater for control group families than for experimental group families, 10.31 years vs. 9.42 years, p = .13.

(6) More of the experimental group respondents indicated difficulty buying clothes for their children, 35% vs. 27%, p = .16.

(7) Experimental group caretakers were more likely to answer that a child or children they care for went through alcohol or drug withdrawal when born.

(8) The state of Kentucky reports five possible outcomes for reports of maltreatment; (1) substantiated, (2) found/substantiated, (3) some indication, (4) unsubstantiated and (5) unable to locate. Substantiated and found/substantiated were collapsed to form a "substantiated" category.

(9) Our analyses did not include children inplacement at the time of random assignment.

(10) Placement spells are defined as any consecutive period of time in substitute care and may consist of several distinct placements (i.e., several different foster homes).

(11) The question on the interview was worded in terms of "social security disability." We intended this to refer to Supplemental Security Income.

(12) The average number of income support programs used was slightly higher for the experimental group than for the control group, 2.21 vs. 1.98, p = .13.

6.3 The New Jersey Families

As in Kentucky, we describe the sample of New Jersey families based on information from our first interviews with caretakers (n = 328, 74% of the 442 net study cases). Again, we describe the sample as a whole, identifying the variables on which the experimental and control groups were statistically different.

Table 6-5 summarizes a number of characteristics of New Jersey caretakers and families. Most (88%) of the caretakers were women. The sample was about evenly divided between whites and African Americans. Forty-seven percent were white, 42 percent African American (not Hispanic), 9 percent Hispanic, and 2 percent other. On average there were 4.7 persons in these families, 1.8 adults and 2.9 children. The average age of the respondents was 39 (n = 324, s.d. = 10.8), the youngest child in the family was, on average, 7.1 years old (n = 328, s.d. = 5.4), and the oldest child in the family, 12.5 (n = 328, s.d. = 4.3). The distribution of the age of the youngest child was 15 percent under 1 year, 26 percent between 1 and 4, 37 percent between 5 and 12, and 23 percent 13 and over. The distribution of the age of the oldest child was 2 percent under 1 year, 5 percent 1 to 4, 30 percent between 5 and 12, and 63 percent 13 and over.

About 9 percent of the respondents had no high school education, 40 percent some high school, 26 percent high school graduation or a GED, 20 percent at least some college education, and 4 percent had special education or vocational schooling (0.9% were unknown). Thirty percent of the respondents were married, 34 percent divorced or separated, 6 percent widowed, and 30 percent never married. At the time of the first interview, 41 percent were employed, 18 percent reported that they were unemployed and looking for work, and 41 percent were unemployed and not looking for work. Two hundred ninety-one respondents provided information about their household incomes, with significant differences between the experimental and control groups (p = .03). Fewer control group cases were at the middle of the income spectrum.¹³

Most (71%) of the respondents were birth mothers, 10 percent were biological fathers, 11 percent grandmothers, and the rest were other relatives, including step-relatives. Four of the respondents were adoptive mothers and two were adoptive fathers. As to household composition at the time of the first interview, 34 percent of the families were headed by birth mothers with no other adults in the home, 27 percent had a birth mother and one male adult, 8 percent had the birth mother with other extended family, 9 percent were headed by a single father, and 17 percent had another relative caretaker (4% of the families did not fall into one of these categories). Forty-three percent reported that they were living with a spouse or partner. Seventy percent of the respondents rented their homes. On none of these characteristics did the experimental and control groups differ significantly at the time of the first interview.¹⁴

Table 6-5.
Description of the New Jersey families
at time of initial interviews
	N	%
Gender of Caretaker/Respondent	328
Male		12
Female		88
Race of Caretaker/Respondent	327
African American (not Hispanic)		42
Caucasian (not Hispanic)		47
Hispanic		9
Other		2
Respondent's education level	325
Elementary school or less		9.4
Some high school		40
High school graduate or obtained GED		26
College		20
Special education or vocational schooling		4.0
Respondent's marital status	328
Married		30
Divorced		23
Separated		11
Widowed		6
Never Married		30
Respondent's Relationship to youngest child	326
Birth mother		71
Biological father		9.5
Grandmother		11
Other relative		8.3
Household composition	328
Birth mother, no other adults		34
Birth mother& 1 male adult		27
Birth mother& extended family^*		8.2
Biological father^*		8.5
Other relative caretaker^*		17
Other^**		4.3
	N	Mean
Age of respondent	324	39.0
Age of youngest child	328	7.1
Age of oldest child	328	12.5
Number of Children	328	2.9
Number of adults	328	1.8

^*These categories may also include other non-related adults in the home

^**Includes: Nonrelative caretaker, adoptive or step-parent, birth mother& non-related females, or birth mother, and more than one non-related male

Family Problems. Problems identified by New Jersey caretakers are summarized in Table 6-6. Emotional and financial problems were most often cited. Fifty-eight percent of the respondents said they had felt "blue or depressed," 52 percent said they felt nervous or tense, 56 percent were overwhelmed by work or family responsibilities, 33 percent said they had just wanted to give up sometime in the last month, and 27 percent said they had few or no friends. Over half (52%) responded affirmatively to the general question as to whether they experienced not having enough money for food or rent, and on more specific questions about difficulties paying bills, 29 percent said they had difficulty paying rent, 37 percent difficulty paying electric or heat bills, 30 percent difficulty buying food (on this item there was a significant difference between the groups, 26% of the experimental group vs. 36% of the control group, p = .04), and 45 percent difficulty buying clothes for their children. Few respondents reported problems in drinking or using drugs (only 0.9% said they "had too much to drink in the last week" and 0.9% said they used drugs several times in a week). Only 3 percent said they had gotten into trouble with the law. Most (93%) thought they were "doing a pretty good job raising [their] kids."

Table 6-6.
Caretaker problems and strengths,
caretaker initial interview,
New Jersey
Problems	Percent responding yes
Felt blue or depressed	58
Felt nervous or tense	52
Just wanted to give up	33
Overwhelmed with work or family responsibility	56
Felt you had few or no friends	27
Not enough money for food, rent, or clothing	52
Gotten in trouble with the law	3
Had too much to drink in a week	1
Used drugs several times a week	1
Economic Items
Had difficulty paying rent	29
Had difficulty paying electric/heat	37
Had difficulty buying enough food	30
Had difficulty buying clothes	45
Positive Items
Have you felt happy	80
Gotten together with anyone to have fun/relax	46
Doing a pretty good job raising kids	93

Table 6-7 shows problems of children identified by caretakers. About four-fifths of caretakers said at least one child in the family threw tantrums and about three-fourths said a child "gets upset easily." School problems were common; over 40 percent had been absent a lot or failed classes and nearly a third had been suspended. Aggressive behavior was common, 40 percent of caretakers said a child fights a lot with other kids and 56 percent said a child was very aggressive toward them.

Caretaker Abuse or Neglect as a Child. Twenty-eight percent of New Jersey caretakers reported having been abused as a child and 25 percent reported having been neglected. Twenty-one percent answered "yes" to both questions, and overall, 32 percent of the caretakers reported having been abused, neglected, or both as a child. Fourteen percent of the respondents had been in a foster home or institution. There was little difference between the experimental and control groups in these previous experiences.

Table 6-7.
Concerns and problems regarding children,
Caretaker Initial Interview,
New Jersey
(% responding yes regarding any child that the respondent cares for)
	New Jersey
	N	%
Asked about all children...
Child went through alcohol withdrawal at birth	315	5
Child went through drug withdrawal when born	315	6
Child doesn't show much interest in what is going on	321	20
Child is Smaller/Lighter than other children	326	14
Child Get(s) upset easily	325	74
Asked for children over 3 months old...
Is/Are Funny and makes you laugh	325	90
Like(s) to share things with others	321	80
Throw(s) tantrums	324	79
Is/Are shy and withdrawn	325	33
Is/Are outgoing and friendly	324	92
Is/Are good looking	325	99
Fight(s) a lot with other kids	317	40
Has/Have language problems	314	26
Asked for children over 4 years old...
Is/Are very aggressive toward you	304	56
Has/Have a special talent in music	305	44
Like(s) animals	306	87
Is/Are good at sports	302	69
Usually does the right thing	304	65
Hangs with friends you don't like	303	49
In the past 3 months, has any child you care for...
Gone to church regularly	306	37
Been absent from school a lot	300	42
Run away from home overnight	304	26
Been temporarily suspended from school	303	32
Been expelled from school	303	9
Taken care of younger children	288	37
Took something that didn't belong	304	42
Absent from school/no good reason	301	27
Received special education at school	304	55
Failed any classes	294	41
Received counseling	304	66
Asked for any child over age 7...
In the last 3 months, has any child been arrested	283	16
Asked only for children over age 10...
Has child age 11 or older had alcohol problems	237	13
Has child age 11 or older had a drug problem	236	17
Has any girl age 12 to 18 been pregnant	160	4
Has any boy age 14 to 18 fathered a child	75	3

Previous Allegations and Placement. Of the 434 New Jersey families for which we had administrative data, 89 percent had an allegation of maltreatment prior to the date of referral to family preservation services. Sixty-four percent had a substantiated report of maltreatment prior the referral date.¹⁵

We have data for 369 cases on the type of allegation just before the last case opening before referral. Forty-two percent of the cases had allegations of physical abuse, 11 percent of lack of supervision, 20 percent of other neglect, 5 percent of sexual abuse, and 5 percent of emotional abuse (cases could fall in more than one of these categories). In 22 percent of the cases, there was no abuse or neglect found before the case opening.

Similar to the analysis of Kentucky data, we examined reports of maltreatment before the allegation prior to the referral to family preservation services, as an indication of prior involvement with the child welfare system. Of the 386 families with allegations prior to referral, 205 (53%) had a substantiated report of maltreatment before that, indicating that about half of the families had previous involvement with the system.

As to substitute care placement, 191 children in 94 families had previously experienced placement. Eighteen of these children were in 5 adoptive homes and the referral to family preservation services was for the purpose of preserving the adoptive home. For the remaining 173 children, the average length of time between the end of the previous placement and random assignment was 53.5 months. The average length of time in that placement spell was 12.9 months.¹⁶ Seventy percent of the first placements in the previous placement spell were foster family care, the remainder were residential treatment, shelter care, group homes, and institutions. There was a difference between the experimental groups in the previous placement experience of children, with control group children averaging 85 days and the experimental group children averaging 104 days (a nonsignificant difference).

Length of Time from Case Opening to Referral to Family Preservation. On 434 New Jersey cases for which we have administrative data, 13 cases were not open at the time of the referral to family preservation services. Two of these 13 cases were opened within 30 days after the referral, and two were opened within 2 to 6 months after the referral. The remaining 9 cases had not been opened as of the last date of observation for these analyses (August 31, 1998). In 34 percent of the 421 cases open at the time of random assignment, the referral to family preservation services occurred within a month after case opening while in another 33 percent it came between two and six months after case opening. In 21 percent of the cases the referral occurred more than a year after case opening. The administrative data also recorded reports of maltreatment prior to random assignment for 386 families. In 37 percent of these cases, the report occurred in the month prior to referral, in another 28 percent it came between two and six months prior. In 25 percent the report occurred more than a year before referral.

Social Program Participation. Table 6-8 shows the rates of participation by New Jersey families in social programs. About half of the respondents reported having received food stamps; two-fifths, AFDC; a third, social security disability; and a fifth, WIC. About a third had been in a community mental health program and two-fifths had had children in Head Start or another pre-school program. Very few had been in alcohol or drug treatment or marriage counseling. The experimental and control groups differed significantly only with regard to job training, 2 percent of the control group and 8 percent of the experimental group had been in such a program (p = .01).

Table 6-8.
Participation in social programs prior to initial interview,
New Jersey
Program	Percent %
Food Stamps	51
Job Training	6
WIC	23
AFDC	43
Housing Vouchers	16
Social Security Disability	31
Alcoholism Program	7
Drug Treatment Program	6
Marriage Counseling	3
Community Mental Health program	31
Head Start/Pre-school	42

(13) Fifteen percent of control group respondents and 17% of experimental group respondents reported an income less than $5,000; 32% control and 22% experimental reported between $5,000 and $10,000; 15% control and 31% experimental reported between $10,000 and $20,000; 24% control and 18% experimental reported between $20,000 and $40,000, and 14% control and 12% experimental reported an income of $40,000 or more.

(14) Control group respondents more often lived with a spouse or partner, 43% vs. 35%, p = .13.

(15) In the New Jersey administrative data, there are 7 possible outcomes of investigations of maltreatment: abuse/neglect/injury confirmed perpetrator, abuse/neglect/injury unconfirmed perpetrator, abuse/neglect/injury perpetrator unknown, unsubstantiated incident, unsubstantiated incident with concern, incident never occurred, and no outcome. The data above concern only persons who were children at the time of random assignment. The administrative data also record information on previous allegations involving persons who are now adults. Seventy-four adults (persons 18 or over at the time of random assignment) from 51 families had been the subjects of previous substantiated reports of maltreatment.

(16) By a "spell" we mean a period of time in placement which may consist of one or more distinct placements in different foster homes or in other settings.

6.4 The Tennessee Families

As with Kentucky and New Jersey, a description of the Tennessee families was compiled using information from the initial interviews with caretakers (n = 117, 80% of the 142 net study cases). In addition to the description of the sample as a whole, specific characteristics on which the experimental and control groups differ significantly are identified below.

Table 6-9 shows some of the characteristics of the caretakers and families in the Tennessee sample. Slightly more than 93 percent of the respondents were women. Eighty-three percent of the sample was African American (not Hispanic), 15 percent Caucasian, and 1 percent Hispanic. Nine percent of the sample had less than a high school education, 46 percent some high school, 18 percent high school graduation or GED, 22 percent at least some college education, and 4 percent had special education or vocational schooling. Over half the sample (54%) had never been married, 3 percent widowed, 14 percent separated, 13 percent divorced, and 17 percent were married. At the time of the initial interview, approximately 40 percent of the respondents were employed, 24 percent reported they were unemployed and looking for work, and 36 percent reported they were unemployed and not looking for work. Information about household income was provided by 115 of the respondents. Thirty-eight percent reported an income less than $5,000, 24 percent reported between $5,000 and $10,000, 23 percent reported between $10,000 and $20,000, 11 percent reported between $20,000 and $40,000, and 3 percent reported an income of $40,000 or more.

Table 6-9.
Description of the Tennessee families at time of initial interviews
	N	%
Gender of Caretaker/Respondent	117
Male		6.8
Female		93.2
Race of Caretaker/Respondent	116
African American (not Hispanic)		83
Caucasian (not Hispanic)		15
Hispanic		1
Other		0
Respondent's education level	116
Elementary school or less		9
Some high school		46
High school graduate or obtained GED		18
College		22
Special education or vocational schooling		4
Respondent's marital status	117
Married		17
Divorced		13
Separated		14
Widowed		3
Never Married		54
Respondent's Relationship to youngest child	117
Birth mother		84
Biological father		6
Grandmother		4.3
Other relative		5.1
Household composition	117
Birth mother, no other adults		50
Birth mother& 1 male adult		21
Birth mother& extended family^*		14
Biological father^*		6
Other relative caretaker^*		9
Other^**		1
	N	Mean
Age of respondent	116	32.5
Age of youngest child	117	4.0
Age of oldest child	117	10.8
Number of Children	117	3.3
Number of adults	117	1.6

^* These categories may also include other non-related adults in the home.

^**Includes: nonrelative caretaker, adoptive or step-parent, birth mother& non-related females, or birth mother, and more than one non-related male.

There was an average of 4.9 persons in the families, 1.6 adults and 3.3 children. The average age of the respondents was 33 (n = 116, s.d. = 8.5). The age of the youngest child in the family ranged from 0 to 17 with an average of 4.0 years (n = 117, s.d. = 4.2); 33 percent were under the age of one, 25 percent were between 1 and 4, 38 percent between 5 and 12, and 3.6 percent 13 and over. The age of the oldest child in the family ranged from 0 to 17 with an average of 10.8 (n = 117, s.d. = 4.8); 4.3 percent were under the age of one, 6.1 percent were between 1 and 4, 46 percent between 5 and 12, and 44 percent 13 and over.

When asked about their relationship to the youngest child in the home, 84 percent of the respondents reported they were birth mothers, 6 percent were biological fathers, 4.3 percent were grandmothers, one respondent was an adoptive mother, and the rest were other relatives (including aunts, uncles, a sister, and a great grandmother). With respect to the household composition at the time of the first interview, exactly half of the sample was comprised of families headed by birth mothers with no other adult in the home, 21 percent had a birth mother and one male adult, 14 percent had a birth mother and extended family, 6 percent were headed by a biological father, and 9 percent had an other relative caretaker (1% of the families did not fall into one of these categories). Thirty-one percent responded that they were living with a spouse or partner. Seventy-six percent reported that they rented their homes while 24 percent reported owning their home. While there were no statistically significant differences between the experimental and control groups, there was a marginally significant difference with respect to the proportion of respondents living with a spouse or partner. A larger proportion of the experimental group reported living with a spouse or partner (36% vs. 19%, p = .06).

Family Problems. Table 6-10 summarizes the problems and strengths identified by caretakers. When asked about emotional and financial problems within the last month, 61 percent of respondents said they felt "blue or depressed," 53 percent said they felt nervous or tense, 46 percent were overwhelmed with work or family responsibilities, 28 percent said they had just wanted to give up, and 24 percent said they felt they had few or no friends. Over half (56%) responded affirmatively to the general question of whether or not they experienced not having enough money for food or rent. On more specific questions about financial difficulties, 35 percent indicated having difficulty buying clothes, 26 percent buying enough food, 42 percent paying electric or heat bills, and 37 percent paying rent (on this last item, a significantly greater proportion of control group respondents answered affirmatively, 54% vs. 29%, p = .01). Less than 10 percent of the sample reported problems in drinking or using drugs (2.5% said they had too much to drink several times a week, and 7.7% reported using drugs several times a week). Only 4.3 percent had gotten in trouble with the law in the past month. Almost all respondents (97%) thought they were "doing a pretty good job raising their kids."

Table 6-10.
Caretaker problems and strengths,
caretaker initial interview,
Tennessee
Problems	Percent responding yes
Felt blue or depressed	62
Felt nervous or tense	53
Just wanted to give up	28
Overwhelmed with work or family responsibility	46
Felt you had few or no friends	24
Not enough money for food, rent, or clothing	56
Gotten in trouble with the law	4
Had too much to drink in a week	3
Used drugs several times a week	8
Economic Items
Had difficulty paying rent	37
Had difficulty paying electric/heat	42
Had difficulty buying enough food	26
Had difficulty buying clothes	35
Positive Items
Have you felt happy	87
Gotten together with anyone to have fun/relax	56
Doing a pretty good job raising kids	97

Table 6-11 shows problems of children identified by caretakers. About two-thirds of caretakers said at least one child in the family threw tantrums and 60 percent said a child "gets upset easily." As in Kentucky and New Jersey, school problems were common; over a quarter had been absent a lot, nearly 40 percent had failed classes, and over 40 percent had been suspended. Somewhat fewer children in Tennessee displayed aggressive behavior, 18 percent of the caretakers responded yes to the items "fights a lot with other kids" and "is very aggressive to you."

Caretaker Abuse or Neglect as a Child. Approximately 33 percent of Tennessee caretakers reported having been abused as a child and 25 percent reported having been neglected. Twenty-one percent responded "yes" to both questions, and overall, 38 percent reported having been abused, neglected, or both as a child. Twelve percent of the respondents reported having been in a foster home or institution as a child. There were no significant differences between experimental and control groups with respect to these previous experiences.

Previous Allegations and Placement. Of the 144 Tennessee families for which we had administrative data, 117 (81%) had an allegation of maltreatment prior to the date of referral to family preservation services. Sixty-seven percent had a substantiated report of maltreatment prior to the referral date.

Table 6-11.
Concerns and problems regarding children,
Caretaker Initial Interview, Tennessee
(% responding yes regarding any child that the respondent cares for)
	Tennessee
Item	N	%
Asked about all children...
Child went through alcohol withdrawal at birth	105	5
Child went through drug withdrawal when born	105	5
Child doesn't show much interest in what is going on	111	29
Child is Smaller/Lighter than other children	114	19
Child Get(s) upset easily	112	60
Asked for children over 3 months old...
Is/Are Funny and makes you laugh	111	93
Like(s) to share things with others	110	86
Throw(s) tantrums	111	65
Is/Are shy and withdrawn	108	30
Is/Are outgoing and friendly	110	99
Is/Are good looking	112	96
Fight(s) a lot with other kids	109	18
Has/Have language problems	109	25
Asked for children over 4 years old...
Is/Are very aggressive toward you	104	18
Has/Have a special talent in music	104	53
Like(s) animals	104	90
Is/Are good at sports	104	72
Usually does the right thing	104	85
Hangs with friends you don't like	102	44
In the past 3 months has any child you care for...
Gone to church regularly	104	63
Been absent from school a lot	99	27
Run away from home overnight	98	21
Been temporarily suspended from school	96	42
Been expelled from school	96	16
Taken care of younger children	93	71
Took something that didn't belong	102	27
Absent from school/no good reason	96	18
Received special education at school	97	32
Failed any classes	98	38
Received counseling	96	39
Asked for any child over age 7...
In the last 3 months, has any child been arrested	85	27
Asked only for children over age 10...
Has child age 11 or older had alcohol problems	73	3
Has child age 11 or older had a drug problem	70	4
Has any girl age 12 to 18 been pregnant	41	2
Has any boy age 14 to 18 fathered a child	21	0

We have data for 106 cases on the type of allegation just before the last case opening before referral. Seventy-six percent of the cases had allegations of physical abuse, 15 percent lack of supervision, 8 percent neglect, and 2 percent injury. The distribution of last substantiated allegation is 79 percent physical abuse, 12 percent lack of supervision, 8 percent neglect, and 1 percent injury.

Similar to the other states, we examined reports of maltreatment before the allegation prior to the referral to family preservation services, as an indication of prior involvement with the child welfare system. Of the 117 families with allegations prior to referral, 48 (41%) had a substantiated report of maltreatment before that, indicating that about two-fifths of the families had previous involvement with the system.

As to substitute care placement, according to the CORS administrative data, 9 children in 4 families had previously experienced placement. The average length of time between the end of the previous placement and random assignment was 6.27 months. The average length of time in that placement spell was 16.47 months. Data on previous unpaid relative placements were not available.

Length of Time from Case Opening to Referral to Family Preservation. On 147 Tennessee cases for which we had administrative data on case openings, 36 cases were not open at the time of the referral to family preservation services. In 57 percent of the 111 cases open at the time of random assignment, the referral to family preservation services occurred within a month after case opening while in another 20 percent it came between two and six months after case opening. In 14 percent of the cases the referral occurred more than a year after case opening.

Social Program Participation. Table 6-12 shows the rates of participation by Tennessee families in social programs prior to the initial interview. Almost three-fourths of the respondents reported having received food stamps; 61 percent AFDC; 30 percent social security disability, and 43 percent WIC. Fifteen percent reported participation in a community mental health program, 10 percent in a drug treatment program, 7 percent in an alcoholism program, and 38 percent had children in Head Start or another pre-school program. None of the respondents reported participating in marriage counseling. Five percent of respondents said they had participated in job training, with marginally significant differences (p = .06) between the experimental group (3%) and the control group (11%).

Table 6-12.
Participation in social programs prior to initial interview, Tennessee
Program	Percent %
Food Stamps	72
Job Training	5
WIC	43
AFDC	61
Housing Vouchers	7
Social Security Disability	30
Alcoholism Program	7
Drug Treatment Program	10
Marriage Counseling	0
Community Mental Health program	15
Head Start/Pre-school	38

6.5 Summary

In all three states, most of the respondents to the first interview were women and birth mothers of the youngest child in the home. In Kentucky and New Jersey, a little over two-fifths of the respondents were African American, while in Tennessee, 83 percent were African American. In Kentucky, slightly more than half were Caucasian, compared to a little under half in New Jersey and only 15 percent in Tennessee. About half of the respondents in all three states had not graduated from high school. Half of the households in Tennessee were headed by a single birth mother, compared to 43 percent in Kentucky, and 34 percent in New Jersey. The average age of the respondents in Kentucky and Tennessee was about 32, while New Jersey respondents were older, an average of 39. Similar differences held for age of youngest child: an average of 4.0 in Tennessee, 4.6 in Kentucky, and 7.1 in New Jersey. The average number of children in the home was around 3 for all three states.

Approximately half of the respondents in Kentucky and New Jersey answered affirmatively to each of three questions about emotional difficulties: "feeling blue or depressed," "feeling nervous or tense," and "feeling overwhelmed with work or family responsibilities." In Tennessee, rates, of reporting these difficulties were a little higher. Half or more of the respondents in all three states indicated that they did not have enough money for food, rent or clothing. Few respondents reported problems with drugs or alcohol. A third or two-fifths reported that they had been abused or neglected or both as a child.

About two-thirds of the respondents in New Jersey and Tennessee reported they participated in at least one of 5 income support programs: AFDC, food stamps, WIC, social security disability, and housing vouchers. In Kentucky, over 80 percent participated in one of these programs. In all three states, the rate of participation was less than 10 percent for each of the following programs: alcoholism treatment, drug treatment, marriage counseling, and job training. A third or two-fifths of the respondents indicated participation in Head Start or another pre-school program.

In Kentucky and New Jersey, about a fifth of the families had children who had previously been in a foster care placement. In Tennessee, only four families had children who had previously been placed.

The Target Group for Family Preservation Services. The families referred to family preservation services in Kentucky, New Jersey, and Tennessee had a variety of problems with a range of severity. Beyond that, they were a diverse group, varying in such things as family composition, ages of children, previous involvement in the child welfare system, and where they were in the system at the time of referral to family preservation services. Question can be raised as to whether a single model can be expected to be appropriate across such a diverse caseload. Can one expect one approach to work as well with older as well as young children? With cases of abuse as well as chronic neglect and dependency? With cases new to the system as well as those with extensive prior involvement?

We may also inquire as to the extent to which the families served in these states are the families for which family preservation services are intended as outlined in the introduction to the chapter. There are two central elements usually found in specifications of the target group for family preservation: imminent risk of placement and the presence of crisis. The paradigmatic case is one in which an allegation of abuse or neglect has recently been made and the case is referred in the course of investigation of that harm. It is evident that many cases are not in this category, particularly in Kentucky and New Jersey. Some are dependency or parent-adolescent conflict cases. Many do not come from the investigative phase of a case but rather from "on-going" workers. Many do not appear to be in immediate crisis, as suggested by the fact that many cases were referred long after the latest reports of maltreatment and after the most recent case opening. Cases in Tennessee more often conformed to this model, although there were a number that did not.

It is true that family preservation services are often advocated in cases other than abuse and neglect (in fact, Homebuilders began in the context of adolescent mental health problems). Furthermore, the specifications of eligible cases, reviewed at the beginning of this chapter, suggest a fairly wide net, including cases referred from on-going workers. Behavioral Sciences Institute's own criteria for referral contain one item that seems to contradict the criterion of crisis: the requirement that other services have been tried and failed. Adhering to this requirement would tend to put off referral to family preservation beyond the time of immediate crisis.

Beyond ambiguities in target group suggested by state policy and by BSI, there are the observations of referring workers that they sometimes, even often, made referrals that did not meet the imminent risk criterion. Although a family might not have a child at risk of placement, they believed the family would benefit from the service, so found a way to refer it. The data presented in this chapter suggest that the imminent risk and crisis criteria were often not met. It appears that the target group for family preservation has been expanded beyond that originally intended, perhaps first by state policy and certainly by practice in the field. Such expansion of the target group is no doubt quite common for social programs. It is natural to attempt to provide a valued service, viewed as beneficial, to more and more cases.

But there is a reason for relatively narrow, carefully defined, target groups. Specification of the target group is closely intertwined with specification of the goals of a program (in family preservation programs, cases of imminent risk of placement are the target group for a service intended to prevent placement). Clarity of target group allows clarity of goals. Once the target group becomes broadened, there is the risk that goals will become muddied. Two problems may ensue: the service being provided may lose structure, definition, and focus; and it becomes more difficult to achieve demonstrable effects of the service.

So the group of families served by family preservation services in these states reveal a central tension: the urge to serve a wide range of families as against the desirability of maintaining program focus on well defined groups. We have no ready solution to this conundrum, which may be inherent in large scale program implementation. We hasten to note that although we focus here on these three states, it is possible that most, if not all, states implementing family preservation programs face very similar issues.

We will return to the targeting problem in family preservation programs after examining the outcomes of the programs in Kentucky, New Jersey, and Tennessee.

Chapter 7: The Services

In this Chapter, we describe the services provided to families in both the family preservation and control groups. We are concerned with describing the experiences of these families in these programs and with making comparisons between the experimental and control groups. We must determine whether the experimental group received more services and more intense services in order to assess the extent to which the tended experimental conditions held. The interpretation of outcome information depends on a demonstration that experimental services were more extensive than “regular” services. We also attempt to compare the services received by the experimental group to the Homebuilders model, to get at questions of the extent to which the model was implemented. Finally, we describe how the families experienced these programs.

Most of the data come from the second interview with caretakers and caseworkers in which we asked questions about services offered and received during the period since random assignment and from the contact forms completed by workers serving both groups. In the follow-up interview a year after random assignment, caretakers were also asked about services received since the post-treatment interview and we report on analyses of those data at the end of this chapter. Comparisons were made between experimental and control group families as they were initially randomly assigned (the "primary analysis";). "Secondary"; analyses, where violations of random assignment and cases receiving minimal service are dropped, were also performed. Tables showing secondary analyses are provided in Appendix H. In most cases, secondary analyses show similar results to those of the primary analyses. Differences are highlighted in footnotes to the following text.

7.1 Caseworker Activities

Caretakers were asked to indicate whether the caseworker provided help with a number of specific problems. Table 7-1 shows the number of affirmative responses in each group.

Table 7-1.
Caretaker reports of caseworker activities, post-treatment interview
	Kentucky			New Jersey			Tennessee
	C	E		C	E		C	E
	%	%	p	%	%	p	%	%	p
Caseworker helped with money for rent-elect-phone	3	17	0.001	5	4		5	10
Caseworker helped with money for other things	9	35	0.001	10	14		11	19
Caseworker provided transportation	16	42	0.001	12	25	0.003	19	34	0.1
Caseworker discussed proper feeding of child	14	20		5	11	0.006	16	28
Caseworker talked with you about discipline	35	55	0.001	39	60	0.001	46	70	0.01
Caseworker talked with you on relations with spouse	16	18		8	14	0.009	11	34	0.01
Caseworker helped you clean house	2	6		2	5		11	9
Caseworker helped with painting/house repairs	1	1		1	1		5	1
Caseworker discussed how to get childcare	15	18		15	1		14	24
Caseworker helped with welfare/food Stamps	8	14		5	7		11	8
Caseworker advised how to get medical care	12	16		14	13		22	20
Caseworker talked with you how to handle anger	28	43	0.005	29	53	0.001	42	70	0.004
Caseworker advised you on substance abuse	3	7		11	12		11	18
Caseworker discussed with you how to get a better place	11	15		12	6	0.06	11	19
Caseworker advised on job training programs	9	19	0.009	7	10		8	16
Caseworker talked about how to get a paying job	6	17	0.004	5	8		11	18
Caseworker advised on how to continue school	9	18	0.04	5	8		14	23
Caseworker arranged for some childcare	1	3		5	7		6	13
Caseworker told you about other agencies	38	43		42	56	0.01	19	33
Note: C = Control Group, E = Experimental Group

Kentucky. According to caretakers, the most common activities in which workers engaged were discussing discipline and anger management and telling caretakers about other agencies that offer services. On the 19 items on which caretakers were questioned, never did the control group workers reportedly engage in an activity more than the experimental group workers. In the primary analysis, for 7 of the 19 items, experimental group workers reportedly engaged in the activity significantly more often than control group workers (all at p = .01 or less). One additional item showed significant differences in the same direction at p = .05 or lower.¹ A total count of the number of these 19 caseworker activities reported by caretakers also shows significant differences between the experimental and control groups. Caretakers in the experimental group reported an average of 3.9 caseworker activities (n = 148) while caretakers in the control group reported an average of 2.2 caseworker activities (n = 146) (p = .001).² Caretakers were asked which of the caseworker activities were especially helpful. Experimental group caretakers judged significantly more activities to be helpful than did control group caretakers (1.7 vs. 1.0, p = .001). The services most often cited as helpful by experimental group caretakers were, in order, "the caseworker talked with you about discipline,"; "the caseworker talked with you about how to handle anger,"; "the caseworker told you about other agencies,"; and "the caseworker helped you with money for other things [other than rent, electricity, or phone]."; For control group caretakers the most often cited helpful items were "the caseworker told you about other agencies,"; "the caseworker talking with you about discipline,"; and "the caseworker talked with you about how to handle anger."

New Jersey. The most common activities of workers (according to the caretakers) were discussions of discipline and the handling of anger and referrals to other agencies. In the primary analysis, in 2 of the 19 items control group workers more often engaged in the activity (ignoring those items with 1% differences): discussing getting a better place to live (p = .055) and discussing child care (not significant). For 4 of the 19 items, experimental group workers significantly more often engaged in the activity: discussion of discipline, transportation, discussion of how to handle anger, and discussion of other agencies. A fifth item, discussion of proper feeding of the child, was nearly significant at p = .06.

There were significant differences between the experimental and control groups in the average number of activities reported by caretakers, 3.25 for the experimental group (n = 210) and 2.31 for the control group (n = 134) (p = .001).³ When asked which of these activities were especially helpful, experimental group respondents judged significantly more activities to be helpful than did the control group respondents (1.97 vs. 1.11, p = .0001). The items cited most often as helpful were remarkably similar to those in Kentucky. The services most often cited as helpful by experimental group caretakers were, in order, "the caseworker talked with you about discipline,"; "the caseworker talked with you about how to handle anger,"; "the caseworker told you about other agencies,"; "the caseworker provided transportation,"; and "the caseworker helped you with money for other things [other than rent, electricity, or phone]."; For control group caretakers the most often cited helpful items were exactly the same as in Kentucky: "the caseworker told you about other agencies,"; "the caseworker talking with you about discipline,"; and "the caseworker talked with you about how to handle anger.";

Tennessee. Activities most often engaged in were similar to those in Kentucky and New Jersey: talk about discipline and talk about handling of anger. In 15 of the 19 items, experimental group workers were reported to have engaged in the activity more often than control group workers, although the differences were significant on only 3 of these items (talk about discipline, talk about handling anger, and talk about relations with spouse). For the four items control group workers more often engaged in, differences between the groups were small and not significant.

As in Kentucky and New Jersey, there was a significant difference between the groups in the average number of activities reported by caretakers, 4.6 for the experimental group (n = 80) vs. 2.89 for the control group (n = 37) (p = .02).⁴ Experimental group respondents also judged more activities as especially helpful, an average of 1.34 vs. .84 (p = .04). Again, the items cited most often as helpful were similar to those in Kentucky and New Jersey. Both the experimental and control groups most often listed talk about discipline, talk about how to handle anger, and transportation as most helpful, although experimental group respondents cited these activities far more often.

(1) The results of the secondary analyses show slightly greater differences between the experimental and control groups. Here, 8 of the 19 items show significant differences in favor of the experimental group at p = .01 or lower, and an additional 2 items show significant differences in the same direction at p = .05 or lower. See Appendix H-1.

(2) These differences were even larger when violations and minimal service cases were excluded from the analyses (4.6 vs. 2.1, ns of 138 and 110, p = .001).

(3) Differences were even greater when the violations and minimal service cases were excluded (3.57 vs. 1.90, ns of 181 and 115, p = .0001)

(4) In the secondary analysis, the experimental group had an average of 4.99 activities, compared to 2.88 for the control group.

7.2 Social Program Participation

In the second interview, at the completion of family preservation services for the experimental group, caretakers were asked about their participation in the same set of social programs they were asked about in the initial interview (see Chapter 6), except this time they were asked to report their participation since the time of the first interview (Table 7-2).

Table 7-2.
Participation in social programs, post-treatment interview
(percents)
	Kentucky			New Jersey			Tennessee
Program	Control	Experimental	p	Control	Experimental	p	Control	Experimental	p
Food Stamps	60	66		51	48		65	64
Job Training	3	8	0.09	2	3		3	4
WIC	32	45	0.02	22	20		51	41
AFDC	47	49		38	40		49	50
Housing Vouchers	15	20		17	16		11	11
Social Security Disability	39	34		32	28		22	36
Alcoholism Program	5	5		5	5		5	6
tment Program	3	1		6	9		8	9
Marriage Couseling	0	7	0.006 (FE)	2	2		0	1
Community Mental Health program	11	15		21	28		14	18
Head Start/Pre-school	26	21		33	32		25	38

NOTE: "FE" indicates significance determined by Fisher’s exact test

Kentucky. The proportions of involvement were remarkably similar to those in the first interview with less than a 2 percent change for most programs in the participation of both groups combined. Exceptions to this were a 2.6 percent decrease in participation in WIC services (from 41% to 38%), a 4 percent decrease in the proportion of respondents receiving food stamps (from 67% to 63%), and a 6.3 percent decrease for participation in Head Start or Pre-school programs (from 30% to 24%). Similar to results of the first interview, there were significant differences in experimental and control group participation in the WIC program, with 45 percent of the experimental group reporting participation at the post-treatment interview (n = 148) compared to 32 percent of the control group (n = 146) (p = .021). Differences between experimental and control groups were also found with respect to participation in job training and marriage counseling. For job training, 8.1 percent of the experimental group reported participation (n = 148) compared to 3.4 percent of the control group (n = 146) (p = .085). Seven percent of the experimental group (n = 102) but none of the control group respondents (n = 105) reported participation in marriage counseling (Fisher’s exact p = .006). No significant differences were found with respect to the total number of income support programs or treatment programs in which respondents participated since the time of the first interview.

New Jersey. As in Kentucky, the proportions of involvement were similar to those in the first interview except for community mental health programs (26% in the second interview compared to 31% in the first) and using Head Start or another pre-school program (32% vs. 42%). There were no significant differences between the experimental and control groups. There were no significant differences between groups in the number of income support programs and treatment programs.

Tennessee. Participation in social programs at the post-treatment interview was similar to that at the initial interview, except for declines in use of food stamps (from 72% to 64%), AFDC (from 61% to 50%) and head start/preschool (38% to 34%). There were no significant differences between the experimental and control groups in participation in any program post-treatment, nor were there significant differences in the average number of income support or treatment programs.

7.3 Caretakers' Reports of Services

In the second interviews, caretakers were asked if they had received any of a set of specific services in the time since the first interview. Results are shown in Table 7-3.

Table 7-3.
Caretaker report of services, post-treatment interview
	Kentucky			New Jersey			Tennessee
	Control	Experimental		Control	Experimental		Control	Experimental
	%	%	p	%	%	p	%	%	p
Daycare	5	19	0.001	10	7		26	26
Help in Finding a Place to Live	1	4		5	2		17	5	0.04
Staying at an Emergency Shelter	1	1		2	1		6	0	0.03
Medical or Dental Care	8	15	0.07	36	42		34	16	0.03
Transportation	7	16	0.02	14	12		17	19
Education Services/GED	1	4		2	2		9	8
Parent education/ Training classes	13	19		8	11		20	8	0.06
Legal Services	7	11		11	7		9	5
Counseling	35	52	0.003	50	56		9	17
Respite Care	1	1		0	1		0	0
Homemaker Services	1	3		6	3		14	3	0.02
A Parent Aide to Help You	1	4		7	4		11	5

Kentucky. A significantly greater proportion of caretakers from the experimental group reported receiving such services as day care (19% vs. 5%), transportation (16% vs. 7%), and counseling (52% vs. 35%) (all are significant at p = .05 or less). Reported receipt of medical or dental care was also a higher for the experimental group than for the control group (15% vs. 8%) ( p = .07).⁵ In a separate question, caretakers were asked whether the agency provided homemaker services or the assistance of parent aide. Approximately 2 percent of all caretakers reported having a homemaker and about 3 percent reported receiving assistance from a parent aide, with slightly greater but not significantly different percentages reported in the experimental group as compared to the control group. When caretakers were asked whether they did not receive any services they felt were needed, 27 percent of the control group responded affirmatively compared to 19 percent of the experimental group, a difference that was not statistically significant in either the primary or secondary analyses.

New Jersey. There were no significant differences between the experimental and control groups in receipt of any of these services in the primary analyses.⁶ About 4 percent of the caretakers reported having a homemaker, with no significant difference between the experimental and control groups. Control group caretakers significantly more often reported they did not receive services that were needed (56% vs. 42%, p= .01).

Tennessee. On most of the services items, control group caretakers more often reported receiving the service, these differences being significant at .06 or lower for five of the items listed in Table 7-3. Experimental group percentages were higher for only two items, counseling (reported by far fewer caretakers in Tennessee than in Kentucky and New Jersey) and transportation, neither difference being significant. These rather surprising results, indicating more services for the control group, contradict data on caseworker activities presented above and data from the Tennessee caseworkers (discussed in section 7.5). Control group caretakers more often reported they did not receive services that were needed (39% vs. 24%, p = .1).

(5) When violations and minimal service cases are excluded from the analyses, the differences remained significant and most p-values decreased even further. The secondary analyses showed a significantly greater proportion of the experimental group caretakers report attendance at parent education classes (p = .05). See Appendix H-3.

(6) In the secondary analysis (dropping violations and minimal service cases) there was a difference on one item: experimental group caretakers more often received counseling (59% vs. 46%, p = .03). See Appendix H-3.

7.4 Relationship with Caseworker

Table 7-4 shows results from a number of questions in which caretakers were asked about their relationships with caseworkers. In all three states, for most of these questions, caretakers in the experimental group rated their workers significantly more positively than did caretakers in the control group. A greater proportion of experimental group caretakers felt their workers listened to their concerns "most of the time"; other responses were "some of the time"; and "not very often";. Also, a greater proportion of experimental group caretakers felt their workers understood their situation "very well"; as compared to "not very well."; A greater proportion of caretakers in the experimental group reported reaching agreement with their workers on goals "most of the time."

Table 7-4.
Caretakers'reports on relationship with caseworker, post-treatment interview
	Kentucky			New Jersey			Tennessee
	Control	Experimental		Control	Experimental		Control	Experimental
	%	%	p	%	%	p	%	%	p
Worker listened to your concerns most of the time	71	87	0.001	56	78	0.001	71	91	0.02
Worker understood your situation very well	75	90	0.002	62	79	0.001	64	81	0.09
You and worker agreed on goals most of the time	66	76	0.06	40	72	0.001	38	58	0.09
Did worker sometimes talk with you about issues that were not easy to talk about?	27	34		29	44	0.01	22	51	0.003
Caseworker helped you to see your good qualities	67	79	0.03	47	70	0.001	53	82	0.001
Caseworker helped you to see your problems	66	76	0.1	52	72	0.001	50	82	0.001
Did you see your caseworker		0.09			0.003
More often than you wanted	9	18		12	14		19	27
As often as you wanted	70	62		43	59		44	48
Not often enough	21	20		45	27		36	25

In all three states, experimental group caretakers significantly more often than control group caretakers reported that workers talked with them about problems that were not easy to talk about, helped caretakers to "see your problems"; (p = .1 in Kentucky), and helped them see their good qualities. With regard to the frequency of contact with the workers, in Kentucky, approximately 20 percent of caretakers from both the experimental and control groups indicated they did not see their caseworkers often enough. A greater proportion of caretakers in the experimental group indicated they saw their workers "more often than [they] wanted"; (18% vs. 9%) and a greater proportion of caretakers in the control group indicated they saw their workers "as often as [they] wanted"; (70% vs. 62%). In New Jersey, a greater proportion of caretakers in the experimental group responded that they saw their workers "as often as [they] wanted"; (59% vs. 43%) and a greater proportion of caretakers in the control group responded that they saw their workers "not often enough"; (45% vs. 27%). In Tennessee, more experimental group caretakers said they saw their workers more often than they wanted (27% vs. 19%) and more control group caretakers said they did not see their workers often enough (36% vs. 25%), but the differences between the groups on this item were not significant.

In none of the three states did the groups differ in the extent to which they called workers when they had problems.

7.5 Caseworkers' Reports of Services

In the second interview, caseworkers were asked whether they had made referrals to any of 25 services, such as childcare, homemaker services, income programs, treatment programs of various sorts, and health care. Results from these 25 items are shown in Table 7-5.

Table 7-5.
Caseworkers'Report of Services Provided to Family, Post-treatment interview
	Kentucky			New Jersey			Tennessee
	Control	Experimental		Control	Experimental		Control	Experimental
	%	%	p	%	%	p	%	%	p
Childcare or baby sitting	14	8.5		9.3	7.7		2.1	8.9
AFDC or other public income (except SSI)	4.3	4.8		4.3	4		4.2	5.6
SSI for adult or child	6.8	5.5		1.2	0.7		2.1	0
Food stamps	2.5	6.1		4.3	3.3		4.2	6.7
Drug treatment	14	4.2		11	5.5	0.05	0	10	0.03
Alcoholism treatment	14	12		8	5.5		0	6.7	0.09
Legal aid	3.7	7.9		1.8	4.8	0.08	0	5.6
Help with education	28	37		14	26	0.001	4.2	15.6	0.05
Respite care	5	11		5.5	5.5		4.2	4.4
Parent training	48	67	0.09	28	67	0.0001	31	68	0.001
Health care	12	27	0.02	15	18		8.3	22	0.04
Inpatient mental health	9.3	2.4		2.5	5.1	0.14	2.1	0
Outpatient mental health/counseling	39	60	0.07	25	37	0.01	17	20
Health assessment	13	29	0.01	17	21		13	13
Housing financial assistance	3.7	22	0.02	5.5	4.8		0	5.6
Other housig services	1.2	15	0.01	1.9	3.7		0	5.6
W.I.C.	1.9	7.3		3.1	2.6		2.1	3.3
Emergency financial assistance other than housing	16	38	0.02	18	22		6.3	23	0.01
Job training	5.6	1.8		1.2	2.2		0	5.6
Emergency shelter	3.7	1.8		6.2	1.5	0.02	2.1	2.2
Recreational services	12	24	0.08	11	23	0.001	4.2	21	0.008
Family planning	9.9	20	0.1	11	20	0.009	8.3	10
Self help groups	20	23		4.3	8.8	0.056	2.1	8.9
Household management	15	32	0.01	12	28	0.0001	17	29
Homemaker services	3.7	13	0.003	6.8	1.5	0.01	8.3	3.3
Other	9.3	21	0.06	15	16		17	14
N	161	165		162	272		48	90

Kentucky. Caseworkers for the experimental group reported helping their clients with an average of 5.0 of these services, while caseworkers from the control group reported helping their clients with an average of 3.2 of these services (p = .001).⁷ In the primary analyses, 7 specific services were provided significantly more often to the experimental group than to the control group (significance levels were all at p = .05 or less). These services include health care, health assessment, housing financial assistance, other housing assistance, emergency financial assistance, household management, and homemaker services. A greater proportion of caseworkers from the experimental group also reported providing outpatient mental health counseling (p = .07) and selected the unspecified category of "other"; services provided (p = .06). No services were provided significantly more often to the control group than the experimental group.⁸

New Jersey. Experimental group caseworkers reported that on average their clients were provided 3.2 of these services, while control group families were provided 2.3, a difference significant at .001.⁹ When individual services are examined, there were six services that were provided significantly more often to the experimental group (education services, parent training, outpatient mental health, recreational services, family planning, and household management). Three services were provided significantly more often to the control group: drug treatment, emergency shelter, and homemakers.¹⁰

Tennessee. Experimental group caseworkers reported providing an average of 3.2 services, compared with 1.6 for the control group, significantly different at .001. Six individual services were significantly more often provided to experimental group families (drug treatment, help with education, parent training, health care, emergency financial assistance, and recreational services).¹¹ No services were significantly more often provided to the control group.

(7) When violations and minimal services cases were excluded, the difference between the groups was even larger (5.8 vs. 3.1, p = .001).[Back To Text]

(8) In the secondary analyses, excluding violations and minimal service cases, 14 services were provided significantly more often to the experimental group than to the control group (again, significance levels were all at p = .05 or less). In addition to the 9 primary analysis items showing differences in favor of the experimental group (seven significant items plus outpatient mental health counseling and the category of "other";), these were: food stamps (p = .02), parent training (p = .002), and recreational services (p = .002). Again, in the secondary analyses, no services were provided significantly more often to the control group than the experimental group. See Appendix H-5.

(9) The difference between groups was even larger when violations and minimal service cases were excluded: 3.47 vs. 1.94 (p < .0001).

(10) Excluding violations and minimal service cases, only one service was provided significantly more often to the control group, emergency shelter, while nine services were significantly more often provided to the experimental group, the above six plus legal aid, emergency financial assistance, and self help groups.

(11) In the secondary analysis, there was one additional service provided significantly more often to the experimental group: household management.

7.6 Contact Data

Workers serving clients in both the experimental and control groups were asked to complete a one-page contact report following each in-person contact with a family member (see Appendix K). The report was a simple check-off form, asking about who was present in the visit and about the content of the conversation. Although these forms were quite simple and easy to fill out, it proved difficult to get workers to complete them. We implored workers who did not fill out these forms to do so, and we have at least one on a fair proportion of the cases. However, it is likely that for at least some cases on which we have forms that we do not have them for all of the contacts. We are unable to determine how many contacts occurred for which we have no forms. Furthermore, the quality of information may be affected by the fact that some of the forms were submitted after many calls from our office and after long delay. The following analyses were limited to those families with contact reports. Only "primary"; analyses are reported for contact reports.

Some data on contact forms are presented in Table 7-6. Forms were received on between 71 percent and 91 percent of the experimental groups and between 51 percent and 71 percent of the control groups. It should be noted that the lower rate for control group cases is partially due to the fact that there was no contact in the four weeks after the date of random assignment, the period of time for which we requested contact forms for the control group (a period comparable to the 4 week period of services for the experimental group. On average, more contact forms were submitted for the experimental group than for the control group. In addition to the overall number of reports submitted, in all three states the experimental group received significantly more home visits, visits with caretakers, visits with the other parent, and visits with children. The experimental group workers were more likely to involve other adults in the family, non-family members, and other workers. As experimental group families received significantly more contacts than the control group families, they also received significantly more individual activities. For both experimental and control families, in all three states the most common concrete service was the provision of transportation. Purchasing food, child care, and providing clothing, furnishings, and supplies were also common forms of concrete services.

Contact forms also captured general information about the topic of discussion, counseling, or instruction. In all three states, for both experimental and control families, the most common topics of discussion were the discipline of children, goals, and the caretaker’s interaction with the children. Other common topics were the child’s anger management and supervision of children.

Table 7-6.
Contact forms
	Kentucky			New Jersey			Tennessee
	C	E	p	C	E	p	C	E	p
Number of cases with at least one form submitted	111 (63%)	124 (71%)		119 (71%)	250 (91%)		25 (51%)	73 (74%)
Average number of forms per case	3.1	13.8	.001	4.4	12.4	.001	2.5	9.5	.001
Average number of home visits	1.9	10.3	.001	3.6	10.8	.001	2.0	8.0	.001
Average number of visits with caretakers	2.4	12.8	.001	3.8	10.3	.001	2.2	8.2	.001
Average number of visits with the other parent	0.4	2.2	.001	0.6	1.9	.001	0.2	1.7	.01
Average number of visits with children	2.1	10.3	.001	3.6	9.4	.001	1.9	7.2	.001
Concrete Services
Transportation	0.5	3.5	.001	0.9	2.3	.001	0.2	1.9	.001
Buying food	0.1	1.0	.001	0.4	0.8	.003	0.0	0.5	.01
Child Care	0.3	0.4		0.2	1.0	.001	.04	.21	.03
Clothing, furnishings, and supplies	0.2	0.9	.001	0.2	0.6	.01	0.0	0.3	.01
Topics of Discussion
Discipline of children	1.5	7.1	.001	2.2	6.0	.001	1.4	4.7	.001
Goals	1.8	6.0	.001	2.1	6.6	.001	1.8	4.2	.001
Caretaker’s interaction with children	1.5	6.0	.001	2.2	5.7	.001	1.7	4.8	.001
Child’s anger management	1.1	3.9	.001	1.6	4.8	.001	0.8	1.5	.06
Supervision of children	1.1	4.0	.001	1.4	2.9	.001	1.6	2.9	.001

NOTE: C = Control Group, E = Experimental Group

The contact forms contained additional concrete services and topics of discussion (see Appendix H). Only those that were most often reported are shown here. Entries are average numbers of times per family that an item was reported, for those families with at least one form submitted.

Experimental group contacts. We examined further the contact forms for the experimental group to explore some issues in the adherence of programs to the Homebuilders model of service, subscribed to in all three states (see Table 7-7). In addition to other critical elements of family preservation, the Homebuilders model specifies that workers should provide an in-home contact within 72 hours of referral, and family preservation workers should be available 7 days per week. Substantial contact should take place within the first week; Kinney, Haapala, and Booth suggest that the typical case receive 11 hours of service in that time.¹² Concrete services are also an important component of service, particularly early in the case.

In Kentucky, of the 124 experimental families with submitted contact forms, 55 (44%) received an in-home contact within 72 hours, 97 (78%) had contact in the first week. Those 97 families had an average of 5.1 hours of face-to-face contact in the first week. Regarding availability of worker, 18 (1%) of contacts occurred on either Saturday or Sunday. Finally, 34 (27%) of the experimental families received some type of concrete service within the first seven days.

In New Jersey, of the 250 experimental families with submitted contact forms, 73 percent received an in-home contact within 72 hours, 219 (88%) in the first week, and those families had an average of 6.5 hours of face-to-face contact in the first week. Regarding availability of the worker, only 196 (6%) of submitted contacts occurred on Saturday or Sunday. Finally, 38 percent of the experimental families received some type of concrete service within the first seven days.

In Tennessee, of the 73 experimental families with submitted contact forms, 42 (57%) received an in-home contact within 72 hours, 53 (73%) had contact in the first week. We are able to calculate hours of contact for 45 of these 53 cases and these cases had an average of 8.3 hours of face-to-face contact in the first week. Regarding availability of worker, 60 (9%) contacts occurred on either Saturday or Sunday. Finally, 21 (29%) of the experimental families received some type of concrete service within the first seven days.

These data seem to indicate that some "structural"; aspects of the Homebuilders model (contact within 72 hours of referral, amount of contact in the first week, services provided at all hours, including weekends, and concrete services early in the case) are not always upheld in these states. However, it is not possible to draw firm conclusions about this, because of issues in the quality of the contact form data discussed earlier.

Table 7-7.
Experimental group contacts
	Kentucky		New Jersey		Tennessee
	N	%	N	%	N	%
Number of families with contact data	124		250		73
Total number of contact forms submitted	1713		308		690
Contacts in week 1	280	16	753	24	169	25
Contacts in week 2	353	21	667	22	142	21
Contacts in week 3	322	19	601	19	133	19
Contacts in week 4	322	19	515	17	111	16
In-home contact within 72 hours	55	44	183	73	42	57
In-home contact with 7 days	97	78	219	88	53	73
Concrete service within 7 days	34	27	95	38	21	29

(12) Jill Kinney, David Haapala, and Charlotte Booth. (1991). Keeping families together: The Homebuilders model. New York: Aldine de Gruyter.

7.7 Summary of Services

In all three states, the caretaker interview, the caseworker interview, and the contacts data generally confirmed the expectation that the experimental group would receive more services and more intensive services that the control group. An exception is the caretaker reports of services received in Tennessee. Table 7-8 shows a summary of those items on which there were significant differences between experimental and control groups on the primary analyses in any state.

In all three states, the number of experimental group caseworker activities reported by caretakers was greater than that reported by control group respondents, and this was also true of "helpful"; caseworker activities. As for specific caseworker activities, experimental group workers in all three states were more likely to provide transportation, talk about discipline, and talk about how to handle anger. In all three states, the number of specific services received by experimental group families was greater than the number received by control group families. Contact from data confirmed that there was far more contact with experimental group families. The most common concrete service reported on contact forms was transportation; the most common topics of discussion were discipline of children, goals, and caretaker’s interaction with children.

It is of interest that transportation is a theme in a number of sources of information about services. We do not have information on where workers were transporting parents and children, but it is evident that needs for transportation are common in these families, needs that workers are able to respond to. This is a concrete service that provides immediate help and builds relationships. Furthermore, workers told us that they often use the time in the car to good advantage in discussing problems of the family.

The most common subject of counseling, interaction with children and in particular their discipline, reflect central problems in these families, problems of paramount concern to the child protective system. It is, therefore, not surprising that workers were focused on altering parent-child interaction patterns.

Experimental group caseworkers in all three states were more often reported to have talked about difficult issues, to have helped the caretaker to see her/his good qualities and problems, and to have "understood your situation.";

Table 7-8.
Summary of services, post-treatment interview
Caseworker Activities:
Proportion of affirmative answers by caretakers to yes/no questions
	Kentucky			New Jersey			Tennessee
	C	E		C	E		C	E
	%	%	p	%	%	p	%	%	p
Is Caseworker still working with family	79	64	0.006	75	31	0.001	57	34	0.02
Caseworker helped with money for rent, electricity, phone	3	17	0.001	5	4		5	10
Caseworker helped with money for other things	9	35	0.001	10	14		11	19
Caseworker provided transportation	16	42	0.001	12	25	0.003	19	34	0.10
Caseworker discussed proper feeding of child	14	20		5	11	0.06	16	28
Caseworker talked with you about discipline	35	55	0.001	39	60	0.001	46	70	0.01
Caseworker talked with you on relationship with spouse	16	18		8	14	0.09	11	34	0.01
Caseworker talked with you about how to handle anger	28	43	0.005	29	53	0.001	42	70	0.004
Caseworker told you about other agencies	38	43		42	56	0.01	19	33
Caseworker advised on job training programs	9	19	0.009	7	10		8	16
Caseworker talked about how to get paying job	6	17	0.004	5	8		11	18
Caseworker advised on how to continue school	9	18	0.04	5	8		14	23
Caseworker talked about uneasy issues	27	34		29	44	0.008	22	51	0.003
Caseworker helped you see good qualities	67	79	0.03	47	70	0.001	53	82	0.001
Caseworker helped you see your problem	66	76	0.10	52	72	0.001	50	82	0.001
Caseworker understood your situation	75	90	0.002	62	79	0.001	64	79	0.08
Called your Caseworker with problems	53	57		66	61		56	65
	Kentucky			New Jersey			Tennessee
	C Mean	E Mean	p	C Mean	E Mean	p	C Mean	E Mean	p
CT report of # of Caseworker activities	2.18	3.90	0.0001	2.31	3.25	0.001	2.89	4.60	0.02
CT report of # of "helpful"; Caseworker activities	1.04	1.68	0.0001	1.11	1.97	0.0001	0.83	1.33	0.04
Services Provided
Proportion of affirmative answers by caretakers to yes/no questions
Anyone been in job training program	3	8	0.09	2	3		3	4
Anyone been in WIC	32	45	0.02	22	20		51	41
Been in a marriage counseling program	0	7	0.006	2	2		0	1
Anyone receive daycare	5	19	0.001	10	7		26	26
Anyone receive transportation	7	16	0.02	14	12		17	19
Anyone receiving parent education/training	13	19		6	10		20	8	0.06
Anyone receive counseling	35	52	0.003	50	56		9	17
Anyone receive help finding a place to live	1	4		5	2		17	5	0.04
Anyone stay at an emergency shelter	1	1		2	1		6	0	0.03
Anyone receive medical or dental care	8	15	0.07	36	42		34	16	0.03
Anyone receive homemaker services	1	3		6	3		14	3	0.02
Were any needed services not gotten	27	19		56	42	0.01	39	24	0.10
	Kentucky			New Jersey			Tennessee
	C Mean	E Mean	p	C Mean	E Mean	p	C Mean	E Mean	p
Caseworker report of # services provided	3.16	4.99	0.001	2.31	3.17	0.001	1.58	3.19	0.0002

NOTE: C = Control Group, E = Experimental Group

This table only includes items with a primary p-value less than .05 in at least one of the states; p-values greater than .10 are not reported.

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

Insofar as there are differences between groups, we can be reasonably sure that the experimental conditions held. Conclusions regarding adherence to the Homebuilders model are less clear cut. Families did not always receive contact within 72 hours, fewer than expected contacts occurred in the first week of the program, and few contacts occurred on weekends. There was relatively little provision of concrete services early on. These results are not entirely surprising. Social programs are never implemented precisely as they are designed. Perhaps the test of a program conception is that it achieves desired outcomes even when it is not implemented exactly as intended.

7.8 Services during the follow-up period

When caretakers were interviewed a year after random assignment, they were asked some of the same questions about services received, this time since the last interview (since the end of family preservation services for the experimental group and during a comparable period for the control group). Tables 7-9, 7-10, and 7-11 show analyses of these questions.

Caseworker Activities. Caretaker reports of caseworker activities since the post-treatment interview are shown in Table 7-9. In all three states, the experimental group respondents reported more caseworker activities than did control group respondents. In Kentucky, there were five activities the experimental group caretakers significantly more often reported: help with money for rent, electricity, or phone; help with money for other things; transportation; advice on getting medical care; and information about other agencies. In New Jersey, there were two such activities, help in cleaning the house and talk about how to handle anger, with a third item nearly significant, information about other agencies (p = .06). In Tennessee, there were seven activities significantly more often reported by experimental group caretakers: help with money for other things, transportation, talk about discipline, advice on substance abuse, help with relations with spouse, talk about how to handle anger, and information about other agencies. Differences between the groups were not as great as those reported for the treatment period, as is to be expected, since the treatment did not continue during this period.

Table 7-9.
Caretaker reports of caseworker activities, follow-up interview
	Kentucky			New Jersey			Tennessee
	Control	Experimental		Control	Experimental		Control	Experimental
	%	%	p	%	%	p	%	%	p
Caseworker helped with money for rent-elect-phone	1	8	.008	4	5		8	13
Caseworker helped with money for other things	8	16	.05	8	11		14	37	.01
Caseworker provided transportation	11	23	.01	11	18		17	42	.007
Caseworker discussed proper feeding of child	3	8		3	3		11	23
Caseworker talked with you about discipline	24	32		24	34	.08	42	62	.05
Caseworker talked with you on relations with spouse	4	9		8	8		19	39	.04
Caseworker helped you clean house	1	1		0	5	.03 (FE)	8	10
Caseworker helped with painting/house repairs	0	0		0	0		3	4
Caseworker discussed how to get childcare	8	11		8	5		6	16
Caseworker helped with welfare/food Stamps	2	4		3	2		3	4
Caseworker advised how to get medical care	2	9	.01	6	6		11	14
Caseworker talked with you how to handle anger	24	33		16	28	.03	36	59	.02
Caseworker advised you on substance abuse	6	6		7	6		0	18	.009 (FE)
Caseworker discussed with you how to get a better place	8	8		7	4		11	18
Caseworker advised on job training programs	7	9		4	3		8	15
Caseworker talked about how to get a paying job	6	9		3	4		8	18
Caseworker advised on how to continue school	5	6		5	4		17	22
Caseworker arranged for some childcare	1	0		2	2		0	1
Caseworker told you about other agencies	14	24	.05	30	41	.06	8	30	.01

NOTE: "FE" indicates significance determined by Fisher’s exact test.

C = Control Group, E = Experimental Group

Participation in Social Programs. As indicated in Table 7-10, there were no significant differences between the experimental and control groups in any state in involvement in social programs during the post-treatment period.

Table 7-10.
Participation in social programs, follow-up interview
(percents)
	Kentucky			New Jersey			Tennessee
	Control	Experimental	p	Control	Experimental	p	Control	Experimental	p
Food Stamps	50	61	.10	49	45		56	54
Job Training	7	13	.10	10	5	.07	11	12
WIC	24	31		21	18		28	34
AFDC	34	39		37	39		31	34
Housing Vouchers	13	13		16	18		19	18
Social Security Disability	32	32		27	27		19	36	.07
Alcoholism Program	5	6		10	8		6	8
Drug Treatment Program	3	4		10	11		3	9
Marriage Counseling	4	4		5	6		3	3
Community Mental Health program	6	7		29	32		19	22
Head Start/Pre-school	29	35		41	45		50	60

Caretaker report of services. Table 7-11 indicates that there was only one service in the three states on which there was a significant difference between groups in receipt post-treatment; in Tennessee more control group respondents reported having a parent aide. In Kentucky and New Jersey, the proportions of the two groups receiving each service are remarkably similar. Control group families in Tennessee more often received a couple of other services, but the differences were not significant. For the most part, the superiority of the control group in Tennessee in receipt of services observed at the post-treatment interview dissipated at the time of the follow-up interview.

Table 7-11.
Caretaker report of services, follow-up interview
	Kentucky			New Jersey			Tennessee
	Control	Experimental		Control	Experimental		Control	Experimental
	%	%	p	%	%	p	%	%	p
Daycare	13	12		14	13		31	33
Help in Finding a Place to Live	2	2		2	2		14	5
Staying at an Emergency Shelter	2	1		2	3		11	3	.09 (FE)
Medical or Dental Care	6	7		64	62		17	19
Transportation	13	17		13	14		9	18
Education Services/GED	3	2		3	4		9	8
Parent education/ Training classes	14	13		7	8		17	16
Legal Services	6	6		14	18		3	1
Counseling	50	48		52	57		19	25
Respite Care	1	2		2	4		0	0
Homemaker Services	1	2		6	5		6	5
A Parent Aide to Help You	1	3		3	2		14	3	.04 (FE)

NOTE: "FE" indicates significance determined by Fisher’s exact test

Summary of Post-treatment Services. A summary of the significant differences between experimental and control groups on report of services at the follow-up caretaker interview is shown in Table 7-12. In the questions about caseworker activities, there is some indication that experimental group families received more services during the post-treatment period. Since caretakers were asked about the period of time following the last interview, we assume that for experimental group respondents the activities were undertaken by workers other than family preservation workers, perhaps workers in the public agency or workers in other private agency programs to which they might have been referred. Hence, the data may be taken as indicating receipt of somewhat more services by the experimental group families after the end of family preservation services, in accordance with the goal of these programs to connect families with ongoing services. However, this finding was not confirmed by data on social programs or services. It is possible that the finding also reflects something that we have often heard from public agency workers working with family preservation programs, that the family preservation involvement gave them more information about the family and enabled them to plan better for services after family preservation.

Table 7-12.
Summary of services, follow-up interview
Caseworker Activities: Proportion of affirmative answers to yes/no questions
	Kentucky			New Jersey			Tennessee
	C	E		C	E		C	E
	%	%		%	%	p	%	%	p
Caseworker helped with money for rent, electricity, phone	1	8	.008	4	5		8	13
Caseworker helped with money for other things	8	16	.05	8	11		14	37	.01
Caseworker provided transportation	11	23	.01	11	18		17	42	.007
Caseworker talked with you about discipline	24	32		24	34	.08	42	62	.05
Caseworker talked with you on relationship with spouse	4	9		8	8		19	39	.04
Caseworker helped you clean house	1	1		0	5	.03 (FE)	8	10
Caseworker talked with you about how to handle anger	24	33		16	28	.03	36	59	.02
Caseworker advised you on substance abuse	6	6		7	6		0	18	.009 (FE)
Caseworker told you about other agencies	14	24	.05	30	41	.06	8	30	.01
	Kentucky			New Jersey			Tennessee
	C Mean	E Mean	p	C Mean	E Mean	p	C Mean	E Mean	p
CT report of # of Caseworker activities	.97	1.65	.01	1.0	1.3		1.6	3.3	.002
Services Provided: Proportion of affirmative answers to yes/no questions
Anyone been in job training program	7	13	.10	10	5	.07	11	12
Anyone receive a parent aide to help you	1	3		3	2		14	3	.04(FE)
Were any needed services not gotten	22	9	.006	48	38	.10	44	32

NOTE: C = Control Group, E = Experimental Group.

"FE" indicates significance determined by Fisher’s exact test

Tables only include items with a primary p-value less than .05 in at least one of the states; p-values greater than .10 are not reported.

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

Chapter 8: The Outcomes

The outcomes we examined were the placement of children in substitute care following random assignment to the experimental or control group, subsequent reports of maltreatment and a number of measures of child and family functioning. The focus is on comparisons between the experimental and control groups. Analyses we have designated as "primary" were conducted on all randomly assigned cases except those that were determined to be inappropriate referrals. This includes cases in which the assignment was violated (cases assigned to the control group that were given family preservation services) and cases assigned to family preservation that received no or little such service. Insofar as family preservation services have effects, inclusion of these cases in the analysis will tend to reduce the observed differences between the groups. However, the most rigorous approach to analysis requires that we retain these cases in the group to which they were assigned in order to maintain the statistical equivalence of the groups at the outset of the experiment, which is the reason for random assignment in the first place.

It is likely that violations and minimal service cases differ in systematic ways from other cases (perhaps not detected in the measurements of the study), hence, switching them to the other group would result in groups that were not equivalent at the beginning. It can be argued that inclusion of minimal service cases in the experimental group is quite proper on other grounds: the implementation of any program will involve some cases that do not receive the service, and estimates of impact ought to take that into account. We did conduct analyses ("secondary" analyses) in which the violations and minimal service cases were dropped, so as to examine differences between cases that actually received the intended treatment (family preservation or regular services). This analysis must be viewed as only suggestive, since it does not preserve the initial statistical equivalence of the groups created by random assignment. In fact, the results of the secondary analyses were usually similar to those of the primary analysis. We note in footnotes when the secondary analysis differed substantially from the primary analysis. The secondary analysis tables are presented in Appendix I.

Some analyses were also conducted on a more "refined" sample in which we attempted to focus on cases that approached a conception of "ideal" family preservation cases. Family preservation services are designed for families in crisis, presumably the crisis surrounding a recent allegation of maltreatment, the investigation of that allegation, and the threat of removal of a child. Theoretically, this state of crisis makes families more willing to seek and respond to help. As indicated in Chapter 6, many of the families did not appear to conform to this specification of the target group. In Kentucky and New Jersey we looked at two subgroups of cases, those with a recent substantiated allegation recorded in the administrative data (within three months prior to referral to family preservation services) and those in which an investigative worker was involved. In Tennessee, nearly all of the cases came from investigating workers, so we looked at those cases with an allegation within 30 days prior to referral.

8.1 Substitute Care Placement Following Random Assignment

A principal goal of family preservation services is the prevention of placement into substitute care, so that must be the first (though not the last) outcome examined. Placement included foster care, institutions and residential treatment programs, group homes, and adoptive placements.¹ We are initially concerned with the character and timing of the first placement of a child following random assignment. We collected data on placement prior to August 31, 1998 in Kentucky and New Jersey and before August 31, 1999 in Tennessee.² Although data were provided at the individual level, most of the analyses are presented at the family level.³ In Kentucky, the administrative files contained data on 942 children in 306 families, 154 in the experimental group and 152 families in the control group. Eighty-six children in 40 families (26%) in the experimental group experienced placement subsequent to random assignment compared to 60 children in 41 families (27%) in the control group. In New Jersey, administrative data were available on 1230 children in 434 families, 269 in the experimental group and 165 in the control group. One hundred sixteen children in 81 families (30%) in the experimental group were placed compared to 44 children in 39 families (24%) in the control group. In Tennessee, multiple sources of data were used to calculate the rate of subsequent placement. A statewide management information system (CORS) provided information on formal paid placements. Additionally, case record reviews provided information on unpaid relative placements. In Tennessee, placement data were available on 468 children in 140 families, 93 in the experimental group and 47 in the control group. In the analysis of CORS data, forty-six children in 23 families (25%) in the experimental group experienced placement subsequent to random assignment compared to 25 children in 10 families (21%) in the control group. Including unpaid relative placements, 60 children in 29 families (31%) in the experimental group experienced placement subsequent to random assignment compared to 31 children in 13 families (28%) in the control group. These differences were not statistically significant at the family level in Kentucky, New Jersey or Tennessee (see Table 8-1 for types of placements after random assignment).

A comparison of these percents is, however, misleading, because of varying periods of risk of placement. The proper approach to the analysis of such data is survival analysis, in which the proportions of cases placed at each point in time following random assignment in each group are compared, accounting for the numbers of cases that "survive" to that point. We examined survival curves for each group and determined whether these curves were statistically different. Family level analyses were based on the first date of placement of any child in the family if a placement occurred.

Table 8-1.
Type of first placement after random assignment, child level
Kentucky
Type	N	Percentage
Foster care	96	66
Private institution	42	29
Foster care, medically fragile	4	2.1
Child psychiatric hospital	3	2.7
Not specified	1	0.7
Total	146	100
New Jersey
Foster care	76	47.5
Juvenile family crisis	33	20.6
Residential treatment	25	15.6
Group home	13	8.1
Shelter care	4	2.5
Public institution	4	2.5
Relative	3	1.9
Independent living	3	0.6
Maternity home	1	0.6
Total	160	100
Tennessee
Foster care	31	44.3
Relative home	9	12.9
Trial home	6	8.5
Residential	6	8.5
Continuum contract	4	5.7
Non-relative home	4	5.7
Adoptive home	3	4.3
Runaway	2	2.8
Shelter	2	2.8
Independent living	2	2.8
Detention	1	1.4
Total	71	100

Note: Includes only placements recorded in administrative data. There were additional unpaid relative placements (see text).

Kentucky. The family level analysis of subsequent placement is displayed in Figure 8-1.⁴ These survival curves show the proportion of families remaining intact (without placement of a child) at each point in time following random assignment. The curves begin at 1, indicating that at the time of random assignment, all children were at home. The curves then decline as children enter care. The higher curve at any point represents the group with fewer placed children at that point. The curves are adjusted for cases that are "right censored." For example, cases that were not observed for a full year following random assignment are dropped in the calculation of the percentage remaining intact ("surviving") at one year. The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. At the one-year interval, 23 percent of experimental group families and 24 percent of control group families experienced substitute care placement.

Figure 8-1 First Placement after random assignment (families)

"Refined" groups analyses were also conducted, limiting the sample to cases referred by investigative workers and to those families with substantiated allegations within the three months prior to random assignment. No statistically significant differences emerged. In the primary analysis of those families coming from an investigative worker, 26 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year after the random assignment date. For those with recent substantiated allegations, 29 percent of the experimental group and 16 percent of the control group experienced subsequent placement within one year.

An additional "refined" group was available for analysis in Kentucky. Prior to random assignment, workers submitted petitions to the court for placement or some other court ordered intervention on 67 families. Administrative data were available for 60 of these 67 families (29 in the experimental group, 31 in the control group). Survival analyses were conducted to explore the relationship between family preservation services and subsequent placement. At one year after random assignment, 18 percent of the experimental group and 33 percent of the control group experienced placement; a nonsignificant difference.

In addition to the administrative data on placement, in Kentucky the Westat site coordinator attempted to document all placements subsequent to random assignment, based on her contacts with caseworkers. The administrative data file contained placements not recorded by the site coordinator, and vice versa. The only systematic difference between these data sources was the documentation of relative placement. Relatives are generally not paid for placements in Kentucky, so these data were not recorded in the administrative files. Survival analyses were conducted with a combination of caseworker and administrative placement records. If either data source recorded a placement event, that family was coded as experiencing subsequent placement. The first documented date of placement, taken from either source, was selected for analysis. The patterns of placement in these analyses are similar to those reported above. At one year, 27 percent of the experimental group and 32 percent of the control group families experienced placement, a nonsignificant difference.

In addition to survival analyses, placement can be examined in terms of the proportion of time in substitute care subsequent to random assignment. If family preservation services are effective in preventing placements, we would expect them to result in lower numbers of days in foster care. Family preservation might also result in shorter stays in care, once children are placed. Comparison of days in care provides a beginning look at the question of whether family preservation results in lower costs of foster care (of course, a complete cost-effectiveness analysis must also factor in the differential costs of family preservation and regular services).

Table 8-2.
Subgroup analyses, significance levels of differences between experimental and control groups
	Kentucky							New Jersey
	Placement				Subtantiated allegations			Placement				Subtantiated allegations
	In 6 mos				In 6 mos			In 6 mos				In 6 mos
	N	Overall %	p^a	Survivalp	Overall %	p^a	Survivalp	N	Overall %	p^a	Survivalp	Overall %	p^a	Survivalp
Overall	306	18			14			434	18			6
Substance abuse	28	25		.099^b	25			50	12			16	.10
No substance abuse	231	17			13			322	20			4
Problems with bills	137	20			19			192	17			6
Problems with daycare	92	19			14			111	22			8
Depression^d	149	20			16			165	23			7
Problems with punishment	187	19			15			256	20			5
Problems with school	134	14			16			196	22			5
Problems with employment	172	17			16			189	16			6
Single mother	118	18			14			110	16		.05	6

^a Fisher exact, two tail

^b Experimental group more likely to experience placement

^c Control group more likely to experience placement

^d Caretakers with depression scores above median for the state

The proportion of time in care is calculated by dividing the number of days in care by the number of days of possible care (number of days between random assignment and the date of administrative data collection). As the proportions are calculated at the family level, the number of days in care represents the total number of care days summed across all children within a particular family. Similarly, the number of possible care days represents the total number of possible care days summed across all children within a particular family. The number of possible care days is adjusted for a child's eighteenth birthday and for births since random assignment. For both primary and secondary analyses, in both the experimental and control groups children spent an average of 6 percent of the days subsequent to random assignment in care.

New Jersey. The family level analysis of placements is shown in Figure 8-1.⁵ More families in the experimental group experienced placement of a child than in the control group (at one year, 28% of the experimental group vs. 22% of the control group) although the differences are not significant. It might be noted that in the analyses the survival curves for the two groups tend to begin to diverge at about 2-3 months, that is, at about that time more children in the experimental group are being placed. We do not have a ready explanation for this divergence.

Refined groups analyses in New Jersey revealed no statistically significant differences. In the primary analysis of those families coming from an investigative worker, 25 percent of the experimental group and 16 percent of the control group experienced subsequent placement within one year of the random assignment date. For those with recent substantiated allegations, 27 percent of the experimental group and 15 percent of the control group experienced subsequent placement.

As to the proportion of time that children spent in care in New Jersey, experimental group children spent an average of 6 percent of that time in placement, compared to 4 percent for the control group children (not a significant difference).

Tennessee. Survival rates at the family level were first calculated using only the CORS and then including relative placement (the "any evidence" analysis) data. The family level analyses of subsequent placement is displayed in Figure 8-1.⁶ The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. In the analysis of CORS data, 23 percent of experimental group families and 19 percent of control group families experienced substitute care placement within one year subsequent to random assignment. In the "any evidence" analysis, 28 percent of the experimental group families and 23 percent of control group families experienced placement within one year subsequent to random assignment.

As in Kentucky and New Jersey, a "refined" group was available for analysis in Tennessee. Ninety-three families had an allegation within 30 days prior to random assignment. The Wilcoxon statistic for the survival analysis of placement in these families indicates that the survival rates of the two groups are not statistically different. In the analysis of CORS data, 17 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year of random assignment. In the "any evidence" analysis, 22 percent of the experimental group and 21 percent of the control group experienced subsequent placement within one year of random assignment.

As to the proportion of time that children spent in care in Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children. This difference is nonsignificant.

In a number of analyses of subsequent placement in these states, more experimental group families experienced placement than did control group families. In a few analyses, fewer experimental group families experienced placement. However, none of these analyses were statistically significant; in none of these states can the data be taken as firm evidence that family preservation resulted in more placements. Nor is there evidence that it resulted in fewer.

Imminent risk of placement. The family preservation programs in these states are designed to prevent the unnecessary removal of children by serving families with children who are at imminent risk of out-of-home placement.⁷ One way to explore the accuracy of the "imminent risk" designation is to examine the proportion of control group families that experienced placement within a short time after random assignment. Since the control and experimental groups were randomly assigned and are expected to be statistically equivalent before services are begun, the proportion of families experiencing placement in the control group indicates the proportion of referred families that would have experienced placement in the absence of receiving family preservation services. We looked at control group placement rates 30 days after random assignment, believing that time period provided a liberal interpretation of "imminent risk." If a significant proportion of the control group experienced placement within 30 days of random assignment, one could argue that the program was appropriately targeted. At the time of random assignment, referring workers were asked to designate those children who were considered "at risk."

In Kentucky, in the first 30 days following random assignment, in the primary analysis 4 percent of at risk children in the experimental group were placed compared to 3 percent of control group at risk children. At the family level, 6 percent of the experimental group families and 5 percent of the control group families experienced placement within the first 30 days subsequent to random assignment. The percentages were similar in the investigative group (8 percent of the experimental compared with 5 percent of the control group), and among those with recent substantiated allegations (6 percent of the experimental group compared with 3 percent of the control group).

In New Jersey, of those children judged to be at risk, 4 percent of the control group and 3 percent of the experimental group were placed in 30 days. At the family level, 4 percent percent of the families in the experimental group experienced placement of at least one child within one month of random assignment, compared to 6 percent of the control group. Rates of imminent placement were similar in the "refined" group analyses. Of those families coming from an investigative worker, 3 percent of the experimental group and 5 percent of the control group experienced placement within 30 days of random assignment. For those families with a substantiated allegation within three months prior to random assignment, 8 percent of the experimental group and 5 percent of the control group experienced placement within 30 days.

In Tennessee, rates of placement within one month were somewhat higher than in Kentucky and New Jersey. Of those children judged to be at risk, 13 percent of the control group and 11 percent of the experimental group were placed in 30 days. There were no relative placements within the first 30 days subsequent to random assignment. Thus, there are no differences between the CORS and "any evidence" analysis. At the family level in Tennessee, the CORS administrative data indicates that 11 percent of both the experimental and control groups experience placement within 30 days subsequent to random assignment. Rates of imminent placement were similar in the "refined" group analyses. Of those families with a recent allegation (within 30 days prior to random assignment), 7 percent of the experimental group and 12 percent of the control group experienced a CORS placement within 30 days of random assignment.

Although the percentages of placement within one month were somewhat higher in Tennessee, in all three states, these percentages were quite low. The numbers of interest here are those for the control group, indicating the targeting efficiency of the program in these three sites is very low.

(1) The full list of New Jersey service codes that were included is: public institution, teaching family placement, para-foster care income maintenance, juvenile-family crisis shelter placement, relative placement, foster care placement, residential treatment placement, finalized adoption placement, selected adoption placement -- pending, maternity home care, group home placement, independent living, and shelter care placement. Four of these categories did not actually occur in the data: teaching family placement, para-foster care income maintenance, finalized adoption placement, and selected adoption placement -- pending. In Kentucky placement (as reflected in the variable FACTYPE), included: adoption, foster care, private institution/boarding schools, family treatment home, unmarried parent, other, children's psychiatric hospital, and foster care medically fragile. The data did not include adoption, family treatment home, and unmarried parent. In Tennessee, placements included: foster care, relative home, trial home, residential care, continuum contract, non-relative home, adoptive home, runaway, shelter, independent living, and detention.

(2) Cases entered the study at varying points in time. In Kentucky, cases entered between May 7, 1996 and February 13, 1998; in New Jersey, cases entered between November 6, 1996 and February 26, 1998; and in Tennessee, between November 19, 1996 and May 26, 1998.

(3) There are two reasons for focusing on family level analyses. First, we are not confident that the administrative data allow for accurate identification of children to be included in the risk pool (what would be the denominator in a rate of placement calculation). Children are identified as belonging to a family through a case number. The analysis requires that we identify children who are in the home at the time of random assignment (or who are born or return to the home subsequently). In these states, children apparently often retain a family case number even when they are not in the home, and the administrative data do not allow us to verify the location of the child at the time of random assignment (or even sometimes at the time of an event such as placement). This problem is alleviated in analyses at the family level, since we know that the family is at risk of having a child placed (as long as there are any children in the family).

As to the accuracy of the "numerator" in our analyses, we focus on the first event (e.g., placement) in the family, subsequent to random assignment. It is possible that the first event occurs with regard to a child identified with a family but not living in that family at the time of the event. We judge the likelihood of that occurring to be small (the effects of this source of error would be similar in a family and child level analysis). In addition, subsequent events involving other children identified with the family but not in the family at the time of the event would not affect the family level analysis, while they would create inaccuracies in a child level analysis.

The second reason for focusing on the family level has to do with a "clustering" effect in the child level analysis. Clustering refers to the lack of independence between children within the same family of observations of such things as placement. If one child is removed from the home, the remaining children are more likely to experience placement. The "clustering effect" leads to an underestimate of the significance levels when analyses are conducted at the child level. Conducting the analyses at the family level is one approach to resolving this dilemma.

We did conduct a few analyses at the child level, when we wanted to take into account child characteristics, but it should be remembered that significance levels in those analyses are downwardly biased.

(4) In Kentucky, the ratio of assignment to experimental and control groups was 50-50.

(5) In New Jersey, approximately 60% of the cases were assigned to the experimental group.

(6) In Tennessee, approximately two-thirds of the cases were assigned to the experimental group.

(7) Kentucky policy specifies that imminent risk includes children who are at risk of commitment as dependent, abused, or neglected; who are identified through the Regional Interagency Council, an interdepartmental unit, as severely emotionally disturbed; or whose families are in conflict such that they are unable to exercise reasonable control of the child. Both the referring worker and family members shall believe that without immediate intensive intervention, out-of-home placement is imminent. At the time of this study, New Jersey targeted family preservation services for families at imminent risk of having at least one child enter placement. The referring worker must have based the assessment of imminent risk on a face-to-face interview with the family no more than 5 days prior to the referral. The family must need services immediately and the worker must determine that other, less intensive, services have been used, are not appropriate, or are not available. In Tennessee, CPS intake workers complete a risk assessment form to identify high, intermediate, low, or no risk. High risk cases are identified as cases where "the child or children in the home are at imminent risk of serious harm if there is no intervention in the situation." A typical high risk case might involve such factors as: 1) a vulnerable child; 2) a history of previous maltreatment; 3) an active perpetrator who has continued access to the child; and 4) no available support or family strengths to offset the stated risks.

8.2 Hazard Analyses of Placement

Hazard analyses permit the examination of the effects of multiple independent variables (in addition to experimental group membership) on rates of placement. They also provide somewhat more precise estimates of the effect of experimental group membership, since they control for the effects of the other variables in examining experimental-control group differences. In addition, they allow for the examination of "interactions" between other variables and experimental-control group membership, to see if the effects of experimental group membership differ for subgroups of the sample. We conducted Cox regression analyses of placement hazards using as predictor variables case characteristics available in the administrative data. Case characteristics in the administrative data are quite limited. Unlike the survival analyses which were conducted at the family level, hazard analyses were done at the child level because we wanted to include in them characteristics of children.

Kentucky. We examined the effects of the child's age, prior placement, and prior allegation of maltreatment, together with experimental group membership, on rates of placement. Only prior allegation was significant, increasing the hazard rate by 68 percent. Interactions of the three variables with experimental group membership were not significant.

New Jersey. New Jersey hazard analyses indicate that older age and prior placement increase the hazard rate significantly (p < .05; prior placement by 77% and each year of age by 8%). Minority status is nearly significant (p = .08), increasing the hazard rate by 37 percent. In an equation with these three variables, experimental group status shows a significant effect in increasing the hazard rate of placement (since this is a child level analysis, significance levels are downwardly biased due to clustering effects). We also examined the interaction of age, prior placement, and minority status with experimental group. The interactions for age and minority status were not significant (that is, the effect of experimental group did not differ by age or by minority status). The interaction with prior placement was close to significant in some of the equations we tried. Examining the differences in placement rates between the experimental and control groups by whether or not the child had had a prior placement indicates that among those with a prior placement, there is little difference in placement rates (23% for the family preservation group and 22% for the control group) while there is a significant difference for those without prior placement (13% for the experimental group and 7% for the control group). (It should be noted that this simple three-way cross tabulation does not account for varying lengths of the observation period, which is controlled in the hazard analysis.) This suggests that children without prior placement who receive family preservation are more likely to be placed than if they did not receive family preservation.

The results do not, however, suggest that family preservation is better for cases with more prior exposure to the child welfare system.⁸

Hazard analyses were also performed to examine the effect of county on placement. These analyses were conducted at the family level. Burlington county was chosen as the reference category, as it had the highest rate of placement. Thus, rates of placement in the other New Jersey counties are compared to the placement rates of Burlington. In addition to the county variables, experimental group and interactions of county with experimental group were entered into the regression equation. The hazard of placement for families was decreased by 42 percent for Camden county, 71 percent for Ocean county, 78 percent for Monmouth county, 61 percent for Essex county, 71 percent for Bergen county, and 75 percent for Passaic county. There were no significant effects of experimental group or of county-experimental group interactions. This indicates that even after removing county variation, there are no significant differences between the experimental and control groups, nor does the effect of experimental group vary by county.

Tennessee. We examined the effects of the child's age, race, prior placement, prior allegation within 30 days of random assignment, prior substantiated allegation within 30 days of random assignment, and experimental group membership on rates of placement. Similar to Kentucky and New Jersey, we also explored interactions between experimental group membership and child characteristics. No significant interactions emerged. Only prior substantiated allegation had a significant effect on the likelihood of placement subsequent to random assignment. In the analysis of the CORS administrative data, a substantiated allegation within the last 30 days prior to random assignment increased the hazard rate by 209 percent. When unpaid relative placements were included ("any evidence") prior substantiation increased the hazard by 173 percent.

(8) When a variable representing the designation of a child at risk is introduced into the equation, it has highly significant effects in predicting placement.

8.3 Allegations of Maltreatment Following Random Assignment

Subsequent maltreatment of children is a second important outcome to be examined. Family preservation programs are intended to lower the risk of harm to children while keeping them at home, and subsequent maltreatment is an indicator of such risk. Furthermore, the justification for family preservation programs rests on the belief that the safety of children is not compromised when their families are referred to these programs, so examination of subsequent maltreatment rates is important to determine whether children, in fact, are safe in these programs.

As with placement, data on subsequent maltreatment come from the administrative data files of the states. As is almost always the case in studies like this, our data do not record actual maltreatment, but only investigated reports of maltreatment. Some abuse and neglect goes unreported, and, because not every report is investigated, there are cases of harm that are reported but not investigated.

As with the analyses of subsequent placement, survival graphs were developed to compare the timing of subsequent, substantiated allegations of maltreatment.⁹ Again, survival analyses were conducted at the family level for both the primary and secondary analysis groups as well as for the "refinement" groups.

Kentucky. One hundred thirteen children in 51 families (33%) in the experimental group were the subjects of investigated allegations of maltreatment following random assignment, compared with 103 children in 47 families (31%) in the control group. The differences were not statistically significant at the family level. The distribution of the various types of allegations is as follows: 4 percent dependency, 8 percent emotional, 49 percent neglect, 43 percent physical abuse, and 16 percent sexual maltreatment. As families can be the subjects of multiple allegations on any given day, these percentages do not sum to 100. Not all investigations result in substantiated allegations. Sixty-six children in 30 families (20%) in the experimental group were the subjects of substantiated allegations of maltreatment compared with 68 children in 28 families (18%) in the control group. These differences were not statistically significant at the family level. The distribution of substantiated allegations is as follows: 6 percent dependency, 2 percent emotional, 55 percent neglect, 38 percent physical abuse, and 10 percent sexual maltreatment.

Figure 8-2 displays the survival curves for substantiated allegations in the primary analysis. At one year subsequent to random assignment, 19 percent of the experimental group and 18 percent of the control group families experienced substantiated reports of maltreatment. Although a higher percentage of families in the "refined" analyses experienced substantiated allegations of maltreatment, similar patterns emerged. For the investigative group, 23 percent of both the experimental and control groups experienced substantiated allegations of maltreatment in the one-year interval. For those families with a substantiated allegation within the three months prior to random assignment, 22 percent of the experimental and 24 percent of the control group experienced substantiated allegations of maltreatment within a year subsequent to random assignment. For the group on which petitions had been submitted to court for placement or other orders, 19 percent of the experimental group and 32 percent of the control group experienced a substantiated allegation within one year subsequent to random assignment, a nonsignificant difference.

The survival analyses indicate that experimental and control group families had a very similar likelihood of substantiated reports of maltreatment subsequent to random assignment.

New Jersey. One hundred fifty-four children in 81 families (30%) in the experimental group were the subjects of investigated allegations of maltreatment following random assignment, compared to 80 children in 48 families (29%) in the control group. In none of the survival analyses conducted were there significant differences between the experimental and control groups. Within one year, about 29 percent of the families had an allegation of maltreatment concerning at least one child in the family. Figure 8-2 shows substantiated allegations at the family level. About 12 percent of families in both groups have substantiated allegations within one year.

Patterns of substantiated allegations were similar for the "refined" group analyses, none of which showed significant differences between groups. Of those families coming from an investigative worker, 7 percent of the experimental group and 10 percent of the control group had a substantiated allegation within one year subsequent to random assignment. For those families with a substantiated allegation within three months prior to random assignment, 10 percent of the experimental group and 17 percent of the control group had a substantiated allegation within one year subsequent to random assignment.

Tennessee. Allegation data were available for 482 children in 144 families. Sixty-four children in 36 families (38%) in the experimental group were the subjects of investigated allegations of maltreatment following random assignment, compared with 61 children in 26 families (54%) in the control group. The differences were not statistically significant at the family level. The distribution of the various types of allegations is: 66 percent physical abuse, 20 percent supervision/neglect, 2 percent sexual abuse/medical, and 12 percent other (includes allegations such as failure to thrive, truancy, and unruly child). Forty-four children in 25 families (26%) in the experimental group were the subjects of substantiated allegations of maltreatment compared with 42 children in 18 families (38%) in the control group. These differences were not statistically significant at the family level. The distribution of the various types of substantiated allegations is: 66 percent physical abuse, 20 percent supervision/neglect, 1 percent sexual abuse/medical, and 13 percent other.

Figure 8-2 displays the survival curves for substantiated allegations in the primary analysis. At one year subsequent to random assignment, 24 percent of the experimental group and 25 percent of the control group families experienced substantiated reports of maltreatment.

Figure 8-2. First substantiated allegation after random assignment (families)

Survival rates were also calculated for those families with an allegation within 30 days prior to random assignment. Significant differences emerged for subsequent allegations and near significant differences emerged for subsequent substantiated allegations. Of those families with a recent allegation, 28 percent of the experimental group and 52 percent of the control group experienced an allegation within one year subsequent to random assignment. Similarly, 18 percent of the experiment group and 30 percent of the control group experienced a substantiated allegation within one year subsequent to random assignment. These differences suggest that in Tennessee family preservation reduced the likelihood of subsequent maltreatment for those families with recent allegations.

(9) Analyses were also done on all allegations, whether substantiated or not. The results were very similar, although, of course, rates for all allegations were higher.

8.4 Sub-group analysis

In Kentucky and New Jersey, we examined a number of subgroups of cases to determine whether we could detect differences between experimental and control groups on placement and substantiated allegations subsequent to random assignment within each subgroup. The number of cases in Tennessee was not sufficient to support subgroup analysis. The results are shown in Table 8-2. Most of the subgroups were defined in terms of problems existing at the time of the initial interview. For both placement and substantiated allegations the table shows the number of cases in each subgroup, the percentage of cases in the subgroup experiencing the event within 6 months, the significance of the difference between the experimental and control groups in the occurrence of the event within six months, and the p value for the analysis of differences in survival curves. The first row of the table shows the results for the Kentucky and New Jersey samples as a whole. Except for substance abuse, the definitions of the subgroups were taken from the initial caretaker interview. Very few caretakers acknowledged substance use in the first interview, so that subgroup was determined from information in both the caretaker and caseworker initial interviews.

As can be seen, nearly all of the experimental-control group comparisons shown are not significant. Of the 36 comparisons in the table, only one is significant at .05, that for substantiated allegations in the single mother group in New Jersey.¹⁰ Among single mothers, those in the experimental group were less likely to have a substantiated allegation than those in the control group.

In the analysis so far, efforts to find subgroups for which family preservation service was related to reduced placement have been unsuccessful.

(10) The six months analyses and survival analyses are obviously not independent.

8.5 Case Closing Subsequent to Random Assignment

Kentucky. Of the 183 cases that were open in the public agency at the time of the referral to family preservation services, 84 (46%) were closed some time after the referral and 99 (54%) remained open as of August 31, 1998 (the last date of observation for these analyses). Survival analyses were performed to examine the lengths of time between the referral to family preservation services and the first closing of the case. As shown in Figure 8-3, no significant differences were found between the experimental and control groups.

Figure 8-3. First Case Closing After Random Assignment

Of the 84 cases that were open at the time of the referral to family preservation services and closed some time after that referral, 3 cases were re-opened again before August 31, 1998. Two of these 3 cases were in the experimental group, one having been closed for 6 days and the other for over a year before the case was re-opened. The third case was a control group case and was closed for slightly over 3 months before being re-opened.

New Jersey. Of the 434 cases with administrative data, 219 (51%) were closed subsequent to the referral to family preservation services.Figure 8-3 shows the results of a survival analysis in which we examined the time to case closing for the 421 cases that were open at the time of the referral to family preservation services. There was no significant difference between the experimental and control groups in the rate or timing of case closings after the referral date.

Of the 219 cases that were closed after the referral to family preservation services, 48 cases (22%) were reopened. There was no significant difference between the experimental and control groups in the proportion of cases that were reopened (21% in the experimental group, 23% in the control group).

Tennessee. Of the 147 families with case opening and closing data, 111 were open in the public agency at the time of the referral to family preservation services. Of these 111 families, 96 (87%) were closed some time after the referral and 15 (14%) remained open as of August 31, 1999 (the last date of observation for these analyses). Survival analyses were performed to examine the lengths of time between the referral to family preservation services and the first closing of the case. As indicated by the survival curves in Figure 8-3, no significant differences were found between the experimental and control groups. Of the 96 cases that were open at the time of the referral to family preservation services and closed some time after that referral, 17 cases were re-opened again before August 31, 1999. There was a significant difference in the rate of reopening. Eight (12%) of the 66 experimental group cases that were closed subsequently reopened, compared with 9 (30%) of the 30 control group cases (p < .05).

8.6 Family and Child Functioning-Caretaker Interviews

Family preservation services are intended to result in improved functioning of children and families. This goal is sought both for its own intrinsic value as well as an intermediate objective in the prevention of subsequent maltreatment and placement; parents who are functioning better and better parent-child relationships should result in lower risk of abuse or neglect.

In our interviews with caretakers and caseworkers we asked a number of questions tapping various aspects of functioning. We asked most of these questions in all three interviews with caretakers (at the beginning of service, four to six weeks after service began, and one year after the beginning of service) and in the two interviews with caseworkers (at the beginning of service and four to six weeks later). In the initial interview, we usually asked respondents to answer in terms of circumstances in the last three months. In the post-treatment and follow-up interviews, we asked in terms of "since we last talked to you [at the time of the initial interview or the post-treatment interview]." To indicate the effects of family preservation services, we can compare the experimental and control groups on the responses to these questions in the second and third interviews and on change between interviews. We report on the responses to a number of individual items in our interviews. In addition, we combined the responses to many questions into summated scales.¹¹We examined differences between experimental and control groups in each state in the average levels of these scales at post-treatment and at follow-up and we examined changes over time in these averages using multivariate repeated measures analysis.¹²The results of the analyses of the scales are shown in Tables 8-3, 8-4, and 8-5and in Figure 8-4.

We report on the differences between the experimental and control groups as they were initially formed (the "primary analysis"). Results of the secondary analyses (dropping violations and minimal service cases) were usually similar. They are reported in footnotes when they were materially different.

We looked at a large number of differences between groups on functioning variables and as will be seen, some of these comparisons revealed statistically significant differences, usually favoring the experimental group. However, as will also be seen, the differences are not consistent across states or across time.

(11) Often we used average responses or proportions of positive responses rather than sums of responses to items. This was done in order to have scores for individuals when there were a few missing items on the scales. If an individual had too many missing items (usually 1/3rd or more) the score was declared missing. Rules for the calculation of all scales are given in Appendix J.

(12) In multivariate repeated measures analysis, three main hypotheses are tested, first, that the scores for the experimental group, averaged over the three points in time are equal to those of the control group, (the "group" hypothesis); second, that the averages of the groups at each point in time are the same (the "time" hypothesis); and third, that there is no interaction between time and group. It is the third hypothesis that is central, indicating whether the groups change in different ways

8.6.1 Life Events

In the caretaker interviews, there was a 15 item "life events" inventory asking about the occurrence of both positive and negative events in the last 3 months in the initial interview or since the previous interview in the post-treatment and follow-up interviews (Appendix K, Initial Caretaker Interview, p. 7). We formed three scales from this inventory, positive life events, negative life events, and a scale of those life events that might reflect depression in the caretaker (we had a more formal depression measure as well, described below). In none of the three states were there statistically significant differences between the experimental and control groups in the scores on these measures at the post-treatment or follow-up interviews nor were there differences between the groups in changes over time (see Tables 8-3, 8-4, and 8-5). It will be noted that generally the levels of these scales dropped in the second interview compared to the first and then increased again in the third interview. This is likely due to the differences in the time periods referenced in the questions asked at the three points in time (the last three months in the initial interview, last 4 to 6 weeks in the post-treatment interview, and last 10 to 11 months in the follow-up interview).

Table 8-3.
Kentucky family and child functioning scales
	Post-Treatment				Follow-Up			Multivariate Repeated Measures
								Means				Univariate ps			Univariate ps-Time		Univariate ps- Grp-time interaction
		N	M^a	p^b	N	M	p	N	Initial	Post	Follow	Grp^c	Time^d	Time -Grp^e	Initial v. later^f	Post v. Follow^g	Initial v. later	Post v. Follow
Positive life events	C	146	.12		119	.22		108	.16	.12	.21	.042	.001	.41		.001
Positive life events	E	148	.14		130	.21		117	.18	.14	.20	.042	.001	.41		.001
Negative life events	C	146	.04		119	.07		108	.08	.03	.07	.40	.001	.001	.001	.001
Negative life events	E	148	.03		130	.09		117	.10	.03	.09	.40	.001	.001	.001	.001
Life events depression	C	145	.35		119	.37		107	.45	.36	.38	.40	.001	.38	.001
Life events depression	E	147	.36		130	.40		117	.50	.37	.42	.40	.001	.38	.001
Economic Functioning	C	142	.17		118	.17		105	.22	.15	.18	.08	.001	.27	.001
Economic Functioning	E	144	.22		127	.20		111	.32	.23	.20	.08	.001	.27	.001
Punishment	C	143	.16		113	.15		101	.22	.17	.15	.49	.001	.36	.001	.09
Punishment	E	147	.17		121	.15		109	.25	.18	.15	.49	.001	.36	.001	.09
Child aggression	C	146	1.36		119	1.24		108	1.56	1.44	1.29	.84	.001	.96	.001	.05
Child aggression	E	148	1.33		130	1.29		117	1.57	1.44	1.32	.84	.001	.96	.001	.05
School problems	C	112	.22		96	.25		78	.29	.24	.27	.14	.03	.41	.01
School problems	E	101	.20		97	.19		77	.26	.21	.18	.14	.03	.41	.01
Child withdrawn	C	146	.89		119	.99		108	1.08	.89	1.00	.99	.001	.29	.001
Child withdrawn	E	148	.93		130	.93		117	1.08	.97	.93	.99	.001	.29	.001
Stolen things or arrested	C	146	.34		119	.39		108	.31	.34	.39	.76	.75	.07			.03
Stolen things or arrested	E	148	.32		130	.31		117	.44	.35	.32	.76	.75	.07			.03
Child substance abuse	C	146	.07		119	.03		108	.05	.07	.04	.40	.35	.67
Child substance abuse	E	148	.04		130	.02		117	.03	.03	.03	.40	.35	.67
Child problems	C	146	2.20		119	1.89		108	2.44	2.23	2.18	.87	.001	.47	.001
Child problems	E	148	2.05		130	2.04		117	2.56	2.14	2.07	.87	.001	.47	.001
Negative child behaviors	C	140	.34		107	.33		98	.37	.34	.34	.95	.001	.47	.001
Negative child behaviors	E	139	.34		120	.34		106	.39	.34	.33	.95	.001	.47	.001
Positive child behaviors	C	142	.71		109	.67		99	.70	.69	.67	.78	.31	.84
Positive child behaviors	E	142	.71		121	.68		109	.69	.69	.67	.78	.31	.84
Household condition	C	142	.02		119	.02		102	.05	.02	.02	.29	.002	.45	.001
Household condition	E	147	.02		129	.01		111	.03	.01	.01	.29	.002	.45	.001
Depression (SCL-90)	C	145	.79		119	.67		107	.95	.77	.70	.67	.001	.31	.001
Depression (SCL-90)	E	146	.74		130	.79		115	.96	.74	.83	.67	.001	.31	.001
Positive child care practices	C	140	.85		107	.82		94	.87	.85	.83	.55	.09	.97	.06
Positive child care practices	E	143	.82		116	.81		103	.85	.84	.81	.55	.09	.97	.06
Negative child care practices	C	141	.14		109	.12		97	.17	.15	.12	.57	.001	.22	.001	.05
Negative child care practices	E	144	.13		117	.13		104	.20	.14	.13	.57	.001	.22	.001	.05

^a Means of control and experimental groups

^b Test of hypothesis of equivalent group means

^c Test of hypothesis that group means, averaged over time, are equal

^d Test of hypothesis that means at three points in time, averaged over the groups, are equal

^e Test of hypothesis of no interaction between group and time, that is, that the pattern of means over time is the same for both groups

^f Test of hypothesis that time one is equal to average of time two and time three

^g Test of hypothesis that time two is equal to time three

Table 8-4.
New Jersey family and child functioning scales
	Post-Treatment				Follow-Up			Multivariate Repeated Measures
								Means				Univariate ps			Univariate ps-Time		Univariate ps-Grp-time interaction
		N	M^a	p^b	N	M	p	N	Initial	Post	Follow	Grp^c	Time^d	Time -Grp^e	Initial v. later^f	Post v. Follow^g	Initial v. later	Post v. Follow
Positive life events	C	133	.15		107	.23		83	.19	.16	.25	.05	.001	.99		.001
Positive life events	E	210	.13		166	.21		129	.17	.13	.22	.05	.001	.99		.001
Negative life events	C	133	.04		107	.09		83	.13	.04	.10	.46	.001	.49	.001	.001
Negative life events	E	210	.05		166	.11		129	.11	.04	.09	.46	.001	.49	.001	.001
Life events depression	C	133	.42		106	.47		83	.48	.38	.44	.93	.001	.33	.001
Life events depression	E	210	.39		165	.42		128	.52	.39	.41	.93	.001	.33	.001
Economic Functioning	C	132	.34	.02	107	.36		83	.39	.33	.33	.07	.06	.31	.06
Economic Functioning	E	209	.25	.02	167	.31		129	.29	.24	.30	.07	.06	.31	.06
Punishment	C	131	.25	.04	105	.21		80	.31	.25	.22	.03	.001	.93	.001	.03
Punishment	E	209	.20	.04	167	.18		129	.27	.20	.17	.03	.001	.93	.001	.03
Child aggression	C	134	1.68	.09	107	1.38		84	1.89	1.77	1.37	.05	.001	.26	.001	.001
Child aggression	E	210	1.48	.09	167	1.32		130	1.62	1.44	1.28	.05	.001	.26	.001	.001
School problems	C	118	.22		96	.34		69	.36	.26	.36	.009	.001	.84	.01	.003
School problems	E	197	.20		157	.28		121	.29	.19	.26	.009	.001	.84	.01	.003
Child withdrawn	C	134	.40		107	.41		84	.62	.42	.37	.38	.001	.51	.001
Child withdrawn	E	210	.40		167	.38		130	.50	.41	.33	.38	.001	.51	.001
Stolen things or arrested	C	134	.37		107	.42		84	.58	.35	.44	.60	.001	.99	.001	.05
Stolen things or arrested	E	210	.29		167	.48		130	.55	.31	.41	.60	.001	.99	.001	.05
Child substance abuse	C	134	.16		107	.20		84	.26	.13	.18	.94	.01	.71	.02	10	.
Child substance abuse	E	210	.17		167	.27		130	.23	.14	.22	.94	.01	.71	.02	10	.
Child problems	C	134	2.64		107	2.88		84	3.00	2.70	2.99	.57	.001	.45	.003	.09
Child problems	E	210	2.64		167	2.83		130	3.03	2.62	2.73	.57	.001	.45	.003	.09
Negative child behaviors	C	130	.33	.04	105	.34		81	.41	.34	.35	.005	.001	.84	.001	.001
Negative child behaviors	E	207	.28	.04	163	.32		126	.35	.28	.30	.005	.001	.84	.001	.001
Positive child behaviors	C	132	.74		106	.75		83	.73	.75	.77	.63	.01	.69	.02	.02
Positive child behaviors	E	208	.73		163	.76		128	.72	.73	.77	.63	.01	.69	.02	.02
Household condition	C	134	.06		107	.04		84	.06	.06	.03	.78	.05	.38	.04
Household condition	E	210	.05		167	.05		129	.06	.05	.04	.78	.05	.38	.04
Depression (SCL-90)	C	134	1.00	.08	105	.85		83	1.01	.84	.82	.92	.001	.71	.001
Depression (SCL-90)	E	209	.83	.08	166	.82		127	1.04	.89	.77	.92	.001	.71	.001
Positive child care practices	C	128	.76		103	.76		77	.79	.77	.77	.43	.11	.67	.06
Positive child care practices	E	206	.77		163	.80		124	.82	.78	.79	.43	.11	.67	.06
Negative child care practices	C	129	.18	.02	101	.14		76	.21	.18	.14	.07	.001	.34	.001	.06
Negative child care practices	E	207	.14	.02	162	.13		124	.19	.13	.13	.07	.001	.34	.001	.06

^a Means of control and experimental groups

^b Test of hypothesis of equivalent group means

^c Test of hypothesis that group means, averaged over time, are equal

^d Test of hypothesis that means at three points in time, averaged over the groups, are equal

^e Test of hypothesis of no interaction between group and time, that is, that the pattern of means over time is the same for both groups

^f Test of hypothesis that time one is equal to average of time two and time three

^g Test of hypothesis that time two is equal to time three

Table 8-5.
Tennessee family and child functioning scales
	Post-Treatment				Follow-Up			Multivariate Repeated Measures
								Means				Univariate ps			Univariate ps-Time		Univariate ps-Grp-time interaction
		N	M^a	p^b	N	M	p	N	Initial	Post	Follow	Grp^c	Time^d	Time -Grp^e	Initial v. later^f	Post v. Follow^g	Initial v. later	Post v. Follow
Positive life events	C	37	.14		36	.31		28	.20	.15	.32	.53	.001	.76		.001
Positive life events	E	80	.18		74	.31		61	.24	.19	.31	.53	.001	.76		.001
Negative life events	C	37	.05		36	.14		28	.14	.06	.15	.13	.001	.77	.01	.001
Negative life events	E	80	.05		74	.11		61	.10	.04	.12	.13	.001	.77	.01	.001
Life events depression	C	31	.36		36	.35		28	.49	.40	.34	.85	.001	.66	.001
Life events depression	E	80	.34		74	.34		61	.50	.37	.34	.85	.001	.66	.001
Economic Functioning	C	37	.25		36	.33		28	.43	.25	.31	.06	.001	.86	.001
Economic Functioning	E	80	.18		73	.22		59	.31	.16	.19	.06	.001	.86	.001
Punishment	C	37	.13		36	.10		28	.21	.12	.11	.45	.001	.11	.001	.09		.10
Punishment	E	76	.13		71	.07		54	.28	.15	.08	.45	.001	.11	.001	.09		.10
Child aggression	C	37	.81		36	.86		28	.89	.86	.86	.97	.11	.18	.09
Child aggression	E	80	.86		74	.59		61	1.05	.93	.61	.97	.11	.18	.09
School problems	C	32	.17		35	.20		22	.35	.25	.25	.11	.003	.65	.002
School problems	E	65	.15		63	.18		46	.23	.15	.20	.11	.003	.65	.002
Child withdrawn	C	37	.27		36	.28		28	.68	.25	.82	.94	.001	.10	.01
Child withdrawn	E	80	.38		74	.23		61	.59	.41	.23	.94	.001	.10	.01
Stolen things or arrested	C	37	.19		36	.47		28	.50	.25	.50	.27	.004	.66	.04	.007
Stolen things or arrested	E	80	.19		74	.34		61	.43	.18	.31	.27	.004	.66	.04	.007
Child substance abuse	C	37	.03		36	.03		28	.00	.04	.04	.44	.97	.31
Child substance abuse	E	80	.03		74	.04		61	.08	.03	.05	.44	.97	.31
Child problems	C	37	2.08		36	2.03		28	2.39	2.14	2.04	.49	.02	.70	.005
Child problems	E	80	1.72		74	1.70		61	2.33	1.84	1.80	.49	.02	.70	.005
Negative child behaviors	C	34	.21		35	.22		26	.33	.23	.23	.52	.001	.81	.001
Negative child behaviors	E	72	.21		71	.19		55	.30	.22	.20	.52	.001	.81	.001
Positive child behaviors	C	34	.81		36	.90	.07	26	.83	.80	.90	.69	.03	.26		.01
Positive child behaviors	E	72	.83		70	.86	.07	54	.81	.83	.85	.69	.03	.26		.01
Household condition	C	36	.07		34	.10		25	.13	.08	.14	.09	.28	.38
Household condition	E	78	.06		71	.06		58	.09	.07	.06	.09	.28	.38
Depression (SCL-90)	C	37	.76		36	.83		28	1.00	.81	.72	.89	.008	.64	.002
Depression (SCL-90)	E	80	.70		74	.73		61	1.03	.71	.73	.89	.008	.64	.002
Positive child care practices	C	36	.82		34	.96		24	.90	.89	.94	.70	.05	.71		.01
Positive child care practices	E	71	.88		67	.93		48	.90	.86	.95	.70	.05	.71		.01
Negative child care practices	C	35	.09		33	.07		25	.11	.09	.08	.38	.001	.09	.001		.03
Negative child care practices	E	72	.09		66	.06		50	.18	.09	.07	.38	.001	.09	.001		.03

^a Means of control and experimental groups

^b Test of hypothesis of equivalent group means

^c Test of hypothesis that group means, averaged over time, are equal

^d Test of hypothesis that means at three points in time, averaged over the groups, are equal

^e Test of hypothesis of no interaction between group and time, that is, that the pattern of means over time is the same for both groups

^f Test of hypothesis that the average initial score is equal to the average of post-treatment and final scores

^g Test of hypothesis that average post-treatment score is equal to the average final score

Figure 8-4. Child and Family Functioning over Time

Figure 8-4_2. Child and Family Functioning over Time Figure 8-4_3. Child and Family Functioning over Time Figure 8-4_4. Child and Family Functioning over Time Figure 8-4_5. Child and Family Functioning over Time Figure 8-4_6. Child and Family Functioning over Time

8.6.2 Problems

In the second and third interviews, caretakers were again asked questions about problems in the family. These questions paralleled those asked in the first interview, except this time caretakers were asked to respond to questions with regard to the time "since we last spoke to you" (seeTables 8-6 and 8-7). We analyze each of these problem items separately. Kentucky. At the post-treatment interview, on 8 of the 9 problem questions, there were no differences between the experimental and control groups in responses. On the question of whether the caretaker felt overwhelmed with work or family responsibilities, a greater proportion of caretakers in the experimental group responded affirmatively at post-treatment than did caretakers in the control group (47% vs. 37%, p = .08).¹³ In addition to the items about problems, caretakers were asked three questions about positive aspects of their lives: "gotten together with anyone to have fun or relax," "felt happy," and "felt that considering everything you're doing a pretty good job raising your kids." The primary analysis revealed that 89% of caretakers in the experimental group responded that they "felt happy" since the last interview as compared to 82% of caretakers in the control group (p = .09). For both the experimental and control groups, 64% responded affirmatively to the question of "getting together with anyone to have fun or relax," and over 90% responded affirmatively that they were "doing a pretty good job raising [their] kids." There were no significant differences between experimental and control groups on these last two items in either the primary or secondary analyses.

At the follow-up interview, there were no significant differences between experimental and control groups on any of these items.

Table 8-6.
Caretaker problems & strengths, caretaker post-treatment interview
(% responding yes)
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	%	N	%	p	N	%	N	%	p	N	%	N	%	p
Problems
Felt blue or depressed	145	41	147	37		133	44	210	48		37	46	80	40
Felt nervous or tense	145	46	147	46		131	49	210	44		37	41	80	40
Just wanted to give up	145	17	147	15		133	26	210	20		35	29	79	23
Overwhelmed with work or family responsibility	145	37	148	47	0.08	133	50	210	44		37	32	80	33
Felt you had few or no friends	145	14	147	18		133	20	209	18		37	38	79	19	0.03
Not enough money for food, rent, or clothing	145	39	148	44		133	52	210	40	0.03	37	46	80	43
Gotten in trouble with the law	144	3	147	3		133	2	210	1		37	3	80	0
Had too much to drink in a week	145	2	148	1		133	2	209	2		37	5	80	3
Used drugs several times a week	145	1	148	0		133	2	209	1		37	5	80	3
Economic Items
Had difficulty paying rent	141	13	144	20		133	29	209	18	0.02	37	24	80	15
Had difficulty paying electric/heat	142	20	144	28		132	30	209	26		37	27	80	21
Had difficulty buying enough food	145	15	146	17		132	28	209	22		37	22	80	11
Had difficulty buying clothes	145	17	146	21		132	47	208	33	0.01	37	27	80	24
Positive Items
Have you felt happy	145	82	147	89	0.09	131	82	210	81		37	84	80	93
Gotten together with anyone to have fun/relax	145	64	148	64		133	65	210	59		37	38	80	75	0.001
Doing a pretty good job raising kids	144	94	147	91		130	88	209	91		36	92	79	96

Table 8-7.
Caretaker problems & strengths, caretaker follow-up interview
(% responding yes)
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	%	N	%	p	N	%	N	%	p	N	%	N	%	p
Problems
Felt blue or depressed	119	44	130	42		107	55	166	48		36	44	74	47
Felt nervous or tense	119	47	130	53		105	53	166	47		36	31	73	38
Just wanted to give up	119	18	130	23		105	27	164	25		36	19	73	14
Overwhelmed with work or family responsibility	119	39	130	42		105	51	165	50		36	44	74	36
Felt you had few or no friends	119	24	130	25		107	15	166	24		36	8	74	19
Not enough money for food, rent, or clothing	118	44	130	46		106	46	166	45		36	42	74	49
Gotten in trouble with the law	119	6	130	4		107	1	166	4		36	3	74	3
Had too much to drink in a week	119	3	130	3		106	0	166	2		36	3	74	7
Used drugs several times a week	119	3	130	0		106	0	166	1		36	3	74	4
Economic Items
Had difficulty paying rent	118	20	127	20		107	34	167	27		36	39	74	20	.04
Had difficulty paying electric/heat	118	19	126	25		107	36	167	37		36	42	74	32
Had difficulty buying enough food	119	14	129	15		107	35	167	26		36	19	73	14
Had difficulty buying clothes	119	15	128	19		107	42	167	35		36	31	73	21
Positive Items
Have you felt happy	119	89	130	89		106	83	166	87		36	92	73	89
Gotten together with anyone to have fun/relax	119	73	130	69		107	57	166	64		36	61	74	69
Doing a pretty good job raising kids	112	91	123	93		104	90	166	92		36	92	74	96

NOTE: "FE" indicates significance determined by Fisher's exact test

New Jersey. On 8 of the 9 problem questions, there were no significant differences between the experimental and control groups in responses at post-treatment. On the overall question about the economic condition of family, "have you felt you just didn't have enough money for food, rent, or clothing?" 52% of the control group said yes, compared to 40% of the experimental group significantly different at p = .03, although in the secondary analysis the difference was not significant.¹⁴ On none of the positive questions were there significant differences between groups in either the primary or secondary analyses.

At follow-up differences between groups approached significance for only one item, "felt you had few or no friends." More experimental group respondents replied affirmatively to this item (24% vs. 15% p = .07).

Tennessee. There was a significant difference between groups on only one of the nine problem questions at post-treatment. Fewer experimental group respondents reported they felt they had few or no friends (19% vs. 38%, p = .03). As to positive items, experimental group respondents far more often reported they "had gotten together with someone to have fun or relax" (75% vs. 38%, p < .001). At follow-up there were no differences between groups on any of these items.

(13) This difference was slightly greater and statistically significant in the secondary analysis (48% vs. 35%, p= .04).

(14) In the secondary analysis, fewer experimental group respondents reported health problems (12% vs. 21% for the control group, p = .04).

8.6.3 Economic Functioning

In addition to the general item in the problem inventory on not having enough money for food, rent, or clothing, we asked four specific questions about difficulties in paying for the essentials of living (rent, electric service and heating, food, and clothes) (see Tables 8-6 and 8-7). These items were examined individually and were combined into an overall economic functioning scale. (See Tables 8-3, 8-4, and 8-5 for analyses of the scale).

Kentucky. On the scale, primary and secondary analyses revealed no significant differences in the average proportion of affirmative responses to the four items either at post-treatment or at follow-up.¹⁵¹⁶ There also were no differences between groups in changes over time.

New Jersey. On the economic functioning scale, the experimental group had a lower average proportion of affirmative responses to these items at post-treatment (.25 vs. .34, p = .02) although the difference was not significant in the secondary analysis. The difference was not significant at follow-up nor were there significant differences in change over time. There were, however, significant differences on two of the specific items at post-treatment. Control group respondents more often reported difficulties paying rent (29% vs. 18%, p = .02) and also more often reported difficulties in buying clothes (47% vs. 33%, p = .01). These differences were not significant in the secondary analysis. At follow-up, there continued to be a difference in regard to buying food, although it was not significant (26% of the experimental group vs. 35% of the control group p = .12). At follow-up, the groups were similar on the other three items.

Tennessee. Control group respondents more often reported problems on the economic functioning scale at both post-treatment and follow-up, although the differences were not significant. There were also no significant differences between the groups in change over time. On individual items, there were no significant differences between groups at the post-treatment interview. At follow-up, significantly fewer experimental group respondents reported difficulties paying rent (20% vs. 39%, p = .04). There were no significant differences on the other three items at follow-up.

(15) The control group had a slightly lower average proportion of affirmative responses to these items at post-treatment (.17 vs. .22, p = .16).

(16) In the primary analysis, at post-treatment, a greater proportion of the experimental group reported difficulties paying rent (20% vs 13%, p= .13) and electric or heat bills (28% vs. 20%, p = .11). In the secondary analysis, differences were smaller and p-values for both items were above .20.

8.6.4 Household Condition

Caretakers were asked 10 questions about problematic conditions in the home (e.g., nonfunctioning heating, plumbing, or electrical systems; peeling paint; broken windows or doors).

Kentucky. In Kentucky, the experimental and control groups did not differ on the average proportions of the presence of such conditions at post-treatment or follow-up, nor did they differ on change in these proportions over time (see Table 8-3). On only one of the specific items were there any differences at post-treatment. Five percent of caretakers in the control group reported having broken windows or doors that were not fixed as compared to 1 percent in the experimental group (Fisher's exact p-value = .034).¹⁷ There were no significant differences on any of the individual items at follow-up.

New Jersey. In New Jersey the experimental and control groups did not differ on the average proportions of the presence of such conditions at post-treatment or follow-up, nor did they differ on change in these proportions between interviews (see Table 8-4). There were no significant differences between the groups on any of the ten individual items at post-treatment. At follow-up, 8 percent of the experimental group respondents reported "a lot of peeling paint" compared to 2 percent of the control group respondents (p = .04). Also, 4 percent of the experimental group and none of the control group respondents reported that cooking appliances did not work (Fisher's exact p-value = .09).

Tennessee. There were no significant differences between experimental and control groups on the overall scales of household condition at either post-treatment or follow-up, nor were there differences in change over time (see Table 8-5). At post-treatment, more control group respondents reported bare electrical wires (8% vs. 0%, p = .08; Fisher's exact p-value = .03) while more experimental group respondents reported living in an "unsafe building because of illegal acts (5% vs. 0%, p = .05). There were no significant differences on any individual items at follow-up.

(17) This difference was maintained but not significant in the secondary analysis (5% vs. 1%, Fisher's exact p-value = .077).

8.6.5 Child Care Practices

Caretakers were asked a series of yes-no questions about child care practices at the end of treatment and in the last three months at follow-up (both positive and negative). The results from these questions are shown in Table 8-8 and 8-9. In addition, three scales were formed using these items: positive child care practices (5 items), negative child care practices (10 items), and punishment (5 items, all of which are also included in the negative child care practices scale). Results from the scales are shown in Tables 8-3, 8-4 and 8-5.

Table 8-8.
Caretaker reports of child care practices, post-treatment interview
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	%	N	%	p	N	%	N	%	p	N	%	N	%	p
Lost temper when child got on nerves	145	43	147	46		132	63	209	59		37	27	77	35
Found that hitting child was good	143	6	147	7		131	12	208	8		37	19	76	8	0.09
Hitting child harder that meant to	143	8	147	6		131	10	208	5		37	5	75	8
Out of control when punishing child	144	24	146	24		131	40	209	30	0.05	36	11	76	12
Have you praised your children	144	94	146	92		132	92	209	92		37	92	76	91
Listened to music together w/child	144	86	146	86		132	86	209	82		37	92	77	90
Tied child with cord- string-belt	142	0	146	0		132	1	209	0		37	0	76	1
Gone to amusement park, pool, picnic	143	71	145	65		130	46	208	37	0.08	37	43	76	68	0.01
Uncomfortable hugging child	126	5	134	6		95	5	144	6		33	9	72	4
Encouraged child to read book	137	92	140	90		126	82	202	91	0.02	34	94	72	96
Have children handled household chores	127	79	125	80		126	75	200	83	0.09	34	91	72	93
Not let children into the house	138	2	137	2		127	8	202	4		33	3	72	1
Punished for not finishing food	139	7	139	1	0.02	126	6	204	5		33	0	71	0
Blamed child w/ things not their fault	142	39	142	33		128	28	204	20	0.07	35	31	75	44
Let child to play where not allowed	138	2	138	4		123	1	200	4		33	0	71	1
Unable to find someone to watch children	144	9	146	12		131	21	206	12	0.04	35	20	73	27

NOTE: "FE" indicates significance determined by Fisher's exact test

Table 8-9.
Caretaker reports of child care practices, follow-up interview
	Kentucky					New Jersey					Tennessee
	Control		Experimental			Control		Experimental			Control		Experimental
	N	%	N	%	p	N	%	N	%	p	N	%	N	%	p
Lost temper when child got on nerves	113	44	121	41		106	60	167	57		36	22	72	18
Found that hitting child was good	113	6	121	4		105	5	167	4		36	8	71	6
Hitting child harder that meant to	113	3	121	5		105	6	167	1	.06	36	6	71	3
Out of control when punishing child	112	21	121	23		105	32	167	25		36	14	71	7
Have you praised your children	113	89	121	91		106	92	167	94		36	97	72	93
Listened to music together w/child	113	82	121	84		106	81	167	86		36	94	69	93
Tied child with cord- string-belt	113	0	121	2		106	1	167	1		36	0	70	0
Gone to amusement park, pool, picnic	113	74	121	70		106	48	167	44		35	94	70	90
Uncomfortable hugging child	102	4	110	5		74	5	116	6		35	6	66	3
Encouraged child to read book	107	83	115	81		103	87	160	89		34	100	67	96
Have children handled household chores	101	75	104	75		103	70	160	83	.02	34	94	63	89
Not let children into the house	106	0	115	0		103	3	162	3		33	3	64	3
Punished for not finishing food	109	3	116	2		103	2	162	2		33	3	65	3
Blamed child w/ things not their fault	109	34	116	42		103	21	162	23		33	15	66	14
Let child to play where not allowed	109	1	114	0		101	2	160	1		33	0	65	3
Unable to find someone to watch children	111	3	122	0		103	17	165	18		33	21	67	19

Kentucky. In the post-treatment interview, "punishment for not finishing food" was the only item for which there were significant differences between the experimental and control groups, with a greater proportion of the control group responding affirmatively. There were no items on which there were significant differences at follow-up.

There were no significant differences between the experimental and control groups with regard to the positive and negative child care practice scales at the time of the second or third interviews. Both groups responded affirmatively to over 80 percent of the positive items and less than 15 percent of the negative items at both points in time. There were also no significant differences in change over time, in both groups there was a decline in the number of negative child care practices over time. With regard to the 5 items that pertain to punishment, caretakers in the experimental group responded affirmatively to a greater average proportion of punishment items than did the caretakers from the control group (.25 vs .20, p = .067) at the initial interview. At post-treatment, the average proportion of punishment items answered affirmatively were nearly the same (.17 for experimental group vs. .16 for the control group). Thus, from the first interview to the second, the reduction in the average proportion of punishment items endorsed was greater for the experimental group than for the control group (.09 fewer vs. .04 fewer, p= .054).¹⁸ However, across the three points in time, there were no significant differences in change.

New Jersey. On two of the items there were significant differences between the experimental and control groups at post-treatment: "have things sometimes gotten out of control when you punished your children?" happened more often in the control group and "have you encouraged your child to read a book?" which was done more often by experimental group respondents. At follow-up, there were two other items with differences: more control group respondents said they hit their child harder than they meant to (6% vs. 1%, p = .06) while more experimental group respondents said they had the children handle household chores (83% vs. 70%, p = .02).

On the scales, there were no significant differences between the experimental and control groups with respect to the positive child care practice items at either post-treatment or follow-up. At both points in time, both groups responded affirmatively to over 75 percent of the items. There were significant differences between the experimental and control groups with respect to the negative child care practice items at post-treatment. Caretakers in the experimental group responded affirmatively to 14 percent of the items whereas caretakers in the control group responded affirmatively to 18 percent of the items (p = .02). The difference disappeared at follow-up. At post-treatment, the experimental group significantly less often used punishment (.20 vs. .25, p = .04), but the difference was not statistically significant in the secondary analysis (p = .08) or in the primary analysis at follow-up (p = .15). There were no significant differences between groups in the change in changes over time in proportion of negative child care practice or punishment items.

Tennessee. At post-treatment, on one item there was a significant difference between experimental and control groups; more experimental group respondents indicated they had gone to an amusement park, pool, or picnic (68% vs. 43%, p = .01). There were no items with significant differences at follow-up. There were no significant differences in any of the three scales at either post-treatment or follow-up, nor in changes over time. The experimental group had a higher negative child care practices score at the first interview, and declined more than the control group, resulting in a nearly significant multivariate time-group interaction (p = .09) and a significant univariate comparison of the first interview score with the average of the scores from the later two interviews (p = .03).

(18) In the secondary analysis, there was again a .09 reduction in the average proportion of punishment items endorsed by the experimental group and a .04 reduction for the control group (p = .03).

8.6.6 Caretaker Depression

In both the initial and post-treatment interviews we administered the SCL-90¹⁹ depression scale to measure the level of depression of the caretaker.²⁰ In none of the three states were there significant differences between the groups in scores on this scale at the post-treatment or follow-up interviews or in changes over time (see Tables 8-3, 8-4, and 8-5). Scores at post-treatment were, on average, less than those in the initial interview for both groups and the reduction was greater for the experimental group (Kentucky: .23 less for the experimental group and .14 less for the control group; New Jersey: .19 less for the experimental group, .09 less for the control group; Tennessee: .32 less for the experimental group, .19 less for the control group), though the differences were not statistically significant.

(19) Derogatis, L. R., Lipman, R. S., & Covi, L. (1973) SCL-90: An outpatient psychiatric rating scale -- preliminary report.Psychopharmacology Bulletin, 9 (1), 13 - 28

(20) Reliability analysis yielded a Cronbach's alpha of .92 at time one and .93 at time two in Kentucky and .95 at time one and .94 at time two in New Jersey.

8.6.7 Child Behavior

In all three interviews, we asked 35 questions about specific child behaviors, both positive and negative. Questions were phrased in terms of "any of the children" and some questions were age specific. Responses to these questions were used to form various scales: aggression (3 items), school problems (5 items), positive child behaviors (10 items), and negative child behaviors (21 items, including the aggression and school problems items). Analyses of these scales are shown in Tables 8-3, 8-4, and 8-5.

Kentucky. Neither the primary nor the secondary analyses revealed any significant differences between the groups in scores on any of these scales at post-treatment or at follow-up or in the change over time. Specific items on whether the child was withdrawn, had stolen things or been arrested, or had engaged in substance abuse did not reveal significant differences between groups at post-treatment or follow-up. For the scale of having stolen things or been arrested, the experimental group scored higher at the initial interview and declined between the first and second interview, while the control group increased between the first and second interviews and again at the follow-up interview. As a result, the multivariate interaction between time and group was nearly significant (p = .07) and the univariate difference between the groups in the difference between the first interview and the average of the later two interviews was significant at p = .03.

New Jersey. In the primary analysis there were no significant differences between groups on these scales at post-treatment or at follow-up, except for the overall negative child behaviors scale at post-treatment, on which the experimental group was lower (an average of 28% of the items vs. 33%, p = .04).²¹ For none of these scales was there a significant difference between groups on change over time.

Specific items on whether the child was withdrawn, had stolen things or been arrested, or had engaged in substance abuse did not reveal significant differences between groups at post-treatment or follow-up or in change over time.

Tennessee. There were no significant differences between the experimental and control groups in the average scores on these scales at post-treatment or at follow-up, nor were there significant differences in change over time. The difference for positive child behaviors at follow-up was nearly significant, with the control group scoring higher (p = .07). The specific items on whether the child was withdrawn, had stolen things or been arrested, or had engaged in substance abuse did not reveal significant differences between groups at post-treatment or follow-up or in change over time.

(21) This difference was also significant for the secondary analysis (28% vs. 33%, p = .006).

8.7 Overall Assessment of Improvement by Caretakers

In the post-treatment interview, caretakers were asked about general changes in their families' lives since entering the study. Results are shown in Table 8-10 and 8-11. At post-treatment, in Kentucky and New Jersey, relative to control group caretakers, a significantly larger proportion of experimental group caretakers generally thought there was "great improvement" in their lives. This difference was significant in both the primary and secondary analyses. In the Tennessee secondary analysis, results tended in the same direction, though not significantly (p = .09). At follow-up, differences between the groups in Kentucky and New Jersey had nearly disappeared. In Tennessee, control group respondents more often thought there was "great improvement," although it was not a significant difference.

Table 8-10.
Caretakers' assessments of overall change since first interview, post-treatment interview
	Kentucky		New Jersey		Tennessee
	Control %	Experimental %	Control %	Experimental %	Control %	Experimental %
Primary analysis:	p = .02		p = .001		p = n.s.
Great Improvement	16	22	9	16	32	32
Some Improvement	31	42	41	52	32	42
Same	42	29	34	20	22	14
Somewhat or a great deal worse	12	6	16	12	14	13

Table 8-11.
Caretakers' assessments of overall change since post-treatment interview, follow-up interview
	Kentucky		New Jersey		Tennessee
	Control %	Experimental %	Control %	Experimental %	Control %	Experimental %
Primary analysis:	p = n.s.		p = n.s.		p = n.s.
Great Improvement	34	36	30	28	53	36
Some Improvement	37	38	36	42	31	41
Same	18	16	17	16	8	15
Somewhat or a great deal worse	11	9	16	13	3	7

8.8 Information from Caseworkers on Family and Child Functioning

The caseworker interviews also contained questions regarding child and family functioning, in an effort to provide another perspective on these issues. In interpreting caseworker reports, it should be noted that experimental group caseworkers were Homebuilders workers, while control group respondents were the public agency workers responsible for the cases at the time of the interview. It is likely that there are differences between these groups of caseworkers in the knowledge they have of the cases, since Homebuilders workers had much more intensive involvement and that involvement began before the first research interview. In addition, it may be that there are systematic differences in these groups of workers in the approaches they take to the assessment of family problems. Hence, interpretations of comparisons between responses of workers serving each of the groups must be made with caution.

8.8.1 Caretaker Functioning

Caseworkers were asked nine questions tapping various aspects of caretaker functioning on a five-point scale from 0 for not adequate to 4 for very adequate. Table 8-12 provides a list of these nine questions and a summary of the results from the initial and post-treatment interviews.

Table 8-12.
Caseworkers' assessments of caretakers' parental functioning
Kentucky
	Control		Experimental		p
	N	Mean	N	Mean	p
Initial:
Caretaker ability to provide food	130	2.68	114	2.96	0.02
Caretaker ability giving affection	132	2.63	125	2.82
Caretaker respect for child's opinions	119	2.38	106	2.58
Respond patiently to child's questions	122	2.16	110	2.44	0.06
Respond to child's emotional needs	137	2.15	122	2.35
Provide learning opportunities	127	2.17	110	2.35
Setting firm/consistent limits/rules	130	1.68	116	1.88
Adequate supervisor/responsible childcare	140	2.14	123	2.39	0.10
Attending to children's health needs	135	2.76	114	3.00	0.08
Caretaker functioning, 9 items, average of nonmissing items, higher = better	118	2.25	102	2.48	0.06
Post-treatment:
Caretaker ability to provide food	145	2.88	154	2.97
Caretaker ability giving affection	147	2.82	157	2.81
Caretaker respect for child's opinions	135	2.58	144	2.45
Respond patiently to child's questions	138	2.43	148	2.34
Respond to child's emotional needs	145	2.28	156	2.28
Provide learning opportunities	144	2.38	154	2.42
Setting firm/consistent limits/rules	145	2.09	150	1.99
Adequate supervisor/responsible childcare	152	2.50	158	2.59
Attending to children's health needs	150	2.93	157	3.08
Caretaker functioning, 9 items, average of nonmissing items, higher=better	142	2.56	151	2.55
New Jersey
Initial:
Caretaker ability to provide food	119	3.24	224	3.20
Caretaker ability giving affection	120	2.88	229	2.62	0.03
Caretaker respect for child's opinions	118	2.42	219	2.32
Respond patiently to child's questions	117	2.44	220	2.27
Respond to child's emotional needs	118	2.37	228	2.23
Provide learning opportunities	114	2.83	220	2.50	0.005
Setting firm/consistent limits/rules	126	2.11	228	1.93
Adequate supervisor/responsible childcare	130	2.80	238	2.71
Attending to children's health needs	125	3.34	214	3.17
Caretaker functioning, 9 items, average of nonmissing items, higher = better	107	2.65	211	2.44	0.02
Post-treatment:
Caretaker ability to provide food	137	3.36	246	3.34
Caretaker ability giving affection	141	2.93	256	2.70	0.04
Caretaker respect for child's opinions	130	2.55	247	2.42
Respond patiently to child's questions	140	2.51	248	2.37
Respond to child's emotional needs	149	2.43	258	2.37
Provide learning opportunities	137	2.89	247	2.60	0.01
Setting firm/consistent limits/rules	147	2.37	252	2.14	0.06
Adequate supervisor/responsible childcare	149	2.95	258	2.79
Attending to children's health needs	148	3.35	252	3.25
Caretaker functioning, 9 items, average of nonmissing items, higher=better	140	2.79	249	2.66	0.10
Tennessee
Initial:
Caretaker ability to provide food	38	2.79	53	3.11
Caretaker ability giving affection	42	2.76	60	2.92
Caretaker respect for child's opinions	34	2.23	52	2.77	0.01
Respond patiently to child's questions	32	2.22	53	2.57
Respond to child's emotional needs	40	2.05	59	2.47	0.04
Provide learning opportunities	39	2.64	56	2.55
Setting firm/consistent limits/rules	36	2.33	57	2.01
Adequate supervisor/responsible childcare	44	2.32	61	2.95	0.005
Attending to children's health needs	43	2.65	59	3.18	0.03
Caretaker functioning, 9 items, average of nonmissing items, higher = better	30	2.53	51	2.60
Post-treatment:
Caretaker ability to provide food	41	2.98	74	3.32	0.06
Caretaker ability giving affection	45	2.73	80	2.95
Caretaker respect for child's opinions	40	2.35	74	2.84	0.01
Respond patiently to child's questions	38	2.26	76	2.67	0.04
Respond to child's emotional needs	42	2.26	81	2.59	0.06
Provide learning opportunities	44	2.64	78	2.64
Setting firm/consistent limits/rules	43	2.04	79	2.38
Adequate supervisor/responsible childcare	46	2.52	82	2.93	0.04
Attending to children's health needs	45	2.96	78	3.13
Caretaker functioning, 9 items, average of nonmissing items, higher=better	42	2.51	77	2.82	0.04

Note: Scale for individual items: 0-4, where 0 = not adequate, 4 = very adequate

Kentucky. At the initial interview, significant or nearly significant differences were found on three items, with the experimental group scoring more adequate on average: ability to provide food (p = .02), responding patiently to child's questions (p = .06), and attending to children's health needs (p = .08). On a scale averaging the nine ratings for each case, the difference between means of the experimental and control groups approached significance, with the experimental group having a higher mean (p = .06). At post-treatment there were no significant differences in the primary analysis. However, in the secondary analysis, caretakers from the experimental group were rated higher (more adequate) than those from the control group with respect to whether they attended to the children's health needs (p = .04). As for the scale averaging the nine ratings, no differences were found between the experimental and control groups at post-treatment. Looking at change over time, on one item, respecting child's opinions, the ratings for the control group increased over time (.19 change), whereas the ratings for the experimental group decreased slightly over time (-.06 change), a difference that is significant (p = .05). The differences between groups in change on the overall scale averaging the nine ratings was not significant.

New Jersey. At the initial interview, on two items there were significant differences between the experimental and control groups, the control group scoring more adequate on average: caretaker's ability in giving affection (p = .03) and the caretaker's ability to provide learning opportunities (p = .005). On the scale averaging the nine ratings for each case, there was a significant difference between means of the experimental and control groups, the control group having a higher mean (p = .02). At post-treatment, the control group scored higher (more adequate functioning) on the same two items as before. On the scale of nine items the control group scored slightly higher, although the difference was nonsignificant. As to change over time, on one item ("respecting child's opinions"), the control group had, on average, more positive change than the experimental group. The difference in degree of change was significant at .05 (this result also held in the secondary analysis, p = .05). Differences between groups in change on the overall scale were not significant.

Tennessee. At the initial interview there were four items on which the groups were significantly different, the experimental group scoring higher on all four: caretaker respect for child's opinions (p = .01), response to child's emotional needs (p = .04), adequate supervision (p = .005), and attending to the child's health needs (p = .03). At post-treatment, five items had differences between groups significant at .06 or lower, all favoring the experimental group: caretaker ability to provide food, respect for child's opinions, response to child's emotional needs, adequate supervision, and respond patiently to child's questions. The average of all nine items was also significantly different for the groups. On one item, setting firm and consistent limits, there was a significant difference in the amount of change over time, the experimental group increased by an average of .31, while the control group declined by an average of .29 (p = .01). On the scale of nine items there was no significant difference between the groups in change over time.

8.8.2 Household Condition

As in the caretaker interview, we asked caseworkers about conditions in the home. Caseworkers were asked 13 yes-no questions, some positive and some negative. These items were combined in a scale which indicated that in Kentucky and New Jersey at post-treatment, control group families had, on average, a significantly better household condition than did experimental group families (Kentucky: p = .014; New Jersey: p = .02). In both states, for both groups the analysis of change over time indicated a slight improvement in the condition of the household. The difference between the experimental and control groups in change over time was not significant in either state. In Tennessee, there was no difference between groups at post-treatment or in change over time (both groups declined by .01).

8.8.3 Caretaker Problems

Caseworkers were asked a number of questions about problems experienced by children, caretakers, or other adult household members (question 19 on the initial caseworker interview, question 17 on the post-treatment caseworker interview). Twenty-one of these problems concerned the caretakers.

Kentucky. At post-treatment, in the primary analysis, caseworkers reported that the experimental group caretakers had, on average, 31 percent of the problems compared to 25 percent for the control group, a difference significant at p = .0005.²² There were no significant differences in change in caretaker problems between the interviews in either the primary or secondary analyses.

New Jersey. At the post-treatment interview, on average, in the primary analysis caseworkers reported that experimental group caretakers had 23 percent of the problems compared to 21 percent of the control group, a nonsignificant difference.²³ There were no significant differences in change in caretaker problems between the interviews in either the primary or secondary analyses.

Tennessee. At post-treatment, caseworkers reported that experimental group caretakers had 18 percent of the problems compared to 21 percent of the control group, a nonsignificant difference. There was a significant difference between the groups in change over time, the experimental group improving more than the control group (-.08 vs. -.03, p = .05).

(22) In the secondary analysis, the difference was maintained and remained significant (31% vs. 24%, p = .0004).

(23) In the secondary analysis, the average percents were 24% for the experimental group and 21% for the control group (p = .06).

8.8.4 Child Problems

Twelve of the items on the caseworker problem inventory concerned the children. In Kentucky at post-treatment, the percentage of child problems for the experimental group was, on average, 27 percent compared to an average of 25 percent for the control group, a nonsignificant difference.²⁴ There were no significant differences in change in child problems between interviews in either the primary or secondary analyses.

In New Jersey at post-treatment, the average of the percentages of child problems was 25 percent for the experimental group and 27 percent for the control group, a nonsignificant difference.²⁵ There were no significant differences in change in child problems between interviews in either the primary or secondary analyses.

In Tennessee, the average percentages of child problems in the two groups at post-treatment were very close (18% for the control group, 19% for the experimental group). The difference between the groups in change over time was not significant.

(24) In the secondary analysis, however, the difference increased and approached significance with 29% for the experimental group and 24% for the control group, p = .06.

(25) The difference for the secondary analysis was also not significant (25% vs. 28%, p = .12).

8.9 Predictors of Outcomes

We performed regression analyses on a number of family functioning outcomes measured at the post-treatment interview and at follow-up. The analyses were intended to control for the effects of a number of variables, thereby providing more sensitive tests of the effects of family preservation, and to examine the effects of the variables on the outcomes. The dependent variables in these analyses were some of the scales of functioning discussed above: caretaker depression, child aggression, punishment, child school problems, difficulty paying bills, positive life events, negative life events, positive child behaviors, negative child behaviors, household condition, positive child care practices, and negative child care practices. Independent variables in these analyses were assignment group (experimental or control), caretaker's age, caretaker's race, family composition, caretaker's educational attainment, caretaker's employment status, residential stability, use of income support programs, caretaker's history of abuse and/or neglect, regular access to an automobile, and time to interview (days between random assignment and post-treatment/follow-up interview). The analyses also included the initial scores for the dependent variable, thereby controlling the level at post-treatment or follow-up for the initial value. Interactions between control variables and experimental group were also examined, only a few were found to be significant.²⁶

Caretaker's age, caretaker's race, family composition, caretaker's educational attainment, caretaker's employment status, use of income support programs, caretaker's history of abuse and/or neglect have all been examined in previous studies of outcomes in child welfare and have often been found to be predictive. Residential stability and regular access to a car have been less often examined. Since transportation and housing assistance are commonly provided in family preservation service models, the inclusion of such variables seems justifiable. Moreover, prior research does support a relationship between residential stability and major depression²⁷ and child adjustment.²⁸ Similarly, transportation (or lack there of) has been found to be related to participation in social programs²⁹ and family functioning.³⁰ We included time to interview because of the fact that that varied considerably and might have affected the degree of change that we were observing.

Regression analyses were conducted at the family level for both the post-treatment and follow-up measures. The coefficients are displayed inTable 8-13 and 8-14. All of the coefficients are shown for the initial measure of the outcome variable and for experimental group. Coefficients for other variables are shown if they were significant at p = .1 or lower. Most of the analyses are ordinary least squares regressions, logistic regressions were used for dichotomous or highly skewed variables. Generally, the initial measure was the strongest predictor. Although the size of these coefficients decreased between the post-treatment and follow-up interview, the majority of such coefficients remained significant. The positive direction of the coefficients indicates that caretakers with higher initial values also had higher post-treatment and follow-up values.

Table 8-13.
Regressions of post-treatment family and child functioning scales
(Regression Coefficients)
Dependent Measures	Initial Measure	Experim. Group Assignment	Care- taker Age	Single Mother	Ethnic Minority	Abuse Neglect History	Education	Employ- ment	Income Support	Access to Car	Housing Stability	Time to Interview
Caretaker Depression
Kentucky²	.596**	-.053	.083			.144**						.078
New Jersey²	.690**	-.202**		-.048		.102*		-.170**
Tennessee²	.606**	-.064
Child Aggression
Kentucky	.522**	-.051
New Jersey	.589**	-.044				.101*		-.079
Tennessee	.533**	-.004
Punishment
Kentucky¹	9.81**	1.16					2.02*
New Jersey	.529**	-.079
Tennessee	.281**	-.012
Child School Problems
Kentucky	.539**	-.020	.111
New Jersey	.381**	-.041						.107
Tennessee	.654**	-.073
Difficulty Paying Bills
Kentucky	.608**	.023
New Jersey	.632**	-.061					-.080			-.088	.107*
Tennessee	.513**	-.020		.155							.142
Positive Life Events
Kentucky	.218**	.055					.160**	.124*
New Jersey	.330**	-.074			-.108*			.104				-.139*
Tennessee	.330**	.020
Negative Life Events
Kentucky¹	1.85*	.833				2.69**
New Jersey	.278**	.008			.132*			.097
Tennessee	.100	-.768**					-.290**		-.707**
Positive Child Behaviors
Kentucky	.489**	-.002	-.089					.120*
New Jersey	.579**	-.032
Tennessee	.525**	.098	-.187*
Negative Child Behaviors
Kentucky	.592**	-.012
New Jersey	.581**	-.078
Tennessee	.647**	-.016
Household Condition
Kentucky¹	9.01**	.961					.414*					.948
New Jersey¹	5.66**	.744							2.01*
Tennessee	.429**	-.004
Positive Child Care
Kentucky	.401**	-.041		.113*						.128*
New Jersey	.566**	.007			-.081
Tennessee	.575**	.069			.268**
Negative Child Care
Kentucky	.569**	-.078							-.115*
New Jersey	.571**	-.119*	.083	.116*
Tennessee	.371**	-.136										-.177

¹ Logistic regression, Exp (B) displayed

² Depression scores transformed using log transformation

* p < .05, ** p < .01. All coefficients for experimental group assignment and initial measure are shown, regardless of significance. All other entries without stars are significant at 0.10.

Table 8-14.
Regressions of follow-up family and child functioning scales (Regression Coefficients)
Dependent Measures	Initial Measure	Experim. Group Assignment	Care- taker Age	Single Mother	Ethnic Minority	Abuse Neglect History	Education	Employ- ment	Income Support	Access to Car	Housing Stability	Time to Interview
Caretaker Depression
Kentucky²	.552**	.068	.116*
New Jersey²	.518**	-.051					-.124*
Tennessee²	.443**	-.011
Child Aggression
Kentucky	.363**	.022
New Jersey	.417**	.031	-.120*
Tennessee	.347**	-.226*										.173
Punishment
Kentucky¹	6.67**	.750										.977*
New Jersey	.288**	-.085			-.126		-.108
Tennessee	.271*	-.146										.293**
Child School Problems
Kentucky	.254**	-.147*	.253**					.180**
New Jersey	.272**	-.098
Tennessee	.451**	-.002					.290**
Difficulty Paying Bills
Kentucky	.396**	-.016					.127*
New Jersey	.537**	-.001				.108*	-.108*
Tennessee	.290**	-.132
Positive Life Events
Kentucky	.260**	-.044			.117		.227**
New Jersey	.081	-.049	-.165*
Tennessee	.261*	-.081						-.198		.304*
Negative Life Events
Kentucky¹	1.71*	1.18
New Jersey	.356**	.023			-.176**
Tennessee	.116	-.127				.272*						.236*

Positive Child Behaviors
Kentucky	.250**	.028					.182**
New Jersey	.294**	-.002					.165*
Tennessee	.192	-.064	-.231*								.301**
Negative Child Behaviors
Kentucky	.385**	-.058						.137*
New Jersey	.404**	-.016			-.121*
Tennessee	.344**	-.097										.200
Household Condition
Kentucky¹	3.86*	1.38		.271
New Jersey¹	3.59**	1.40							2.51*
Tennessee	.045	-1.24**	-.317									.207
Positive Child Care
Kentucky	.370**	.014	-.193**					.133*
New Jersey	.164*	.044	-.200**
Tennessee	.110	-.056
Negative Child Care
Kentucky	.340**	.020
New Jersey	.311**	-.050			-.135*			-.143*
Tennessee	.195	-.085										.296*

¹ Logistic Regression, Exp (B) displayed

² Depression scores transformed using log transformation

* p < .05, ** p < .01. All coefficients for experimental group assignment and initial measure are shown, regardless of significance. All other entries without stars are significant at .1.

In regard to the post-treatment analyses, experimental group families generally had better outcomes, but the differences were significant in only three analyses. In New Jersey, the experimental group had lower depression scores and lower negative child care practices than the control group when controlled for the other independent variables. In the analysis without controlling for the other variables, the result for depression was in the same direction, but not significant (p = .08). The result for negative child care practices without the control variables was also in the same direction and significant (p = .02). In Tennessee the experimental group had fewer negative life events in the regression analysis. The difference between groups in the uncontrolled analysis was not significant.³¹ Three differences significantly in favor of the experimental group in the uncontrolled analyses were no longer significant in the regression analysis, all in New Jersey: caretaker use of punishment, negative life events, and positive child behaviors.

At the follow-up interview, the regression analysis indicates that family preservation clients had lower levels of child aggression in Tennessee, fewer school problems in Kentucky, and fewer problematic conditions in the home in Tennessee. There were no significant differences between groups in the uncontrolled comparisons.

Regarding the remaining independent variables, there was little consistency in whether or not a variable had an effect and even in the direction of the effect. The following discussion focuses on those variables significant at p = .05 or lower. At post-treatment, the variables that most often showed effects were education and the caretaker having a history of being maltreated. Caretaker education was related to three post-treatment outcomes in Kentucky. More education was associated with more punishment, more positive life events, and worse household condition. In Tennessee more education was related to fewer negative life events. In Kentucky, having a history of maltreatment was related to higher depression and more negative life events. In New Jersey, history of maltreatment was related to higher depression and children being more aggressive. Income support, ethnic minority, and caretaker employment all were predictors in 3 of the 36 post-treatment regressions. Time to interview was significant in only one of the regressions.

At follow-up, time to interview emerged as a predictor in 4 of the 36 regression equations, in all cases related to an increase (worsening). Other variables often related to outcome were caretaker age and education. In New Jersey, older caretakers had fewer positive life events and had children who were less aggressive. In Tennessee, older caretakers had children with fewer positive behaviors. In Kentucky, older caretakers had higher depression scores, less often engaged in positive child care practices, and had children with more school problems.

In New Jersey at follow-up, caretaker education was related to 3 outcomes. More education was related to lower depression, less difficulty paying bills, and more positive child behaviors. In Kentucky, more education was related to more difficulty paying bills, more positive life events, and more positive child behaviors. More education in Tennessee is related to more child school problems. In Kentucky, caretaker's employment is related to more negative child behaviors, more child school problems, and more positive child care practices. Caretaker employment in New Jersey is associated with fewer negative child care practices. Ethnic minority caretakers in New Jersey had fewer negative life events, engaged in fewer negative child care practices, and had children with fewer negative behaviors.

The 72 regression equations for post-treatment and follow-up contain a fair number of significant coefficients, but there is little consistency across states or across outcomes.

In summary, regression models were constructed to explore the relationship between caretaker demographic characteristics and experimental group and family functioning. Other than the initial value of the measures, relatively few significant relationships emerged. Moreover, these relationships were not consistent across the states. As to the effects of family preservation services, these data do not support a strong relationship between these services and better family functioning.

(26) The significant interactions with experimental group were as follows. For depression at post-treatment in New Jersey, there was an interaction of experimental group with single motherhood; for single mothers, there was no relationship between experimental group and depression, for other caretakers, the control group had higher depression scores. Also for depression at post-treatment in New Jersey, there was an interaction with employment; for those employed at the initial interview, there was no difference between the experimental and control groups, for those unemployed, the control group had higher depression scores. For negative life events at post-treatment in Tennessee, there was an interaction with income support; for those not receiving income support the control group had more negative life events, for those receiving income support, there was no difference between the experimental and control groups in negative life events. For household condition at follow-up in Tennessee, there was an interaction between age of caretaker and experimental group; in the control group there was no relationship between age and household condition while in the experimental group, older caretakers had worse household conditions.

(27) Brown, D., Ahmed, F., Gary, L., & Milburn, N. (1995) Major depression in a community sample of African Americans. American Journal of Psychiatry 152(3), March 373-378.

(28) Humke, C. & Schaefer, C. (1995) Relocation: A review of the effects of residential mobility on children and adolescents. Psychology; a quarterly journal of human behavior, 32(1), 16-24.

(29) Honig, A. & Pfannestiel, A. (1991) Difficulties in reaching low-income new fathers: Issues and cases. Early Child Development &z Care77, 115-125.

(30) Baxter, A., & Kahn, J. (1999) Social support, needs and stress in urban families with children enrolled in an early intervention program.Infant-Toddler Intervention 9(3), September 239-257.

(31) The differing results for the uncontrolled analysis and the regression analysis may be due to the significant interaction in the regression equation of experimental group and income support.

8.10 Summary of Outcome Data

Information from the caretaker interviews, the caseworker interviews, and the administrative data were analyzed for indications of differences between the experimental and control groups subsequent to the referral to the family preservation program. Tables 8-15 and 8-16 contain a summary of those outcomes on which we found significant differences between the experimental and control groups in any state for the primary analyses (p < .05). Items in bold are those on which the experimental group had better outcomes, those in italics are those on which the control group had better outcomes.

In none of the three states were there significant differences between the experimental and control groups on family level rates of placement or case closings. Subsequent maltreatment was generally not related to experimental group membership, except for one subgroup in Tennessee. In Tennessee, in those families with an allegation within 30 days prior to random assignment, the experimental group children experienced fewer substantiated allegations than children in the control group.

In Tables 8-15 and 8-16 there are a number of child and family functioning items in which the experimental group displayed better outcomes than the control group in one of the states. It should be noted that the results have not been adjusted for the multiplicity of significance tests performed. That is, these significant items surfaced out of a large number of items and scales examined. In such a situation it is to be expected that some items will show significant differences simply by chance, so the appearance of a few significant differences should not be taken as an indication of superiority of one group over another, particularly when the results are not confirmed in more than one state. On only two items were differences found in two states: caretakers' assessment of whether goals had been accomplished and their assessment of overall change. We are inclined to believe that family preservation programs as represented in these states do result in higher assessments by clients of the extent to which goals have been accomplished and of overall change, since differences on those items were found in both states. Beyond that, we are unable to claim consistent evidence of positive effects of family preservation services.³²

There are a few items on which the control group had better outcomes, nearly all of them on measures provided by caseworkers. We are not inclined to read too much into these results, since experimental group caseworkers generally knew the families better and there may well have been significant differences in the ways that workers serving the two groups saw families and judged their functioning.

Table 8-15.
Summary of outcomes, post-treatment interview
Caretaker Interview: Proportion of affirmative answers to yes/no questions	Kentucky			New Jersey			Tennessee
	Control %	Exp %	p	Control %	Exp %	p	Control %	Exp %	p
Is apartment/house rented (vs. owned)	75	89	0.005	70	68		69	75
Got together with anyone to have fun	64	64		65	59		38	75	0.001
Felt had few or no friends	14	18		20	18		38	19	0.03
Had difficulty buying clothes	17	21		47	33	0.008	27	24
Out of control when punishing child	24	24		40	30	0.05	11	12
Punished for not finishing food	7	1	0.02	6	5		0	0
Unable to find someone to watch child	9	12		21	12	0.04	20	27
Encouraged child to read a book	92	90		82	91	0.02	94	96
Have goals been accomplished	63	77	0.02	52	71	0.001	81	84
Assessment of overall change:			0.02			0.001
Some Improvement	16	22		9	16		32	32
Great Improvement	31	42		41	52		32	42
Same	42	29		34			22	14
Same Somewhat or a great deal worse	12	6		16	12		14	13
Caretaker Scales:
Difficulty paying bills (proportion of 4 items)	0.17	0.22		0.34	0.25	0.02	0.25	0.18
Negative child care practices (proportion of 10 items)	0.14	.0.13		0.18	0.14	0.02	0.09	0.09
Punishment (proportion of 5 items)	0.16	0.17		0.25	0.20	0.04	0.13	0.13
Negative child behaviors (proportion of 21 items)	0.34	0.34		0.33	0.28	0.04	0.21	0.21
Change in proportion of punishment items from Initial to Post-treatment interviews	-0.04	-0.09	0.05	-0.05	-0.07		-0.07	-0.13
Change in proportion of negative child care practices from Initial to Post-treatment interviews	-0.02	-0.06	0.04	-0.04	-0.05		-0.01	-0.08	0.02
Caseworker Scales:
Ability giving affection (higher = more adequate)	2.83	2.83		2.93	2.70	0.04	2.73	2.95
Providing learning opportunities for child (higher = more adequate)	2.38	2.42		2.89	2.60	0.008	2.64	2.64
Respecting child's opinions (higher = more adequate)	2.58	2.45		2.55	2.42		2.35	2.84	0.01
Responding patiently to child's questions (higher = more adequate)	2.43	2.34		2.44	2.27		2.26	2.67	0.04
Adequate supervision / Responsible child care (higher = more adequate)	2.50	2.59		2.80	2.71		2.52	2.93	0.04
Household condition (proportion of 13 items, higher = worse condition)	0.10	0.13	0.01	0.09	0.11	0.02	0.12	0.12
Caretaker problems (proportion of 21 items, higher = more problems)	0.25	0.31	0.0005	0.21	0.23		0.21	0.18
Caretaker functioning (higher = better)	2.56	2.55		2.79	2.66	0.10	2.51	2.82	0.04
Respecting child's opinions (change in average ratings from Time 1 to Time 2)**	0.19	-0.06	0.05	0.27	0.04	0.05	0.06	0.14
Setting firm/consistent limits/rules (change in average ratings from Time 1 to Time 2) **	0.35	0.22		0.33	0.25		-0.29	0.29	0.01
Caretaker Problems (Change in proportion of 21 items; lower = less at Time 2)	-0.06	-0.04		-0.05	-0.04		-0.01	-0.08	0.05

NOTE: This table only includes items with a primary analysis p-value less than .05 in at least one of the states; p-values greater than .10 are not reported.

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

** Scale for change in ratings: -4 = ability decreased greatly over time, 0 = no change in ability over time, +4 = ability increased greatly over time

Table 8-16.
Summary of outcomes, caretaker follow-up interview
Proportion of affirmative answers to yes/no questions
	Kentucky			New Jersey			Tennessee
	Control %	Exp %	p	Control %	Exp %	p	Control %	Exp %	p
Has spouse held full time job	81	78		86	68	.05	100	85
Had difficulty paying rent	20	20		34	27		39	20	.04
Have children handled household chores	75	75		70	83	.02	94	89

NOTE: This table only includes items with either a primary p-value less than .05 in at least one of the states; p-values greater than .10 are not reported

Items in bold indicate significant findings in favor of the experimental group whereas italicized items indicate significant findings in favor of the control group.

(32) The reader is reminded of the findings reported in Chapter 7 indicating that experimental group caretakers generally had more positive views of service and of their relationships with workers than control group caretakers.

Chapter 9: Conclusions

In the late 1980s and early 1990s, family preservation programs became a popular response of states to rising rates of foster care placement of children. It was commonly assumed that many children were unnecessarily removed from their parents and that intensive services could prevent those placements while protecting children from harm. Early evaluations suggested these programs had considerable promise but these studies were criticized for flaws in research design. Later, more rigorously designed studies began to cast doubt on the extensive claims of success. The largest of these studies were in California, New Jersey, and Illinois. No placement prevention effects were found in California and Illinois, while the study in New Jersey found short term effects that dissipated with time.¹ However, these studies were also criticized, most notably for not having examined those programs thought to be most likely to be effective.

This evaluation of family preservation programs was designed to overcome shortcomings of previous studies. It assessed the extent to which key goals of the programs are being met: the goals of reduction of foster care placement, maintaining the safety of children, and improving family functioning.² It studied the Homebuilders model of service, thought by many to be the most promising.

The design for this evaluation was an experiment in which families were randomly assigned to either a family preservation program (the experimental group) or to other, "regular" services of the child welfare system (the control group). Families were followed for over a year after random assignment. Data collection involved multiple interviews with caretakers and caseworkers and examination of administrative data on placements, reports of maltreatment, and case openings and closings. This report concerns programs in three states, Kentucky (Louisville and Lexington), New Jersey (seven counties), and Tennessee (Memphis). The programs in these states followed the Homebuilders model of family preservation (sometimes labeled "intensive family preservation").

(1) J. Littell and J. Schuerman. (1995). A Synthesis of Research on Family Preservation and Family Reunification.http://aspe.hhs.gov/hsp/cyp/fplitrev.htm.

(2) We did not assess the extent to which reducing placement was an appropriate goal in particular cases; preventing placement and preserving families whenever possible is a well accepted value of the child welfare system.

9.1 Outcomes

9.1.1 Placement

We are unable to conclude that the family preservation programs in these three states achieve the objective of reducing placement of children in foster care.³ A summary of various analyses of placement rates at various points in time following random assignment is shown in Table 9-1. In none of the three states were there significant differences in placement rates over time for the samples as they were originally randomly assigned (the "primary" analysis). Since some of the families in the control group were actually provided family preservation services ("violations") and some of the families in the experimental group did not receive services or received only minimal services ("minimal service" cases), we also conducted analyses in which we dropped those cases ("secondary" analyses). Results of the secondary analyses were quite similar to the primary analyses, also showing no significant differences between the groups.

Since it was thought that the samples included families that did not fit the conception of cases best suited for the program model, we attempted to identify subgroups that might better fit criteria for referral. This selection was based on the idea that the service is most useful for families in crisis. Hence, we focused on cases referred in the course of an investigation of abuse or neglect and cases with recent substantiated allegations of maltreatment, on the grounds that these groups of cases might reflect families in crisis. These "refined groups" analyses also failed to show differences between the experimental and control groups on placement rates over time.

In Kentucky and Tennessee, we obtained data from case records and caseworkers on placements with relatives that were not recorded in the administrative data. Adding those data to our analyses, there were again no differences between experimental groups. Although not statistically significant, some of the differences between groups appear to be fairly substantial, particularly at the one-year point. However, there is no consistent pattern to these differences, sometimes the experimental group percentage is higher, sometimes it is the other way around.

Table 9-1.
Summary of Placement Data, Survival Analyses
Percents of families experiencing placement
of at least one child within specified periods of time
Kentucky
	One month		6 months		One year
	E	C	E	C	E	C
Primary analysis	6	5	18	18	23	24
Secondary analysis	4	4	13	17	20	24
Refined Analyses:
Investigative cases, primary	8	5	16	14	26	15
Recent substantiated, primary	6	3	17	12	29	16
Petition cases, primary	7	10	14	26	18	33
Including relatives, primary	8	9	21	25	27	32
Including relatives, secondary	5	9	14	25	22	32
New Jersey
Primary analysis	4	6	19	16	28	22
Secondary analysis	3	6	18	16	26	22
Refined Analyses:
Investigative cases, primary	3	5	18	13	25	16
Recent substantiated, primary	8	5	20	12	27	15
Tennessee
Administrative data, primary analysis	11	11	22	19	23	19
Administrative data, secondary analysis	7	12	18	19	19	19
Including relatives, primary	11	11	26	21	28	23
Including relatives, secondary	7	12	20	19	23	21
Refined Analyses:
Recent investigation, CORS	7	12	15	15	17	15
Recent investigation, includes Relative	7	12	18	18	22	21

Since these programs were intended to prevent the placement of children, the target group for the services was families in which at least one child was "in imminent risk of placement." We found that, by and large, the families served were not in that target group. This is shown by the placement rate within a short period of time in the control group, indicating the placement experience in the absence of family preservation services. In all three states, the placement rate in the control group within one month (a liberal definition of "imminent") was quite low. It would, therefore, have been virtually impossible for the programs to be effective in preventing imminent placement, since very few families would have experienced placement within a month without family preservation services.⁴ It should be noted, however, that the rates of eventual placement in the control group were higher, about one-fifth to one-fourth within one year. Hence, it would have been possible for family preservation to have shown effects on placement over time, but those effects were not observed.

There was one group that it seemed might represent better targeting, the "petition" cases in Kentucky. Prior to random assignment, workers submitted petitions to the court for placement or some other court ordered intervention on 67 families. It might be supposed that this group would be more likely to have children placed. Although more of the control group families in this group experienced the placement of a child within one month than other subgroups in Kentucky, that proportion was still quite low (10%), suggesting that focusing on groups such as this (cases with court involvement) would not resolve the targeting problem.⁵

(3) The language we use here is carefully chosen. Technically, we cannot conclude that the programs had no effect.

(4) It would be unreasonable to expect that targeting would be perfect, that is, that all cases referred for services were at imminent risk of placement. But how high should the targeting rate be? The answer to that question depends on the impact of the program, its costs, and the cost of placement. If the impact of the program is large (that is, it substantially reduces the rate of placement in those cases in which placement would have occurred) or if it is relatively inexpensive relative to the cost of placement, the targeting rate can be lower. Some algebra indicates that the ratio of cost of FPS to placement cost averted (per case served) must be less than the proportion of cases in which placement was averted. For example, if the targeting rate was .5 and the success rate was .4, then the proportion of cases served that result in placement avoidance will be .2 (the product of .5 and .4). The ratio of the cost of FPS to the cost of placement must then be less than .2 for FPS to be cost effective.

(5) This group also showed the largest difference between the experimental and control groups in percentages of families experiencing placement at one year, a difference of 15% favoring the experimental group. However, the difference is not significant. Furthermore, there are other differences in the table almost as large, some favoring the control group.

9.1.2 Child Safety

In general, the rates of substantiated allegations of abuse or neglect were quite low. In most of our analyses, there was little difference between the family preservation and control groups in the incidence of reports of maltreatment subsequent to random assignment. An exception was the group of cases in Tennessee with prior allegations of harm within 30 days before random assignment. For this set of families, the control group had a significantly higher rate of subsequent substantiated allegations.

The findings of little difference between the experimental and control group can be read in two ways. It indicates that families served by family preservation were no more likely than families not receiving the service to be subjects of allegations of harm. In this sense, children were, by and large, kept safely at home while receiving family preservation services. However, children in both groups were primarily in their homes, and family preservation did not result in lower incidence of maltreatment compared with children in the control group.

9.1.3 Subgroups

In Kentucky and New Jersey, we examined a number of subgroups of families to determine whether we could detect differences between experimental and control groups on placement and substantiated allegations subsequent to random assignment within each subgroup. Most of the subgroups were defined in terms of problems of the family, for example, substance abuse, financial difficulties, and depression. The number of cases in Tennessee was not sufficient to support subgroup analysis. In only one subgroup was a significant difference found between experimental and control cases: among single mothers in New Jersey, those in the experimental group were less likely to have a subsequent substantiated allegation than those in the control group. No subgroups were found in which there were effects on placement. Hence, the effort to find subgroups for which family preservation service was successful in reducing placement was not successful.

9.1.4 Case Closing and Subsequent Reopening

Family preservation services are sometimes thought to lead to quicker case closings in the public agency and less frequent subsequent involvement with the child welfare agency. Administrative data on case closings and subsequent case openings were examined to determine the effects of these services on case closings and subsequent reopenings. There were no significant differences between experimental groups in rates of case closing over time in the three states. In Kentucky, only three of the cases that were closed had reopened at the time we collected administrative data, two in the experimental group and one in the control group. In New Jersey, the difference between groups in proportion of cases reopened was not significant (21% in the experimental group, 23% in the control group). In Tennessee, significantly more of the closed control group cases reopened (9 of 30, 30%, compared to 8 of 66, 12% of the experimental group).

9.1.5 Family and Child Functioning

We interviewed caretakers at three points in time, shortly after the beginning of service (the "initial" interview), four to six weeks later (at the end of service for families receiving family preservation services, called the "post-treatment interview"), and again a year after services began (the "follow-up interview"). Caseworkers for both experimental and control group families were interviewed at the first two of these points in time. In these interviews, we examined a number of areas of family and child functioning that might have been affected by family preservation services. We looked at both levels of functioning at post treatment and follow up and changes over time in levels of functioning. We examined responses to some of the individual items in the interviews, and we combined responses into various scales measuring dimensions of functioning. The following are the areas examined.

Caretaker interview:

Life events. An inventory of recent positive and negative life events was used to construct three scales: positive life events, negative life events, and depression.
Problems. Nine items, examined individually.
Economic functioning. Four items on difficulty in paying for rent, electricity and heat, food, and clothes were examined individually and combined in a scale.
Household condition. Ten items, examined individually and combined in a scale.
Child care practices. Fifteen items, examined individually and in three scales: positive child care practices, negative child care practices, and punishment.
Caretaker depression. Scores on the SCL-90 depression scale.
Child behavior. Thirty-five questions comprising scales for aggression, school problems, positive child behaviors, and negative child behaviors.
Overall assessment of improvement. A single question.

Caseworker interview:

Caretaker functioning. Nine five-point scale questions, examined individually and averaged.
Household condition. Thirteen questions combined in a scale.
Caretaker problems. Twenty-one questions combined in a scale.
Child problems. Twelve questions combined in a scale.

The results of the measures of functioning are summarized in Tables 9-2 and 9-3. In a few of these areas of functioning, in one or the other of the states, families in the experimental group appeared to be doing better post-treatment. There were very few differences at the year follow-up and in changes over time. Those differences that did appear (primarily at post-treatment) were not consistent across states and were not maintained. At best, it can be said that family preservation services may have small, apparently short-term, effects on some areas of functioning. There was one item with some consistency, the overall assessment of improvement by caretakers. At post treatment, in Kentucky and New Jersey, a significantly larger proportion of experimental group caretakers generally thought there was "great improvement" in their lives. This difference was significant in both the primary and secondary analyses. In the Tennessee secondary analysis, results tended in the same direction, though not significantly (p = .09). At follow up, differences between the groups in Kentucky and New Jersey had nearly disappeared. In Tennessee at follow up, control group respondents more often thought there was "great improvement" (p = .055).

9.2 Targeting

The findings of no effects of family preservation programs on placement rates and of problems in targeting these programs are not new, they have been observed in a number of rigorously designed experiments.⁶ Partially as a result of these previous findings, efforts were made in this project to improve targeting. In New Jersey and Kentucky (but not in Tennessee) a screening instrument developed by the evaluators was employed to encourage referral of cases with a risk of imminent placement and to discourage referral of cases not at risk of placement. It is evident that this effort did not work; evidently, the screening instrument was a weak "intervention" in the problem of targeting.

Clearly, referring agents sent families to the programs that did not fit the criterion of imminent risk of placement. Our interviews with referring workers, discussed in earlier chapters, reveal some of the reasons. Workers acknowledged that they often did not refer cases that were at risk of placement, rather they used the programs for families that they thought could benefit from them. Evidently, they believed that in cases where placement was needed, family preservation services were not appropriate, contrary to the assumptions of the designers of these programs. But the programs were valued, and they were used to help families in the context of a generally service-poor child welfare system.

Table 9-2.
Summary of family and child functioning outcomes, data from caretaker interviews
Differences between experimental and control groups at post treatment, follow up, and change over time
Area	Post treatment	Follow up	Change over time
Life events
Positive life events scale (6 items)	No significant differences between experimental and control groups in any state	No significant differences between experimental and control groups in any state	No significant differences between experimental and control groups in any state
Negative life events scale (8 items)	No significant differences	No significant differences	No significant differences
Depression scale (4 items)	No significant differences	No significant differences	No significant differences
Family problems, 9 individual items	KY: no significant differences NJ: fewer experimentals did not have enough money for food, rent, or clothing TN: fewer experimentals had few or no friends	No significant differences	NA
Economic functioning (4 items)
Individual items	KY: no significant differences NJ: experimental group had less difficulty paying rent and buying clothes TN: no significant differences	KY: no significant differences NJ: no significant differences TN: fewer experimentals had difficulty paying rent	NA
Scale	KY: no significant difference NJ: experimental average lower (better) TN: no significant difference	No significant differences	No significant differences
Household condition (10 items)
Individual items	KY: experimentals had fewer broken windows or doors NJ: no significant differences TN: more experimentals in unsafe building because of illegal acts	No significant differences	NA
Scale	No significant differences	No significant differences	No significant differences
Child care practices (15 items)
Individual items	KY: fewer experimentals used punishment for not finishing food NJ: experimentals less often got out of control when punishing child and more often encouraged child to read a book TN: more experimentals went to amusement park, pool, or picnic	No significant differences	NA
Positive scale (5 items)	No significant differences	No significant differences	No significant differences
Negative scale (10 items)	KY: no significant difference NJ: experimentals lower (better) TN: no significant difference	No significant differences	No significant differences
Punishment (5 items)	KY: no significant difference NJ: experimentals lower (better) TN: no significant difference	No significant differences	No significant differences
Caretaker depression scale (13 items)	No significant differences	No significant differences	No significant differences
Child behavior
Aggression scale (3 items)	No significant differences	No significant differences	No significant differences
School problems (5 items)	No significant differences	No significant differences	No significant differences
Positive child behaviors (10 items)	No significant differences	No significant differences	No significant differences
Negative child behaviors (21 items)	KY: no significant differences NJ: experimental group lower (better) TN: no significant differences	No significant differences	No significant differences
Overall assessment of improvement	KY: experimentals, greater improvement NJ: experimentals, greater improvement TN: no significant difference	No significant differences	NA

Note: Changes over time were not determined for individual items so entries for those cells are designated "NA."

Table 9-3.
Summary of family and child functioning outcomes, data from caseworker interviews
Differences between experimental and control groups at post treatment and change over time
Area	Post treatment	Change over time
Caseworker report of caretaker functioning (9 items)
Individual items	KY: no significant difference NJ: control group higher (better) in ability in giving affection and providing learning opportunities TN: experimental group higher (better) on five items	KY: respecting child's opinions: experimental group declined, control group increased NJ: control group had more positive change in respecting child's opinions TN: experimental group more positive change on setting firm and consistent limits
Scale	KY: no significant difference NJ: no significant difference TN: experimental group higher (better)	No significant differences
Caseworker report of household condition Scale (13 items)	KY: control group better NJ: control group better TN: no significant difference	No significant differences
Caseworker report of caretaker problems Scale (21 items)	KY: experimentals more problems NJ: no significant difference TN: no significant difference	KY: no significant difference NJ: no significant difference TN: experimentals declined more
Caseworker report of child problems Scale (12 items)	No significant differences	No significant differences

There are other possible explanations for the low placement rate in the control group. It is possible that in cases assigned to the control group, workers on those cases exerted efforts to prevent placement of the child. Placement prevention as a central value may pervade the system (perhaps more during the time we were collecting these data than now, it is possible that the Adoption and Safe Families Act has shifted emphasis away from this value). Of course, in this regard the philosophy of family preservation seems to have been widely adopted, even though rigorous evaluations have not shown placement prevention effects of its services.

But there are still other aspects of the targeting problem. Homebuilders has developed into a quite generalist program, used in a wide variety of cases. In Kentucky and New Jersey there is considerable heterogeneity in the cases referred to these services, in both characteristics and problems of families and in where the case is in the child welfare system. Families come from both the investigative and on-going phases of cases.⁷ It seems likely that many of those referred from on-going caseloads are not referred because of likelihood of placement but because the case is not going well and everything else has been tried.⁸ Families do not always appear to be in crisis, another important criterion for referral. Furthermore, a number of cases do not involve abuse or neglect, but rather are cases of child dependency or of parent-adolescent conflict. And the cases involve a wide range of ages of children at risk. It could be argued that this variation is detrimental to the development of programs. No one program can expect to be successful in all cases. Having such variation inevitably results in a lack of focus and prevents the development of specialized expertise in handling particular cases. The lack of focus and expertise is likely to affect the outcomes that can be expected. Furthermore, the variation in the character of cases must contribute to variations in outcomes.

A natural response to this state of affairs is that we must tighten up the targeting, demanding strict adherence to referral criteria. Our attempts to assist states to do this were clearly unsuccessful. We suggest that it will be extremely difficult to achieve the goal of better targeting. There are a number of reasons for this skepticism. Referring workers acknowledged that they often referred families that were not at risk of placement, at least not those at imminent risk of placement. We cannot fully explain why workers did not follow the rules for referrals, but we can propose some conjectures. Workers believe that they remove children from the home only when that is absolutely necessary, when no service can prevent placement. In this sense, one might conclude that family preservation values have come to pervade the system, there are few unnecessary placements, leaving few placements to be prevented with intensive services⁹ However, these services are valued by referring workers, they are responses to the needs of families (families other than those with children about to be placed), and services to meet those needs are scarce. Hence, family preservation programs are used for very real needs of families in the child welfare system.

Beyond this dynamic, there is the general tendency to expand the benefits of a good program. If a program is believed to be beneficial, it is often assumed that it will be useful for an ever-expanding range of cases. Evidently, this occurred in the states we studied. Expansion of the target group is aided by the fact that target group definitions usually have one or more vague terms that allow for the expression of discretion (e.g., most people’s problems can be conceptualized as "crises").

Finally, our efforts to identify particular groups of families for which the programs are successful at preventing placement were mostly unfruitful. Hence we are unable to satisfy the demands of policy makers and practitioners for guidance on specific groups that might be targeted.

These circumstances, together with the fact that referrals to family preservation programs involve judgments that cannot be completely systematized or circumscribed, lead to our skepticism about the likelihood of improving targeting of these programs. Furthermore, it is possible that the programs are, by and large, being used in those circumstances for which they are best suited.¹⁰

(6) J. Littell and J. Schuerman. (1995). A Synthesis of Research on Family Preservation and Family Reunification.http://aspe.hhs.gov/hsp/cyp/fplitrev.htm.

(7) A number of cases in Kentucky and New Jersey were referred to family preservation to assist in the return home of children from foster care. These reunification cases were excluded from the experiment, but they may have contributed to the diffusion of the program.

(8) The fact that the case is not going well and that everything else has been tried may or may not mean that placement is likely. Note that "everything else has been tried" is sometimes specified as a criterion for Homebuilders referral. As we noted earlier, this criterion conflicts with the objective of immediate response to crisis.

(9) However, it is clear that there is great variation among jurisdictions, workers, judges, and other decision makers in the circumstances in which children are removed from their homes (J. Schuerman, P. Rossi, and S. Budde. (1999). Decisions on Placement and Family Preservation. Evaluation Review 23:599-618).

(10) We note one effort to solve the targeting problem in the family preservation program in Detroit. As part of an experimental evaluation of the Families First program, judges were asked to identify cases in which they intended to remove a child from the family, but which they deemed could be diverted to family preservation. After screening by project personnel, a group of such cases was randomly assigned to family preservation or to other services, presumably placement. Results of the study have not been published to date. Our understanding is that the group selected for random assignment was a relatively small portion of all families designated for placement. Furthermore, although the procedure was very promising from the standpoint of tightening up the evaluation, it is unlikely that it could be implemented widely or consistently to solve the targeting problem.

9.3 Possible Alternative Explanations of the Findings

Positive findings of experimental evaluations provide evidence for the validity of a theory of intervention and confirm the effectiveness of a particular implementation of that theory. Null findings are more ambiguous, they do not necessarily disprove an intervention theory and may not even be evidence of ineffectiveness of implementation. One cannot be sure whether the results are due to problematic program conceptions, inadequate program implementation, unique contextual problems, or flawed evaluation procedures. The findings of this study will be questioned, as have those of the previous studies, for various supposed methodological and implementation shortcomings. We consider here some of the factors that might have affected the findings, beginning with problems in the implementation of the evaluation.

Violations of experimental assignment. In all three states, there were violations of experimental group assignment, that is, families assigned to the control group that were given family preservation services. This was particularly a problem in New Jersey, where 14% of the control group families received family preservation. The dictates of rigorous analysis required that we retain these cases in the control group (we also conducted "secondary" analyses in which we dropped these cases from analysis and there were few differences between our primary and secondary analyses). Violation cases could significantly affect the findings. For example, they could represent cases that would have experienced placement in the absence of the service. To the extent this was the case, the placement rate in the control group would be underestimated. This could affect the conclusions about both the effective targeting rate and experimental-control group differences in placement.

We attempted to examine the extent to which violations might have affected the results in New Jersey (there were too few violations in Kentucky and Tennessee to have significant effects). Even if all of the violations had been placed early on, the proportion of families in the control group experiencing placement would not have reached levels that one would consider close to adequate targeting. Sensitivity analysis in which all violations are assumed to be placed early suggests that under this extreme assumption there would have been differences in placement rates favoring the family preservation group early on but these differences dissipate over time.¹¹ Hence, at the very least, violations could not affect a conclusion that family preservation does not appear to prevent long placements of a year or more.

Inclusion of minimal service cases in the analysis. Some families in the experimental group did not receive family preservation services or received only small amounts of service. These cases were included in the primary analysis and it might be argued that this reduced the apparent effects of the service and that we should have eliminated these cases from analysis to produce a fair estimate of effects. We did drop these cases from our "secondary" analysis, and found few differences compared to the primary analysis. In addition, it should be observed that programs will always have minimal service cases, cases in which the family cannot be found, declines service, or otherwise refuses to cooperate. Retaining them in the analysis is appropriate in determining the average effects of the service over a group of cases thought to need the service. Theoretically, one might be able to reduce the size of the minimal service group through better targeting, but in practice, it is likely to be difficult to identify a substantial proportion of these cases prior to referral.

The "John Henry" effect. The John Henry effect is reputed to be present in some experimental evaluations. This is the situation in which workers in control group cases exert special efforts on behalf of families, providing them far more service than would have been provided in normal circumstances (so the control group is not a "regular service" group). There are a couple of possible reasons this might occur. A worker might be unhappy with the experiment in general and with the assignment of this particular case to the control group in particular, and exert special effort in response. Alternatively, workers might feel the families assigned to the control group really need the experimental service, the prevention of placement is very important, so efforts are made to emulate Homebuilders. (This may be a special case of experimental leakage.)

In Kentucky and New Jersey, there is no evidence in the data on services to suggest this happened. Families in the experimental group did receive much more service than the control group. It is possible that the control group received more than "regular services." We cannot determine that. So it is possible that there is a threshold of services that has placement prevention effects and that was reached by the control group. If this were the case, it would indicate that the desired results can be obtained without intensive family preservation services.

In Tennessee, there is some evidence that families in the control group may have received as much, or perhaps more, service than the experimental group. This is seen in a specific set of questions asked of the caretakers about services received, and is not confirmed in other evidence regarding services provided to the two groups. Nonetheless, we cannot be as confident in Tennessee that experimental group families received much more service than the control group. Since the outcomes of the two groups were similar, this could again be taken as an indication that the results could be obtained without the family preservation services we studied.

Effects of the experiment on the nature of the referred group. It is possible that instituting the experiment caused a change in the character of cases referred to the program. In particular, agencies and workers were required to refer more cases in order to fill the control group as well as the experimental group. This resulted in dipping further into the pool of cases, perhaps taking "less severe" cases, those with less risk of placement. Anticipating this problem, we endeavored to select sites for the experiment in which demand considerably exceeded supply, however, we cannot be sure that we succeeded in this regard. It is also possible that workers referred different cases because of the chance that they would be assigned to the control group and not receive family preservation services. Or they may have changed referral practices to sabotage the research.

We cannot be sure that these factors were not present in referrals of families to the experiment, but we have no strong evidence that they were a strong influence. Operating against such dynamics were the desires of workers to provide significant services to families.

The program implementation was flawed. The family preservation programs in Kentucky, New Jersey, and Tennessee claimed adherence to the Homebuilders model of service. However, it is possible that the implementation did not adequately follow that model, with the result that this evaluation was not a fair test of the model. We attempted to measure certain aspects of model adherence and found some variation from the prescribed ideal. One cannot expect any implementation of a model to adhere totally to it, adaptations must be made to local conditions, the character of individual cases, and to the styles of individual workers. Models of social service do not provide for the same response in all cases nor can they be used to prescribe exactly what should be done in each case. Even for the best specified model, judgment abounds in its application, such that there might be legitimate disagreements as to whether it was applied in a particular case. In fact, one might hope that a model would be "robust" for at least small violations of it, having benefit even when it is not applied in an ideal way.

In the end, it is a matter of judgment as to whether the model was adequately adhered to in these three states. The fact that we have three states with similar findings, that is, similar degrees of adherence to the model, is again relevant. Was the model violated in all three states? Possibly, but that would then suggest the difficulty, perhaps the unlikelihood, of adequately implementing it elsewhere.

Contextual factors caused the model to fail. It is possible that a variety of contextual factors caused the outcomes that we observed. There are a multitude of possible such factors: the political and economic climate, the climate in the agencies, administrative barriers, approaches of judges, competence of workers, availability of other services, etc. These influences would weigh on both the experimental and control groups, presumably in equivalent ways, but they could prevent any new approach from having effects different from usual treatment. While we cannot exclude such factors as explanations for our results, again the fact that we have three states with similar results is relevant. Multiple sites make it less likely that the same contextual factors are explanations of the findings. Furthermore, social programs must operate in less than ideal contexts, to be effective, their conceptualizations must take into account these circumstances.

One set of contextual factors may have prevented positive effects of family preservation services: broad social problems of poverty, racism, inadequate housing, inadequate education, and substance abuse. Perhaps it is unrealistic to expect a short term program to solve such serious problems.

The program conceptualization is flawed. It is always possible that findings such as ours are the result of program design that is flawed. Obviously, this is the interpretation that is most difficult for program advocates to contemplate. But it is possible that the intervention activities of family preservation programs, even if carried out in an ideal way, are inadequate to achieve their goals. We note here one specific aspect of these programs that is often criticized and blamed for perceived failures: their brevity. It is often suggested that a program only four weeks in length, even if it is very intense, cannot expect to have significant effects on very serious individual and family problems which are often of long duration, therefore requiring much longer interventions. Going even further, it is possible that the available intervention technology is simply inadequate in the face of the problems it is expected to solve.

(11) Under the assumption that all violations would have been placed in the first month, 27% of the control group would have been placed in the first six months, compared to 19% of the experimental group. At one year, the proportions would have been 29% in the control group and 28% in the experimental group.

9.4 What to Make of These Findings

The findings of this study are not new. As in this investigation, a number of previous evaluations with relatively rigorous designs have failed to produce evidence that family preservation programs have placement prevention effects or have more than minimal benefits in improved family or child functioning. The work reported here may be thought of as three independent evaluations, in three states, adding to the set of previous studies with similar results. While the findings of this study can be questioned (as have those of the previous studies), the accumulation of like findings from a number of studies in several states, with varying measures of outcome, is compelling.

The results do not indicate that family preservation services are detrimental to families. Generally, families in these programs did not do worse than those in the control groups. Nor should the findings be taken as showing that these programs serve no useful purpose in the child welfare system. The findings can be seen as a challenge to keep trying, to find new ways to deal with the problems of families in the child welfare system. The findings indicate the grave difficulties facing those who devise approaches to those problems, failure in such undertakings should not be surprising, and those who risk trying to find solutions should not be punished when evaluations such as this indicate they may have come up short.

The foremost of these issues concerns the objectives of the programs. A number of observers have suggested that placement prevention be abandoned as the central objective in intensive family preservation services in favor of other objectives, notably the improvement of family and child functioning. We have suggested above that targeting these services on families at risk of placement is unlikely to be successful, so if these services are to continue, they will continue to serve "in-home" cases, families in which there has been a substantiated allegation of abuse or neglect or serious conflicts between parents and children but in which children remain in the home. Although the focus of concern in child welfare policy has long been on foster care, in most jurisdictions there are more cases opened for in-home services than for foster care (a relatively small proportion of indicated reports of maltreatment eventuate in removal of the child from the home and even fewer result in long term placement). Many, if not most, of these "intact" families need help. Relatively intensive and relatively short-term services such as those provided by family preservation programs are one source of such help. In this respect, family preservation programs can be thought of as an important part of the continuum of child welfare services.

Another question that program designers must address is that of specialization. We did not find subgroups for which the programs were successful, but as indicated above, these programs are quite generalist in character, and thus may sacrifice some of the benefits of specialization. Among those benefits are a clearer focus of services, tighter target group definition, specification of service characteristics such as length and intensity based on needs of the target group, and the development of more specific competencies on the part of workers. Specialization could be in terms of problems (e.g., substance abuse) or characteristics of clients (young, isolated mothers). There are clear drawbacks to specialization, including the tendency to define problems in terms of the service one offers. Furthermore, limiting target groups inherently limits the impact of programs. Nonetheless, it may be better to mount a series of small programs rather than putting all of one’s resources into large, undifferentiated efforts.

Another issue that program planners must address is that of length and intensity. These aspects of services are generally considered to be inversely related. Because of cost, long-term services cannot be as intensive as short-term efforts. The Homebuilders model pushes the combination of intensity and short term to what seems to be the limit: no more than two cases per worker at a time, 10 to 20 hours of work on a case per week for one month, a period of time much shorter than the planned service period in traditional social services. This is a bold departure from the usual way of doing things. It is based on ideas of crisis intervention. At the time of crisis, people are ready to change and ready to make use of intensive help to change. While crises can happen at any time, child welfare clients are thought to be most likely to be "in crisis" at the time of, or shortly after, an investigation of child maltreatment. Hence, the prototypical family preservation case is a family referred by an investigative worker.

The extent to which the intensive-short-term-crisis approach fits the needs of child welfare clients needs to be reexamined. Families encountering the child welfare system have often been there before and have usually been involved with other public or private service programs, so that being investigated and threatened with removal of a child is more an element of on-going experiences than a crisis. Furthermore, the lives of these families are often full of difficulties — externally imposed and internally generated — such that their problems are better characterized as chronic, rather than crises.

Families with chronic difficulties can no doubt benefit from short-term, intensive services, but those services are unlikely to solve, or make much of a dent in the underlying problems. As an example, substance addiction is a chronic problem in many child welfare families, one that cannot usually be successfully treated in a month’s time, however intensive the treatment. Of course, the hope is that family preservation programs will be able to connect families with on-going services to treat more chronic problems, but that appears to happen far less than needed. The central point here is that we need a range of service lengths and service intensities to meet the needs of child welfare clients.

Perhaps the best summary of the status of family preservation programs was provided by McGowan in 1990:

Family preservation services must not be viewed as a panacea. These are categorical programs able to help only one segment of the total range of families and children in need of support and are organized to provide limited types of case services. They cannot address the socioeconomic forces that contribute to tensions and inadequacies in family functioning nor can they provide the long-term assistance and/or specialized treatment required by some parents and children. Thus it is essential to maintain realistic expectations of what these programs can and cannot do.¹²

(12) Brenda McGowan. (1990). Family-based services and public policy: Context and implications. In J. Whittaker, J. Kinney, E. Tracy, and C. Booth (eds.). Reaching high risk families: Intensive family preservation in human services. (pp. 81-82) New York: Aldine de Gruyter.

Topics

Child Welfare | Child Maltreatment | Adoption & Foster Care