First Year Impacts of the Heritage Keepers Life Skills Education Component

Acknowledgments

We are grateful to the many people who contributed to this report on the first-year impacts of the Heritage Keepers® Life Skills Education Component in Edgefield, South Carolina. We especially thank Anne Badgley, Sally Raymond, Jerry Raymond, Sheila Whittington, Priscilla Montoya, Susan Swanson, and all the Heritage Keepers® program staff in both Edgefield and Charleston, South Carolina. They generously allowed us to visit their program and observe their operations, and they provided thoughtful and insightful comments on their program design and operations. We also thank the Edgefield County School District staff who supported this evaluation, especially Robert Heflin, Virginia Culbertson, Suzanne ONeal, Greg Thompson, David Mathis, and Dr. Sharon Keasley.

Over the course of the Title V, Section 510 evaluation, we have received valuable comments and guidance from members of our Technical Work Group  Marilyn Benoit, Sarah Brown, Ron Haskins, Jim Jaccard, David Larson (1998-2000), Joe McIlhaney, Robert Michael, Kristin Moore, Susan Philliber, Robert Rector, David Rowberry, Freya Sonenstrin, Marta Tienda (1998-2000), John Vessey, Stan Weed (1998-2000), and Brian Wilcox.

Rebecca Maynard was project director and co-principal investigator at the time of the data collection for this report, and we are grateful for her intellectual and project leadership. Linda Mendenko and Milena Rosenblum shared various responsibilities for the design and oversight of the survey data collection for the evaluation of the Heritage Keepers® Life Skills Education Component. Ece Kalay provided database management and programming support for the evaluation. Walt Brower provided editorial support for the report, which was produced by Jennifer Baskwell, Bryan Gustus, Bill Garrett, and Jill Miller. Chris Trenholm and Amy Johnson provided important guidance during the analysis and preparation of the report. Alan Hershey and Ken Fortson also provided valuable comments on drafts of the report.

Finally, we are grateful for the support of the U.S. Department of Health and Human Services, which oversees this project. We are especially grateful to Meredith Kelsey, the project officer for this study, and to Martha Moorehouse and Barbara Broman for their thoughtful comments on the draft report.

Although we gratefully acknowledge the input of these and many other individuals, we are responsible for any errors or omissions in the report. Any opinions expressed in the report are our own and do not necessarily reflect the views of the U.S. Department of Health and Human Services.

Melissa Clark (mclark @ mathematica-mpr.com) Barbara Devaney (bdevaney @ mathematica-mpr.com)

Conclusions and Discussion

This study of the Life Skills Education Component of the Heritage Keepers® Program in Edgefield, South Carolina examines the first-year impacts of the Life Skills Education Component on the health and sex education services youth received and on intermediate outcomes thought to be related to sexual activity. Since all youth in both the treatment and the control groups participated in the schools mandatory Abstinence Education Component and may have been exposed to various elements of the Community Education Component, the impact estimates presented in this report represent the incremental effects of providing the Life Skills Education Component to youth. By design, the study does not provide evidence on the effects of the classroom-based Abstinence Education Component all youth received, or of the Community Education Component to which the youth may have been exposed.

The results of this study indicate that the Life Skills Education Component had a positive impact on the perceived helpfulness of program services but no impact on reported participation in health, family life, and sex education classes. The program generally had no impact on intermediate outcomes thought to be related to teen sexual activity. The one exception was a positive impact on support for abstinence among middle school youths five closest friends, which is consistent with the programs goals of influencing youth peer groups. However, there were no impacts on this measure for high school youth or on the other measure of peer influences and relations examined.

There was generally more evidence of impacts on both service receipt and intermediate outcomes among middle school youth in the later (2003) enrollment cohort than among youth in the earlier (2001) enrollment cohorts. This suggests that the programs effectiveness in influencing these outcomes may have improved over time. Larger impacts for the 2003 cohort are also consistent with the revision to the Life Skills Education Component that occurred at that time, when program staff modified the curriculum so that each curriculum topic was related explicitly to the underlying message of abstinence until marriage. In addition, the program had larger impacts on services received among middle school youth with views less supportive of abstinence than among youth with views more supportive.

Limitations of this Study. Several factors are important to keep in mind in interpreting these results. First, mean outcomes for youth in the control group were near the top of the scales for several measures, leaving only limited opportunity for the Life Skills Education Component to further influence these outcomes. The high mean outcomes among the control group could suggest that the core Abstinence Education Component, in which all youth participated, had a positive impact on youths intermediate outcomes or that youth in the study sample would have had generally high mean levels of these outcomes even in the absence of either component. Either way, given the generally high mean values for intermediate outcomes among the control group, there may have been only limited opportunity for the Life Skills Education Component to influence these intermediate outcomes further.

Second, as mentioned earlier, a potential limitation of the random assignment research design is that there may be unmeasured spillover effects of the Life Skills Education Component. If the program had some influence on students in the control group simply through changes in school culture or effects on their peers in the program group, estimated impacts of the programs effects may be understated.

Third, a rigorous evaluation of the Life Skills Education Component required a random assignment design; however, this design may have changed the selection procedures typically used by program staff. Since the Life Skills Education Component usually does not have sufficient resources to include all eligible youth, program staff typically select a subset of youth for the program. They attempt to select a broad representation of the school youth, but with a focus on including trendsetters. For the purposes of the impact evaluation, however, all youth who applied for the program were randomly assigned to the program or to the control group. The impact estimates from this report therefore represent the average impacts among all youth who applied to the program. To the extent that the programs impacts differ for different types of youth, the impact estimates for the full program group may not readily generalize to impacts for the types of youth typically selected to be in the program.

The Title V, Section 510 evaluation design may also have affected Heritage Keepers® Program operations more generally. One of the three components of the Heritage Keepers® Program is the Community Education Component, which involves parent education, assembly speakers, faith community initiatives, media initiatives, and Family Assets and Character Councils. Because of their inclusion in the national evaluation, program staff sometimes attempted to limit what they perceived as potential contamination of the control group, either by reducing their exposure to the Community Education Component or by not including many of the usual elements of the program in Edgefield in the first place. Either way, this could lead estimated impacts of the Life Skills Education Component to be somewhat overstated relative to impacts under normal program operations if the full the Community Education Component also has positive effects on youth; alternatively, it could lead estimated impacts to be somewhat understated if the Life Skills Education Component is more effective if youth are also exposed to the full intended intensity of the Community Education Component.

Finally, the Life Skills Education Component is a multiyear intervention in which youth are intended to remain for up to six years. The study of the first-year impacts is based on data collected only 6 months to one year after youth enrolled in the study sample, and youth had therefore received only a fraction of the intended intervention. A future study, based on data collected between 2 and 5 years after youth enrolled, will examine the programs impacts after youth have had the opportunity to participate for the full intervention period. Findings from this study will provide evidence on whether the Life Skills Education Component is a useful and effective addition to the core classroom-based Abstinence Education Component all youth receive, and will provide guidance to staff of Heritage Keepers® and similar programs on how best to allocate their resources to promote abstinence among youth.