Willing, Able -> Ready: Basics and Policy Implications of Readiness as a Key Component for Scaling up Implementation of Evidence-Based Interventions

DEPARTMENT OF HEALTH & HUMAN SERVICES

Office of the Secretary

This brief is one in a series exploring issues related to the implementation of evidence-based interventions. It describes the basics of readiness using the R=MC2 (Readiness = Motivation × General Capacity and Intervention-Specific Capacity) heuristic, examines some of the policy implications of readiness, and identifies directions for future research. “Readiness” refers to the extent to which an organization is both willing and able to implement a particular practice. An emerging body of scholarly work identifies three components of readiness that organizations should address when implementing new EBIs: (a) motivation of people within the organization to adopt new EBIs, (b) general organizational capacities, and (c) intervention-specific capacities. Motivation includes beliefs about an intervention and support for the program which contribute to the desire to adopt a practice. General capacity speaks to different aspects of organizational functioning such as culture, climate, staff capacity, and leadership. Intervention-specific capacity describes human, technical, and fiscal conditions such as knowledge, skills, and intervention-related abilities that are important to the successful implementation of a particular practice.

About This Issue Brief

This issue brief was written by Allison Dymnicki, Ph.D., of the American Institutes for Research, Abraham Wandersman, Ph.D., of the University of South Carolina, David Osher, Ph.D., of the American Institutes for Research, Violanda Grigorescu, M.D., of the Centers for Disease Control and Prevention, and Larke Huang, Ph.D.,of the Substance Abuse and Mental Health Services Administration

Executive Summary

As more federal agencies recommend that organizations implement evidence-based interventions (EBIs), it is critical to know whether an organization is ready to implement these approaches. “Readiness” refers to the extent to which an organization is both willing and able to implement a particular practice. An emerging body of scholarly work identifies three components of readiness that organizations should address when implementing new EBIs: (a) motivation of people within the organization to adopt new EBIs, (b) general organizational capacities, and (c) intervention-specific capacities. Motivation includes beliefs about an intervention and support for the program which contribute to the desire to adopt a practice. General capacity speaks to different aspects of organizational functioning such as culture, climate, staff capacity, and leadership. Intervention-specific capacity describes human, technical, and fiscal conditions such as knowledge, skills, and intervention-related abilities that are important to the successful implementation of a particular practice.

Although a scientific body of knowledge about readiness already exists, there is a need to develop more concrete recommendations for federal agencies and practitioners to use when implementing EBIs. This brief establishes the basics of readiness using the R=MC² (Readiness = Motivation × General Capacity and Intervention-Specific Capacity) heuristic, examines some of the policy implications of readiness, and identifies directions for future research.

Key Take Away-Messages

• When considering an organization’s readiness for implementing EBIs, it is helpful toconsider the constructs of motivation, general capacity, and intervention-specificcapacity. This relationship is represented in the heuristic R=MC².
• Policymakers may include targeted questions about readiness in funding opportunityannouncements (FOAs) and develop criteria to evaluate answers to these questions toincorporate information about readiness when choosing grantees.
• Practitioner organizations (e.g., community-based organizations, schools) should assesstheir readiness before implementing an EBI to better understand their technical assistanceneeds. They should also assess organizational readiness throughout the life cycle of aprogram to foster continuous program improvement.
• Policymakers interested in future research on readiness could consider determiningcommonalities and differences in FOAs across federal agencies, and researching thedifferential effectiveness of different types of technical assistance provided based on thereadiness of an organization.
• Researchers should explore the weights of various subcomponents of the R=MC² heuristic (i.e., the relative importance of different aspects of motivation) to establish highpriority areas for training and technical assistance (TTA).

Introduction

Program administrators and staff are increasingly called to use evidence-based interventions (EBIs) and practices¹ to improve outcomes for children and youth across a variety of areas. However, an often unmeasured but necessary precondition for undertaking program improvement activities is that an organization be “ready” for the change; that is, staff are motivated and capable of making the adjustments required. Readiness (or lack thereof) is key to whether a change will take hold successfully or fail to catch on. Readiness to implement EBIs effectively influences whether the time, energy, and money dedicated to new programs will be well spent. 

The purpose of this brief is to describe the role of readiness in implementing and scaling up EBIs. Issues that are related to implementation readiness and EBI scale up are relevant to producing knowledge about what works, and for moving science into practice in socially significant ways. Readiness as conceptualized here, involves the extent to which the members of an organization as well as the organization as a whole are motivated, individually as well as strategically, and have the capacity to implement an intervention with quality. This brief is targeted primarily to staff working in federal agencies but it is also relevant to federal and state policymakers, practitioners, researchers, evaluators at all levels, and consumers—those who will ultimately benefit from the successful implementation of EBIs at scale.

¹ We define “evidence-based interventions” as programs, practices, or policies that have been proven to positively change the problem being targeted by a body of scientific knowledge, usually including some form of evaluation (Kratochwill & Shernoff, 2003).

What Are the Core Components of Readiness?

This brief builds on the readiness literature and in particular a conceptualization of readiness proposed by Scaccia and colleagues (2014). Three components of readiness should be considered when implementing an EBI that is new to an organization: motivation, general capacity, and intervention-specific capacity.² The three components can be depicted as R=MC² (Readiness = Motivation × General Capacity × Intervention-Specific Capacity).

² The focus of this brief is on the readiness of an organization and/or the community where it is located. Equally important, but beyond the scope of this brief, is the readiness of an EBI to be scaled up and the quality of evidence that suggests that a practice is “evidence-based.”

To What Extent Is the Organization Motivated To Implement the Intervention?

Scaccia et al. (2014) define motivation as being influenced by the “perceived incentives and disincentives that contribute to the desirability” to use an intervention. We see motivation as both individual and organizational. Motivation is specific to the intervention being implemented. It includes beliefs about and support for the intervention—such as collective expectations, attributes of an intervention, anticipated outcomes of an intervention, pressures for change, and emotional responses. Table 1 presents a list of the subcomponents of motivation³.

Table 1. Subcomponents of Motivation

Source: Scaccia et al. (2014)

³ For each of the three readiness constructs, the key subcomponents identified from a systematic review of the readiness literature are presented. This is not an exhaustive list of all subcomponents that could comprise each readiness construct.

What Is the General Capacity of the Organization That Will Implement the Intervention?

General organizational capacities are related to maintaining a functioning organization (e.g., sufficient staffing, effective organizational leadership) and connecting with other organizations and the community. This includes the context, culture, current infrastructure, and organizational processes of the organization where the EBI will be implemented. General capacities are associated with the ability to implement any intervention (Flaspohler, Duffy, Wandersman, Stillman, & Maras, 2008) and include human, fiscal, technical, and evaluative categories. Table 2 presents a list of general capacities.

Table 2. Subcomponents of General Capacities

Source: Scaccia et al. (2014)

What Are the Capacities Needed To Put a Particular Intervention in Place?

Intervention-specific capacities are the human, technical, and fiscal conditions important for successfully implementing a particular intervention with quality (Flaspohler et al., 2008; Scaccia et al., 2014)⁴ . Although table 3 presents global constructs associated with intervention-specific capacity, each new program, practice, or policy has its own set of knowledge and skills required to implement it with quality.

Table 3. Subcomponents of Intervention-Specific Capacity

Source: Scaccia et al. (2014)

⁴ In Scaccia et al. (2014), this is referred to as “innovation-specific capacity.” We use the term “intervention-specific capacity” synonymously throughout this brief.

Understanding Organizational Readiness as R=MC2

Organizational readiness for an intervention is a function of three components: (1) motivation, (2) general organizational capacity, and (3) intervention-specific capacities. Each component contributes to an organization’s readiness and is abbreviated by R=MC² (Scaccia et al., 2014). One implication is that if any of the components is zero or near zero, the relationship is multiplicative and the organization is not ready to implement an EBI and any attempt to implement the EBI before the missing element is addressed will likely be unsuccessful.

For organizations to implement EBIs at scale successfully, it is important for funders, researchers, and practitioners to understand readiness and how to create it.

What Does Readiness Look Like During Different Phases of Implementation?

The National Implementation Research Network framework describes four phases of implementation:

1. Exploration (when people explore the possibility of making use of an innovation)
2. Installation (where the goal is to acquire or repurpose the resources needed to do the work ahead)
3. Initial implementation (when the innovation is being used for the first time)
4. Full implementation (reached when 50% or more of the intended practitioners, staff, or team members are using an effective innovation with fidelity and good outcomes

(National Implementation Research Network, n.d.)

Readiness at one phase does not ensure readiness for the next phase. Communities and organizations need to be ready to enter each phase of implementation: exploration, installation, initial implementation, full implementation. It would be useful for key stakeholders (e.g., leadership, implementers, community members, and clients being served) to discuss readiness regularly. Because readiness is dynamic (i.e., it evolves during a project period and can increase, stay the same, or decrease over time in different phases), these discussions enable project staff to consider readiness components as new challenges arise and to modify the existing approach to meet these needs. Readiness indicators can be assessed through a range of methods, including surveys of providers, clients served, or community members; focus groups eliciting group perspectives; and observation of program components. In the next section, we discuss the implications of the readiness heuristic for staff working in federal agencies.

Implications of Readiness for Staff Working in Federal Agencies

The importance of readiness when implementing an EBI raises a variety of questions for practitioners, funders, training and technical assistance (TTA) providers, and researchers, including:

• How can funding opportunity announcements (FOAs) measure the readiness of grantees to implement EBIs with quality?
• How can TTA resources be used to assess and support implementation readiness?
• How can funders partner with TTA providers and practitioners to enable more organizations to become ready to implement EBIs with quality? Attention to the readiness of grantees during the grant making process and through TTA may improve the likelihood that grantees will succeed in implementing their proposed interventions with fidelity and can achieve the outcomes projected in their applications.

A framework can be helpful in providing a big picture for integrating research and practice to achieve outcomes and to illustrate the roles of practitioners, researchers/evaluators, TTA providers, and funders in this process. Wandersman and colleagues (2008) at the Centers for Disease Control and Prevention and elsewhere developed the Interactive Systems Framework for Dissemination and Implementation (ISF) to help bridge research and practice in innovative ways. The ISF describes three interacting systems: the Delivery System is the organization(s) or community setting that actually implements interventions (e.g., mental health centers, schools). The Delivery System needs sufficient general capacity, motivation, and intervention-specific capacity (i.e., readiness) to be ready to implement an intervention with quality to achieve outcomes. The Support System (e.g., training and technical assistance centers) uses strategies like TTA to strengthen the Delivery System’s ability to be ready to implement interventions with quality (Wandersman et al., 2012). The Synthesis and Translation System synthesizes the products of research and translates them into user-friendly formats that practitioners in the Support and Delivery Systems can easily access and understand. One of the benefits of using a framework such as the ISF to understand readiness is its ability to show how different systems in an organization and community interact to promote readiness. For example, the Support System can provide targeted TTA based on the Delivery System organizations’ readiness needs (e.g., TTA can be provided specifically to improve the motivation of frontline staff). See the sidebar for more information about the ISF and how it is being used to synthesize and translate research to be ready for practitioners’ use and how funders use it to diagnose major gaps in the logic that would lead to outcomes.

Interactive Systems Framework for Dissemination and Implementation (ISF)

  The ISF has been applied in many fields of prevention and treatment, including to summarize the literature; for example, home visiting programs (Paulsell, Del Grosso & Supplee (in press); child abuse prevention (Brodowski et al., 2013), and diagnosing needs for TTA (e.g., a CDC teen pregnancy prevention initiative, Lesesne et al., 2008). Below, we provide a quick example of the three systems and how they interact, drawing upon Lesesne et al. and other developments in teen pregnancy prevention. There are hundreds of articles and multiple evidence-based programs about teen pregnancy prevention. Organizations in the delivery system (e.g., schools, Boys and Girls Clubs) are not likely to ask staff to review the literature. A synthesis and translation of the literature would be helpful. The U.S. Department of Health and Human Services has gathered information on 31 evidence-based programs for teen pregnancy prevention (see http://www.hhs.gov/ash/oah/oah-initiatives/teen_pregnancy/db/programs.html ). However, having access to a website does not mean that the organizations in the delivery system are ready to implement any of the 31 evidence-based programs.

Questions remain, such as “Do organizations have the general capacity to be good host organizations for an evidence-based program? Do they have the innovation-specific capacity required to deliver a specific evidence-based program that meets the requirements for quality implementation, and are they motivated to implement the evidence-based program with quality?” The support system serves as an intermediary system that can provide TTA to help delivery system organizations become ready for implementation.

How To Assess Implementation Readiness Through Funding Opportunity Announcements

In various venues, federal staff have expressed a strong interest in asking about readiness appropriately in FOAs. Table 4 presents sample questions that could be included in FOAs that assess the three readiness components along with criteria that federal staff might use when reviewing FOA responses.

Federal staff could use these criteria in several ways to score applicants. Many of these questions and criteria already exist in the problem statement, project design/goals and objectives, management plans, and management capacity sections of FOAs. Therefore, these questions and criteria could be taken from other sections and included in a new form tied to all federal organizations; they could become a Government Performance and Results Act indicator for organizational functioning. (Readiness would be one part of this.) When scoring, one suggestion is to use a tiered scoring approach in which different applicant scores are associated with different levels of funding, TTA support, and evaluation requirements. Each agency should consider the implications for proposal length and overall scoring when adding this type of information to FOAs.

Table 4a. Examples of Readiness Questions Related to the Motivation Construct To Include in Funding Opportunity Announcements

Table 4b. Examples of Readiness Questions Related to the General Capacity Construct To Include in Funding Opportunity Announcements

Table 4c. Examples of Readiness Questions Related to the Intervention-Specific Capacity Construct To Include in Funding Opportunity Announcements

⁵ For example, if an organization already has an established mentoring structure in place for new staff and is proposing to implement a new substance abuse program that all staff will be trained in, an applicant could describe how the proposed approach would build on that mentoring structure. For example, if all new staff are required to attend monthly 1-hour PD sessions, one topic of a PD session could be on the new substance abuse program.

⁶ Depending on what the proposed approach includes, this could be described as (a) weekly staff meetings for groups of people being trained to allow time for problem-solving implementation challenges as they arise, or (b) monthly data meetings for organization staff to review program data and make appropriate changes to the program.

What Is the Role of TTA in Supporting Implementation Readiness?

TTA providers need to use their limited resources and capacities strategically. Analysis of readiness (specifically motivation, general organizational capacity, and intervention-specific capacity) can help to align the strengths and limitations of a grantee with the types of TTA that are deemed appropriate (Wandersman, Chien, & Katz, 2012). Conversely, organizational staff can assess the readiness of their organization to offer different types of support strategies (e.g., online vs. in-person, capacity to train all staff in an organization at once vs. capacity to train a select group of people in multiple cohorts) and develop their implementation plan accordingly. For example, TTA providers could ask organizational staff to answer a brief set of questions such as, “Will there be an implementation team to oversee the implementation of this proposed approach? Who will be part of this team? What kind of training do members of this team need to be successful at implementing this approach? Are there opportunities for ongoing staff collaboration after the training that can be used to reinforce and apply skills learned?” The TTA being provided could then be customized based on responses to these questions (e.g., if the assessment indicates that general capacity needs to be built, then tools, training, and technical assistance (TA) will be used to enhance that specific need). If a TTA provider is working with a site for an extended period of time, readiness assessments could be completed several times during a project and two-way conversations between the TTA provider and TTA recipients could inform how to improve different aspects of readiness.

How Do We Partner With and Foster Readiness in Settings With Limited Capacity and Willingness?

Some organizations with critical needs and who receive federal funding may not currently have the capacity to implement EBIs with quality.⁷ This problem can be addressed by working with these organizations to first build their capacity through a readiness development phase of the initiative. For example, the Substance Abuse and Mental Health Services Administration (SAMHSA) has done this with its Circles of Care program, which helps tribal communities build a capacity to implement systems of care and in so doing, to compete for grants under the Comprehensive Community Mental Health Services for Children and Their Families Program. The Maternal Infant and Early Childhood Home Visiting Program is another example. Lessons learned from this effort emphasize the importance of developing trusting relationships in which federal staff and TA providers work closely with grantees during the project period to provide support in three major areas: (a) developing a needs assessment and plan for responding to identified needs, (b) developing and carrying out their implementation plans, and (c) designing and carrying out a locally driven rigorous evaluation.

Another example includes the Permanency Innovations Initiative (PII) which is a multi-site federal demonstration project designed to improve permanency outcomes among children in foster care who face the most serious barriers to permanency (Permanency Innovations Initiative Training and Technical Assistance Project & Permanency Innovations Initiative Evaluation Team, 2013).

An overarching objective of PII is to develop interventions that are supported by solid evidence of effectiveness and are ready for replication, adaptation, and broad-scale rollout. PII grantees are guided through four implementation stages: exploration, installation, initial implementation, and full implementation. A specific focus of PII is on building readiness during the exploration stage where grantees receive technical assistance to coordinate a teaming structure, select and promote buy-in for an intervention, and plan for implementation and evaluation of the intervention. These activities are intentionally intended to create readiness for change within grantee organizational structures, ensure the appropriateness of the selected target population for intervention, assess the needs of the target population, and determine the feasibility of the interventions meeting those needs.

⁷ An important first step is to assess the potential of organizations to develop readiness (DiMaggio & Powell, 1983; Levin & Minton, 1986). Several measures to be used for this purpose are listed in Additional Resources.

Maternal, Infant, and Early Childhood Home Visiting Program

Visiting Program (MIECHV), administered by the Health Resources and Services Administration (HRSA) in collaboration with the Administration for Children & Families (ACF), implemented a provision of the Patient Protection and Affordable Care Act (2010), which expanded home visiting to states, territories, and tribes (Supplee, et al., 2013). Through this effort, three percent of funds were set aside for tribes, tribal organizations, and urban Indian organizations and administered by ACF. To promote the capacity of the tribal MIECHV grantees, ACF required grantees to develop a needs assessment to assess their capacity to implement home visiting programs and use the results to create a plan for responding to identified needs. Ongoing TA and support continued throughout the implementation process by having ACF staff and TA providers work closely with the grantees as they developed and carried out their implementation plans.

Tribal grantees were also required to conduct a rigorous evaluation of their home visiting program, its implementation, or both, because of limited evidence in the field about the http://homvee.acf.hhs.gov/Tribal_Report_2012.pdf http://homvee.acf.hhs.gov/Tribal_Report_2012. The Office of Planning, Research & Evaluation (OPRE) within ACF funded the Tribal Home Visiting Evaluation Institute (TEI) to provide TA in designing locally driven, rigorous evaluations of home visiting. The TEI provides individualized, culturally relevant TA that empowers grantees to conduct research and evaluation that is meaningful for the tribe and meets FOA requirements for rigor (i.e., credibility, applicability, consistency, and neutrality). The TEI builds on ACF’s past and continuing efforts to build evaluation and research capacity within American Indian/American Native communities to improve their early childhood programming. The approach TEI takes to helping grantees develop an evaluation plan that links their community needs to the intended impact of the home visiting model enables communities to build readiness within a prescribed framework.

Directions for Future Research

Although scholarly work has identified several core components of readiness and this brief presented some policy implications, gaps still exist in the field’s understanding of how readiness relates to scaling up and implementing EBIs. Future research directions could include (a) how to measure the relative importance of motivation, general capacity, and intervention-specific capacity in grant applications (and their subcomponents) and (b) how to understand differences in readiness based on scale of implementation (e.g., communities vs. states) and different types of organizations (e.g., child welfare vs. substance abuse treatment providers, faith-based coalitions vs. not-for-profit organizations, small businesses vs. national providers, or urban vs. rural settings). One future direction identified by our federal coauthors involves the analysis of performance data (e.g., Government Performance and Results Act indicators) collected in cross-site, national, and local evaluations, and how readiness constructs can be investigated using these data. This type of program data, which is widely available, should be assessed for its quality. If the quality of the data is confirmed, they could be used along with quantitative and qualitative data collected during the implementation and evaluation of an EBI to increase our understanding of how different aspects of readiness play out in practice. A second future direction involves developing some readiness criteria among different programs that could be useful to explore differences in readiness and its importance in the successful implementation of EBIs. For example, future work could identify criteria for working with communities that have limited infrastructure—in other words, criteria that communities should meet before being selected for implementing an EBI. Based on information gleaned from several federal staff with expertise in this area, these criteria might include (a) strong leadership support from a senior and midlevel staff member, (b) alignment of the intervention with their strategic plan, and (c) evidence of organizational sustainability (e.g., resources are in place for the organization to function for the next 5 years).

Knowledge Sharing Among Staff at Federal Agencies about Readiness

Several approaches to practice and intentional research could enhance knowledge on these topics. Although there has been a focus on linking science with practice across agencies, grant programs, and grantees, this effort has been sporadic. An assessment of FOAs across federal agencies could help ensure the proper exploration of commonalities and differences in effective strategies and performance measurements across varying types of applications (e.g., intervention programs vs. surveillance systems). Research could compare the effectiveness of TTA being provided based on an organization’s readiness by assessing the relationship of TTA to program outcomes for communities with different levels of initial readiness. This could inform best practices for infusing readiness based on certain organizational or community characteristics.

Conclusion

To increase the number of grantees that will succeed at implementing EBIs and achieving positive outcomes, it is important to understand what grantees should have in place before and during implementation. Scholarly work suggests that in order to be successful grantees should be willing (motivated) and able (have the general and intervention-specific capacities). Recognizing the importance of readiness has several policy implications for federal funders; these include assessing readiness during the grant application process, using readiness data to deploy TTA strategically, and building readiness in organizations with limited capacity. More communication, collaboration, and systematic evaluation among federal partners about these and other policy implications can help deepen the evidence base on readiness and promote the successful implementation of EBIs.

Additional Resources

References

Brodowski, M. L., Counts, J. M., Gillam, R. J., Baker, L., Spiva Collins, V., Winkle, E., et al. (2013). Translating evidence-based policy to practice: A multilevel partnership using the Interactive Systems Framework. Families in Society: The Journal of Contemporary Social Services, 94(3), 141–149.

DiMaggio, P. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological review, 48, 147-160.

Flaspohler, P., Duffy, J. L., Wandersman, A., Stillman, L., & Maras, M. (2008). Unpacking prevention capacity: The intersection of research to practice models and community-centered models. American Journal of Community Psychology, 41, 182–196.

Hoagwood, K., Schoenwald, S. K., & Chapman, J. E. (2003). Dimensions of Organizational Readiness-Revised (DOOR-R). Unpublished instrument.

Kratochwill, T. R., & Shernoff, E. S. (2003). Evidence-based practice: Promoting evidence-based interventions in school psychology. School Psychology Quarterly, 18(4), 389.

Lesesne, C. A., Lewis, K. M., White, C. P., Green, D. C., Duffy, J. L., & Wandersman, A. (2008). Promoting science-based approaches to teen pregnancy prevention: Proactively engaging the three systems of the interactive systems framework. American Journal of Community Psychology, 41(3-4), 379-392.

Lewin, A. Y., & Minton, J. W. (1986). Determining organizational effectiveness: another look, and an agenda for research. Management science, 32(5), 514-538.

Lynne, D., Finnerty, M., & Boyle, P. (2005). Implementing EBPs: Early results and the use of the General Organizational Index and other tools in implementing EBPs. Paper presented at the Fifteenth Annual NRI Conference on State Mental Health Agency Services Research, Program Evaluation, and Policy. Baltimore, MD.

Patterson, M., et al. (2005). Validating the organizational climate measure: links to managerial practices, productivity and innovation. Journal of Organizational Behavior, 26, 379-408. Retrieved from https://wiki.uio.no/admin/amunder/images/a/a0/OCM-1-.pdf

Paulsell, D., Del Grosso, P., & Supplee, L. (in press). Supporting replication and scale-up of evidence-based home visiting programs: Assessing the implementation knowledge base using the Interactive Systems Framework for Dissemination and Implementation. American Journal of Public Health.

Permanency Innovations Initiative Training and Technical Assistance Project & Permanency Innovations Initiative Evaluation Team. (2013). The PII approach: Building implementation and evaluation capacity in child welfare (Rev. ed). Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau, and Office of Planning, Research and Evaluation.

Scaccia, J., Cook, B., Lamont, A., Wandersman, A., Castellow, J., Katz, J., & Beidas, R. (in press). A practical implementation science heuristic for organizational readiness: R = MC². Journal of Community Psychology.

Shea, C. M., Jacobs, S. R., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014). Organizational readiness for implementing change: a psychometric assessment of a new measure. Implementation Science, 9(1), 7-22.

Supplee, L. H., Harwood, R. L., Margie, N. G., Meyer, A. L. (2013). New opportunities and directions in home visiting research and evaluation. Zero to Three, 33(3), 45-50.

Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support (tools,) for implementing innovations with quality: Tools, training, technical assistance, quality assurance/quality improvement. American Journal of Community Psychology, 50(3-4), 445–459.

Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, S., et al. (2008). Bridging the gap between prevention research and practice: The Interactive Systems Framework for Dissemination and Implementation. American Journal of Community Psychology, 41(3–4), 171–181.