Monitoring Fidelity and Outcomes

Outcomes
A fundamental principle of an evidence-based culture is continuous quality improvement to strive for the best outcomes for children, youth and families. 

John Weisz, Ph.D. summarized the following methods that practitioners can use to assess and monitor treatment when measuring outcomes:

• Parent Ratings
• Child Reports
• Teacher Reports
• Trained Observer Ratings
• Peer Observer Ratings
• Peer Sociometric Choices
• Child Performance/Task/Test
• Diagnostic Interview – P/C
• Global Assessment Ratings/Mental Health

Standardized instruments that match these topical areas include the Child Behavior Checklist with a parent, teacher, or trained observer rating; and the Youth Satisfaction Survey for Families as reported by parents and children.  For a list of specific instruments that can be used to assess outcomes, see the OJJDP Resource Guide in the Practitioners’ section on Assessing Youth and Family Needs.

Miller, Duncan, & Hubble (2005) have proposed the following 3-step process for practitioners to become “outcome-informed” in their practice (see reference for full text article):

• Select instruments (finding measures of process and outcomes that are valid, reliable, and feasible)
• Pilot the tools chosen, gather data
• Develop a feedback process (scoring and discussing results with clients at each session).

John Lyons has proposed that a “Total Clinical Outcomes Management Framework” be adopted to monitor your practice. This framework encourages practitioners to capture the progress that a youth and family are making and use this information to manage ongoing treatment.
 
Fidelity
As evidence-based practices are selected and implemented with shared decision-making, practitioners must be trained and aware of how to implement “true to the intervention.” Adhering to the actual model or treatment protocol increases the likelihood of obtaining the outcomes that are expected.

Fidelity measures and scales are available for some, but not all, evidence-based practices. Although fidelity scales do not exist for some children’s EBPs, practitioners are still encouraged to use these interventions. 

Gorman-Smith (2006, working paper), proposes that practitioners and supervisors can use one of several methods to measure adherence to an intervention protocol:

• Practitioners complete checklist targeted to the main content areas; recipients of intervention also complete the same checklists
• Direct observation of the implemented intervention (supervisor’s review)
• Videotaped observation of intervention session (reviewed and coded later) (pg. 8).

The strategic use of repeated evaluations of programs using fidelity scales, either for an individual program or on a statewide level, is based on the general principle that whatever is attended to is more likely to be improved.

Helpful Links:

• To become better informed around outcomes, see Saggese’s (2006) article, “Maximizing Treatment Effectiveness in Clinical Practice: An Outcome-Informed, Collaborative Approach.”
• For information about issues related to assessing the fidelity of Wraparound services using the Wraparound Fidelity Index, see the article in Focus, entitled “Ensuring Fidelity to the Wraparound Process” from the Portland State University’s Research and Training Center on Family Support and Children’s Mental Health.
• For information about assessing the fidelity of the Incredible Years program and to see some of the measurement instruments, visit the Incredible Years website.

References:
Miller, S. D., Duncan, B. L., & Hubble, M. A. (2004).  Beyond integration: The triumph of outcome over process in clinical practice. Psychotherapy in Australia, 10(2), 2-19. http://www.talkingcure.com/documents/BeyondIntegration.pdf