Monday, 25 May 2015 10:26

Realist evaluation: Assessing national programs in unique communities

Written by    Volume: 21  Issue: 5
Rate this item
(0 votes)

Canada is a diverse nation, in language, culture, geography, and, ultimately, the needs of its citizens. Such diversity can pose a challenge to a government that seeks to develop, implement, and evaluate national-level programming to local communities. Ultimately, programs and policies must be tailored to meet the unique needs of a community.

That presents a conundrum: the more a program is locally relevant, the harder it is to assess effectiveness at the national level.

In academic terms, this conundrum is part of the positivist vs. realist debate. Thankfully, there can be a reconciliation and way to move forward – where scientific rigour and local relevance can be of equal consideration in evaluation.

Realist evaluation offers a lens on establishing evidence; pushing beyond a focus on outcomes, and broadening to include both the context and processes (or mechanisms) affecting such outcomes. In fact, Health Canada has been exploring and researching this approach, particularly with respect to health promotion, and has added realist evaluation (RE) and syntheses to its methodological repertoire.

What is RE?
A mantra commonly used to describe RE is “what works, for whom, how, why, and under what circumstances.” This highlights the important role of context – namely, characteristics of the population or service provider, and of the setting. In addition, RE seeks to uncover the processes or mechanisms that bring certain outcomes. As a result, RE is frequently described as a theory-based approach to evaluation, including theory of change related to participants and theory of change related to the implementation of programs and policies.

The nuance of RE is the recognition that certain contexts may facilitate or hinder processes that bring about outcomes. This latter point is particularly relevant to diversity of the Canadian context – if evaluations uncover common mechanisms while capturing contextual features that support or hinder outcomes, one can identify successful processes that can be targeted across the contexts known to support outcomes (instead of applying them invariably across all contexts). Thus, realist evaluation has potential to focus on both the nation (common mechanisms) and the community (unique contextual features).

Current examples
An example of program implementation informed by a RE has been demonstrated by the Scottish government. The goal was to implement midwife-led care as a result of research demonstrating reductions in routine childbirth interventions and improvements in patient-reported care experiences.

Using interviews, focus groups, and multiple case studies, researchers found that consultant midwives varied in their approach to implementation of care in the team context according to the strength of the medical culture. In contexts with an ingrained medical culture (e.g., with little support for midwives), the appointment of midwives by administration demonstrated a high-level commitment to implementation of midwifery care and midwives relied on more robust change strategies (e.g., audits, debates, hands-on-training). In contexts where midwives were supported by obstetricians and physicians, midwife-lead care could emerge through empowerment of midwives to practice more autonomously.

Overall, the study was able to identify a set of mechanisms (e.g., commitment, resistance, feeling pressure) that would have differential impacts on outcomes according to the unique context of each hospital.

RE has also informed policy development. For example, the Australian Institute of Criminology undertook a realist review on effective crime prevention strategies. In addition to identifying which interventions appeared to be successful in preventing crimes, the authors were able to distill why and in what circumstances programs worked. Through a systematic realist review, they were also able to identify the necessary requirements for implementing these programs.

The findings from this review have led to the development of factsheets and handbooks that local governments can use to select, adapt, and implement interventions within their regions. In this way, if policymakers and practitioners understand how and why programs work in different contexts, better decisions can be made with regard to which programs or policies to use and how to adapt them in different contexts.

Embarking on RE requires reflection, multidisciplinary collaboration and time. Not surprisingly there can be systemic barriers to such an approach that will need to be addressed, overcome, or circumvented. RE is inherently messy as it is about complexity – there can be multiple mechanisms operating in an orchestrated fashion. Finally, conclusions from RE are tentative directions and not considered stable and infallible.

As a nation recognizant of both unique and common identities, striking the right balance between both program and policy is crucial. RE offers a relevant framework that targets mechanisms/processes, as well as uncovers and documents contextual features responsible for supporting particular outcomes.

The broader application of RE could be facilitated through the identification of necessary infrastructure (e.g., cross discipline teams). Several scholars within the CUEE are actively pursuing research into this burgeoning evaluation approach to determine its application in governmental evaluation.

Read 570 times Last modified on Tuesday, 26 May 2015 07:24
Kelly McShane, Sofia Puente-Duran and Fiona C. Thomas

Kelly McShane is Assistant Professor in the Department of Psychology at Ryerson University. Sofia Puente-Duran and Fiona C. Thomas are PhD students in the department.

Leave a comment

Make sure you enter the (*) required information where indicated. HTML code is not allowed.

Share this article

copy link

 bookmark article



Related Articles

Now that Harper is out, what does PSAC want from Trudeau?
The Public Service Alliance of Canada was one of the most vociferous opponents of the Harper government during Election 42. Now that itu2019s all over, and PSAC finds itself governed by a Prime Minister thatu2019s promised u201cReal Change,u201d you could be forgiven for thinking that the war is over.  read more...

From the Editor: Where you sit is where you stand
Itu2019s hard to believe we are still talking about this, but the numbers confirm the anecdotal evidence. The reality is that government departments, even after years of efforts in weeding them out, harbour more jerks than what is endurable.  read more...

Civility Matters! An interview with Craig Dowden
Craig Dowden recently published Civility Matters! An Evidence-Based Review on How to Cultivate a Respectful Federal Public Service. Patrice Dutil, the Editor of Canadian Government Executive, sat down with Dr. Dowden to learn more.  read more...

Think you're not a cheater? 50% of job seekers are.
The Public Service Commission catches 1 in 900 people lying on their resumes. The truth is, nearly half of us are doing it.  read more...

Nearly one-third of public sector execs are actively "disengaged"
Nearly 1/3 of Canadian public servant execs are feeling disengaged from their jobs, making attempts to reform the public service difficult.  read more...

Copyright © 1995 - 2017 1618261 Ontario Inc. O/A Navatar Press