Friday, 30 January 2015 08:36

Reaching organizational goals through improved evaluation capacity

Written by    Volume: 21  Issue: 1
Rate this item
(1 Vote)

Over the last four decades, public sector organizations have institutionalized program and policy evaluation practices to varying degrees. As a management tool, evaluation has the potential to provide timely, useful information to decision-makers concerned with resource optimization and program outcomes.

However, piecemeal approaches to evaluation are commonplace in various public administrations, including the federal government. Such approaches tend to produce operational recommendations that hinder the ability of evaluators to support management and contribute meaningfully to organizational learning, accountability and improvement.

Building capacity
Organizations interested in developing a more strategic approach to program evaluation typically focus their efforts on staff training and budgetary allocations. Although these measures are an important starting point, the institutionalization of evaluation requires additional effort.

Researchers have turned their attention to this issue over the last decade and have dubbed the process "organizational evaluation capacity building." Evaluation capacity can take on different forms, depending on an organization’s context; our own research has focused, therefore, on the development of an empirically-based description of organizational evaluation capacity in the federal government.

Our organizational evaluation capacity framework includes six main dimensions. The first three focus on the organization’s ability to produce high-quality evaluation studies, while the others describe the organization’s ability to use evaluation for organizational learning and accountability purposes.

Human Resources, the first dimension, refers to the technical and interpersonal competencies of evaluators as well as evaluation leadership within the organization. The second dimension, Organizational Resources, focuses on the systems and structures available to evaluators to carry out their duties, such as the financial resources allocated to evaluation studies and the availability of quality performance measurement data. Evaluation Planning and Activities, the third dimension, describes how evaluation is planned and conducted in the organization. This dimension includes, for example, the use of consultants and other external resources to complement internal capabilities and skill sets.

Evaluation Literacy is the fourth dimension and the first to describe organizational capacity to use evaluation. It refers to the widespread knowledge of evaluation and its associated concepts, such as results-based management, across the organization. Organizational Decision Making characterizes the fifth dimension and focuses on the integration of evaluation findings in broader organizational processes and decision-making, such as program resourcing or deficit reduction activities. Finally, Learning Benefits describes how evaluation is used for learning and improvement in the organization, which is the ultimate objective of evaluation itself and evaluation capacity building more specifically.

In addition to the six dimensions and their associated sub-dimensions, the framework also outlines four levels of evaluation capacity: low, developing, intermediate and exemplary capacity. Each of the sub-dimensions, therefore, is described qualitatively at each of those four levels in order to profile and highlight the incremental steps required to move from one level of capacity to the next.

Managers interested in assessing their own organization's evaluation capacity may use this organizational profile framework to situate their organization across the four levels for all six dimensions. Alternatively, they may also be interested in using an organizational self-assessment instrument, which operationalizes the original framework and facilitates the assessment process.

Although the instrument is meant to be used by federal government departments and agencies, it is possible to adapt the tool to other organizational contexts. We have already adapted the instrument to the not-for-profit sector and the Quebec provincial government, and we are continuing this work to render the tool suitable for use in the Ontario public health sector. Fundamentally, the six dimensions apply to most public sector settings and it is possible to use the instrument in all of these, albeit in a slightly modified form that accounts for local contexts and terminology.

Organizational improvement
Our research shows that organizations that have a good sense of their evaluation capacity are better able to devise strategies to address any major gaps and continue on their path towards institutionalization.

We also know, based on our own sectoral comparative analyses, that organizations with higher evaluation capacity get the greatest "bang for their evaluation buck"; not only are their studies of greater quality and more timely, they also better anticipate senior decisio- makers' information needs. This, in turn, leads to greater evaluation use and organizational learning.

The link to improved program design and administration, and to a more effective and efficient use of organizational resources, makes this endeavour wholly appealing, not only to evaluators, but to managers across all levels of public sector organizations.

Read 4446 times Last modified on Wednesday, 04 February 2015 08:35
Isabelle Bourgeois and J. Bradley Cousins

Isabelle Bourgeois is a professor of program evaluation at l’École nationale d’administration publique in Gatineau. J. Bradley Cousins is a professor of educational evaluation at the Faculty of Education and director of the Centre for Research on Educational and Community Services, University of Ottawa.


  • Comment Link Laurie Monday, 09 March 2015 10:14 posted by Laurie

    Please post your website

  • Comment Link Ricardo Ramirez Sunday, 08 March 2015 11:56 posted by Ricardo Ramirez

    Hello Isabelle and Brad: your entry certainly confirmed our experience establishing 'readiness' with partners prior to introducing Utilization-Focused Evaluation. The readiness steps of UFE are strategically important, and not often understood. We have recently updated our readiness requirements and would be happy to share them further.
    Warm regards,
    ps you can access our UFE primer for free in English, French and Spanish from our website, as well as recent presentations on readiness and mentoring.

  • Comment Link Alison Brewin Sunday, 08 March 2015 11:47 posted by Alison Brewin

    This is very exciting to me. I work as a consultant to non-profits in British Columbia and advocate for the use of evaluation to improve overall organizational performance, accountability and decision-making. Where can I access more information about this framework and tool?

  • Comment Link Curtis Thursday, 05 March 2015 12:26 posted by Curtis

    Sounds like a great resource, could you provide any information on how we may go about obtaining this tool?

Leave a comment

Make sure you enter the (*) required information where indicated. HTML code is not allowed.

Share this article

copy link

 bookmark article



Related Articles

CGE successfully completes 3rd annual summit
Canadian Government Executiveu2019s Third Annual Leadership Summit at the Westin Hotel in downtown Ottawa on November 18th brought in respected keynote speakers that left the audience engaged and informed.  read more...

Prime Minister’s Advisory Committee Report on the Public Service Embraces the Value of Project Management
In March of this year, the Prime Ministeru2019s Advisory Committee on the Public Service, co-chaired in 2015 by former Senator, the Honourable Hugh Segal, and Rick Waugh, former CEO of Scotiabank, released its Ninth Annual Report to the Prime Minister of Canada. The Advisory Committee was established in 2006 to give advice to the Prime Minister on the renewal of the Public Service. The Committeeu2019s objective is to help shape the Public Service into a more effective and efficient institution, distinguished by highly-engaged and highly-skilled people performing critical tasks with professionalism and efficiency.  read more...

Interview: Leading for Results
The Canadian Government Executive Summit is just one week away and, with all the changes taking place in Ottawa, this yearu2019s summit couldnu2019t be taking place at a more opportune time. Recently we had the chance to sit down with John Richard Jones, the group publisher of Canadian Government Executive, to discuss the conference and what it might mean given the political context that Canadian government professionals of all stripes are now facing.  read more...

The logical way to manage change and achieve results
The necessity for government executives to manage change, and achieve results, is an enduring quality of government work. Two fundamental assumptions support this view. Every government program, regardless of its scale or scope, exists to serve a purpose...  read more...

Benefits management: A better method for maximizing performance
Every organization does it, but often with ambivalence: the performance measurement process has become a mechanical exercise dutifully executed to satisfy corporate requirements.  read more...

Copyright © 1995 - 2017 1618261 Ontario Inc. O/A Navatar Press