Navigating criticisms of evaluation methods
5 tips to increase evaluation quality, satisfaction, and deal with criticisms constructively
Ever delivered an evaluation report, only to receive negative feedback on the methods used? It can be disheartening, especially after putting in a lot of work. Usually, by the time the report is written, it's too late to change much about the evaluation design, methodology or methods.
Stakeholders seem to criticise methods most often when presented with findings that are not as positive as they expected, or when the evaluation fails to capture the essence of their narrative, making them feel misunderstood or undervalued. This reaction is quite understandable. Evaluation is a two-way street. As evaluators, we must be prepared to receive and consider feedback.
We can also manage the evaluation process in a way that invites early input and reduces the risk of criticisms arising too late. Here are five tips to increase the chances of stakeholders being satisfied with your final report, and for dealing constructively with criticisms of methods.
Tip 1: Invest in genuine relationships with stakeholders
“If you want a friend, get a dog - not an evaluator”, someone said to me once. I’m pretty sure he meant it in jest (I hope so, cos I laughed 😬). For starters, it makes no sense; dogs are evaluators too. Also, I prefer to think of program evaluators as critical friends.
I believe we all do our best work and communicate best when we’re getting along well and have mutual trust and respect. That doesn’t mean we would or should agree on everything. Nor does it mean shying away from delivering unpopular findings. It does mean we have a solid foundation for giving and receiving feedback and for any disagreements to be fruitful. So tip number one is to invest in genuine relationships. Get to know each other as people as well as colleagues.
Tip 2: Involve stakeholders in the evaluation design process
Tip number two is to engage stakeholders, rights-holders and end-users in designing the evaluation. Their expert knowledge will strengthen the design. It also reduces the risk of methods being criticised at the end of the evaluation. Why? Because co-designing an evaluation involves reaching a shared understanding of the appropriate questions, criteria, evidence and data collection approaches to use in the evaluation. It creates the space for concerns to be raised early, when there’s still time to discuss and fine-tune the methods. Accordingly, stakeholders are likely to feel a greater sense of ownership of the evaluation approach.
Some stakeholders may decline the invitation to participate, but if they are key stakeholders, senior decision-makers or end-users of the evaluation, I recommend being quite insistent that they are meaningfully involved. Those who are responsible for signing off on the final report should also give their explicit endorsement of the evaluation framework.
Tip 3: Use rubrics to ensure all parties have a clear understanding of how judgements are made
Rubrics are a good way to make evaluative judgements transparent. Developing rubrics collaboratively with stakeholders strengthens the validity of the evaluation framework. It also reduces the risk of people feeling blindsided by findings. Why? Because stakeholders who were involved in rubric development can see their values expressed in the rubrics. They are more likely to understand and endorse the basis upon which judgements are made.
Just as stakeholders should be involved in developing rubrics, they should also be engaged in using the rubrics to make sense of the evidence. When they look at the evidence through the lens of the rubrics, they will reach their own evaluative conclusions. Rubrics increase the likelihood that their conclusions will be similar to yours - and if they’re not, rubrics will help to precisely identify points of disagreement.
For example, rubrics help stakeholders point to which sub-criterion, which corresponding piece of evidence, or which rating feels wrong to them. This allows a more focused and useful conversation than general comments about methods.
Tip 4: Get formal sign-off on the evaluation framework
As a good project management practice, I recommend drawing a clear line between the end of the evaluation design phase (getting to an evaluation framework) and the start of the evaluation implementation phase (getting to an evaluation report).
I typically build in a pause-point at the end of the design phase, with a clear expectation that we won’t start to implement the evaluation until we have formal sign-off on the framework.
Agreeing on the evaluation framework includes agreeing that past this point, the methodology cannot be changed without reviewing the time frame and budget. If valid criticisms of the methods are raised after this point, they can be documented in the limitations section.
Tip 5: Pinpoint the reasons for the criticism
Even when you invest in genuine relationships, involve stakeholders in the evaluation design process, engage them in developing and using rubrics, and get formal sign-off on the evaluation framework, satisfaction is not guaranteed. There may still be objections to the evaluation design, methods and/or findings.
So here’s tip number five - pinpoint exactly what aspects of the methodology are being criticised. Here are four common aspects that can draw criticism, and hints on navigating each one.
Some people may criticise the very idea of evaluation being a judgement-oriented endeavour.
Like, why make judgements at all? Why not just present the facts?
When I talk about evaluation, I mean determining the value of a policy, program or intervention, based on:
evidence (e.g. what we can observe about its performance); and
values (aspects of merit, worth or significance to a relevant group of people).
Combining evidence and values involves making judgements. To evaluate is to judge. The judgements should be made on an explicit basis so that they’re transparent, traceable, and challengeable.
Take away the evaluative judgements and what you have left is descriptive evidence of something. I would call that research.
Some may criticise the idea of using rubrics to support judgement-making.
This is related to the first criticism but has distinct elements of its own.
Not all evaluation questions require judgements about value, quality, performance, success, or some other dimension of goodness. But when they do, rubrics are there to help.
To those trained in social science research, there’s something about rubrics that on first acquaintance can feel a bit “subjective”. This is a misconception that I thoroughly addressed in a previous post. In fact, rubrics guard against individual subjectivity by setting out an explicit, agreed basis for making judgements.
If there’s strong opposition to rubrics, there are alternatives. The bottom line is to make and present evaluative judgements in a way that connects them to evidence and logical argument, making the reasoning transparent.
Take away the rubrics (or whichever alternative you use) and what you’ll have left is reduced clarity about how judgements were made.
Some may question the validity of a specific rubric (or something specific within it).
It’s worth paying close attention to these questions. As with a theory of change, we tend to develop rubrics during evaluation design. Later, as we’re collecting and analysing evidence, we may learn things that give us good reasons to revise the theory of change - and perhaps the rubrics too. To avoid accusations of fudging or moving the goalposts, we can transparently document what was updated and why.
An objection to a particular judgement may point to some aspect of the rubric (e.g., a particular criterion or standard) that in hindsight wasn’t defined quite right, and a claim to a legitimate change. Or not. Our job is to remain alert to those cases and to be willing to update the rubric where warranted - and where we don’t consider it’s warranted, to be ready to argue why not.
A common misconception can arise here when people expect criteria to resemble indicators. Indicators are specific and measurable, whereas criteria describe what matters about a policy or program in a context. These descriptions are deliberately broader and less specific than indicators, as befits their intended purpose of helping us to make meaningful evaluative judgements. If indicators are used in the evaluation, they shouldn’t be specified within the rubric but rather subsequently, as part of identifying the evidence needed.
Some may question the reliability of the evidence.
Again, we need to give these objections a fair hearing. There are no perfect methods, and criticisms may point to areas for further investigation (perhaps as part of a future evaluation) and/or limitations to note in the final report.
At the same time, we need to be ready to push back against arguments about methods that are not valid. Common examples include: blanket rejection of qualitative or mixed methods evidence; rigid preference for so-called “gold standard” methods like RCTs or CBA regardless of context; New Public Management dogma (like, “if you can’t measure it you can’t manage it”); or a misunderstanding of what constitutes a valid sample for a specific purpose.
There’s an enormous body of literature on these sorts of topics and it’s a lifelong endeavour keeping on top of it. The reward is better-informed methods selection. (Don’t sweat it, we’re all on a journey). It’s useful to have some literature within reach (like, here’s a paper that argues CBA isn’t the gold standard).
Conclusion
Evaluation isn’t just about methods (how evidence is gathered). Fundamentally it’s about reasoning (how judgements are made), and even more fundamentally it’s about power (who gets to say). If anybody voices a sense of discomfort about methods, it’s worth bearing in mind that the discomfort may also have something to do with the reasoning process and/or who has power in the evaluation.
As an evaluator, it’s essential to be prepared to receive feedback and consider criticisms of evaluation design and methods. These moments should be viewed as opportunities for growth and improvement.
To reduce the risk of criticisms arriving too late, we can invest in genuine relationships, involve stakeholders in the evaluation design process, use rubrics to increase transparency about how judgements are made, and get formal sign-off on the evaluation framework.
If methods are still criticised, seek clarity on what specific aspects are being questioned. Rubrics can facilitate this clarity by helping to distinguish reasoning from methods, by making the reasoning traceable and challengeable, and by giving stakeholders something they can use to pinpoint exactly which sub-criterion or piece of evidence they’re questioning.
As a matter of professional integrity, we mustn’t shy away from reporting unpopular evaluative judgements just because somebody doesn’t like them. But we also need to keep an open mind and consider whether criticisms are valid. Rubrics can be transparently amended if warranted. Extra evidence may be included where appropriate. If errors were made, we must own them and correct them.
These tips will not only help to address criticisms effectively but also increase validity, credibility, stakeholder ownership of the evaluation and ensure transparency and rigour in the evaluation process.
This is my perspective as a consultant. I have a feeling there may be multiple perspectives on this issue. I welcome yours!
Totally aggree re "Tip 2: Involve stakeholders in the evaluation design process". Too often, when I look at evaluability assessments, or what passes under that name, I see negelct of stakeholder interests, a big focus on data and to some extent the Theory of Change. Choice of methods and approach needs involve, and resolve , all three. See my favorite diagram page 14 https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/GL_for_Evaluability_Assessments.pdf