Appraise

Having found the evidence relevant to your project (see Step 2, Accessing Evidence), the next step towards using evidence is to establish whether the evidence is trustworthy and relevant. This means looking at how it has been produced to help you judge its quality and decide whether the findings are appropriate to your setting.  This is called critical appraisal and includes asking some simple questions such as:

• What is the design of the study and methods used?
• What are the main findings/results?
• Are the results relevant locally?

Below you will find further guidance about tools which are used to structure the appraisal process and the specific questions to ask. It is helpful to keep in mind that looking critically at evidence involves using your own judgement to reach a decision about its quality. Whether it is ‘good enough’ will depend on what decision you are hoping it will inform – it may not be clear cut and there is not necessarily a ‘right’ answer.   Whether you are a provider of care for individuals, or a commissioner of care for the local population or a patient or member of the public considering your own health choices, the important thing is looking critically at evidence and not taking it at face value.

Critical appraisal tools

There are many tools available to help you break down the steps of critical appraisal. You can search for these online but a few suggestions are given here:

• A set of widely used and practical tools from the Critical Appraisal Skills Programme for looking at different types of research evidence 
• A tool for appraising surveys from the Centre for Evidence Based Management
• The AACODs tool for appraising grey literature from Monash University

Even if you don’t have time for a complete appraisal, applying some simple questions to help you look at the details of a study can be better than nothing. Below are two sets of simple questions you might use – one for quantitative evidence and one for qualitative evidence, each with four areas to focus on. For quantitative evidence – look at details such as:

1. PEOPLE
   1. Who were the participants? Are people selected from a database or other source that might exclude some people? Is the number of people included explained by the authors? If so, does it meet the stated sample size requirement? If not, does the sample size seem good enough?
   2. If participants were split into two or more groups (as in a trial), were the two groups similar at the outset of the study, in terms of numbers and characteristics such as age, gender or health status, for example?
2. MEASUREMENTS
   1. Is there scope for bias in the measurements used? Were outcome measures subjective such as relying on self-reported symptoms, or objective such as using a recognised scale e.g. EQ-5D quality of life measure? Apply the same to any exposure measurements (e.g. in a cohort or case control study). If a questionnaire was used, was this an existing tool? Were any new tools piloted before use? Were any important outcomes missing?
   2. Was there a baseline measured before the intervention (if applicable)? Was the time between measurements sufficient to see a change and whether this lasted? Was the rate of drop-out from the study reasonable? (or does it suggest a problem with the intervention or methodology?) Does it reduce the sample size significantly?
3. RESULTS
   1. What was the main result? Did they consider the influence of certain variables on outcomes, such as age, gender, ethnicity? If applicable, are there statistics about how likely the results are due to chance factors (look for p values <0.05)? Is there a confidence interval and is this narrow? (this is the range of values which we are 95% confident contains the ‘true’ value for the population. Narrow range=more confidence; wide interval=less confidence because the ‘true’ value could be quite different from the one seen. Close or below zero (or no difference) reject the results.)
4. TRUST
   1. From the above, are you satisfied that the sample is comparable to your population, that measurements are robust, and do you trust the study’s findings? You may want a stronger piece of evidence for certain decisions. But no decision should rest on a single piece of evidence!

For qualitative evidence, look at details such as:

1. PEOPLE
   1. Who were the participants? Were the right people included, i.e., those with characteristics or experience relating to the topic of the study? Is the number of people explained by the authors? Is this adequate for a qualitative study? (remember this is not aiming to be representative)
2. METHODS
   1. How was information gathered from participants? Did this method suit the topic being discussed? i.e., was it suited to individual or group discussion; could other methods have been used? Was this process transparent (are the questions shown)? Was it recorded in some way to aid the analysis process? Did more than one person help analyse the data?
3. RESULTS
   1. What were the main findings? Were these agreed among the researchers? Were these discussed with the participants? Are quotes cited? Do these seem to match the interpretation?
4. TRUST
   1. From the above, is the study credible? Do the findings seem plausible? Do you trust the study? Should it inform your thinking?

All of these questions should be possible to answer by careful reading of the article. They are intended to prompt reflection on the way the evidence has been produced and you will need to reach your own decision about how important each detail is. Remember no evidence is perfect! The aim is to decide is it good enough?