EVIDENCE BASED PRACTICE Evaluating the Quality of Evidence Quality of Study Rating Form (QSRF)
Client type(s): _____________________________________________________________________________
Intervention method(s): ______________________________________________________________________
Outcome measure to compute ES1: _____________________________________________________________
Outcome measure to compute ES2: _____________________________________________________________
Outcome measure to compute ES3: _____________________________________________________________
Source in APA format: ______________________________________________________________________
Criteria for Rating Study
Clear Definition of Treatment
6. Subjects randomly assigned to treatment or control
7. Analysis shows equal treatment and control groups before treatment
8. Subjects blind to being in treatment or control group
Criteria for Rating Study (cont.)
9. Subjects randomly selected for study inclusion
10.Control or non-treated group used
11. Number of subjects in smallest treatment group exceeds 20
12. Outcome measure has face validity
13. Treatment outcome measure was checked for reliability
14. Reliability measure has value greater than .70 or percent of rater agreement greater than 70%
Criteria for Rating Study (cont.)
15. Those rating outcome rated it blind
16. Treatment outcome was measured after treatment was completed
17. Test of statistical significance was made
18. Follow-up was greater than 75%
19. Total quality (add
20. Effect size = (ES1) = SD units = (mean of treatment – mean of control or alternate treatment ) ÷ (standard deviation of control or alternative treatment)
Criteria for Rating Effect Size
21. Effect size (ES2) = Absolute risk reduction =
(percent improved in treatment) – (percent improved in control)
22. Effect size (ES3) = Number needed to treat =
100 ÷ ES2
23. Was the key variable statistically significant p<.05? __________
o Takes the guesswork out of rating effectiveness studies.
o Is designed so best effectiveness studies score highest.
o Includes indices of study quality and treatment impact.
o Total quality points – provides an index of confidence in the studies validity.
o Indices that may be compared across studies to estimate the relative magnitude of a treatment’s effect.
§ Standardized mean difference (ES1)
§ Absolute risk reduction (ARR or ES2)
§ Number needed to treat (NNT or ES3)
o Instructions for scoring.
o Items 1 to 18 assess quality. These are summed in item 19. Item 19 ranges from 0 to 100. The closer to 100, the more confidence the rater can place in the study’s findings.
o Items 20, 21, and 22 are three relative indices of treatment effect size. These indices summarize the impact of treatment in standardized units.
o Explanation of criteria regarding study quality.
o The first section of the QSRF states the identifying information regarding the client type (e.g., depressed, middle-aged men); intervention method (e.g., aerobic exercise four or more times a week); and the outcome measure(s) – preferably ones that can be used to calculate treatment effect size (e.g., score on Beck Depression Inventory).
o Experience with the QSRF’s reliability indicates that reliability will be highest for all-or-nothing points for each item; so give either zero points or the particular point value indicated if the study meets the criterion, as numbered on the form and described in the following list:
1. Who: The authors describes who is treated by stating the subject(s)’ average age and standard deviation of age, and sex or proportion of males and females, and clearly defines clients’ presenting problem(s).
2. What: The authors tell what the treatment involves so specifically that you could apply the treatment with nothing more to go on than their description, or they refer you to a book, videotape, CD-ROM, article, or Web address that describes the treatment method.
3. Where: Authors state where the treatment occurred so specifically that you could contact people at the facility by phone, letter, or E-mail address.
4. When: Authors tell the when of treatment by stating how long subjects participated in the treatment in days, weeks, or months or tell how many treatment sessions were attended by subjects.
5. Why: Authors either discuss a specific theory that describes why they used one or more treatment methods, or they cite literature that supports the use of the treatment method.
6. Subjects randomly assigned to treatment or control: The author states specifically that subjects were randomly assigned to treatment groups or refers to the assignment of subjects on the basis of random numbers, computer algorithm, or accepted randomization procedures. This means that the procedure resulted in subject having an equal chance of being assigned to treatment or control groups.
7. Analysis shows equal treatment and control groups before treatment. Even though subjects have been randomly assigned, unequal treatment and control groups can occur by chance; so, to guard against this, the authors need to make comparisons across treatment and control groups on key client characteristics to see that they are similar prior to treatment (e.g., sex, race, age, economic status, condition, strengths).
8. Subjects blind to being in treatment or control group: Subjects who know they are in a control group can experience effects of being there including demoralization or competition with experimentals. Subjects who know they are in a treatment group can experience powerful healing effects because they expect them. Give points for subjects blinded if two or more groups get some kind of treatment, if controls get some form of sham treatment that is not expected to have and effect but gives assurance to subjects that something is being done, if subjects serve in a delayed treatment control group where they serve as controls but get treatment later, or if subjects truly do not know whether they are in a treatment or control group.
9. Subjects randomly selected for inclusion in study: Selection of subjects is different from random assignment. Random selection means that subjects are taken from some potential pool of subjects for inclusion in the study by using a table of random numbers or other statistically random procedures. For example, if subjects are chosen randomly from among all residents on a psychiatric ward, the results of the study can be generalized more confidently to all residents of that ward.
10. Control (nontreated) group used: Member of a nontreated control group do not receive a different kind of treatment; they receive no treatment. An example of a non treated control group would be a group of subjects who are denied counseling while others are given group counseling. Subjects in nontreated control group may receive treatment at a later date but do not receive treatment while experimental group subjects are receiving their treatment.
11. Number of subjects in smallest treatment group exceeds 20: Those in the treatment group or groups are those who receive some kind of special care intended to help them. It is this treatment that is being evaluated by those doing the study. In order to meet criterion 11, the number of subjects in the smallest treatment group must be at least 21. Here, number of subjects means total number of individuals, not number of couples or number of groups.
12. Outcome measure has face validity: Face validity is present if the outcome measure used to determine the effectiveness of treatment makes sense to you. A good criterion for the sense of an outcome measure is whether the measure evaluates something that should logically be affected by the treatment. For example, drinking behavior has face validity as an outcome measure for treating alcoholism.
13. Treatment outcome measure was checked for reliability: For this criterion to be met, to merely say that the outcome of treatment was measured in some way is not enough. The outcome measure itself must be evaluated to check its reliability. Reliability refers to the consistency of measurement. The reliability criterion here is satisfied only if the author of the study affirms that evaluations were made of the outcome measure’s reliability (for example, inter-rater agreement), and the author lists a numerical value of some kind for this measure of reliability. Where multiple outcome criteria are used, reliability checks of any one of the major outcome criteria satisfy Criterion 13.
14. Reliability measure has value greater than .70 or percent of rater agreement is greater than 70%: The reliability coefficient in Criterion 13 is .70 or greater. Reliability coefficients typically range from -1 (perfect disagreement), through 0 (no pattern of agreement or disagreement), to 1 (perfect agreement).
15. Those rating outcome rated it blind: This criterion concerns the way bias can enter into measurement if the person measuring outcome knows whether the subject being measure is from a treatment or control group, or, worse, the person measuring outcome is in a position to determine the outcome measure. Give the points for this criterion only if the person conducting the outcome measuring did not know which subjects were in treatment or control groups.
16. Treatment outcome was measured after treatment was completed: At least one outcome measure was obtained after treatment was completed. Outcome measure both during treatment and after treatment is sufficient to meet this criterion.
17. Test of statistical significance was made and p < .05: Test of statistical significance are generally referred to by phrases such as “differences between treatment groups were significant at the .05 level” or “results show statistical significance for…” Statistical significance refers to the probability of obtaining an observed difference between treatment or control groups as great as or greater than by chance alone. Give credit for meeting this criterion only if the author refers to a test of statistical significance for a major outcome variable naming the statistical procedure (e.g., analysis of variance, chi-square, t test) and gives a p value, for example p < .05, and the p value is equal to or smaller than .05.
18. Follow-up was greater than 75%: The proportion of subjects successfully followed up refers to the number contracted to measure outcome compared with the number who began the study. Ideally, the two should be the same (100% followed up). To compare the proportion followed up for each group studies (i.e., treatment group(s), control group), determine the number of subjects who initially entered the study in the group and determine the number successfully followed up. (If there is more than one follow-up period, use the longest one). Then, for each group, divide the number successfully followed up by the number who began in each group and multiply each quotient by 100. If the smallest of these percentages exceeds 75%, then the study meets this criterion.
19. Total quality point (TQP) (add 1-18): Simply add the point values for Criteria 1-18 and record the value in Box 19. This value will range between 0 and 100.
20. Effect size (ES1) or standardized mean difference (magnitude of difference between groups in standard deviation units) calculated by the following:
ES1 = mean of treatment – mean of control or alternate treatment
standard deviation of control or alternate treatment
This formula is for computing ES1 (standardized mean difference or difference in standard deviation units) when outcome means of treatment and control groups are given. To compute effect size from information presented in a study’s report, select two means to compare; for example, outcome might be a mean of a treatment group compared with a mean of a nontreatment control group. Subtract the mean of the second group from the mean of the first group and divide this difference by the standard deviation of the second group. ES1 may be a negative or positive number. If the outcome measure’s score gets greater as client outcome improves and ES1 is positive, then the treatment has had a positive effect, proportionate to the size of ES1. In this case, if ES1 is negative, then the treatment harms.
21. Effect size (ES2) or absolute risk reduction:
ES2 = [(number improved in treatment ÷ total number in treatment group) x 100] – [(number improved
in alternate treatment or control ÷ total number in alternate treatment or control) x 100]
Absolute risk reduction (ES2) refers to the event rate in treatment relative to the event rate in the control group. Assume that you are comparing the proportion in a treatment group who are improved against the proportion in the control group who are improved.
22. Effect size (ES3) or number needed to treat:
NNT = 100
NNT (ES3) is the number of clients that a clinician must treat with the experimental treatment in order to create a good outcome or to prevent one bad outcome in comparison to the control treatment. If controls do better, then this number is the number needed to harm.
23. State whether the outcome variable of interest to you is significant at p < .05
Are you overwhelmed by your class schedule and need help completing this assignment? You deserve the best professional and plagiarism-free writing services. Allow us to take the weight off your shoulders by clicking this button.Get help