Some months ago I happened to be seated next to a person involved in the CA test and this persons enthusiasm for, as what I considered a PC term, distracters simply amazed me. I was informed that if not enough of the examinees fell for the distracter the question and the distractor were adjusted to make the distractor more attractive.
Exactly what that tests is beyond me.
NCEES Launches Computer-Based Testing
-
land butcher
- Posts: 1615
- Joined: Fri Jul 26, 2002 7:26 pm
- Location: calif
-
E_Page
- Posts: 2144
- Joined: Thu Jun 23, 2005 6:49 am
- Location: El Dorado County
Here's the logic of the "distractor", also often described as "logical but incorrect" answer:
When an answer is given that is fully correct, along with one or more that are nearly but not quite correct, it is supposed to help differentiate between those who are competent and those who are not quite competent in that the not quite competent person is supposed to select the not quite correct answer.
Sounds good in theory except that the fully correct and the not quite correct answer probably both look good to the not quite competent person, so the choices for a correct answer get narrowed from 4 or 5 to 2.
That leaves the not quite competent person a 50% opportunity to choose the correct answer on each such question he or she guesses on. With no penalty for wrong answers and relatively low cut scores, that still affords the not quite competent person an excellent chance (and I chose that word to give full effect to its meaning here) at winning a license.
Another problem with the distractor is that unless the person formulating the answers and questions with distractors has broad and deep experience on the matter, the distractor may be the best answer in some circumstances, or depending upon local jurisdictional ordinances or customs. The question writer needs to be very careful to write the problem setup and question in such a manner that it fully eliminates the distractor as a possible valid answer. Since BPELSG has eliminated the appeal, there is only one correct answer in the grading plan, and there is no longer any mechanism to either take broader experience than that of the initial question writer or uneliminated circumstances into account when grading, and no means to challenge a question, there is no way to get points for that distractor if it is actually the better answer.
A clear but open ended question requiring an answer in a sentence or short paragraph containing the elements of the answers from 2 or 3 multiple guess questions on the same topic, would far more clearly, effectively, and consistently demonstrate the level of competence of an examinee.
Some questions appropriate to a professional level exam lend themselves well to multiple guess. But it is an exceedingly poor format to model an entire professional level exam on.
When an answer is given that is fully correct, along with one or more that are nearly but not quite correct, it is supposed to help differentiate between those who are competent and those who are not quite competent in that the not quite competent person is supposed to select the not quite correct answer.
Sounds good in theory except that the fully correct and the not quite correct answer probably both look good to the not quite competent person, so the choices for a correct answer get narrowed from 4 or 5 to 2.
That leaves the not quite competent person a 50% opportunity to choose the correct answer on each such question he or she guesses on. With no penalty for wrong answers and relatively low cut scores, that still affords the not quite competent person an excellent chance (and I chose that word to give full effect to its meaning here) at winning a license.
Another problem with the distractor is that unless the person formulating the answers and questions with distractors has broad and deep experience on the matter, the distractor may be the best answer in some circumstances, or depending upon local jurisdictional ordinances or customs. The question writer needs to be very careful to write the problem setup and question in such a manner that it fully eliminates the distractor as a possible valid answer. Since BPELSG has eliminated the appeal, there is only one correct answer in the grading plan, and there is no longer any mechanism to either take broader experience than that of the initial question writer or uneliminated circumstances into account when grading, and no means to challenge a question, there is no way to get points for that distractor if it is actually the better answer.
A clear but open ended question requiring an answer in a sentence or short paragraph containing the elements of the answers from 2 or 3 multiple guess questions on the same topic, would far more clearly, effectively, and consistently demonstrate the level of competence of an examinee.
Some questions appropriate to a professional level exam lend themselves well to multiple guess. But it is an exceedingly poor format to model an entire professional level exam on.
Evan Page, PLS
A Visiting Forum Essayist
A Visiting Forum Essayist