But once you match up

[austin_dern]

A newspaper article clipped outside someone's office offered this not really novel insight -- multiple choice tests can punish students who think deeply about questions. Their given example:

Which of these animals moves in a way unlike any others?

a) dog. b) frog. c) kangaroo. d) rabbit.

The intended answer was frog, but every one of these is a defensible answer. Still, it's not like defective questions -- ones with many distinct defensible answers -- are unique to multiple choice (or to true/false) questions. I don't use them often myself; it's hard to think of questions that are mathematical enough and test more than trivia and which have several plausible answers. But they are easy to grade, always important, and there's no fussing about partial credit except when a question is defective. That questions can't be too deep can be an advantage; it lets the instructor ask a wide variety of topics. Mix it with some short-form answers and a couple particularly deep questions and you can have a good testing regimen.

Still, I tend to pick questions that look as much as possible like homework questions. I also try to make the exams the ``answer five of these six'' sort; it doesn't take too much more work to think of another question, and students seem to like the sense that they can pick out the hard question and avoid it. Usually one question ends up being the default skipped question that all but a few oddballs don't do. The very best students tend to try all the questions and mark off the one they feel they did worst on; the worst students just grab at everything in the hopes something will stick. The bizarre thing to me is the students' choice of skipped question is almost never the one I'd pick as the hardest and most worthy of skipping.

To sum up, if there's one thing I could advise students who want to make the lives of their instructors easier, please, just get your exam questions perfectly right. That's the easiest thing to grade. Second easiest is when the student skips the question altogether; it'd be fast if it weren't for checking that the student didn't do something weird like write the answer on the wrong page. (Students are capable of all manner of bizarre behavior given the chance.) Just a thought, for considerate students.

Trivia: NASA's Boeing 737-130 Transport Systems Research Vehicle first flew on 9 April 1967. Source: Airborne Trailblazer: Two Decades with NASA Langley's 737 Flying Laboratory, Lane E. Wallace. NASA SP-4216.

Currently Reading: Nathaniel's Nutmeg: How One Man's Courage Changed The Course of History, Giles Milton.

Comments

Hot-dog, jumping frog, Albuquerque!

[tracerj.livejournal.com]

The intended answer was frog? But... frogs and kangaroos and rabbits all hop but dogs don't do that as a means of locomotion, yet all four can swim, so the answer would seem to be dog. Of course, kangaroos are the only ones that increase speed by lengthening gait rather than quickening the pace.... Oddly, I cant find a defense for rabbits.

Talk about defective question. I'd have battled bitterly over that one. Then again, that's part of why my teachers hated me, I was as good at arguing with them as they were with arguing with other teachers.

Re: Hot-dog, jumping frog, Albuquerque!

[austin-dern.livejournal.com]

The unique trait about rabbits is they're the ones who (routinely) move while underground. The thinking behind the choice of frog as the intended answer was that frogs were the only ordinarily amphibious critters.

Of course the question was picked as lead for the article because it was hopelessly malformed. More ordinary questions on, for example, comparing the density of common objects (``which is densest, a block of wood, of plastic, of rock, or of water?'') leave less wiggle-room, though there's still quite a bit left in there.

Re: Hot-dog, jumping frog, Albuquerque!

[bunny-hugger.livejournal.com]

My first choice would have been "dog." As for considerate vs. inconsiderate students... the worst are those who, having not attended class in several weeks, encounter a truth table problem on the midterm and decide to come up with their own way of setting up a truth table, using bizarre extra columns or arrows pointing this way and that. I used to pick through such messes to see if anything stuck, but by my third time or so grading logic exams I decided that if they can't set it up right, they won't get credit for it. I have better things to do with my time. Early on I realized that I should provide the columns of T's and F's on the left of the table FOR them, as well as drawing guide lines for the rows; this was intended to prevent people from managing to set up the rows technically correctly but in a nonstandard order because that makes it really slow to grade. But my providing that part of the table for them still doesn't head off those people who want to invent their own truth table format entirely.

Re: Hot-dog, jumping frog, Albuquerque!

[austin-dern.livejournal.com]

Oh, gah! Truth tables. I had a lot of them for a logic course I was adjunct instructor for a year and change ago. They're great questions -- simple to set up, unambiguous answers, easy to grade -- as long as students figure out how to set the pattern of true and false for the propositions. Even after I pointed out the rules, and pointed out why being systematic in giving T and F to the propositions helps immensely, and showed the two obvious systems to use (alternating most rapidly for the first or for the last proposition), there were four or five students who just would not do it. I think one of them managed to turn a truth table with three propositions into something with 24 rows. Gah.

I never started drawing skeletons of the tables in for students ... perhaps foolishly I admitted to my students they didn't have to follow this system, and you could get the correct result by other routines. It pains me to lie to my students and pretend there's only one correct approach to something or other, but sometimes it really does save confusion on their parts and frustration on mine.

On the midterm that class I made another classic mistake; one of the three-proposition truth tables was a tautology. Some of the students panicked when every single row came up 'T' and changed the answer just so something would be false. I keep forgetting not to toss out really easy questions in the middle of an exam.

Re: Hot-dog, jumping frog, Albuquerque!

[bunny-hugger.livejournal.com]

I never lied about it; I told them that this was just a convention, but I insisted they use it anyway (and made it pretty much impossible for them not to). Every logic textbook I've ever seen (I believe the total I'm familiar with is four, but two of those have the same author) has the rightmost row as TFTF... I think it's reasonable to insist that people use the standard because that's what people will expect to see when they glance at the table.

I should point out that the class I'm talking about is actually mostly an informal reasoning course, and so people (who thought they wouldn't have to do anything that looks too much like math) absolutely panic when we do the brief section on truth tables and Venn diagrams.

Re: Hot-dog, jumping frog, Albuquerque!

[austin-dern.livejournal.com]

Good justification. If I teach a course like that again I'll have to use it.

It sounds like your class was the sort of thing mathematics and physics majors use to satisfy the liberal arts requirement without really learning something new ... come to think of it, so was one of my courses this last term (a history of algorithms course). People get bizarrely terrified when things look too much like mathematics; it's a shame there's not more ways to sneak up and present the subject without hitting ``panic words'' like theorem or proof.

[bunny-hugger.livejournal.com]

Conversely, one of my math/science requirements as an undergrad was satisfied by taking symbolic logic, also required for my major. >:) Obviously, that's a much more formal course than the "logic and reasoning" thing I taught at MSU. At the time (and maybe still) the math and philosophy departments took turns teaching it.

[austin-dern.livejournal.com]

Philosophy, I've come to conclude, doesn't get enough respect from the mathematics and physics majors, who get really unjustifiably snotty when the field is mentioned. Historic ties alone should justify more respect, but there's this elitist drive most majors get. I suspect it's overcompensation for middle school.

[(Anonymous)]

I also try to make the exams the ``answer five of these six'' sort; it doesn't take too much more work to think of another question, and students seem to like the sense that they can pick out the hard question and avoid it...the very best students tend to try all the questions and mark off the one they feel they did worst on; the worst students just grab at everything in the hopes something will stick.

I remember going through the entire range on these types of tests (and most often, they were given in a mathematics course); the ones where I tried to answer all the questions were rare, but happened earlier on in my mathematics career.

I think the worst I ever did was for a calculus exam I was hopelessly unprepared for. I answered all five (20) point questions; two reasonably, one grasping at straws, and two with Greek tragedy, in Greek lettering. I remember hoping that, explicitly against the spirit of the test, they'd just grade all five questions and give me the four highest scored. In my mind, this would have added up to some sort of statistically-higher-than-average-made-up-answer-grade based on pity for me. At least, was the ill-conceived plan I came up with somewhere in the middle of this exam. Unfortunately, the graders saw through my plan; if I answered all five, they just took the scores from the first four questions.

That exam didn't go too well.

I remember feeling a little bad for the graders of some of my mathematics exams. Grading computational answers where it's possible to go down the wrong track, double back, and still get the right answer despite your best efforts to the contrary can't be easy. At least with essays I felt more in control on the occasions when I knew I was spouting nonsense and hoping for the best. But then, I did do better at essay exams overall.

-Skyler

[austin-dern.livejournal.com]

Well, now, not specifying which one to skip is invoking undefined behavior, after all ... usually when I get that I'll grade them all and drop the low score, but I'm an old softie.

But ... yeah; desperation grabs at points just don't fool the instructor. Really. You may get some points out of pity, but is it better to fail with a 45 than a 40? Does it matter?

[xolo.livejournal.com]

I'd have gone with Kangaroo, since they use their tails to help them move at low speed.

[austin-dern.livejournal.com]

That's a good reason, though their high-speed hopping is the best reason to pick that. The gait there is a fundamentally different pattern than other animals running. I've been to a few talks about this, actually; it turns out you can make a ridiculously simple model of quadruped motion, and the various common walks and hops -- including the pronging of gemsbok and similar animals -- correspond well to the natural oscillation modes.

[oliver-otter.livejournal.com]

Kangaroos also hop in time with the resonance of their chest cavities. Each hop provides a deep breath of air (for both cooling and metabolism) without any extra effort.

[austin-dern.livejournal.com]

I didn't know that; that's rather neat. Why don't they mention that when kangaroos appear on nature shows, instead of asserting that kangaroos can't move their hind feet separately?