The Acommodations + Accessibility

If you’ve never heard of "Accommodations + Accessibility" it’s probably because you don’t require their services, and this post is not for you. If you want to know more, check out their website. They were formerly known as Accessibility Resources (AR), and before that, Student Accessibility Services (SAS), and before that, Specialized Support and Disability Services (SSDS). Yes, it's hard to keep track after so many different names over the years.

This post has been written for students in my classes who write their exams under the supervision of Accommodations + Accessibility. Specifically, it’s to explain some gaps in their procedures, and how we can work together to ensure that exams run as smoothly as possible for you, for me, and for the rest of the students in the class. It also relates to the reason why Van Halen insisted that there be no brown M&Ms in their dressing room.

The Letter
First, I need your letter. You know the one. The Letter of Accommodation. This introduces you, and lets me know that you're going to write your exams with Accommodations + Accessibility. Log in to ClockWork, complete all the required information, and then it will send me an email notification. You must let me know--as soon as possible, if not sooner. Do not give me your letter a day before an exam. Do not give me your letter a week before an exam. (Accommodations + Accessibility actually requires me to submit exams to them at least one week in advance, and you have to register for exams at least a week in advance.) I've had students write exams with Accommodations without giving me the letter. This is not good. In the worst case, you show up and there's no exam for you!

Once, after I admonished a student about not giving me the letter sooner, they told me "I'm writing the exam at Accessibility Resources because of my ADHD, and my not giving you the letter is a manifestation of that disorder." Wha--? Do not be that person. Do not make excuses. Take responsibility for yourself.

(If you have already registered with Accommodations + Accessibility and I have been sent your letter, thanks!)

The Exams
I send my exams to Accommodations + Accessibility via their secure website, called ClockWork. (I believe I was the first instructor ever to email them exams in PDF format over 17 years ago to help accommodate a vision-impaired student.) After exams are written, I schlep to the Accommodations + Accessibility office to pick up the exams. I go to this extra effort myself (I do not send TAs to pick up exams) for a couple of important reasons:

1) I calculate a lot of exam statistics. Really, a lot. If I do not include your exam with the rest of the class when the exams are scored by TSQS, I have to mark it by hand. That means it’s not included in any of the exam statistics for the class. I don’t like having incomplete data; I want to get the most complete picture of a class’s performance possible--not leaving anyone’s data out. Plus, I hate hand-marking multiple choice exams and am prone to errors, despite my best efforts.

2) I process and post exam results quickly. Very quickly. My goal is to be faster than anyone else on campus. I have, on more than one occasion, posted exam results the same day (my record is 1 hour after the exam ended). To do that, I bring all of the exams to TSQS personally as soon as possible after the exam. However, if students are writing their exams at Accommodations + Accessibility, they are often given extra time, or write on a different day. Everything gets held up by this.

What if you’re sick or something comes up and you don’t write your exam at Accommodations + Accessibility after all? One thing’s for sure: I don’t know about it. They don't call or email me to say that you didn’t show up. All I know is that I don’t have your exam. This is bad when it comes to midterms, but it’s even worse for final exams. Say you miss a final exam for a legitimate reason. You do what you’re supposed to do: go to your Faculty office and apply for a deferral of the final exam within two working days of the originally scheduled final exam date (NOT the date you write the exam with Accommodations + Accessibility). Great. But in the meantime, I don’t know where the hell your exam is. Do you have it? Does Accommodations + Accessibility still have it? Did you even write it? I don’t know. I would love to process the final grades--students are starting to pester me about why the results haven’t been posted yet--but I can’t, because I’m still waiting for your exam.

If you do NOT write your exam with Accommodations + Accessibility for any reason (incapacitating illness, severe domestic affliction, religious belief, or you just decided to write it with the rest of the class in the classroom), TELL ME as soon as possible. Email is preferable; this gives me a record that I can refer to, if need be.

If I have sent you an email asking you to read this post, now’s the time for you to send me a reply email acknowledging that you have read and understood this post, and agree to the conditions that I have specified. This is the part that relates to Van Halen. For safety reasons, the band wanted a way to ensure that venue operators would read their contract all the way through. One clause stated that the band was to be provided with M&Ms backstage--with the brown ones removed. If there were brown M&Ms, their contract was not read completely. Snopes explains it all. So, if I don't get an email from you, it's like there are brown M&Ms. Thanks.

If I haven’t asked you to read this post, well...

Why aren’t you studying?

(Updated 9/29/2024: updated Accommodations + Accessibility name and took out much that has become irrelevant in the past seven years.)

The Exam Statistics: The Q Score

This is my final post on the topic of exam statistics. Previously, I described my use of the mean, difficulty scores, and point-biserial correlation. This time: the dreaded Q score. (Just to clarify: I'm not describing the other "Q Score", which represents the public's familiarity with--and appeal of--a person, product, company, or television show. That's not dreaded at all.)

The dreaded Q score is not a statistic that I regularly receive with all my other exam stats. I have to put in a special request. It's extra work for the people over at TS&QS, which means there's an additional cost that must be paid by the Department of Psychology. TS&QS has to go into their database of exam results for my class and perform a statistical comparison between two (or more) given exams.

Here's where the dread comes in: Why would I want to statistically compare two (or more) students' exams? If I suspect them of cheating, that's why. Sometimes, the cheating is blatantly obvious. The cameras in the classrooms (you know about those, right?) may clearly show one person peering over at the exam of another. Other times, it's not so obvious. Why is that guy jittering in his seat, looking everywhere except at his exam? Maybe he's nervous, or has exam anxiety. Why is that girl acting squirrelly, shifting her eyes back and forth? Maybe she drank too much coffee, has caffeine overload, and now really, really has to pee. Whatever the case, the exam proctors will not interrupt any student taking the exam. Nope. We'll just let you do what you do. If that's cheating, so be it.

However, at the end of the exam, the answer sheets from any suspicious students are set aside. (Think you can fool us by not leaving at the same time, or handing in your exams to different proctors? Tsk. You don't know how many eyes are watching, do you?) Those answer sheets will be analyzed, and I will get the dreaded Q score. I don't want to say too much about how it works, so suffice it to say that it gives a probability that cheating has occurred, compared to chance. Maybe I'll write more about how I have to deal with cheating in another post, but for now, let's just say it involves a lot of dread.

Although I have caught several cheaters over the years, I'm glad I've never had to deal with anything like what happened in Professor Richard Quinn's class recently. Yeesh.

(Cartoon by Frank Cammuso. It's important to give credit where credit is due. Otherwise, it's like, um...cheating.)

Why aren't you studying?

The Exam Statistics: The Point Biserial Correlation

I'm continuing my explanation of the reams of statistics I get about multiple choice exams. Last time, I explained exam item difficulty scores. (Fascinating, no?) This time: point biserial correlation coefficient, or "rpb". That is, "r" for the correlation coefficient (why, oh why is it the letter r?) and "pb" to specify that it's the point biserial and not some other kind of correlation. Like, um, some other kind.

If I've constructed a good exam item, it should be neither too hard nor too easy. It should also differentiate among students. But I can't tell how well it does that just by looking at the difficulty score. Instead, there's a more complex measure, the rpb. In general, I need a correlation index for a categorical variable with a continuous variable. More specifically, I want to correlate the categorical variable of a test item (i.e., whether a student answered the test item correctly or incorrectly), with the continuous variable of the student's percent score on the examination. Got that? I didn't think so.

Let me try again. Student A did well on the exam, getting 90% correct. Student B did not do so well, getting only 50%. If I look at any given exam question, in general, student A should be more likely to answer it correctly than student B. This is not the same as difficulty, because I'm not simply looking at what proportion of the class answered the question correctly. I'm correlating each student's score with their performance on each question. The key to all this is the word "should" in the sentence above.

If an exam item is poorly constructed for whatever reason, good students may do worse on it than students who did worse on the exam. That is, the better you are overall, the less likely you are to answer it correctly. That is not supposed to happen. The rpb gives me this information for each question on the exam. Experts in exam construction recommend that the rpb should range from 0.30 to 1.00. Anything question getting a rpb lower than 0.30 means that I will take a look at it and try to figure out why that's happening.

And if the rpb is negative, well...it's a negative correlation. That's the worst case I described: better students are doing worse answering this question; and poorer students are doing well. I won't use any questions getting a negative rpb again unless I can figure out why it's happening. Maybe I can tweak the question, maybe I have to rewrite it to ask about the same knowledge in a different way. Or maybe I'll just give up entirely, go and get a coffee, and check out some LOLcats.

Why aren't you studying?

The Exam Statistics: The Difficulty

In my last post, I discussed how I analyze the mean in my (multiple choice) exams. This time, I'm going to look at difficulty. This is not directly related to the mean. Huh? Isn't it the case that, the more difficult the exam, the lower the mean? Well, yes. But that's not the "difficulty" I'm writing about.

Among all the pages and pages of results I get from Test Scoring & Questionnaire Services is the "DIF" score or difficulty of each question. It's actually the proportion of the class who answered that question correctly. DIF=1.000 means that everyone got it right, but DIF=0.250 means that only 25% of the class did. But it's not really "difficulty," is it? If a question is really difficult, fewer people will answer it correctly and the number should decrease. So, really, it shouldn't be called difficulty, it should be called easiness. But, look, it's just called "difficulty," OK?

You might be thinking that I want everyone to answer every question correctly, right? Um, sorry to rain on your ice cream, but...no. It's really, really unlikely that everyone was able to learn absolutely everything in the course, and was also able to remember and apply that knowledge on an exam perfectly correctly for every question. What an exam should do is assess each student's learning of the material, and provide some way of differentiating among all students. If all questions are answered correctly, the exam itself has failed.

I went to a seminar last year at which a renowned expert in testing and exam question construction gave a talk. After it was over, I talked to him about DIF scores--specifically, what should they be? The general rule is that an exam question is doing a good job of differentiating among students if it's at least 0.300. That is, at least 30% of the class should be getting each question correct. There is no guideline for the upper end, but at another seminar, I heard an instructor say that she liked to put at least one DIF=1.000 question on each exam as a confidence booster. Yup, a gimme. I thought that was a pretty nice thing to do, so I try to include at least one high DIF question on every one of my exams, too.

So difficulty is related to the mean in that, the higher the DIF, the higher the mean on the exam overall. The mean is good for evaluating the overall performance of the class. But I also need to evaluate the questions on my exams, so I get the DIF score for each one. If the DIF is too low, the question either gets killed (*snff*), or rewritten to clarify it. Oh, and if I ever get DIF=0.000, it means I've keyed in an incorrect answer. Ooops.

Why aren't you studying?

The Exam Statistics: The Mean

With the first round of (multiple-choice) midterms over, I'm now swimming in data. I want to tell you about some of the stats I go through to assess and improve my exams. Unfortunately, I'm too late to celebrate (the first) World Statistics Day. But I don't feel too bad. At least statistics has a day. It's not like there's a "Psychology Month" or anything. Oh, look--yes there is. And I'm late for that, too. Moving on...

This installment is about the (arithmetic) mean, or, if you insist, the "average." I post the class mean of every exam because you demanded it! Really, though--what use is it to you? For classes that don't grade on the curve, you don't need to know the mean (or standard deviation) to determine your absolute standing in the class. Just take your percentage correct, and see what grade that corresponds to in the syllabus. Right?

Yes, that's important. But don't you want to know how everyone else did, too? Sure you do. "Did everyone think that exam was a killer, or just me?" We want to compare ourselves to other people. Some students even want to know what the top score was. "Did anyone get 100%?" "Am I the best in the class?"

The mean also serves another purpose, when there are multiple forms of an exam. In larger classes, multiple forms of an exam are used to discourage cheating (or at least, to make it more difficult). Typically, there is one form that has the questions arranged in order of topics (e.g., questions based on the first lecture and textbook chapter first, followed by questions on the second lecture and chapter, etc.). The other forms will have the questions in a random order. Are students who get the scrambled forms at a disadvantage? Or, put another way, is there a benefit to answering questions in a sequence that reflects the arrangement of the learning materials? If so, that wouldn't be fair, would it?

The data from every exam includes the means from each form. They are usually a little bit different. But is that difference a fluke, or is it due to the ordering of questions? Hmm, sound like a job for...statistics! The data also includes the results of an ANOVA (analysis of variance) that compares the means to each other. That is, are any differences statistically significant? The answer: No. I've never had a difference at p < 0.01 or even p < 0.05. That means any differences are small; they are due to chance.

The bottom line: It doesn't matter which form you get. Isn't science cool?

Why aren't you studying?

How to Get Free Marks

You want free marks? Sure, no problem--all you have to do is prove to me that I'm wrong.

It doesn't happen often, but it does happen. Earlier this term, Melissa T. asked me why the teaching assistant had marked an answer wrong on one of her assignments. She had written an alternative answer that sounded plausible to me, so I told her about my free mark policy. Melissa came back a couple of days later with a strong journal article supporting her point, and she ended up with a free mark! (To be fair, my TA stuck to the marking guide I provided, so it's not the TA's fault Melissa initially lost a mark. It's impossible to create a marking guide that covers every possible answer, and every variation on an answer. But that's another post for another day.)

You might have learned something in another one of your classes that contradicts what I say (or what the textbook says). This shouldn't be surprising--"facts" change all the time in science. (Like, old-time doctors used to prescribe smoking. And cocaine. Nice going, stupid old-time doctors!) The instructor of one of your other classes might be up on some brand-new study that hasn't worked its way into the textbook I use, into my lectures, and into my exams. So why should you be penalized for knowing more? Or, perhaps, for knowing something in more detail?

So, to get a free mark, all you have to is:

• find an error in the marking of an exam or assignment (sorry, no free marks for correcting an error in the textbook--that is worth $2, however--and no free marks for pointing out an error in my lectures, although you will have my genuine gratitude)
• find good (i.e., peer-reviewed, primary source) evidence to support your answer (sorry, Googling some random website or even finding empirical evidence cited in a textbook is not good enough--it's got to make it through the peer-review process)
• find current evidence (sorry, digging up a peer-reviewed paper from 1847 may not qualify if subsequent research has undermined it)
• give me your supporting evidence at least one day before the final exam
  

This is a great win-win situation: you end up getting a free mark, and I end up with a exam or assignment question that's more relevant, up-to-date, and fair.

Why aren't you studying?

The Marking

Just a short post to interrupt my marking. (Hmm, procrastination--another good topic a future blog post...)

Marking is not the most enjoyable thing. This fact has nothing to do with the quality of the answers. It's just awfully repetitive. I get to read answers to the same essay question over and over and over... This is by choice, mind you. When it comes to written/short-answer/long-answer I try to mark as many of the same question as I can, so that I can be as consistent in marking as possible.

So I get a fresh cup of coffee (hmm, coffee--another good topic for a future blog post...), put on some quiet music and concentrate on marking. This is difficult, as interruptions and distractions are many. Luckily, it's just me and the baby this afternoon. Awesomely, I've discovered the perfect music solution for both of us: Rockabye Baby!

These are baby lullabies of rawk songs, and improbably include songs from:

• The Beatles
• U2
• Bob Marley
• Coldplay
• No Doubt
• Metallica
  
• Led Zeppelin
• Radiohead
• Green Day
• Nine Inch Nails
• AC/DC
• and more.

You've never lived (as a parent) until you've heard American Idiot, Enter Sandman, or Highway to Hell played as a gentle, soothing lullaby. Insidious, bizarre, and just about the only way to simultaneously satisfy both a cranky 3-month-old and a cranky psychology instructor. (Country music fans, don't feel left out: check out Hushabye Baby.)

Why aren't you studying?

The Begging

Usually, I like to get email from students. It's great to have the chance to help explain a concept or idea that someone's having trouble with (or to correct an error I might have made--oops!). Sometimes, I get links to interesting websites that relate to a course. I even get some from students after they've finished I course I taught. So they're still thinking of me. >snff<

Hi,

I really loved your course, u r a great instructor. I just wanted to wish you a nice holiday.

BTW, I was just wondering about how you round up the marks. I have a total of 54.2%. Would it be possible to get that rounded up to an A+? I really want to get into medical school--it's been my dream since I was a fetus, and the only thing standing in my way is my mark in your course, which is an F- right now.

Thanks!!!

Now, it seems like the person just wants to know about how marks get rounded up. (I apply standard rounding, also known as the common method.) But, er, it's not just asking about that, is it? It's begging.

There are two main problems I have with this. First, no. No, nyet, nein. I'm not going to change your mark because you asked. That's not how it works. It's embarassing for you to ask. So I'm not going to respond. If you've got a legitimate concern about the calculation of your grade, that's one thing.

Second, by the time the final is over, it's too late. There are so many things I could do to help you during the term. Did you come to my office hour to get help with things you didn't understand? Did you look over your midterms? Did you spend time checking out my study resources? Did you at least send me questions by email? No? Why not? It sure looks to me like you're either doing so well that you don't need any help, or you're not particularly engaged in the course.

After the final is over, there's nothing more I can do. If you fail the course, you can apply to rewrite the final, which you know if you've read the Calendar. But no, I can't simply bump you up a grade.

OK, so I don't want to be mean. I realize that in this age of instant gratification, after seeing your final exam mark, many students just want to know their grade in the course. It does take quite a while to get from me, through all the bureaucracy, and into your transcripts online. But you should be able to calculate your grade yourself: Just get your overall weighted total, and look up in the syllabus what grade that corresponds to. No curve--what grade you get simply depends on your overall percentage. Unfortunately, I'm not allowed to send marks/grade information over email. So you're going to have to be patient.

Why aren't you studying?

The Exam Structure

You know that it's important to look at your midterms to see where you went wrong, don't you? Just sayin'...

When you look at your exam, there are things you should look for:

• On how many questions did you have to guess the answer?
  

• Which lectures/chapters give you problems?

• Did you devote less time to studying those? (Were you absent for those lectures?) Maybe a neuroscience chapter was particularly challenging, because of all the anatomical terms. On the other hand, maybe you got those questions all correct because you spent so much time on them--at the expense of other material.
  

• What kind of questions gave you the most difficulty?

Let me explain what I mean by "kind of question." There's a way of categorizing different multiple-choice questions called Bloom's Taxonomy. I use a slightly less complex version that has three categories of questions:

1. Factual: assess your knowledge of definitions, terms, and other facts. Answering these questions requires remembering of learned information. For example:
   
   Who wrote The Principles of Psychology in 1890?
   (a) Wilhelm Wundt
   (b) Sigmund Freud
   (c) William James
   (d) B. F. Skinner
   
   
2. Application: evaluate your ability to apply learned factual knowledge to a new situation, or to solving a problem. You'll have to have the background knowledge, but also the ability to see how it relates to a previously unseen example. For example:
   
   Assuming the Weber fraction is 0.03, given a 1,000 g weight, what is the JND?
   (a) 1,000 g
   (b) 3 g
   (c) 30 g
   (d) 0.03 g
   
   
3. Conceptual: tests your capacity to see patterns, relate knowledge from different areas, evaluate evidence. Not only do you have to have the background knowledge, you've got to be able to analyze it, synthesize it with new information, and evaluate what you know. For example:
   
   Which of the following phenomena of operant conditioning contains within it a form of classical conditioning?
   (a) latent learning
   (b) overjustification effect
   (c) secondary reinforcement
   (d) delay of gratification
   

Multiple choice exams usually contain a large number of factual questions. There will be a few application questions, and also a few conceptual ones. It's hard to write good conceptual questions, and they can be quite challenging to answer. If you're not doing well on the factual questions, you're probably going to struggle with the other kinds as well.

But problems just with application and conceptual questions reveal that your understanding isn't as...deep as it could be. You may be great at memorizing definitions, but that's not all there is to taking a (well-designed) multiple choice exam.

(BTW, Bloom's Taxonomy can also be used in short-, medium-, and long-answer exams. In fact, I use it in mine. For example, in my "medium-length" questions, I ask students to 1. define a term, 2. give an example, and 3. explain why it's important or what other concepts it relates to. See? Bloom's Taxonomy.)

In case you must know, the answer to all the questions above is (c). When in doubt, pick (c), right? ;-)

Why aren't you studying?

The Exam Marking

And...done. Done marking midterms, that is. No, not the multiple choice ones. Actual long-answer, hand-written, sweat-stained essay exams. Exam on Thursday, done marking on Saturday. Not too bad. I once finished them in one day. Ah, but I was younger then, with eyes that focused and no daughter who always wants to play My Little Pony with me (I get to be Pinkie Pie!).

I try really hard to get exam results back as quickly as I can. As soon as a multiple choice exam is over, I walk it over to TSQS (Test Scoring and Questionnaire Services, of course). Many instructors just go back to their office and have their administrative assistant schlep them off via campus mail. Me, I prefer to have results back as quickly as I can. My best time ever was 2 hours, from exam over to results posted online.

I've heard that there should be a "cooling-off" period, but I don't know exactly why. Heck, I know a prof who has exams on which you scratch off your answer, like a lottery ticket. If you're right, you get full marks. If you're wrong, you get to scratch off another answer. Get that one right, and you get part marks. It's pretty cool, but it's apparently expensive to get the exams made up that way.

I like the nice people at TSQS. They've even helped me out of tight spots, when I've needed results back in a hurry. Normally, they promise 24 hours, but they regularly beat that, except at busy times, like during final exams.

Do you care either way? Would you like to have scratch-and-win exams? Or do you prefer to wait, even if you know the results have been posted?

Why aren't you studying?

The Exam Aftermath

So, some midterms are over. When I was a student, I would try to reward myself after every exam. I'd often go and buy a stack of comics, or get a big bowl of Smartfood and watch a TV show I'd taped on the VCR.

What do you do after a big exam is over?
And then...

Why aren't you studying?

The Exam Prep: 1

Midterms are fast approaching. Here's some advice that's a bit obvious, but maybe also helpful:

• Do read the textbook. Stunningly obvious, no? Unfortunately, some first-year students don't realize that at university, you're expected to actually read the textbook. In my course syllabi (academic word for "course outline"), I have a table which lists my lectures by week. There's a separate column for assigned readings. Typically, each lecture topic corresponds with a textbook chapter.

• Do read the textbook in the right environment. TV on? Rockin' your iPod? Tickling your sweetie? No. No. No. All of these things use up your attentional resources and distract you from reading your textbook. Sweetie will have to wait.

• Don't use a highlighter. Take a look at past chapters: have you just been "painting" your textbook? Highlighting is laziness; you're just going, "This looks important. I better remember to actually learn that--later." If there's an important term, it's already in bold.

• Do take active notes. After reading a section of a textbook, try to summarize what it was all about in your own words. Don't just copy definitions word-for-word. Yes, this takes a lot of time and effort. Know what? The learning takes place not when you review these notes, but when you make them. If you also have time to review them, so much the better.

• Do review your lecture notes. Sometimes things make perfect sense when you've just heard my explanation of it in class. But a month later, you might look at the same stuff and go, "What?" If you don't understand something, your instructors (or the teaching assistants) can help--if there's enough time left before the exam.

• Do make up flashcards. You may feel like an x-treme keener, but it works. On one side of a card, write an important term. On the other, write the definition. Shuffle the cards, and pick one. Do you recognize the term? Can you write down the definition on a separate sheet of paper? You may be able to do both of these things, and conclude that you're wasting your time. Not so: you added to your learning when you made the flashcards.

• Do get enough sleep. There are so many studies that show how important sleep is to learning (and other things), it's not even funny. And why are you pulling an all-nighter anyway? Aren't you keeping up with the assigned readings?

These should be enough tips to keep you busy until your midterms. Tips for writing exams are coming up!

Why aren't you studying?

The Exam Writing: 1

Midterms are approaching like a freight train, like a airplane coming in for a landing, like a puppy whose leash has broken. Here are some quick exam writing tips, for multiple choice exams mostly:

• If noise bothers you, wear earplugs at the exam. (No, not earphones, sorry.)

• Read the question--no, seriously. Don't skim it, you're bound to miss an important word.

• Think about the answer. Don't read the choices, not yet--they are sure to distract you. Jot your answer on the exam booklet, if possible. Then look at your choices.

• Rule out obviously incorrect distractors. (That's what the incorrect choices on multiple choice exams are called. They're there to distract you. Seems kinda mean. Still, it's better than the term, "this'll-fool-'ems")

• Can't decide on the correct answer? Is it eating you up inside, like last night's extra spicy four-bean chili? Are you starting to obsess over it? You're wasting time. Move on to questions you can actually answer. Come back to more difficult ones later. You can guess on them if you can't figure them out--that's better than leaving them blank (yes, I've seen this happen).
  

• Take extra care when questions have negatives; you have to spend more time working out the negatives, or even double negatives.

• Be careful when deciding to change your answer. Research has shown that changed answers are frequently wrong.

• Put your name on the exam.
  

Why aren't you studying?