Category Archives: Assessment

H.O.T. Multiple Choice Exams

Assessment

Catherine ScottCatherine Scott, Asst Dir for Faculty Dev – Testing, Assessment & Evaluation

What do you remember about taking exams? The stress? The anxiety? Trying to decipher which concepts to memorize? Which information the instructor stressed as important? Or maybe you completely blanked out like I did more times than I can count! As a professional, I can remember taking hundreds of exams, but unfortunately, I can not remember all the information I crammed into my brain to pass them. So, what was the objective of these tests? To make me stressed or help me learn? In college, students tend to approach multiple choice exams the same way they did in high school, utilizing the test taking strategies that help to eliminate the wrong answers, instead of knowing the correct one.  

Quick story. I have two sons, both bright in various areas. My oldest, teachers claimed, had things come to him easily. However, he always stressed taking tests! He would literally get sick and then perform poorly. If you asked him later about the same material, he knew it all. He was even given resources that later were taken away because he proved he did not really need them. My other son struggled in class but aced the tests. I knew something was going on but found it difficult to get him the resources he needed because he always did so well on the tests. He was later diagnosed with an extreme case of dyslexia and what we found out was he is just a great test taker. In fact, many times he was just utilizing the process of elimination and then guessing. I often use my children as a real-life example to remember when I’m assisting an instructor with H.O.T. exams.  

Let’s make those exams H.O.T. 

What does H.O.T even mean? Higher Order Thinking. A concept born out of the research of Benjamin Bloom, a Harvard professor who studied three domains: the cognitive, affective, and psychomotor. Higher order thinking resides in the cognitive domain and can be useful to classify different hierarchical levels of understanding that students can achieve in a course. 

Bloom's TaxonomyBefore incorporating Bloom’s into designing your exams, here are some strategies you can take to strengthen your questions. Each question has a stem, which presents the problem; a correct option, which is the right answer; and the distractors, which are the incorrect options used to distract you from the right answer. 

Tips for strengthening your stem

  1. Articulate stems around your course learning objectives.
  2. Start with lower-level items first to help build confidence.
  3. Test what you really want individuals to learn. 
  4. Ensure that the directions in the stem are very clear; sometimes students do not understand what is being asked. 
  5. Include the central idea in the stem instead of the choices.
  6. Avoid window dressing (excessive verbiage); include only what is necessary to the question. 
  7. Word the stem positively, avoid negatives such as NOT or EXCEPT. If negative words are used, use the word cautiously and always ensure that the word appears CAPITALIZED and boldface. 
  8. Check borrowed items carefully; sometimes we find questions used by others but are not aligned with our own course objectives. 
  9. Get feedback on items; ask others to take the exam and provide feedback. 
  10. Compose test items over time; make sure you review each semester to ensure the content still applies. 

Tips for strengthening your correction option and your distractors 

  1. Develop as many effective choices as you can, but research suggests three distractors are adequate.
  2. Make sure that only one of these choices is the right answer.
  3. Utilize or perform an item analysis to determine the difficulty index. 
  4. Vary the location of the right answer according to the number of choices.
  5. Keep choices homogeneous in content and grammatical structure; students looking to guess will try to determine which responses visually look different. 
  6. Incorporate typical errors regarding the content; which mistakes have students made in the past?
  7. Make them plausible; if not, the answer may seem obvious.
  8. Use humor cautiously, especially if you do not use humor in your lectures. 
  9. “None-of-the-above” should be used carefully.
  10. Avoid giving clues to the right answer.

After you have incorporated the above suggestions, you’re ready to utilize Bloom’s taxonomy to ensure you are measuring higher learning. Use the chart below to help design some of your exam questions. First, which level of Bloom’s will you be measuring?  This will help you determine how students will conceptualize the material, which will help you decide how to ask the questions. 

Bloom's Levels with exam question examples If you have questions or would like to discuss further, please contact us.

Plagiarism: Opportunities for a Teachable Moment

Assessment, ,

Troy Priest  Troy Priest, Senior Instructional Designer

As academics, we understand that our professional reputations depend on us properly using and citing scholarly sources in our writing and research. Every so often we hear of academic reputations being ruined by allegations of plagiarism. 

As instructors we stress the importance of academic integrity to our students, and we require them to adhere to the codes of conduct laid out by the university including avoiding plagiarism in the writing assignments. But are all instances of plagiarism the same? Should we deal with all examples of plagiarism the same way?  How can we use and make them teaching opportunities?

Intentional vs. Unintentional Plagiarism

person typing on computer with question marks around themNot all instances of plagiarism are intentional, but when they are it is usually fairly obvious. The most egregious examples of plagiarism include turning in another student’s paper and claiming it as one’s own work, having someone else write one’s assignment, or copying and pasting sections of text and passing them off as one’s own ideas. These forms of plagiarism are clear-cut examples of academic dishonesty and must be dealt with appropriately.

Unintentional Plagiarism is less nefarious and may provide us with a teaching opportunity. The ability to find and select relevant and reliable scholarly sources; successfully incorporate those sources (including deciding whether to use direct quotations from the source or paraphrase the ideas) to support a coherent and cogent argument; and properly cite those sources according to the style guide of the course discipline can be daunting for students who haven’t practiced these skills. Many students have been ‘taught’ these academic skills, but because they are skills, students need to practice them to be proficient. 

This is particularly true of students who are non-native speakers of English. Many second language learners may struggle with paraphrasing scholarly sources or understanding how citations and sources function within the formation of an academic argument. Therefore, successful integration of sources into their writing may be more challenging. 

Strategies to Prevent Plagiarism 

As a former writing instructor, I have learned it’s important not to make assumptions about what students are able to do with regards to using scholarly sources. Just because they were ‘taught’ how to use sources and citations before doesn’t mean they are necessarily proficient at it. It is a skill that has to be practiced.  For those of us who have spent many years in university, both as students and teachers, citing sources has become so integral to what we do that we often take it for granted. For our students, this may not be the case. Providing students with resources and opportunities to practice is important. 

Some specific strategies and resources to help students are: 

  • Clearly define what you mean by plagiarism. For many of our students it may not be clear what constitutes plagiarism or what is considered appropriate use of sources or citations. 
  • Providing students with models of writing that incorporate and integrate sources and citations effectively. These examples should model the kinds of work and scholarship you require from your students for the course. Using peer-reviewed articles as examples may not be helpful to students.
  • Require rough drafts. Writing is developmental, and breaking up the writing assignment or research project into parts allows you to check in with students and give feedback. This ensures they are on the right track and eliminates surprises when the final paper is submitted. If for example the assignment is worth 20 points, break up and distribute the points for the assignment to each step or submission.
  • Incorporating annotated bibliography into the assignment. Annotated bibliographies  ensures that students have engaged with the texts and are able to paraphrase and/or summarize the ideas of the authors. Paraphrasing and summarizing are advanced linguistic and cognitive skills. These skills can be particularly troublesome for second-language learners.
  • Use plagiarism tools such as Safe Assign as educational resources rather than just detection tools. Safe Assign is integrated in Blackboard and is a useful tool for instructors to check to see if a paper is original or has been plagiarized. However, Safe Assign can also be a learning tool. You can allow students to use these tools to check for plagiarism in their own papers before submitting them to you. This allows them to see if they inadvertently left out a citation or failed to paraphrase or quote a source.
  • Inform students of support services. There are university  resources available to students as well. These include The Writing Center, Academic Success and Tutoring Center, and Stony Brook University Libraries just to name a few.

If you have questions or would like to discuss with one of our instructional designers ways to incorporate these strategies in your course(s), please contact us at celt@stonybrook.edu.

Engaging Students in an Online Math Course During the Pandemic

Assessment, Educational Technologies, Online Learning, , ,

Editor’s note: This is the first of a two-part blog series on choosing the right technology for your teaching needs. In this blog post, Associate Professor of Mathematics, Moira Chas, discusses her experience with finding a way to engage her students during synchronous online lectures. Look for the second blog post by CELT Instructional Designer and Technologist, Jennifer Jaiswal, who will describe how to select the appropriate technology. Email CELT@stonybroook.edu to get a consultation with an instructional designer who can work with you to choose the right tools for your teaching goals.

Dr. Moira ChasMoira Chas, Associate Professor, Mathematics 

Image courtesy of Moira Chas, who is seen in her office with some of the crocheted models she has created to illustrate ideas in topology.

Before the terrible pandemic that turned our worlds upside-down and inside-out, (or better said, outside-in), I used to teach by walking incessantly around the classroom, asking many questions and trying to read in the faces of the students whether they had arrived at the answers. I peeked at the pages they were writing, and if I found cell phones on desks I would point out how unproductive these gadgets can make us. I often brought to the classroom as many “math toys” as possible to make mathematical ideas tangible.  

Chinese abacus
A Chinese Abacus/Shieldforyoureyes from Wikimedia Commons

I remember the last class I taught in person in March 2020: It was about Mathematics in Ancient China. (I was teaching a course in History of Mathematics.) I distributed a few abaci and advised the students not to share them. (It felt terrible to have to recommend such a thing. Normally, I would have them working in groups with all hands on the abacus). The week after, we entered the Zoom-universe. It was hard to lecture there, but at least I was talking to students with whom I had established a connection before becoming a face on their screens. 

The next semester, for reasons that I will spare you, I decided to be a virtual instructor. A significant challenge I experienced was in replicating the immediate interaction of physically being in the same room with my students as they solved problems. I needed a tool that would allow me to conduct polls and to pose open-ended questions in real time while I conducted synchronous online lectures for my courses, which typically have an enrollment of 35 students.

I investigated several platforms, including Slido, Mentimeter, TurningPoint, and Poll Everywhere. In all these platforms, students can type written answers to questions through a web browser. Instructors can see the answers and share them, if they so desire, with the whole class during a synchronous online meeting.

Stony Brook University supports the use of TurningPoint, where each student pays a fee for a license. Currently, the cost is about $10 for a five-month term. One benefit of using TurningPoint is that the student responses can be connected to the Grade Center in Blackboard. For options that result in no extra cost to students, Stony Brook instructors can use Google Forms, but I found this a bit “less interactive.” Google Docs and Google Slides are platforms where students write in a “live” document (In Slides, the instructor can prepare a set of identical slides and assign a group of students to each slide for a problem-solving activity in real time). Zoom also has a polling tool that instructors can use for real time interaction during a meeting. Lastly, Zoom has the chat tool, which also allows for immediate interactivity during a synchronous meeting. 

After exploring all the options, I decided to pay for my own subscription to Poll Everywhere. This tool helped me simulate the real time human interaction of being in the same physical space with students. In fact, this tool proved so valuable that I am planning to keep using it after the pandemic is behind us.

This is how I used it. When teaching during a synchronous meeting, I started a typical lesson with a greeting and  a word cloud that was generated by the students’ answers to a question like, “write down a word that describes how you feel”, or “tell us something you gained and something you lost because of COVID”  I tried to acknowledge the hardship of the moment and, to remind us of hope. 

During the rest of the lecture I would never talk for more than 10 minutes without having the students participate in some way. For instance, when we studied how Ancient Egyptians measured geometric figures, I asked students to answer one of the following questions: “What does measuring a segment mean?” or  “How do you measure a segment?”   At that point in the course, Egyptians were discovering mathematical concepts and I wanted my student to put themselves in the experience of discovery. After reading some answers, I gave my own, or shared some of the students’ responses. I explained why certain answers were inappropriate. Then I asked, “What does it mean to find the area of a plane shape?” Finally, I gave concrete examples of Ancient Egyptian problems where shapes are measured. 

The School of Athens
File:Scuola_di_atene_23.jpg, Public domain, via Wikimedia Commons

Another frequent activity was having the students read a paragraph and explain what it means. For instance, it is said that when the ruler Ptolemy asked Euclid whether there was a way of learning geometry faster than reading The Elements, a 13-book mathematical treatise, Euclid answered: “There is no royal road to geometry.” Then I asked students to write what they thought Euclid meant. 

Sometimes I asked for educated guesses on topics where students were unlikely to know the answer. After a discussion, I would pose the same question again. I tried to use Zoom breakout rooms for group activities, but I did not manage to do it in a productive way. I would often visit a breakout room and find the students in complete silence. Some students expressed frustration at the lack of participation by their classmates.

Every time a beautiful math idea appeared in front of us (and there are so many!) I would point it out and emphasize how lucky we were to be studying such wonders. 

At the end of each lecture, students wrote up a short summary of the lecture and submitted it through the Poll Everywhere tool. However, this could be done with another tool or through Blackboard, which is the learning management system supported by Stony Brook University.

Overall, I think all of us learned about math history and about each other. Reading the students’ answers to my open-ended questions was like visiting their minds, in a way sometimes more effective than my “face reading” during the in-person lectures. Mostly because I could read the answers one by one, (and when reading a whole bunch of faces it is easy to miss a few), and also because all students answered (and words are often more explicit than faces). 

Two books have helped, taught, and inspired me during this time of teaching during the pandemic: James M. Lang’s Small Teaching: Everyday Lessons from the Science of Learning and Dan Levy’s Teaching Effectively with Zoom: A Practical Guide to Engage Your Students and Help Them Learn. Lang has also many useful essays in The Chronicle of Higher Education.

While I write these last words, the students of last semester come to my mind, and I find it hard to believe that I miss them even though I never met most of them in person. This was my first all-virtual teaching semester, and despite all the turmoil of the time we are living in, to my surprise, I enjoyed almost every minute of the experience.

Dr. Chas and Dr. Alan Kim are facilitating the SBU Faculty Writing Group, which meets on Fridays from 1:30 to 2:30 p.m. starting on Feb. 5 through April 30, 2021. Register at this link. The Faculty Writing Group is sponsored and supported by the CELT’s Faculty Commons.

Incorporating Simple Formative Assessment Strategies to Improve Learning and Teaching  

Assessment

Troy Priest  Troy Priest, Senior Instructional Designer

Faculty spend a lot of time creating and grading assessments while providing students feedback on their performance. These can include projects, papers, presentations, exams, and quizzes, and are an integral part of teaching and learning 

formative vs. summative assessment
Formative vs Summative
by @bryanMMathers
is licensed under CC-BY-ND

Summative assessments measure how well students have achieved the course learning outcomes; they make sure that the courses and the students have met the rigorous standards of the programs objectives; and they ensure that when students graduate from Stony Brook University, they are qualified and credentialed within their discipline and major. These are assessments of student learning and are typically what we think about when deciding on how to assess students. 

Formative assessments are often low stakes (or no stakes) assessments or activities that provide feedback to students and faculty in order to better understand what students have learned, but more importantly what they still need to learn or are struggling to comprehend and/or master. They refer to a variety of assessment methods that faculty and students can use to identify what students have learned as well as address any gaps. These are assessments for student learning and their importance can often be overlooked. 

Below are a few simple, formative assessment techniques, most of which can be facilitated and managed by using tools such as Blackboard, Google, or Qualtrics. These suggestions are techniques that are easily implemented and do not require any significant course or assessment redesign.

Assessing formatively before the class

It is useful that we gauge what our students know coming into a course or lesson. Assessing students’ background knowledge helps the instructor to know what content and concepts may need extra emphasis and attention. A background knowledge probe is a short and simple questionnaire that gauges students’ background knowledge on and assumptions about a topic or unit. This can be done prior to the start of the course or unit or could be done at the beginning of a class. 

Assessing formatively during class

In-class assessments provide instant feedback on students’ understanding of material being presented in class. These can include ‘low-stakes’ or ‘no-stakes’ assessments using a polling tool (Turning Technologies clickers, Zoom polls, Google forms, etc.) or an in-class chat tool  (Zoom chat, Google chat, Google Docs, etc.) where faculty can ask questions and gauge what students understand (or don’t) in real time.  

Real-time, in-class formative assessments can be very powerful when lecturing by allowing instructors to adapt and respond to students’ questions as they arise. 

Assessing formatively at the end of class

Quick formative assessments at the end of the class can not only help faculty gauge their students’ learning and mastery of the content, it can also provide them with feedback on how well the lesson or lecture hit the mark. They can also help faculty identify if their assumptions about students’ understanding, experiences, and expectations of the content were correct. Some examples include:

  • One minute papers where students spend only one minute reflecting on a question or concept and summarizing what they have learned or what is unclear.
  • Critical incident questionnaires where students are given five-question surveys at the end of class to reflect on points in the class that were most helpful, most confusing, and most surprising. 
  • Muddiest points where students write short, simple reflections about what they found to be most confusing or most unclear points during class. 

Assessing formatively outside of class

A great way to understand what students have learned or what they are still struggling with is by asking questions through reflection. Some examples using tools available in Blackboard include: 

  • Discussion boards where students can reflect on their learning by responding to open-ended questions and each other. 
  • Journals where students maintain reflections of their learning, ponder unclear points, and make connections. This feature also allows instructors to respond and give feedback on students’ posts.

These are just a few, easily implemented formative assessment techniques that any instructor could implement in their courses. However, if you would like to explore these further or rethink your course or assessment strategies, please contact an assessment specialist or instructional designer in CELT for a consultation. 

Online Assessment, Through an Equity Lens

Assessment, Equity and Inclusion, , ,

Carol Hernandez  Carol Hernandez, Senior Instructional Designer

Online assessment is an equity issue. Right now, the majority of us are doing some, if not all, of our teaching and learning online while at the same time, all of us are living through a global pandemic. But we are not all facing the same barriers. COVID-19 is disproportionately affecting people in low income groups and those who are members of racial and ethnic minorities (Center for Disease Control and Prevention, 2020). In addition to facing an increased risk of becoming ill, and experiencing job, food, and housing insecurities, these circumstances have also exacerbated the digital divide–the gap in technology and internet access that already existed between wealthy and poor households (Ramsetty & Adams, 2020).

person typing on laptop in the dark with a mask
Image by Engin Akyurt from Pixabay

What does this have to do with assessment? Everything. Oftentimes we hear the word assessment, and immediately think of multiple choice exams and grades. This may lead to assuming that exams must be synchronous, proctored, timed, or otherwise “controlled” to prevent cheating and to ultimately ensure academic honesty (Hubler, 2020). However, we may not be aware that these approaches could also place extra burdens on already marginalized students. How? If a student does not have access to technology, Wi-Fi, and other resources, such as a private, dedicated workspace with a video camera and an open schedule that does not include working or caring for family members, a synchronous proctored online exam could be out of reach (if not impossible). A recent Pew Research study found that low-income parents of school-aged children expect to experience at least one of these three digital obstacles: having to complete schoolwork on a cell phone, having to use public Wi-Fi, and not having a computer in the home (Vogels, 2020). 

When designing assessments, it’s important to remember that the types of controls that can be enforced in a physical setting may not be ideal for an online setting. This challenges us to think in new ways about how to conduct assessments that take into consideration not only the online environment, but also the associated factors that result in COVID-19 disparities in the lives of our students. Earlier this year, Montenegro and Jankowski (2020) discussed the need to embed equity into the assessment practice. Now that we are all living through the pandemic, this concept has taken on a greater sense of urgency. 

According to Singer-Freeman, Hobbs, & Robinson (2019), “timed tests that use closed-ended questions and are completed in groups [synchronously] have the greatest potential to reveal false achievement gaps,” and fail to reflect a student’s true competence (p. 15). This points to less equity for students. However, assessments that use open-ended test questions and projects with real-world applications, such as reflective writing, ePortfolios, and research experiences, increase equity for students.  

One way to bring equity into the process of designing assessments is to reframe them as a way to see how students have transformed as a result of learning what we are teaching (Ambrose, Bridges, DiPietro, Lovett, & Norman, 2010). In doing so, we open the door to more creative ways to document student progress. Furthermore, in an online environment, there are arguably an increased number of opportunities for students to show or demonstrate their newly acquired knowledge, attitudes, and skills. Students can create, present, write, problem solve, and collaborate via the tools that are already available to them through Blackboard or Google Suite. 

Here are three approaches to keep in mind when designing equity-minded assessments for the online classroom.

  1. Asynchronous. This avoids having students all try to take the same exam at the same time (Cohn & Seltzer, 2020). Is there a way to keep it open for a period of days so students are able to log in when they have the time and access to Wi-Fi or the technology they will need? Better yet, is there a way to randomize the questions or provide choices of assessment questions or types so that students are not all taking the same assessment?
  2. Open-ended, reflective, relevant to real-world applications. This option takes into account the different challenges that students may be facing and invites them to make the course concepts relevant to their own professional or academic goals. For example, students may interview a practitioner already in the field, may create a plan for addressing a case study, or work with a partner to solve a problem or create a presentation for their peers.
  3. Frequent, low-stakes, iterative, scaffolded. Is it possible to break up a project into a series of low-stakes chunks that students submit steadily throughout the semester? In this way, both you and your students can see their progress increase steadily rather than waiting for one or two high-stakes exams that may not reflect their true competency.

For more ideas and strategies, watch this presentation, Assessment Through a Lens of Inclusion in Times of COVID-19, or email us at CELT@stonybrook.edu and set up a consultation.

References

Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. Jossey-Bass.

Centers for Disease Control and Prevention. (2020, July 24). Health equity considerations and racial and ethnic minority groups. Retrieved from https://www.cdc.gov/coronavirus/2019-ncov/community/health-equity/race-ethnicity.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fcoronavirus%2F2019-ncov%2Fneed-extra-precautions%2Fracial-ethnic-minorities.html

Cohn, J., & Seltzer, B. (2020). Teaching effectively during times of disruption, for SIS and PWR. Retrieved from bit.ly/stanfordteachingdisruption

Hubler, S. (2020, May 10). Keeping online testing honest? Or an Orwellian overreach? The New York Times. Retrieved from https://nyti.ms/2LcnKz7

Montenegro, E., & Jankowski, N. A. (2020, January). A new decade for assessment: Embedding equity into assessment praxis (Occasional Paper No. 42). National Institute for Learning Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/uploads/2020/01/A-New-Decade-for-Assessment.pdf

Ramsetty, A., & Adams, C. (2020). Impact of the digital divide in the age of COVID-19. Journal of the American Medical Informatics Association, 27(7), 1147-1148.

Singer‐Freeman, K., Hobbs, H., & Robinson, C. (2019). Theoretical matrix of culturally relevant assessment. Assessment Update, 31(4), 1-16.

Vogels, E., A. (2020). 59% of U.S. parents with lower incomes say their child may face digital obstacles in schoolwork. Pew Research Center. Retrieved from https://www.pewresearch.org/fact-tank/2020/09/10/59-of-u-s-parents-with-lower-incomes-say-their-child-may-face-digital-obstacles-in-schoolwork/