Issue 34: Are you making these common assessment mistakes? 🎯

This week we're answering the following questions:

• Does incorporating gambling into an LMS increase learner engagement?
• Can spaced practice using a flashcard app improve test performance?
• What are the most common assessment mistakes made by teachers and trainers?

Can Gambling Increase Learner Engagement?

It’s possible that a trip to Las Vegas may have been the inspiration for this week’s first study. Researchers around the world collaborated to answer this pressing question: if an online slot machine is incorporated into a learning management system (LMS), and learners earn “coins” for the machine each time they answer quiz questions correctly, will they be more engaged in the LMS? Researchers expected a positive association between choosing to play the slot machine and the change in the number of daily, voluntary training modules completed.

Results from the experiment were mixed. Learners who played the online slot machine initially showed an increased participation in voluntary daily training and increased performance on the related quizzes, compared to the same timeframe before introduction of the slot machine. However, this effect only lasted a few months before declining to pre-gambling levels. In addition, researchers “also observe[d] a significant decrease in training activity by employees who chose not to play the game, which in our matched samples served to offset the positive motivational effects observed for players” (Kelly, Valtchanov, & Webb, in press). In this study, some learners were so discouraged by the idea of a slot machine that they actually stopped participating in training altogether.

So, should we all incorporate gambling into our LMSs? Maybe not. These researchers do suggest, however, that employers aiming to increase participation in voluntary training should still seek out novel approaches to reward engaged learners. These approaches should have the following characteristics:

• Simple to engage in
• Can be done frequently
• Generates excitement
• Has a low ‘cost’ of play

In short, the extent to which games like these will be associated with increased engagement in the LMS, depends not only on the proportion of employees who choose to play and how much they enjoy playing, but also the reactions of learners who may object to the company’s use of such approaches (Kelly, Valtchanov, & Webb, in press).

Key Takeaway: Slot machines are great for Vegas (and for this lady), but perhaps not for your LMS. (Sorry, gambling fans.) Organizations should look for other approaches with the characteristics listed above to increase engagement.

Read More ($): Kelly, K., Valtchanov, D., & Webb, A. (In press). Behavioral implications of using an online slot machine game to motivate employees: A cautionary tale. Accounting, Organizations and Society.

Spaced, Repetitive Practice with A Flashcard App: An Effective Learning Strategy?

Multiple studies have supported spaced repetition practice (as opposed to massed exposure to content) as an effective learning strategy, especially for the memorization of factual knowledge. In a recent article, researchers confirmed the efficacy of surgeons’ usage of spaced practice with a small study that found a strong, positive correlation between time spent on a flashcard app program (in this case, Anki) and their final orthopedic surgery examination scores. For each surgeon, the more flashcards were reviewed over time, the higher the score on the final examination.

Key Takeaway: Memorizing factual knowledge? Leverage something like a flashcard app and space the practice sessions over time to help build your memory.

Read More ($): Lambers, A. & Talia, A.J. (2021). Spaced Repetition Learning as a Tool for Orthopedic Surgical Education: A Prospective Cohort Study on a Training Examination. Journal of Surgical Education, 78(1), 134-139.

Common Assessment Mistakes

Our intern Kelley contributed this piece. Assessments, tests, exams, and quizzes often carry significant weight for learners and are commonly used as determining factors in decisions regarding program completion, certification, promotions, etc. While it’s crucial that teachers and corporate trainers have a certain level of mastery at developing assessments, researchers have discovered that this isn’t often the case -- regardless of the setting, they’ve found several of the same mistakes in instructor-developed assessments.

The most common mistakes include:

• Implausible or illogical distractors, which are options that can be easily avoided or ruled out when answering a question
• The inclusion of hints or cues, which can provide helpful insight to the correct response to a different question
• An illogical order of options or regular sequence of correct answers
• Highlighting the correct response in the form of the longest, most well-developed option
• Inclusion of phrases like “in your opinion” and “to your best knowledge”

Key Takeaway: Assessment designers, instructors, trainers, and other educators should keep these common mistakes in mind to avoid developing flawed testing instruments.

Read More ($): Simsek, A. (2016). A Comparative Analysis of Common Mistakes in Achievement Tests Prepared by School Teachers and Corporate Trainers. European Journal of Science and Mathematics Education, 4(4), 477–489.

LSW Podcast: Episode 9 Now Available!

Our recent podcast episode with the amazing Mirjam Neelen was released -- check it out on our site or wherever you get your podcasts. This is a can't-miss session with one of the leaders of learning science!

Next week we'll chat with legend Dr. David Merrill, who developed the First Principles of Instruction (discussed in our second issue).

Pets of Learning Science Weekly

This week we’re featuring Bowser and Peach! These French Bulldogs, who celebrated their third birthdays last month, enjoy running/rolling around Seattle with reader Mark D. Peach hurt her back over the summer and is now partially paralyzed, but we're told she loves her new wheels! Thanks for sharing, Mark!

Send us your pet pics at editor@learningscienceweekly.com.

Wondering why we’re including animal photos in a learning science newsletter? It may seem weird, we admit. But we’re banking on the baby schema effect and the “power of Kawaii.” So, send us your cute pet pics -- you’re helping us all learn better!

---

The LSW Crew

Learning Science Weekly is written and edited by Julia Huprich, Ph.D. Our head of growth and community is Julieta Cygiel.

Have something to share? Want to see something in next week's issue? Send your suggestions: editor@learningscienceweekly.com