Methods: Backwards Design, Learning Principles, Alignment of Goals, Instruction and Assessment, Interviews with Experts, User Testing, Cognitive Task Analysis (think alouds)

 

Overview

I made this one hour e-learning module, Beginner Mandarin, with my teammates Elizabeth Onstwedder and Teja Talluri.  It was a terrific opportunity to apply the e-Learning design principles we learned from Clark and Mayer’s book E-Learning and the Science of Instruction.  This project was for Dr. Ken Koedinger’s class e-Learning Design Principles and Methods.  

 

What We Did

We…

  • developed a one-hour e-learning unit and tested it with 29 learners
  • followed a backwards planning framework, starting with goals, then creating assessment and finally instruction
  • worked with a Subject Matter Expert, a Mandarin professor, and conducted cognitive task analysis to determine appropriate goals
  • utilized instructional design principles based on research to improve its effectiveness
  • conducted an A/B test

I was involved in all parts of the process, so I can answer any questions you may have.

 

Learning Objectives 

We chose the following four learning objectives, which would familiarize someone with common Mandarin phrases they could use in an everyday situation.

  1. Given audio of a Mandarin phrase, students will write the English equivalent.
  2. Given an English phrase, students will recall the Mandarin (Pinyin) translation.
  3. Students will apply the ‘ma’ rule to make a yes/no questions.
  4. Students will recall the pitches for each of the four tones in Mandarin.

 

Using Cognitive Task Analysis 

After creating initial assessment questions, we conducted a think aloud with a graduate student with beginner-level Mandarin skills.  With this, we found out what about our questions worked and didn’t work.  We also discovered a problem-solving strategy she used, which we used in an A/B test, which you can find out more about in a following section!

Screenshot 2018-01-09 20.36.01

 

Our Results

We observed that learners performed 52% better from pre-test to post-test.  The tests contained nine multiple choice questions each and were shown to be roughly equivalent in terms of difficulty.

              Screenshot 2018-01-09 20.56.07

Learning Principles We Used

We applied Clark and Mayer’s e-Learning principles where appropriate.  Here are two particular applications, along with a

Multimedia principle – we enabled learners to find the information they needed with the least amount of extraneous mental effort, thus allowing for greater learning, by providing tables of information.

Personalization principle  – we made our learners feel comfortable by explicitly inviting them to the module.  Moreover, we used concise and conversational language throughout, and encouraged our learners at every opportunity.   When learners feel that someone is conversation with them, they try harder to learn the material.

Knowledge-Learning-Instruction taxonomy (citation) – a huge take-away from my time at Carnegie Mellon University is that there is a taxonomy of knowledge that we can target with particular teaching techniques.  For example, being able to recall vocabulary words falls under the umbrella of knowing facts.  One technique we used to support our learners was to use the spacing effect.  If this is a new concept to you, as it was to me before this semester, the spacing effect means that distributing practice over time leads to more long-term retention of what you practice, compared to doing all the practice all at once.  It also happens to be particularly powerful for remembering facts!

Screenshot 2018-01-09 19.50.47

 

Doing Learning Science with an A/B Test

We performed an experiment to explore whether providing a specific listening strategy (detailed below) learners could use for the audio questions would improve their performance.  We were surprised to look at the data and not see a sign of improved learning.  This led us to additional questions: was our population size too small, or could the strategy be so widely known that everyone, including our control group, used it?  Although we could have been disappointed by the result, I think it made a point that Professor Koedinger taught us: actually doing learning science means testing what could potentially improve learning.

Listening Strategy: Play a recording several times and pick out sub-phrases you know to build up understanding of the whole sentence.

Screenshot 2018-01-07 17.00.37

 

 

 

 

 

Bonus Material!

We also taught the rule for forming yes/no questions: first, think of the analogous statement, and second, add the word ‘ma’ to the end of it.  Because the rule’s so interesting, I’d like to share it with you!

Example:

Nǐ xǐ huān jī ròu means You like chicken.

Nǐ xǐ huān jī ròu ma means Do you like chicken?

Can you answer this question?

Screenshot 2018-01-09 20.59.35