The Challenge:

Design and implement a one-hour e-learning module on a topic of your choice, incorporating e-learning principles.  Conduct an A/B test to see whether an innovative online teaching practice leads to statistically significant learning gains.

 

Our Solution:

We designed an e-learning module to teach Mandarin vocabulary and questions.  Then we conducted an A/B test, which showed that our innovative teaching practice had not led to statistically significant learning gains.  Here’s our report.

 

 

DESIGN PROCESS

  1. LEARNING OBJECTIVES

Set the scope to our time constraints

With a total of eight weeks, and no native Mandarin speakers on our team, we knew that setting an appropriate scope for our project would be key.  We quickly contacted a Chinese professor at CMU to get some background info on how she taught non-native speakers.  Then, we decided as a team that we would focus on food vocabulary and the grammar used to ask yes/no questions, because it seemed reasonable that our classmates from different cultures could talk about food in our common space, and asking questions is a good way to learn new vocabulary.

We chose the following four student-centered learning objectives, which specify the given conditions, and the intended student behavior.

  1. Given audio of a Mandarin phrase, students will write the English equivalent.
  2. Given an English phrase, students will recall the Mandarin (Pinyin) translation.
  3. Students will apply the ‘ma’ rule to make a yes/no questions.
  4. Students will recall the pitches for each of the four tones in Mandarin.

 

2. ASSESSMENTS

Align assessment with learning objectives

We created at least two assessment questions per learning objective. This meant we could assess each learning goal at least once in the pre-test before the instruction and once in the post-test at the end of the instruction.

 

3. USER RESEARCH & DESIGN

Find out how experts and novices approach our questions

We conducted think aloud tests with participants and constructed a cognitive task analysis (CTA) to determine the skills and sub-skills that we needed to teach and assess in our module.

One of our participants was a student with beginner-level Mandarin skills.   She was the perfect person to do a think aloud because she was enough of an expert to complete tasks on her own, and enough of a novice to notice the thought processes she used to answer the questions.  Knowing her thought processes enabled us to augment our instruction with specific instruction that matched what she was doing in her head.  There are obvious limitations to this, but in our short amount of time, it was the best we could do.

 

Screenshot 2018-01-09 20.36.01

 

4. IMPLEMENTATION

Development environment

We developed our e-learning module using CMU’s course authoring platform, OLI, which stands for Open Learning Initative.  We chose this platform because of its ability to provide specific feedback for different answer choices, and because of easy integration with another CMU tool, DataShop, which we could use to analyze our participants’ results.

Applying Learning science

We applied Clark and Mayer’s e-Learning principles in the following ways.

  • Multimedia principle – Using the right kind of multi-media enables learners to find the information they need with the least amount of extraneous mental effort, thus reducing their cognitive load.

We provided tables of information.  Although this can seem obvious, applying the principle reinforced best practices for creating online learning.

  • Personalization principle  – Interestingly, when the tone in online learning modules is casual, suggestive and inviting, learners actually learn better!  The reason, according to Clark and Mayer, is that this activates a part of people’s brains that makes people more open to learning.

We explicitly invited learners to the module through a Welcome page.  We also used concise, encouraging and conversational language throughout.

  • Knowledge-Learning-Instruction taxonomy – not all knowledge is of the same ‘type.’  Targeting the learning of certain knowledge with particular teaching techniques is the best way to approach instructional decisions.  The KLI framework was developed in large part due to Ken Koedinger, who just so happened to be teaching our class!  Here’s the seminal paper.

In our module, we were teaching two main types of knowledge: multiple facts (i.e. vocabulary) and a skill (i.e. one grammatical rule).  Facts are well taught with spaced recall, which I’ll talk about below.  Skills are well taught by having students “induce” the rule by seeing multiple, differing examples, which show the “boundaries” of the rule.  For example: seeing the grammatical rule in different yes/no questions, but not in an open-ended question.

  • Spaced practice – This technique is used to improve memory for specific facts.  By distributing practice over time, learners remember these facts better.  Companies such as Duolingo and Membean use spaced practice.

To implement spaced practice in our module, we made sure to include questions about prior topics throughout the module.

Screenshot 2018-01-09 19.50.47

 

5. RESULTS

Evidence of learning

We observed that learners performed 52% better from pre-test to post-test.  We also checked whether the pre-test or post-test was significantly more difficult than the other.  Our results showed that they were not.

              Screenshot 2018-01-09 20.56.07

 

‘Doing Learning Science’ with an A/B Test

We performed an experiment to explore whether, by providing a specific listening strategy (detailed below), learners’ learning gains would improve.  As it so happened, data analyzed with a t-test showed that there wasn’t evidence of improved learning.

This led us to additional questions, like: could the strategy be so widely known that everyone, including our control group, used it?  Although we could have been disappointed by the result, my take-away was one of Professor Koedinger’s closely held beliefs: we can’t just apply learning science, we also need to do learning science.

Listening Strategy: Play a recording several times and pick out sub-phrases you know to build up understanding of the whole sentence.

Screenshot 2018-01-07 17.00.37

 

6. CONTENT

In order to verbally say yes/no questions, first think of the analogous statement, and then add the word ‘ma’ to the end of it.

Example:

Nǐ xǐ huān jī ròu means You like chicken.

Nǐ xǐ huān jī ròu ma means Do you like chicken?

Can you answer this question?

Screenshot 2018-01-09 20.59.35

 

Time: 8 weeks

Team: Elizabeth Onstwedder, Teja Talluri

Role: Instructional designer

Methods: cooperating with subject matter experts, UX research (think alouds), cognitive task analysis, QA testing, A/B testing

Tools: Open Learning Initiative (CMU), DataShop, iMovie