Evaluating the effectiveness of game-based learning

by | Jul 1, 2020 | Serious Games | 0 comments

Since 2019, the Humanitarian Leadership Academy (Academy) has partnered with Quicksand, India to create new game-based learning solutions on Gamoteca that incorporate more human-to-human engagement and empathy building within e-learning. The challenge at hand is not only to create these new innovative learning experiences, but also see if they could lead to higher user engagement and better learning  outcomes.

The key goal of the game-based learning experiences was to scale human engagement in learning via role-playing, scenarios, peer-to-peer learning and simulation exercises. 

Certain  game concepts were developed through co-creation workshops using design thinking with local partners (in Kenya, the Philippines, Bangladesh, Jordan and India), drawing on local knowledge, expertise and stories. Some of these outputs were further developed in close collaboration with the initial creators of the concept to ensure their authenticity. 

Finally the games were developed iteratively on the Gamoteca platform and rolled out via Kaya:

    • An Exercise in Coaching and Mentoring:  A game that aims to bring to life skills taught in the Coaching and Mentoring course on Kaya, such as active listening and the structuring of open and closed questions. The format leveraged a video-based interaction between two players, a ‘coach’ and ‘coachee’, in order to provide a structured, prompt-based interaction that facilitated the use of skills taught in the original course. The goal of the game was to create practice-based learning opportunities for the concepts covered in the course material.
    • Challenge your Gender Assumptions: A game inspired by from a workshop in Amman, Jordan, that looked to inculcate gender sensitivity through role-playing. The game and its exercises were adapted to engage a broader, global audience.
    • The Flood: A role-playing game in which the user takes the role of a person whose town has been affected by a storm and flooding. It looks to situate the user in a challenging environment and guide them to strategize and make effective decisions, while building empathy with fellow community members affected. Users take the role of someone who has been affected by the storm, and makes decisions towards becoming a volunteer.  

Findings from the qualitative evaluations:

Creating learning content required an iterative approach to evaluating the challenges and efficacy of the new games. This assessment was approached through qualitative and quantitative methodologies.

Qualitative testing of the first versions of the selected learning resources was conducted with roughly 30 participants. Each session followed a fixed process of introduction, experiencing the game, and a final feedback and discussion session.

  1.       Group-based learning is a key driver of engagement

Games that involved more than one player were seen to facilitate increased engagement with the learning material. Interacting with another person within the experience led to higher attentiveness in most learners, while also giving them a sense of being part of a larger exercise.

 “I feel this is a cool way to teach a skill that people often find boring. When learning doesn’t have a two way engagement, the receiver often feels she/he is not contributing anything to the learning process. Also, in this game you get to engage with mentors across the globe.”  

Participant after having played ‘An Exercise in Coaching and Mentoring’.

  1.       Interactivity provides agency

Exercises and activities within the games —individual as well as with other learners — were found to give learners a sense of agency and ownership over the journey, allowing them to define their own learnings and key takeaways from the experience. In most cases, these still aligned with the broader learning objectives established for each game.

“This is definitely a fun and interactive way to learn. I feel the visual and audio element in the game helped in relating to the situation at hand. This could be a great tool to use in our work, too, when we train the youth on relationship building, so definitely  see myself using this in my training.”

Participant after having played An Exercise in Coaching and Mentoring

  1.       Immersiveness

Given that most of the outputs tested were narrative-driven experiences, the extent of their immersiveness was a key factor to gauge through the testing process. The games’ immersiveness was seen to depend on how convincing the narrative was, as well as the integration of activities and decision-making points.

The more compelling the narrative, the deeper the learning. A believable, nuanced narrative that recreates relatable characters, events, and environments, helps draw the learner in further into the experience, allowing for more impactful learning.

“Having a game like this adds a little more excitement to a learning experience. The point where I had to make a decision about whether (as a person affected by a flood) I should be with two of my children or go searching for my third was very powerful, it gave me a very concrete idea of what it could be like to be in such a situation.”

Participant after having played The Flood : Multiplayer

  1.       Role-playing helps heighten immersion

Role-playing allowed learners to empathise and understand another person’s context. With sufficient context, learners were seen to relate better and invest in their characters and corresponding activities.

The more compelling the narrative, the deeper the learning. A believable, nuanced narrative that recreates relatable characters, events, and environments, helps draw the learner in further into the experience, allowing for more impactful learning.

“The game made me think about being in someone else’s shoes and what their background story is.”

Participant after having played The Flood : Multiplayer

  1.       Alignment with learning objectives

The identified learning objectives were qualitative in nature, and included practical coaching skills, building empathy in relief contexts, and decision making. Data to measure these was collected through conversations and group discussions at the end of the participants’ learning journey.

“That it gave an insight into something that can be so complex in such a simple way.”

A participant after having played An Exercise in Coaching and Mentoring

Findings from the surveys:

Additionally, there is an ongoing quantitative survey measuring the effectiveness of the games and their comparability to traditional e-learning methodologies. At the time of publishing this article, there were a over 100 responses.

The results of the survey show that learners are keen to engage with game-based learning and that found the methodology comparable if not better than traditional e-learning one

 

62% found the content useful and very useful

  • Relevance and usefulness of the content 62% 62%

63% would recommend a learning game to one of their colleagues

  • Would recommend 63% 63%

58% described these games as being better or far better when compared to previous methods of e-learning they had experienced

  • Comparison to previous methods of e-learning 58% 58%

73% would engage with this type of learning in the future

  • Willingness to engage with game-based learning in the future 73% 73%

If you want to learn more about the development of the games and the evaluation methodology, you can read the full report here.