Jumple: Virtual classroom for the pandemic era
Jumple: Virtual classroom for the pandemic era
Embedded Files
Research Project
Research Project
Role: design Research, user Research, game UX Design, UI design, design for AI, gesture UI
Team: designer(myself), and three engineers
Context: a project for a design and engineering convergence class in the graduate engineering course at Seoul National University
Tools: Figma, Adobe Illustrator, Python
Duration: Sept 2020~2021, final user testing early 2022
Ideation
Ideation
image source: https://blog.tensorflow.org/
Challenge
With the current development of AI pose estimation technology, we wanted to develop a system that can help people exercise.
Idea Selection
Through rigorous discussion of ideas, We chose to design a system that young children's physical education, which was suffering during the pandemic. While games such as Nintendo and Playstation do exist, these require hardware specifically made for pose detection. With AI pose estimation, we only need a webcam on a laptop or smartphone.
Our goal was to design and develop a system that can help elementary students exercise during physical education.
Our goal was to design and develop a system that can help elementary students exercise during physical education.
User Interview
User Interview
Image Source: [언택트시대, 체육쌤스토리] '위기는 기회다' 기백쌤 스토리, https://www.chosun.com/
In order to find out the difficulties that physical education during the pandemic, we conducted an expert interview with a physical education teacher, and asked the following Questions:
What does your day to day teaching look like?
What is difficult about teaching physical education during the pandemic?
What have you done to solve that problem?
Did it work? Why was that difficult to solve?
Figuring out user pain point through user journey mapping
As a result of user research, we identified the following problems:
Retain Students interest: students quickly lost interest without real time feedback from the teacher or interaction with other students.
Initiating physical activity: the teacher found it challenging to design activities that the students can perform within limited tools and space.
Provide educational content: integrating appropriate educational content with course material.
Accessibility to technology: students need easy access to devices and technolgy.
Market Research
Market Research
We also looked at different apps that used pose estimation technology and its capabilities. Then we also compared what is possible for us to create.
we found that most of the current services were aimed at physical training, rather than enjoying the experience. During the pandemic, many children grew obese because of sedentary behavior. Therefore, we wanted to create an experience that aimed toward the enjoyment of physical activity, rather than professional training.
Image source left: Likefit, right: Homecourt
We wanted to create a service that not only provided physical training,
We wanted to create a service that not only provided physical training,
but something children could enjoy by moving their bodies and connecting with other people.
but something children could enjoy by moving their bodies and connecting with other people.
Initial idea visualization of game concept
Initial idea visualization of game concept
students and teachers can engage in physical activity, play games online, through interacting on objects on the screen
We asked ourselves how the system could be integrated into the existing education system.
We asked ourselves how the system could be integrated into the existing education system.
UX scenario storyboard
UX scenario storyboard
Wireframes
Wireframes
Game Experience Design
Game Experience Design
We decided to design and prototype five games that children can play in a virtual physical education classrrom.
game 1: Catch the Stars
goal: catch 10 stars as quickly as possible
game 2: Pop all the Balloons
goal: pop all the balloons on the screen in the given amount of time
game 3: Play Volleyball
goal: practive hitting the ball at the red line
game 4: Match Body Parts
goal: move the right body part on the circle 10 times, as quickly as possible
game 5: Play Volleyball (2 Players)
goal: pass the ball to the other person's screen
The final game lets you play a game with other people from a remote location, by passing the ball to the other person's screen.
The final game lets you play a game with other people from a remote location, by passing the ball to the other person's screen.
Game play screens in order:
UI design
UI design
Initial prototype:
Game UI iterations:
From the initial prototype, I developed the game UI, considering visibility and mood.
Considering that the background color can change depending on where the user is, I tested out different colors, layout and typography hat helps users see the visual information on the screen.
For real time stats on the screen, white letters on dark background with transparency was chosen to provide visibility while not disrupting what is going on in the background.
Each type of information was separated on four corners of the screen rather than having all the information at one place for visual clarity.
For directional text, black letters with neon colored outlines was chosen to provide maximum visibility.
Also, we had to guide the user to show their body above the knee, so that the pose estimation library can detect the person.
As a solution, I added a visual indicator that asks the user to keep their knees above the line.
The visual assets were developed in bitmap style for two reasons:
To achieve a playful, game-like look
To make up for the technical limitations which tends to blur the edges of the pictures. The pixelated outlines help the image seem less blurry.
final UI assets:
Evaluation
Evaluation
The Game was evaluated by 72 elemetary school students in Seoul, Korea as part of their physical education class. We asked the students which game was the most fun, and also which game provided the most exercise.
Findings:
Currently Jumple may not provide enough movement to count as exercise, however it can motivate children to move their body and avoid sedentary lifestyle.
Interaction using different parts of the body has the potential to provide more exercise.
What I learned as a designer from the Experience
What I learned as a designer from the Experience
Designing for body gestures in an AR environment
As an interaction designer, I had the unique opportunity to design and prototype body gestures as input.
Here are what I learned to consider:
Optimal visibility considering distance from the screen
User experience for distanced hardware setup
UI visibility on AR
Collaborating with Engineers
As the only designer in the team of four, I had to learn quickly how to work with engineers.
Here are the following ways I learned to communicate:
Visualize ideas early in the process so that engineers and I are on the same page
Research the most recent technology to present ideas but also to understand what the team is capable of
When prototyping, communicate clearly about what needs to be built with pictures and diagrams
Designing for best usability within the technical limitations
The goal of this project was not concept design, but a real functioning prototype that could be implemented with AI technology. Therefore, from project idea to technical implementation, I had to find the best design solution that provided ease of use and visual attractiveness within the bounds of technical limitations. Through this challenge, I learned to iterate and test many different ideas, and find design choices beyond my comfort zone.
Final thoughts and future work
Final thoughts and future work
This project started from finding a useful application for the latest AI technology. This is a different method of approach from human centered approach which starts from evaluating user's need. Nevertheless, I learned how to connect it back to user needs, using different methods to ideate.
Read more about the project from a paper we wrote:
Soohyun Shin, Jaekyung Cho, and Seong-Woo Kim, "Jumple: Interactive Contents for the Virtual Physical Education Classroom in the Pandemic Era," ACM Augmented Humans International Conference, February 22–24, 2021, Rovaniemi, Finland
soohyunshin@snu.ac.kr
© 2023 by Soohyun Shin
Page updated
Google Sites
Report abuse