UX RESEARCH FOR OMSCS
Online Learning
UX research, participatory design
Team: Bianca Copello, Lindsay Kelly, Danielle Schechter
Advisor: Betsy DiSalvo
Time: 11/2018 - 12/2018
Problem Space
Georgia Tech established the Online Master of Science in Computer Science (short as OMSCS) program to offer accredited advanced computing education. Georgia Tech believes the following advantages of this program can help more people overcome their physical barriers to get the best education in the world:
It's much cheaper than the on-campus master program.
It allows students to study part-time and better fits in their schedule.
The degree they get after graduation has no difference from the on-campus CS degree. They don't have to list it as an online degree in their resume.
However, the enrollment data has shown that this program doesn't meet the expectations to attract a larger population of applicants to apply and increase the diversity of the students enrolled.
Problem 1: The program fails to attract a larger population of applicants
Though we assumed the program would help students all over the world to overcome physical barriers to participate in the program, The real data shows most of the students enrolled are actually from Georgia.
Problem 2: The program fails to attract more diverse applicants
The data shows that the number of female students and underrepresented minorities is even smaller than the on-campus program.
So, in our research, we need to figure out why people still choose not to apply for an online master's program when some of the obstacles are thought to be removed. We'll also provide design implications to redesign the technology they used for the actual online classes in order to clear up the doubts of the prospective applicants.
Approach
We decided to use qualitative research methods to collect data about potential users. We targeted at the people who have a bachelor degree in CS but haven't gone to a grad school yet. Our methods include participatory design and semi-structured interviews. Then we analyzed the data we collected to generate informative insights and design implications for OMSCS administrators.
My Contribution
Participatory design & interview session: I co-moderated the PD session with my teammates and conducted the interview session as well.
Coding & affinity diagram: I cooperated with my teammates to analyze the data we've collected.
Brainstorm & discussion: I constantly discussed with my teammates to brainstorm the research insights and design implications.
Evaluation of current design: I evaluated the information design of current OMSCS website and the online class technology they used for OMSCS to assist in brainstorming.
Data Collection
Participatory Design
Our participatory design session was conducted with four volunteers who have been in the industry for a range of approximately 3-19 years, all currently working in technology-related fields at the same company.
The PD session consisted of four activities. They were aimed to learn more about their professional experience, career goals and how they perceive the significance of the master's degree.
Activity 1: Professional Collage
Participants are asked to create a collage from magazine images that represented who they are professionally. After they completed the "current day" collage, we ask them to make another collage that represents who they want to be professionally in the future, thinking 5-10 years out.
Participants were working on the collages.
"Present" & "future" collages
We used this activity to relax our participants and made them comfortable to share their career experiences and challenges they encountered. Then we probed how the grad school can help them achieve their career goal.
Activity 2: Professional Timeline
Participants are given a set of stickers with life milestones printed on them, such as "start of grad school", "first job", "first promotion", "first time I led a meeting", "getting into trouble", "first baby", and so on. They are asked to place these milestone stickers on a large piece of paper. The paper has drawn on it a simple timeline representing birth to death.
Participants were working on the timelines.
One participant's timeline
This helped us understand their career goals and professional experience as well.
Activity 3: Introducing OMSCS
We give participants paper and pens and ask them to write down everything they know about the OMS CS program. They are encouraged to write down things they thought to be true, even if they were not sure.
We designed this activity to understand people's assumptions about the online master's program. This could help us identify people's misconception of the OMSCS program. We would tell us about what this program really is after they listed their assumptions.
Activity 4: Designing for OMSCS
At last, we introduced the problems of OMSCS (limited applicants & diversity issues). We ask participants to brainstorm how OMSCS program could change to solve these problems. -We ask participants to create an online advertisement to present their ideas.
Participants were working on advertisements.
One participant mentioned masking gender or other identities to make the online learning environment "free from chauvinists & judgement"
One participant thinks it should highlight less of the technical stuff and more about how people are connected and work together
Initial Takeaway
1. Don't think grad school is useful for technical skills. Participants were very clear about the fact that they don’t see much value in graduate school in terms of learning technical skills. They believe they should have the ability to be self-taught.
2. They want to learn collaboration & communication skills. Some of them said that those are the kind of skills they’d want to learn in graduate school or through formal classes, presumably because it’s something they don’t feel they can do as easily on their own.
Semi-Structured Interviews
Our team conducted semi-structured interview sessions to further collect user data. We framed our questions into three sections:
1. We asked about their challenges and experience in the CS undergrad. Then we asked them to talk about how their undergrad degree helped them with their current career.
2. We asked them about their future career goals and how they want to achieve it.
3. We asked them about their online learning experience and how they perceive OMSCS.
Our team recruited and talked with four interviewees. Combined with the interviewees that other teams in this research project had talked with, we had data from 28 interviewees to analyze. They all had studied in a CS-related major in undergrad. And our interviewees are diverse since female students, international students and underrepresented minorities were included in the research.
Analysis I
We uploaded the transcripts of our interviews and PD session to a qualitative coding platform, Dedoose. Then we coded the transcripts. We also got the chance to look at the 28 interviewees' transcripts and how they got coded.
Specifically, we pulled data from the following codes: 1) “reasons for grad school”, 2) “reasons against grad school”, and 3) “online versus classroom learning”. We used an affinity map to derive bottoms-up insights from over 150 pieces of qualitative data that we analyzed. We created the affinity map using the RealtimeBoard tool.
The affinity map
Research Insights
We generated our informative insights after reading and refining the highest level of sticky notes in the affinity map. Here are our five insights:
1. Participants want their learning experiences to enable human-to-human interaction.
For example, participants mentioned that they wanted a sense of community in the program. They also said they wanted to engage with teachers and other students while learning.
“I think if they emphasized kind of the social aspect of it...establishing that community despite the online program, I think that would do a lot to persuade me!”
“Having a human guide the [learning] process I think is really important, educationally...I would not trust a scripted program to guide me through this.”
2. Participants are interested in learning interpersonal skills that they can’t learn independently.
They believed they could learn technical skills independently without the help of an online degree. However, many felt that it would be interesting to work with a team on projects to get more of the team experience and build interpersonal skills like effective communication and leadership skills. But some pointed that they would be interested in grad schools if they provide some more advanced topics, like machine learning or artificial intelligence.
“I’d like to go to grad school if it taught me more management tools or people collaboration skills, things that I can’t learn on my own easily.”
3. Participants perceive long lectures to be boring and want more engaging, dynamic learning content.
For example, one participant said her first impression of OMSCS is that it “would be impersonal and boring.” They would assume OMSCS program only consists of a bunch of videos. They easily get tired and lose focus after watching too many videos. Participants said they wanted a more engaging, dynamic, and responsive learning environment.
“People get bored after listening to eight-to-ten lectures.”
“Codecademy is different where you can directly compile and learn on the platform.”
4. Lack of self-motivation is an impediment to those considering an online graduate program.
Some participants had tried online learning programs or MOOCs in the past and said they found it to be harder to stay engaged and motivated, compared to in-classroom learning.
“I always stop in the middle of the [online] class, like don’t do it anymore or lose motivation.”
“If I lived in Montana or something, I would want to take the online Master’s to get a degree in a good school even though it might be self-motivated, which I do poorly at, [and which is] the reason I would want to do it in person.”
5. Participants have mixed opinions about whether grad school is worth it from a time, money and value perspective.
The skeptics of grad school don't think the tradeoff is worthwhile to help them achieve their career goal. They think the time and money cost might be a barrier for them to choose a master's program. However, there are some people think grad school can make their resume look good.
Pro: “It's just another way of getting the experience you would have gotten in the industry anyway.”
Con: “It's good for your resume. [It] makes you look more qualified...I feel like an MS in CS would make me look a bit more well-rounded to potential employers.”
Analysis II
Persona
In order to better understand this problem, we created 4 user profiles to analyze their needs and concerns.
Evaluation of Current Design
I evaluated the current design of OMSCS against the research insights we gathered from the potential users. We aimed to find the design opportunities by analyzing the pros and cons of the current version. In order to do so, I looked at the official website of OMSCS, their orientation materials and talked to some students who have been familiar with the program.
1. Human-human interaction
OMSCS programs have used a lot of tools to support the human-human interaction, which made the online collaboration possible.
-
It appears that the program already uses Piazza to allow students to have online discussions and seek feedback and answers from their instructors.
-
Students can use Webex to attend virtual office hour with TAs and teleconferences.
-
The program also has a student-run Google+ community for people to meet up.
Shortcoming:
Although those tools could make online social interaction possible, it still can’t make up for the lack of face-to-face interaction. It's unclear how they can support the active learning environment that the physical classroom can have.
On the other hand, since the lectures are basically pre-recorded videos, the in-class participation and live interaction with the professor and other students might not be possible in this case.
2. Classes Provided in OMSCS
The classes provided in the Georgia Tech OMSCS program are virtually the same as the ones in the on-campus program, with the exception of being taught online. For instance, they are taught by the same professors and assessed in the same way.
Part of the classes provided in OMSCS
Shortcoming:
While the interpersonal skills are highly in demand by the interviewees we talked with, our research showed that no classes in this program are specific about the social skills or team management. How the online platform could help the students build up their interpersonal skills is also not clear to the prospective students.
3. How they teach online
The OMSCS program mainly uses Udacity as the online platform to deliver its lectures.
-
The lessons consist of a series of several short videos. They use various multimedia elements to demonstrate the contents they try to teach.
-
The video can be replayed and paused so the students can watch them again if they find something confusing.
-
Udacity also involves some assessments & exercises in the lessons, which gives the users opportunities to interact instead of passive listening.
A class example on Udacity
A in-class programming exercise on Udacity
Shortcoming:
However, all of the videos are pre-recorded. It doesn't allow the students to actively participate in the lecture. Some real-time engaging activities can't be supported by this platform.
Besides, our research insight showed the online program is an opportunity to incorporate more active and dynamic learning activity, like games or interactive manipulatives. Our interviewees believe it has the potential to create a learning activity that people can't have in a physical classroom. However, the platform doesn't seem to support or invest in such kind of activities.
4. Motivation
The lessons in OMSCS are mostly self-paced. It doesn't have a fixed time so that they can better fit in students' schedule.
All of the videos in the lesson are available from day 1.
Shortcoming:
But the flexibility of the class could cause the students to lose motivation and easily get distracted. Though the program suggests students should have a regular routine to study in the class, it doesn't have any tools to help students make their own schedule and motivate them to follow the timeline.
On the other hand, the online community can help students feel the existence of their classmates. However, if the users don't join those online groups, they wouldn't feel the peer motivation.
Design Implications
In order to generate design implications, we looked at the personas and the current design. Then each of us brainstormed at least 5 ideas to address the issues we found in our research. Then we synthesize them together as the design implications.
1. Emphasize the existing human interactions in the program, as well as provide in-person community-building opportunities to closer resemble that of the on-campus experience.
Proposed solutions:
-
Use existing Marketing channels in order to bring attention to the fact that online classes have TA’s and other interpersonal communication methods
-
Facilitate interactions with professors, add some live interaction hours with professors and ensure faculty responds to questions in a timely manner.
-
Create location-based ‘study’ groups within the program in order to facilitate real-world interactions.
-
Create ‘class hubs’ where students can stream the content at the same time, perhaps provide TA’s to the locations with the most students
2. Ensure skills being taught are not only the in-demand technical skills, but also important interpersonal skills that will set students apart when seeking career advancement, such as managerial positions.
Proposed solutions:
-
Use existing Marketing channels to emphasize that what is being taught in the program has real-world applications.
-
Focus Marketing on demonstrating the group-building and leadership skills that can be developed while enrolled in OMSCS.
-
Research where Computer Science majors are lacking in terms of growth of their career and how that can be implemented into the OMSCS curriculum.
3. Provide flexible and personalized programming that can be adjusted to fit into student’s varied and often hectic lifestyles.
Proposed solutions:
-
Instead of relying on traditional one-way video lectures, redesign classes to provide more interactive components inserted within lectures, such as questions, assessments, and even games. Programs like Kahoot can help with these goals.
4. Create incentives and support structures to help students stay motivated and improve OMSCS student retention.
Proposed solutions:
-
Create a structure for targeted interventions for students who are falling behind can help provide individual accountability
-
Provide a virtual social element setting so users can see how many of their classmates are currently in class can also provide social motivation.
-
Allow the users to view the progress of other students in order to know if they are left behind.
-
Create a scheduling tool to allow students to manage and plan their classes in a weekly routine. Provide reminder and notifications based on the schedule.
5. Allow the program to be flexible so students can create the program that will best benefit them.
Proposed solutions:
-
Offer a wide variety of career and skill-building paths so students can carve their own way. This is especially true for Master’s students because it is likely they have experienced a real-world environment and they know what they need in order to be successful in their career.
-
Allow students to test out of core courses so they can focus on more advanced or specialized topics and not feel like they are repeating what they already know.
We presented our research insights and design implications to the administrators and designers of OMSCS. They're going to work on the redesign of the online learning experience and the marketing materials based on our research.
Research Limitations
1. Since we tried to include all kinds of potential students of this program in our research, we were unable to have enough data on female students and underrepresented minority students to address the diversity issue. Further research is needed to focus on this issue specifically.
2. Based on our research, we found for the international students who live in the states, their major concern is the visa issue. Unfortunately, the OMSCS program can't sponsor international students' visa. At this moment, we had to exclude international students from the targeted users. But we might be able to consider students who live outside the U.S. as potential users. Future research is needed.
