How I Use User Testings to Make Design Decisions

Team: with 1 UX researcher, 1 product owner, and 1 developer

My Role: UX design, user testing

Time: 03/2020 - 04/2020


At Walmart, one of my projects was to build a conference room management platform to help engineers and technicians to monitor the current status of each conference room within the offices.

As identified by the user research, the engineering team needed an access control system to set the permission for different users on this platform. Here are three types of users that will be using this platform:

1. Owner: Owners have full permission. They can make edits on the platform and manage other users.

2. Admin: Admins can only make edits on the platform.

3. Guest: Guests have the view only permission.

So, the page I designed was a user management system that grants the owners of this platform with the abilities to manage the permission level of other users. It should include the following features:

Add in new users and set permission for new users

Delete old users

Change users' role (among owners, admins, and guests)

My Solution & Contribution


Note: The screens shown in my portfolio have been modified to remove confidential information. All of the user data shown in the design are only placeholders. They are not the real user data.

Designed functionalities for both single-edit and bulk-edit to provide flexibility and scalability

To allow the owners to edit the mass amounts of user data on this platform, I designed both experiences for single-edit and bulk-edit. This will benefit the product team when they want to scale this platform to a larger audience.

Validated by user testings

The whole design was repeatedly tested with users. All of the design iterations and decisions were based on the testing results. I designed and conducted all of my user testing studies.

Provided responsive web design solutions

I designed breakpoints for different screen sizes and resolutions so the design can work in different situations.

My Approach

Low-fi to high-fi design
Prepare for user testings
Design Prep
Understand research
Task flow
User Testing
Usability test
Design iterations
Annotation sheet
Two Concepts of Design

There's a lot of stories in the process of designing this page. But the most memorable one for me was the design process for "Add new users" flow. I learned how to use user testings to make the crucial design decisions while designing this flow. I will talk about how I designed this flow in the following section.

Task Flows

I started the design process by drawing the task flows. I used flow charts to lay out the tasks that users need to perform when using this system.

I came up with my solution and discussed it with my product team. Then we found out another solution. So, here are two ways of designing this:

As highlighted in purple, the major differences between these two flows were whether we should let the owners search and add in users to the system first or choose the role they want to add first.

To better think of these two concepts, I started to visualize them and made two mockups to demonstrate my ideas.

Concept A: Role First

In this concept, users will see a dropdown menu to ask them to choose the role they want to add first. Then they search and add in people altogether. But they can only add in people with the same role at a time.

If we ask the owner to add 2 persons as a guest, 4 persons as admins. Here is the flow: (screen shown below is only a mid-fidelity version used for usability testing)

Here, users are back to the main page. If they, as in this case, want to add other people in a different role, they have to click "add new" button and go through the flow again.  

This flow is repetitive to the one I showed above so I won't go into details here.

Concept B: People First

In this concept, Users will add in all the people they need first. Then they can click radio buttons to assign them with the corresponding permission one by one.

If we ask the owner to add 2 persons as a guest, 4 persons as admins. Here is the flow: (screen shown below is only a mid-fidelity version used for usability testing)

Clicking these radio buttons repeatedly

Which One Is Better?

In order to figure out which version is better, I listed out my own assumptions of these two flows in the form of pros and cons.

My next step is to figure out if their assumptions are true and what the users' priority is. Basically, it's impossible to make a perfect solution so I need to understand users either prefer to add persons with mixed roles altogether or to assign permission in bulk.

User Testing

I decided to conduct a user testing to validate my assumptions and figure out users' preference between two concepts. Below is how I designed my testing plan.

User Testing Method


Moderated in-person user testing sessions


Participants tested for each concept

Qualitative methods with follow-up questions

Due to the timeline, I was not able to recruit lots of testers so I mainly focused on a qualitative approach to probe users' likes, dislikes, and reasoning behind. 10 participants were shown with concept A while the other 10 participants were shown with concept B.

User Testing Tasks

Based on my assumption, the concept B (adding people with the mixed permission together) might have a larger drawback when there is a large quantity of data but might have advantage in a smaller dataset. At the same time, the drawback of concept A (choose permission first then add people) might only show up when users need to add in people with mixed permission.

So, I need to design two tasks to cover different scenarios.

Task 1: Fewer people with same permission

You are the owner of V2 conference room. You plan to add the following people as “Guests” of this platform:

  • Ricky Yu

  • James West

Task 2: More people with mixed permission

You are the owner of V2 conference room. You plan to add the following people into the platform and assign them with the corresponding permission.

  • Ricky Yu - Guest

  • James West - Guest

  • David Hicks – Admin

  • Anusha Dandekar – Admin

  • Helen Sheperd – Admin

  • Alex Hammer - Admin

I designed two prototypes for both concepts to cover the two tasks above. The prototype is made in Invision and similar to the low-fi mockups shown earlier.

Follow-up Questions

In order to collect qualitative data, each participant was asked the following questions after the tasks:

  1. On a scale of 1-7 (1 being Extremely dissatisfied to 7 being Extremely satisfied), how satisfied are you with this task?

  2. What did you like about this concept?

  3. What did you not like about this concept?

(The satisfaction score is only used to ask participants to describe their experience and allow me to further probe into what they think. Because of the small sample size, the data is not used for any decision-making process.)

Testing Results and Analysis

The user testing sessions helped me collect sufficient data to generate insights. What I found out was both of the concept had a 100% success rate so what really mattered at this point was their preference. So, I feel it is necessary to drill down into qualitative data.


I analyzed the qualitative data by using affinity diagram method. I grouped users' feedback to identify common themes. Then I reorganized them into a new table of pros and cons.

Our early assumptions have been mostly validated in this testing. But for concept B, there are two more additional problems that we identified from this study, which we didn't expect before.

For concept A, although users think it is repetitive to go through the flow if they need to add in people with different permission, they didn't express a strong frustration towards it. Conversely, users got annoyed when they had to select roles for multiple users in concept B.

Tester for concept B

Although she has been given the UI of concept B, she actually used it in the way how concept A was supposed to work. She didn't add in every person all at once. She manually grouped persons with same permission together. Then she added all of the guest people in. Then she opened the flow again to add in all of the admin people.

She explained, "It's very hard for me to track the roles for each user. It will become a problem for me when I have to add in so many users. I can't assign roles in bulk. So, I decided to add in the guests first then the admins."

In this way, I deemed the lack of bulk edit in concept B as a larger usability issue. At the same time, concept B has additional problems which concept A doesn't have. At the same time, users expressed that it's unlikely that they would add in people with mixed permission together but it's very likely that they will add in lots of data at once. This further favors concept A.


So, I decided to go with concept A in my final design based on the user testing data.

Concept A (role first) wins

Final Design

I translated the insights I collected from my user testing to the final design. Then I upgraded my low-fi design to high-fi. In this section, I will showcase my final design decisions.

Data is pre-segmented by role so users can see how many people are in each role and view them quickly

When users hover above the question mark, they'll see a tooltip describing the capabilities of this role

Bulk actions

  • the button will become active when the checkboxes are selected

  • it includes bulk delete and assign role in bulk

Users can click on radio buttons to change the role

Single edit - delete

Flow for Adding New Users

I listened to my testing result and used the pattern in concept A for the final design.

Flow for Single Edit - Delete

This is the flow for single eidt- delete. A modal will show up to ask users to confirm.


The flow for the single edit - change role is similar to the delete flow. Users can click on the radio button to change the role. For the loading speed of this page, I won't show the details here.

Flow for Bulk Edit - Assign Role

This is the flow for bulk edit - assign role. When the checkboxes are selected, the bulk action is activated. After the role is assigned, a snackbar will pop up to notify users.


The flow for the bulk edit - delete is similar to the assign role flow. For the loading speed of this page, I won't show the details here.

Final Version & Next Steps

I also provided designs for different breakpoints and annotation sheets to deliver specs and interactive states of each component. Eventually, I finished the hand-off with the devs and wrapped up this section.


Designed by Fangxiao Yu,

last updated in February 2020