OVERVIEW
About this project
Microsoft SharePoint is an application primary used by enterprises to share and manage content as well as collaborate across organizations through building websites.
My team and I partnered with the SharePoint design team to investigate any usability issues within their website builder and determine if SharePoint can expand their reach into the educational sphere. This project was sponsored by our client Microsoft.
My role:
Project management, scenario & study design, recruitment, test & interview moderation, and data analysis
Team:
Mia Zhu, Fengyi Xu, Tyson Florence
Duration:
10 weeks
PURPOSE & GOALS
-
Discover any opportunities for growth when using the SharePoint website builder by users with different skill levels.
-
Determine features that university students and administrative staff will benefit from in their day to day activities through interviews and a survey.
THE STUDY
Research questions & analysis
RECRUITMENT
I created a screener survey, recruited and scheduled 10 participants for our usability study that were involved within educational contexts.
User groups​:
-
4 UW administrative staff
-
6 UW students
Experience levels:​
-
2 experts
-
8 novices
-
All have experience with website creation or management​
-
PROCEDURES
When, where, & how
Test set up
-
Used Validately to record test sessions
-
Provided a materials folder with all the needed materials to complete objectives
-
Completed sessions in 1 hour
Test procedure
-
1 moderator & 1 note taker per test
SharePoint Interaction Map
TEST DESIGN
Scenario
We provided a general scenario to our participants so that they could easily connect the test objectives and tasks to a real life situation. For the scenario we asked participants to imagine that they are working for the UW Graduate & Professional Student Senate and they are tasked with creating a new website for the organization.
TEST DESIGN
Main objectives
Tasks were organized by objective to help participants move through the study, and to test specific aspects of usability within SharePoint. Each objective builds upon the previous one and increase in difficulty as participants continue through the test.
1. Set Up Profile Page
T1. Add text
T2. Add an image
T3. Add links
T4. Add a video
2. Create a News Post
T5. Post news on the homepage
T2. Add quick links to the new news page
T3. Create a document library & add files
3. Post a New Event
T8. Create a new event page & post events
T9. Link event page to the homepage navigation bar; post the events on the homepage
4. Change Layout
T10. Change the homepage layout
POST TEST
Data analysis
Grouped & coded observational notes
Ranked issues by severity and frequency
For the data analysis, we grouped and coded notes recorded from our ten testing sessions. Afterwards, I grouped similar comments following the 10 Usability Heuristics for User Interface Design and extracted the main issues by task so that we could easily calculate the frequency of each issue. We worked as a team to combine like issues and create an overarching heuristic category for each main issue and noted potential recommendations.
To provide further evidence for each issue, I created a second affinity diagram of our post interview comments to determine if there is a correlation in needs between students and administrative staff. ​
Heuristic categorization of issues & solutions
Affinity diagram of post-test interview comments
USER INSIGHTS
I compiled a list of participant's insights about using the SharePoint communication website builder. I found that both students and administrative staff enjoyed changing the layout of the website and found the events calendar and news page most valuable.
Students
-
Found SharePoint to be a powerful tool and quick to learn
-
Are open to using SharePoint, especially if it is free
-
Wanted more customization within layouts and color schemes
-
Wanted more integration with social media platforms
"SharePoint is really good for creating a resume, portfolio and something like this (website creation)."
-Participant 5 (novice user)
Administrative staff
-
Liked using it in terms of ease
-
Felt empowered when using it
-
Wanted hover pop up descriptions that are well integrated with less sensitive computers since some admin use older systems
"I can do this myself and don't have to rely on an outside department!"
-Participant 6 (novice user)
FINDINGS
Genreral findings
After calculating the success rate for each task we noticed that while novice participants needed more prompts in the earlier tasks they tended to need less prompts as the test continued into the more difficult tasks.
The opposite was found for expert participants. We infer this might be due to the experts having a predefined idea of the user flow of a similar application due to their previous experience with competitor products.
FINDINGS
Issues & recommendations
We compiled a list of 5 major findings and 8 minor ones discovered during the usability tests. I listed the issues found based on the user flow when using the application and the severity and frequency scale.
prevents the participant from completing their task
delays user significantly but finally allows them to complete the tasks
delays user briefly
KEY ISSUES
1. Confused about how to get started
Most participants are unsure how to start adding content to their homepage.
-
Only 1 out of 10 participants were able to find the edit button on their first try.
-
Most of the participants clicked on the + New button first before finding the correct user flow.
P6: “I’m tempted to click + New.”
P4: [Clicks "+ New"] “I don’t think this is what I’m supposed to do.”
Recommendation:
-
Automatically activate editing status when users enter the application and stay within the editing mode.
2. Unclear web part names & incons
Participants were confused about the naming conventions of web parts,(e.g. Links vs Quick Links). They felt that there was a disconnect between the web part names and icons, as well as an overwhelming amount of web part options.
-
None of our participants expanded the web parts menu
-
Very few participants caught the nuances between the different hover descriptions​​​
-
When hovering over the label = name of the label pops up​ vs. when hovering anywhere else = the web part description pops up
P5: “There was maybe too many buttons, maybe simplify?”
P2: “Some of the naming conventions could use a little bit of work”; “Some of the icons also could be a little more descriptive”
Recommendations:
-
Refining labels & icons to match the user's vocabulary & adding a more responsive hover description
-
Increasing the visual hierarchy of the expand web parts button will help users feel less overwhelmed by options
-
Regroup and rename categories by like functions and include categories within the add web parts menu
3. Confused about add icons
Participants cannot differentiate between the add web parts icon vs the add section icon. They tend to have slips and click on the wrong button.
-
5 out of 8 participants were confused about the difference between the add web parts and the section layout icon
P7: “I kept trying to add content from the left one (button) when I actually needed to be doing it from the middle.”
P3: “I want to blame the color scheme on why I can’t find things.”
P3: “I sorta assumed they were the same button.”
Recommendation:
We recommend to move the section layout button to a more intuitive location such as the left column where the move and delete section buttons are found. Changing the shape or color of one of the buttons would be another option to alleviate any confusion.
PRODUCT SURVEY
Understanding user needs
We released a product survey to determine what features admins and students at the University of Washington would like to see on a website building application after completing our usability tests. Below are our main findings:
-
Most of the participants are students with at least some experience posting content on a website
-
The most popular website builders for students and admins are Wix, Wordpress, Squarespace & Google sites
Most wanted functions
ACADEMIC
CAMPUS
SOCIAL
FINDINGS SUMMARY
Revisiting research questions
1. Efficiency & effectiveness
How easily and successfully can users create, edit, and organize the web parts in their own communication sites?
Participants were able to easily create and edit simple content, but struggled with organizing web parts, especially web parts that required multiple steps. Users could benefit from more obvious and distinct web part choices, and the simplification of multi-step web parts.
2. Learnability & intuitiveness
Can users perform common tasks with little or now assistance?
Participants rated the difficulty of the study objectives as primarily neutral, and were able to complete the tasks at a rate of 90% or higher with a single prompt (which we related primarily to learnability). Additional improvements may come from more consistent information architecture and enhanced interaction within the application.
3. Usefulness & satisfaction
Does the program provide the functions users need & expect when creating a website/page?
In general yes, however, users could benefit from increased functionality through additional web parts related specifically to academia.
CONCLUSION
Study constraints
Recruitment
Due to the limited recruitment time, our screener criteria for admin staff was broad and included administrative staff who qualified as both students and admin (e.g. teaching assistants). We would have benefited to narrow down the definition of an administrator to gather more focused data.
Study duration​
Our study duration was set for an hour, but we had two participants who were unable to complete the study within the time frame which resulted in missing data for the last task. Going forward, it will be beneficial to pilot the study with more individuals to gain a better understanding of time needed.
Survey sample
Our survey sample was (n = 47) and could be larger to yield more accurate data.
Study design
Our study was designed for a lab setting with a high level of structure and set tasks. While this design was ideal to meet our study goals, it did limit our ability to gather data in a more natural setting and did not allow users to freely explore the website builder.
Team
UW Student research team with our partners from SharePoint. With special thanks to our partners at SharePoint!
Appendix
Below you can view the full usability study kit, our usability study report ,which describes the full study and findings in detail, and our presentation.