A Comprehensive Guide to Performing Website Usability Tests

Originally posted on June 18, 2014
Current and revised version are updated on March 24, 2023

Written by Erin Lentz, Executive Director of Design at ArtVersion
Photos courtesy of Women of Color in Tech

Enhancing Accessibility, Usability, and Inclusion 

Abstract:

This white paper presents a comprehensive methodology for conducting website usability tests. The aim is to provide a framework for assessing and improving the accessibility, usability, and inclusion by covering the entire user experience of a website and ensuring that it meets the needs and expectations of diverse user groups. The paper outlines the steps involved in planning, executing, and analyzing the results of usability tests, as well as the tools and techniques required for the effective test implementation.

1. Introduction

1.1. Background and Objectives

The rapid growth of the internet has highlighted the importance of website usability, accessibility, and inclusion. As more users rely on websites for information, services, and transactions, organizations must ensure that their websites are user-friendly and accessible to diverse user groups, including those with disabilities. This paper presents a systematic approach to performing usability tests for websites with the aim of improving their overall performance and user satisfaction.

1.2 Scope

This white paper covers the following topics:

• The importance of usability, accessibility, and inclusion 
• The stages of a usability test
• The different types of usability tests and their objectives
• Tools and techniques for conducting usability tests

2. The Importance of Usability, Accessibility, and Inclusive User Experience

2.1 Usability

Usability is a measure of how easy and efficient it is for users to interact with a website. It encompasses several aspects, including learnability, efficiency, memorability, error prevention, and user satisfaction. A website with good usability ensures that users can quickly and easily find the information or complete the tasks they are seeking, resulting in a positive user experience.

2.2 Accessibility

Accessibility refers to the design of websites, tools, and technologies to be usable by people with disabilities. It is essential to provide equal access and opportunity for all users, regardless of their abilities. Websites should adhere to accessibility guidelines, such as the Web Content Accessibility Guidelines (WCAG), to ensure that they are inclusive and can be used by a wide range of people.

2.3 Inclusive User Experience

User experience (UX) encompasses all aspects of a user’s interaction with a website, including emotional and psychological factors. It considers the overall experience, from the visual design and information architecture to the ease of navigation and usability. A good user experience leads to increased user satisfaction, loyalty, and engagement.

3. Stages of a Usability Test

3.1 Planning

• Define the goals and objectives of the usability test
• Identify the target user groups and create user personas
• Develop test scenarios and tasks that reflect real-world situations
• Determine the type of usability test to be conducted (e.g., moderated, unmoderated, remote, or in-person)
• Choose the appropriate tools and techniques for conducting the test
• Recruit participants that match the target user groups
• Prepare a test plan and schedule

3.2 Execution

• Set up the testing environment, ensuring it is representative of the actual user context
• Conduct a pilot test to identify any issues with the test plan or setup
• Facilitate the test sessions, guiding participants through the test scenarios and tasks
• Record user interactions, feedback, and observations for later analysis
• Collect any additional data, such as questionnaires or interviews, to gain further insights into the user experience

3.3 Analysis and Reporting

• Review and analyze the collected data, identifying patterns, trends, and areas for improvement
• Quantify the findings, such as task completion rates, error rates, and time on task
• Create a report that summarizes the findings, provides actionable recommendations, and highlights the impact of the proposed changes
• Share the report with relevant stakeholders and discuss the next steps for implementing the recommendations

4. Types of Usability Tests

4.1 Moderated Testing

In moderated testing, a facilitator guides participants through the test scenarios and tasks while observing and taking notes. This allows for real-time interaction and clarification of any issues or questions that arise during the test.

4.2 Unmoderated Testing

Unmoderated testing involves participants completing the test scenarios and tasks independently, without direct guidance from a facilitator. Data is collected through screen recordings, audio, and post-test questionnaires.

4.3 Remote Testing

Remote testing can be either moderated or unmoderated and is conducted with participants in different locations, typically using video conferencing and screen-sharing tools. This approach allows for testing with a diverse group of users from different geographic locations.

4.4 In-person Testing

In-person testing involves participants and facilitators being physically present in the same location, allowing for direct observation and interaction. While potentially more resource-intensive, in-person testing can provide valuable insights into user behavior and body language.

5. Tools and Techniques for Conducting Usability Tests

5.1 Testing Platforms

Various platforms are available for conducting usability tests, including UserTesting, Optimal Workshop, and Lookback. These platforms offer tools for remote testing, screen recording, note-taking, and data analysis.

5.2 Screen Recording and Analytics

Tools such as Hotjar, FullStory, and Crazy Egg provide screen recording and analytics capabilities, allowing researchers to capture user interactions, heatmaps, and click maps for further analysis.

5.3 Surveys and Questionnaires

Surveys and questionnaires can be used to collect additional feedback from participants regarding their experience and satisfaction with the website. Tools like SurveyMonkey, Google Forms, and Typeform offer customizable templates for creating user surveys.

———————

User Testing Questions for Screen Recording and Surveys

Developing effective user testing questions is essential for gathering meaningful insights into user behavior and preferences. The following categories of questions can be used to guide the creation of screen recording and survey questions during usability tests:

A. Task-Based Questions

Task-based questions focus on the specific tasks that users must complete during the test. These questions help identify any issues or barriers that users may encounter while performing tasks on the website.

Examples:

• Were you able to find the information you were looking for quickly and easily? If not, what challenges did you face?
• Did you experience any difficulties while completing the checkout process?
• How would you rate the ease of use of the website’s search function?

B. Navigation and Layout Questions

These questions explore how users perceive and interact with the website’s navigation and layout, helping to identify any areas for improvement.

Examples:

• How would you rate the clarity of the website’s navigation menu?
• Were you able to understand the structure of the website easily?
• Did you find the layout of the website to be visually appealing and easy to read?

C. Content and Information Questions

Content and information questions evaluate the quality, relevance, and presentation of the website’s content.

Examples:

• Did you find the information on the website to be straightforward and easy to understand?
• How would you rate the quality and relevance of the website’s content for your needs?
• Were there any content elements that you found to be confusing or misleading?

D. Satisfaction and Overall Experience Questions

Satisfaction and overall experience questions help gauge users’ perceptions of their experience with the website, providing insights into areas for improvement or enhancement.

Examples:

• How satisfied were you with your overall experience on the website?
• Would you recommend this website to others? Why or why not?
• What aspects of the website did you like the most, and what areas do you think could be improved?

—————

5.4 Accessibility Evaluation

Accessibility evaluation tools, such as Axe, WAVE, and Lighthouse, can be used to assess a website’s compliance with accessibility guidelines and identify areas for improvement.

Axe, WAVE, and Lighthouse are popular accessibility evaluation tools that can help you assess your website’s compliance with accessibility guidelines and identify areas for improvement. To effectively utilize these tools, follow the steps outlined below:

Axe

Axe is an open-source accessibility testing tool developed by Deque Systems. It is available as a browser extension for Chrome, Firefox, and Edge, as well as a command-line interface (CLI) tool and a library for integration with other testing frameworks.

Steps to use Axe:

a. Install the Axe browser extension:

• For Chrome: https://chrome.google.com/webstore/detail/axe-devtools-web-accessib/lhdoppojpmngadmnindnejefpokejbdd
• For Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/
• For Edge: https://microsoftedge.microsoft.com/addons/detail/axe-devtools-web-accessib/cplneopgfnjigeoabhfeonpfnhidakea

b. Open the website you want to test in the browser with the installed extension.

c. Open the developer tools (Ctrl+Shift+I or F12) and navigate to the “Axe” tab.

d. Click “Analyze” to run the accessibility test.

e. Review the results, which will display a list of accessibility issues, their severity, and guidance on how to fix them.

f. Address the identified issues and retest the website to ensure that the problems have been resolved.

WAVE

WAVE is an accessibility evaluation tool developed by WebAIM. It is available as a browser extension for Chrome and Firefox and as a web-based tool for evaluating public websites.

Steps to use WAVE:

a. Install the WAVE browser extension:

• For Chrome: https://chrome.google.com/webstore/detail/wave-evaluation-tool/jbbplnpkjmmeebjpijfedlgcdilocofh
• For Firefox: https://addons.mozilla.org/en-US/firefox/addon/wave-accessibility-tool/

b. Open the website you want to test in the browser with the installed extension.

c. Click the WAVE icon in the browser toolbar to run the accessibility test.

d. Review the results, which will display a list of accessibility issues, their severity, and guidance on how to fix them.

e. Address the identified issues and retest the website to ensure that the problems have been resolved.

Lighthouse

Lighthouse is an open-source, automated tool developed by Google for improving the quality of web pages. It includes accessibility testing, among other performance metrics. Lighthouse is available as a built-in feature in the Chrome Developer Tools, as a Chrome extension, as a CLI tool, and as a library for integration with other testing frameworks.

Steps to use Lighthouse:

a. Option 1: Using Chrome Developer Tools

• Open the website you want to test in the Chrome browser.
• Open the developer tools (Ctrl+Shift+I or F12) and navigate to the “Lighthouse” tab.
• Select “Accessibility” from the list of categories and click “Generate report.”

b. Option 2: Using the Lighthouse browser extension

• Install the Lighthouse Chrome extension: https://chrome.google.com/webstore/detail/lighthouse/blipmdconlkpinefehnmjammfjpmpbjk
• Open the website you want to test in the Chrome browser.
• Click the Lighthouse icon in the browser toolbar and select “Accessibility” from the list of categories. Click “Generate report.”

c. Option 3: Using the Lighthouse CLI

• Install Node.js (https://nodejs.org/) and the Lighthouse CLI using the command: npm install -g lighthouse
• Run the accessibility test using the command: lighthouse –output=html –output-path=./report.html –only-categories=accessibility 
• Replace with the website URL you want to test.

d. Review the results, which will display a list of accessibility issues, their severity, and guidance on how to fix them.

e. Address the identified issues and retest the website to ensure that the problems have been resolved.

By utilizing Axe, WAVE, and Lighthouse for accessibility testing, you can ensure that your website complies with established accessibility guidelines, such as the Web Content Accessibility Guidelines (WCAG). Regular testing and addressing identified issues will help create a more inclusive and accessible web experience for all users, regardless of their abilities or needs.

—————

Accessibility and Inclusivity Questions

Creating practical user testing questions for accessibility tests is a critical aspect of the usability testing process. By focusing on task-based navigation and layout, content and information, inclusivity, satisfaction and overall experience questions, researchers can gather valuable insights into user behavior and preferences. These insights can be used to identify areas for improvement, enhance the overall accessibility, usability, and user experience of the website, and ultimately, drive user satisfaction and engagement. Implementing well-structured questions during usability tests ensures that organizations can effectively address their users’ needs, leading to more successful and user-centric website designs.

These questions assess the accessibility and inclusivity of the website, ensuring that it caters to users with diverse abilities and needs.

Examples:

• Were you able to use the website’s features and functions without any difficulties, regardless of any disabilities or impairments you may have?
• Did you find the website’s text and images to be accessible and easy to read or understand?
• Were there any elements of the website that you found to be inaccessible or difficult to use?

—————

Conclusion

Conducting website usability tests is crucial for ensuring that websites meet the needs and expectations of diverse user groups. By following the outlined steps for planning, executing, and analyzing usability tests, businesses can identify areas for improvement and enhance the overall accessibility, usability, and user experience of their websites. Employing the appropriate tools and techniques can streamline the testing process and provide valuable insights into user behavior, resulting in more satisfied and engaged users.

In conclusion, the importance of website usability testing cannot be overstated in today’s digital space. By focusing on critical aspects such as accessibility, usability, and user experience, organizations can ensure that their websites cater to a wide range of users and promote inclusivity. As technology continues to advance and user expectations evolve, regular usability testing will be essential for maintaining a competitive edge in the online marketplace.

Additionally, the integration of user testing questions into screen recording and survey tools during the testing process provides a deeper understanding of user behavior and preferences. The larger the sample group is, and the more reverse group is, the better data can be gathered. By addressing task-based, navigation and layout, content and information, accessibility and inclusivity, and satisfaction and overall experience aspects, designers can create more user-centric designs that drive user satisfaction and adherence.

The use of accessibility evaluation tools, such as Axe, WAVE, and Lighthouse, is vital for ensuring that websites adhere to established accessibility guidelines, creating a more inclusive web experience for all users. Regular testing and addressing identified issues will help websites maintain compliance with these guidelines and demonstrate a commitment to inclusivity.

Ultimately, conducting website usability tests and regularly implementing the findings will lead to the development of more successful and user-centric websites that cater to wide user groups. By following the comprehensive framework outlined in this paper, organizations can systematically approach usability testing, enabling them to continually optimize their websites and deliver exceptional user experiences. This iterative process will not only improve user satisfaction and engagement but also foster trust.

Furthermore, the insights gained from usability testing can provide valuable input for future website enhancements and redesigns. By prioritizing user needs and preferences, industries can make data-driven decisions that align with their target audience’s expectations.

In summary, conducting website usability tests is a vital component of a successful online presence. Investing in usability testing will yield long-term benefits, as organizations can continually optimize their websites to deliver the best practices that likely will improve user experience, fostering lasting relationships with their audience and driving success in the digital realm.