INTRODUCTION TO UX BENCHMARKING.

Testing UX Is an effective method for understanding how people use and think about an interface, be it a website, software, or mobile app.

Benchmarking is becoming an integral part of the plan to systematically improve the user experience. Much has to do with running an effective test. 

For starters, effective benchmarking user experience means first understanding the term. And also what is user experience and then move on to methods, metrics and analysis.

What is user experience?

I am not going to offer an official definition. Here's a definition I'm using, similar to Tullis and Albert's: User experience is a combination of all the behaviors and relationships that people have when interacting with an interface. They include:

• Ability to complete tasks;
• The time it takes to complete tasks or find information;
• The ability to find products or information;
• Attitude towards appearance;
• Attitude towards trust;
• The perception of lightness, usefulness and satisfaction.

These are also many of the classic usability testing metrics. But they include broader metrics regarding relationships, branding, loyalty, and appearance. As such, we borrow heavily on usability testing techniques and terminology.

What is benchmarking and why should you do it?

A benchmark is a standard or benchmark against which performance can be compared or measured. This gives a good idea of the goal. 

In the case of computers, a benchmark is usually a score that measures the performance of software or hardware. To set standards for future tests or tests to evaluate performance. Likewise, UX benchmarking involves evaluating an interface using a standard set of metrics to measure its relative performance.

One of the hallmarks of measuring user experience is to see if the design effort has actually led to quantitative changes over time. Regular follow-up research is a great way to keep track of this. Reference tests are most effective when they are performed at regular intervals or after significant changes in design or function. The graphic shows the same tasks as the automotive website for three years: 2011, 2012 and 2014.

Picture 1. Performance metrics for five objectives from a UX study for an automotive website.

The benchmark shows where a website or product falls with any meaningful comparison. Can be compared with:

• Earlier versions of a product or website;
• Competitions;
• In relation to the industry;
• Industry standard such as NPS or call rate;
• Other products in the same company.

What can you compare?

While almost everything can be compared, the most common interfaces that have performance ratings are:

• Websites (B2C & B2B): Shopping experience at Walmart.com, Costco, or GE.com
• Desktop software (B2C and B2B): QuickBooks, Excel or iTunes;
• Web Apps: Salesforce.com or MailChimp
• Mobile sites: PayPal mobile site;
• Mobile Apps: Facebook, Snapchat or Chase Mobile Banking App;
• Physical devices: remote controls, car entertainment systems, or medical devices;
• Internal applications within the company: applications for reporting expenses or HR systems;
• Service Experience: Support Calls or Out-of-the-box Solutions (OOBE).

Two types of reference studies.

There are two main types of research: retrospective and task-based.

Retrospective: Participants are asked to recall their most recent experience with the interface and answer questions. We used this approach for the Consumer Software and Business Software Benchmark reports. With this approach, you don't need access to software. However, you are limited to existing users who need to recall past activities.

Task-based: Participants are asked to try and perform prescribed tasks on the interface, simulating actual use in a controlled environment. This is a common usability test setup that we most often use when working with clients. With this approach, you get more detailed task interactions and can test with new and existing users. But you need access to software or applications, and you need to define objectives and criteria for success.

Retrospective and objective-based research focuses on different experiences, as shown in Figure 2.

Figure 2: Retrospective and task-based benchmarking studies focus on different parts of the experience.

Fortunately, retrospective and problem-based research can be used in UX tests. We use this approach whenever possible. We start by asking current clients to reflect on their experiences, and then asking new and existing clients to try and accomplish the tasks.

We used a mixed approach in our UX Hotel UX study. Four hundred and five participants contacted one of five hotel sites and answered a series of experience questions, including SUPR-Q (retrospective). Another 160 members who booked the hotel online were randomly assigned to perform two tasks on the same hotel sites. This gave us a more complete picture of the user experience.

Various modes of UX Benchmarking.

When running a task-based UX benchmark, you must choose between different types of modes: moderated or unmoderated testing.

Moderated testing requires a moderator or moderator with a participant. Moderated testing can be done in person or remotely using monitoring software such as GoToMeeting or WebEx.

Unmoderated testing looks like a poll. Participants independently manage the study by following directions to answer questions and attempt to complete tasks. Software like the MUIQ, Loop11 or UserZoom platform helps automate the process and collect a rich set of data, including time, clicks, heatmaps, and videos. There are also some low-cost solutions, such as using a survey platform to get participants to complete their tasks.

Moderated and unmoderated testing has many advantages and disadvantages. The main difference is that unmoderated testing allows you to quickly collect data from more participants in more locations. 

Metrics.

Control studies are often called summative assessments, where the emphasis is not on finding problems, but on assessing current experience. This experience is quantified using both broader learning level metrics and detailed task level metrics.

Research-based metrics.

These metrics are usually collected at the end or beginning of a study (task-based or retrospective).

SUPR-Q: Provides an assessment of the overall quality of the website user experience, as well as an assessment of usability, appearance, trust and loyalty.

SUPR-Qm: a questionnaire for users of mobile applications (in the press).

SUS: a measure of perceived usability; suitable for software.

NPS: customer loyalty indicator for all interfaces; better for consumers.

UMUX-Lite: compact measure of perceived usefulness and perceived lightness.

Brand Attitude or Brand Rise: Brand has a significant impact on UX metrics. Measuring a brand before and after research helps determine how positive / negative experiences affect brand attitudes.

Task-level metrics.

For research with assignments, the following are the most common metrics.

Attitude: Perception of SEQ lightness and confidence collected after the assignment.

Actions: completion metrics, task execution time, errors (collected from task).

Summary.

A UX test provides a quantifiable interface such as websites, mobile apps, products, or software. A good benchmark shows how the performance of an interface is judged against a meaningful comparison to an earlier point in time, competition, or industry standard. Benchmarks can be retrospective where participants reflect actual use. Or task-based, where participants try to complete tasks in simulated use. Collecting data for UX testing involves the same modes as usability testing: moderated or unmoderated approaches. Benchmarking data should come at the research level (SUPR-Q, SUS, NPS) and, if there are tasks, at the task level (performance, time, errors, SEQ).

Based on materials from the site: measuringu.com.

✔️ What does effective UX benchmarking mean?

This means understanding benchmarking, user experience, moving to methods, metrics and analysis.

✔️ What does UX benchmarking include?

UX benchmarking involves evaluating an interface using a standard set of metrics to measure its relative performance.

✔️ What are the main studies called?

Basic research is divided into 2 types: retrospective and task-based. These types focus on different experiences.

✔️ What types of benchmarking modes are there?

These are moderated and unmoderated tests. The upside is that unmoderated testing allows you to quickly collect data from more participants in more locations. 

✔️ What is UX Assessment?

UX score is a metric that categorizes every session recorded by Real User Monitoring as satisfactory, disappointing, acceptable. A lot of other related data is taken into account to determine the UX score for each user session.

✔️ What is Benchmarking?

Benchmarking is the process of measuring and comparing usability metrics to baseline values. Comparative studies are conducted on a regular basis (monthly, quarterly, annually). Such tests help to track changes in product quality over time.

✔️ How is competitive UX analysis done?

Competitive analysis involves five important steps.
Competition research.
Study of competitors' products.
Identifying competitors' UX problems.
Identifying the flaws in the visual design of the site or application of competitors.
Preparing of report.