Use the System Usability Scale Analysis Toolkit

The System Usability Scale

The System Usability Scale (SUS) is a popular measurement tool for perceived usability. It consists of a 10-item Likert scale questionnaire, where participants' responses for the 10 five-level items range from 'Strongly disagree' to 'Strongly agree'. The results are then calculated into a single 0 - 100 score, the SUS score. The sample average of several SUS scores is called the SUS study score. This SUS study score can be used for the usability evaluation of systems that requires any amount of human interaction as well as comparative usability studies, e.g. comparing the usability of interaction techniques by comparing multiple SUS study scores.

The SUS questionnaire is straightforward in its application for researchers and usability practitioners, validated through years of research and its application, available in numerous languages, and easy to understand for participants.

Throughout the years, researchers proposed approaches to contextualize SUS scores, trying to answer the question of what a specific SUS study score actually means. As they follow neither a normal nor a uniform distribution, they cannot be interpreted linearly, making it challenging to interpret them directly. Consequently, researchers calculated percentile curves of SUS scores from SUS study datasets, tried to contextualize SUS scores on adjectives, grading, net promoter score, quartile and acceptability scales, calculated at which point SUS scores become conclusive, and investigated the dimensionality of the SUS questionnaire.
The System Usability Scale Questionnaire. Source: "SUSapp: A Free Mobile Application That Makes the System Usability Scale (SUS) Easier to Administer" - Xiong et al. 2020

The System Usability Scale Analysis Toolkit

By calculating single SUS study scores, calculating comparative and iterative multi-variable SUS study datasets, contextualizing SUS scores on contextualization scales, comparing SUS scores on percentile curves of SUS study meta-analyses, calculating a datasets' conclusiveness, and plotting and visualizing all of this information inside an open source web-based tool, the SUS analysis toolkit allows researchers and practitioners to quickly and correctly apply the insights gained through several years of research that would otherwise have to be gathered from the literature and applied manually.

Hereby, a special focus lies on producing "camera ready" quality figures for SUS studies that can be used in scientific publications and presentations.

Exemplary use cases of the SUS Analysis Toolkit:
  1. Calculating the SUS study score of a usability study for a product currently in development. Here, the "Single Study Dashboard" can calculate the SUS study score (and additional metrics like Media, SD, Min, Max, Quartiles), per Item averages, and the conclusiveness of the score based on the number of participants. It contextualizes the scores on the contextualization scales and provides a "dashboard" plot combining the most important information.
  2. Calculating and comparing SUS study scores from iterative (e.g. formative) usability studies during product development. Here, the "Multi Study Dashboard" can calculate the SUS study datasets metrics for each of the SUS study scores, contextualizes them, shows per Item comparisons and the conclusiveness of each iteration, and visualizes differences based on the SUS study dataset percentile curve.
  3. Comparing "SUS study scores" used as one of the independent variables in a scientific study to compare the perceived usability of different implementations, variations or products using the "Multi Study Dashboard" the same way as for the iterative study use case.
Exemaple SUS score comparison plot generated with the SUS Analysis Toolkit. Source: "TrainAR: A Scalable Interaction Concept and Didactic Framework for Procedural Trainings Using Handheld Augmented Reality" - Blattgerste et al. 2021
Exemaple percentile curve commparison generated with the SUS Analysis Toolkit. Source: "TrainAR: A Scalable Interaction Concept and Didactic Framework for Procedural Trainings Using Handheld Augmented Reality" - Blattgerste et al. 2021

Project

The toolkit is hosted online  under https://sus.mixality.de. The full source code is available at https://github.com/jblattgerste/sus-analysis-toolkit/

If you find problems, bugs or want to provide feedback or suggestions, contact us through email or post your feedback as an issue on the GitHub project. 

Feel free to contribute to our project through suggestions, feedback or by contributing to our open source GitHub repository through pull requests.

Acknowledgement

Developers

Jonas Blattgerste, M.Sc.

Email: jonas.blattgerste@hs-emden-leer.de

Prof. Dr. Thies Pfeiffer

Email: thies.pfeiffer@hs-emden-leer.de

Jan Behrends, B.Sc.

Email: jan.behrends@stud.hs-emden-leer.de