Web-Based User Interface Evaluation with Questionnaires

Last updated: 2009-05-15 by Gary Perlman (www.acm.org/perlman/ | perlman@acm.org)

Abstract

question.cgi is a customizable Web-based perl CGI script to administer and collect data according to a few "standard" user interface evaluation questionnaire forms (QUIS, PUEU, CSUQ, and PUTQ). The script can be customized and run from a remote server, www.acm.org/perlman/question.cgi, from which data are emailed in a simple format. The script accepts options as part of its form data: The script uses these options to present the questionnaire on the Web, gather ratings, general comments, and question-specific comments, and email the results to the designated survey administrator. This document is a user manual that allows the online reader to design and create their own questionnaire. It is available at: www.acm.org/perlman/question.html. The script is freely available, AS-IS, in a shell archive at: www.acm.org/perlman/question.sh.

Keywords

Subjective ratings, Usability evaluation, User satisfaction, User acceptance, User interface questionnaire, Online survey administration, Remote evaluation, Measurement, World-Wide Web, CGI, form, Human-computer interaction, Human factors

Introduction

Questionnaires have long been used to evaluate user interfaces (Root & Draper, 1983). Questionnaires have also long been used in electronic form (Perlman, 1985). For a handful of questionnaires specifically designed to assess aspects of usability, the validity and/or reliability have been established, including some in the following table, some of which are discussed in Chapter 6 of Tullis & Albert, 2008.

Acronym Instrument Reference Institution Example
QUIS Questionnaire for User Interface Satisfaction Chin et al, 1988 Maryland 27 questions
PUEU Perceived Usefulness and Ease of Use Davis, 1989 IBM 12 questions
NAU Nielsen's Attributes of Usability Nielsen, 1993 Bellcore 5 attributes
NHE Nielsen's Heuristic Evaluation Nielsen, 1993 Bellcore 10 heuristics
CSUQ Computer System Usability Questionnaire Lewis, 1995 IBM 19 questions
ASQ After Scenario Questionnaire Lewis, 1995 IBM 3 questions
PHUE Practical Heuristics for Usability Evaluation Perlman, 1997 OSU 13 heuristics
PUTQ Purdue Usability Testing Questionnaire Lin et al, 1997 Purdue 100 questions
USE USE Questionnaire Lund, 2001 Sapient 30 questions

There are other questionnaires, including instruments from the HFRG (Human Factors Research Group):

Also, see LaLomia & Sidowski, 1990, 1991, 1993 for reviews of measurements of satisfaction, literacy, aptitude, attitude, and anxiety.

NetRaker (netraker.com) once provided online usability evaluation, with a free trial version. Now there are many online survey services, most offering free simple surveys or trial periods:

Note: Although these instruments are online, many are copyrighted and some require a fee for non-educational use. See the references at the end of each form for more information.

These questionnaires are similar in format, so a Web-based system to present them has been developed (as seen in the Example links above). The system, question.cgi, is a perl-based (version 4 or 5) CGI script that presents a form for a questionnaire and emails data to a specifiable address. Many options for the format of the questionnaire are read from the incoming form data, so a surveyor can adapt an existing questionnaire (although in ways that may change the validity or reliability of the standard forms of the questionnaires). The rest of this document describes how to use the standard forms of the surveys and how to adapt them.

Standard Forms of Questionnaires

Three options to question.cgi specify all you need to evaluate a system using a Web form.
  1. By default, the CSUQ is used, but you can supply a form= option: 
    form=QUIS
form=CSUQ
form=PUEU
form=PUTQ
  2. You can specify a system name (e.g., My Web Site, WordProcessorII) that will be used in the form and reported in the data. 
    system=HCI_Bibliography_Search
system=VR_Applet
  3. You can specify an email address to which the data will be sent when the rater submits their data. 
    email=perlman@acm.org

Combining Options

Options can be appended to a URL or supplied as form data (either hidden or visible).

Options Reference and Design Form

The following options can be supplied as form data or on the URL. The default values of the options depend on which form is in use. The default or modified values are kept in the HTML as hidden values; they can be viewed by looking at the HTML page source.

GENERAL OPTIONS
OPTION EXAMPLE DESCRIPTION
The name of the system to be evaluated. This is used as part of the Subject: line in the emailed data.
The type of object to be evaluated (by default "System") This is used as part of the form and in some questions.
Definite article to precede the object at the start of a sentence. Used for translation to languages with gender (e.g., French Le, La).
Pronoun to precede the object at the start of a sentence. Used for translation to languages with gender (e.g., French Cet, Cette).
The email address to send the data to.
The form of the questionnaire. Allowed values are:
  • QUIS: Questionnaire for User Interface Satisfaction
  • PUEU: Perceived Usefulness and Ease of Use
  • NAU: Nielsen's Attributes of Usability
  • NHE: Nielsen's Heuristic Evaluation
  • CSUQ: Computer System Usability Questionnaire (and the 3-item After-Scenario Questionnaire, ASQ)
  • PHUE: Practical Heuristics for Usability Evaluation
  • PUTQ: Purdue Usability Testing Questionnaire
  • USE: USE Questionnaire
If unspecified, the CSUQ is presented. If any other value is provided, it causes a server error.
The title at the top of the form. This is used as part of the Subject: line in the emailed data.
This is used as the Sender: of the data.
This overrides the paragraph of help at the top of the form. Supplying this requires a fair amount of expertise with HTML forms.
RATING SCALE OPTIONS
I like to use seven-point rating scales because they allow three levels of either positive or negative ratings; two levels seems too few. I prefer to range from 1-7 (bad-good) instead of the more techno -3...+3.
OPTION EXAMPLE DESCRIPTION
The value at the low end of the rating scale. This value can be negative.
The value at the high end of the rating scale.
The label at the low end of the rating scale (for questions not assigned a special label).
The label at the high end of the rating scale (for questions not assigned a special label).
The label to indicate Not Applicable.
The value to send when NA is chosen.
The number of lines in the comment area. If 0, then no comments can be added to questions. If 1, then a single input field is provided. If 2 or more, a textarea is provided.
NEGATIVE/POSITIVE POINTS
I like to ask users about the N most negative / positive points about a system (first negative, then positive). I usually get N/2 answers, so 3 or 5 is a good number to ask for.
OPTION EXAMPLE DESCRIPTION
Number of negative points about system for rater to supply.
Number of positive points about system for rater to supply.

Data Format and Analysis

Data is sent to the email address in a name=value format, with one name and value per line. Missing comments on specific questions and missing negative/positive points are omitted. You are on your own for the analysis. Obviously, you must coordinate with the questionnaire form.

Here is an example mail message with questionnaire data. 

From httpd@turing.acm.org Mon Oct  5 12:47:00 1998
Date: Mon, 5 Oct 1998 12:46:59 -0400
Subject: question.cgi: Questionnaire System Evaluation
Sender: perlman@acm.org
From: perlman@acm.org (question.cgi)
Apparently-To: perlman@acm.org

system=question.cgi
date=98-10-05 12:46:58
comment=I like the generalization of this system.<br>Thanks.
q1=1
q2=0
q3=2
q4=1
q5=1
q6=0
q7=-1
c7=I found the concept novel & difficult to grok.
q8=0
q9=NA
q10=0
q11=-1
q12=-1
q13=0
q14=0
q15=1
q16=1
q17=1
q18=2
c18=It has more than expected, and yet, not enough.  Sigh.
q19=2
negative1=no analysis tools
negative2=hard to completely customize
negative3=questionnaire terminology does not closely match my domain
positive1=nice looking forms
positive2=lots of options
positive3=based on established questionnaires
positive4=lets me automate data collection