Formative and Summative Usability Testing

I bumped into this document and it suggests the use of (a) Formative and (b) Summative Usability Testing.

To be honest, is the first time that I bumped into this kind of terminology, and I while reading about it, it's not quite clear to me how can I use these tools to assess usability.

Can anyone give me an example of each?

Answer

Short version:

Do Formative Usability Testing at the start of the design phase, testing with paper-prototypes and similar. Do this to discover insights and shape the design direction.

Do Summative Usability Testing during latter half of the development phase, testing with actual working prototypes. Do this to determine metrics (time on task, success rates), which you would then compare test results from improved versions against.

---

Longer version:

In an article by Jeff Sauro where he discusses whether the two terms are useful he defines them as:

Most usability testing involves finding and fixing problems as part of an iterative design process to make an interface more usable. It is typically called a Formative Usability Evaluation. In contrast, a Summative Usability Evaluation describes the current usability of an interface—as measured by things like task times, completion rates and satisfaction scores. Summative tells you how usable an interface is and formative tells you what isn't usable.

In other words - Formative reports on what is broken, and Summative is for what is working.

---

Akendi describes Formative and Summative usability testing on their site, and has this to say:

Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process and - if applied early in the development process - provides valuable insights of where users have difficulty reaching their user goals with the technology (website, desktop GUI design, hardware product) or service.

They go on to describe using sketches or paper prototypes for the formative testing to identify if they are heading in the right direction.

And then this…

Summative usability testing is a Quality Assurance (QA) type of test usually performed later in the development process. A similar usability test protocol is used as in formative usability testing but now this setup is used to do formal user acceptance testing before the product is released to the target audience.

… which works on the assumption that the design direction is the right one and that there is potential for a usable solution, and thus the summative usability testing determines whether the execution actually delivers on that promise.

---

UserFocus have a summary of Formative and Summative usability testing in an article about Common Industry Format for reporting usability tests.

The CIF makes a distinction between "formative" and "summative" usability tests.

Formative tests are carried out:

• During the development of a product;


• To mould or improve the product;


• Virtually anywhere (you don’t need a lab);


• With the test administrator and the participant co-present.

The outputs from a formative test may include:

• Participant comments in the form of a "thinking aloud" narrative (for example, attitudes, sources of confusion, reasons for actions);


• Photographs and highlights videos;



• Usability problems and suggested fixes.

In contrast, summative tests are carried out:

• At the end of a development stage;


• To measure or validate the usability of a product;


• To answer the question: "How usable is this product";


• To compare against competitor products or usability metrics;


• To generate data to support marketing claims about usability;


• In a usability lab;



• With the participant working alone.

The outputs from a summative test may include:

• Statistical measures of usability (for example, success rate, average time to complete a task, number of assists);


• Reports or white papers.

---

If you don't do Formative Usability Testing, you might end up designing something that simply will never be a usable solution (e.g. a baby crib made from barb wire and broken glass, vs. one of wood and fabric). If you don't do Summative Usability Testing then you won't find out if your potentially usable solution has any problems or not (e.g. a baby crib made of wood and fabric, but with very awkward latches, lead paint, and nasty nails sticking out everywhere).