I am trying to find examples of user behaviour research that combines both quantitative and qualitative data. It seems to me that most of the research leans heavily or exclusively towards one approach or the other. So you often see a large-scale survey study drawing some very specific conclusions, or heat-map or eye tracking studies that make some very general conclusions. The extent to which you can interpolate or extrapolate these results surely depends on if you are able to link them to a specific context. Considering that there are obvious advantages and disadvantages to each approach, what is the major concern with researcher running studies to collect both types of information? I don't by the argument of cost or time, because if the information being collected is not accurate or cannot be put into context correctly then it is a much bigger waste of time.
One of the key take-home messages for UX practitioners from Comparative Usability Evaluation 8 is to:
"Combine qualitative and quantitative findings in your report. Present what happened and support it with why it happened."
Because by applying qualitative and quantitative research methods you are able to treat a problem by linking the symptoms (what happened) to the root cause of the problem (why it happened).
Why isn't quantitative and qualitative data collected at the same time in more UX research/studies?
Answer
You’ll find a taxonomy and descriptions of different ways of integrating qualitative and quantitative methods in the “Mixed Methods Procedures” chapter in John Creswell’s (2013) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (SAGE Publications). The chapter includes examples of actual studies. The book is intended for social science field research, but the chapter can be applied to UX work too. For example, Luke Swartz’s thesis Why People Hate the Paperclip: Labels, Appearance, Behavior, and Social Responses to User Interface Agents is an example of a “sequential exploratory strategy.”
To summarize, you integrate the qualitative and quantitative results by:
Using the qualitative data to explain what you found in the quantitative data.
Using the qualitative data to decide what to manipulate and measure quantitatively.
Use one method to validate the results of the other (where one makes up for the weakness of the other).
Use whichever method is most suitable for each component of your theory you’re testing.
Qualitative and quantitative methods have different needs, so they are often conducted separately, almost as if they were separate studies. They often use different samples or sampling procedures. However, you can do mixed-methods in a single round of usability testing:
You encourage the user to think aloud, which (I’ve always assumed) has little impact on the relative quantitative performance (e.g., time to complete task, number of errors, eye-gaze durations). To get accurate quantitative results, avoid interrupting the user, perhaps doing so only if they get stuck for standard period of time.
At the end of the test, you can use a quantitative survey (e.g., SUS). After that you conduct a qualitative interview/debrief (which may involve going back into the app to discuss what happened at various places).
You perform qualitative analysis of the video tapes, the eye-tracking data (which may also be analyzed quantitatively), and interview results. Because qualitative analysis is time-consuming per user while quantitative methods need a relatively large sample size, you may want to perform qualitative analysis on subset of your users.
You integrate the results from the methods using one or more of Creswell’s strategies.
No comments:
Post a Comment