StackExchange rolls out a new top bar on meta-StackOverflow. This is a complex innovative design containing several new (as in unfamiliar to current user base) elements.
Should one present a new UI design to users/testers as a whole as opposed to user-testing each new element and only then A/B testing at the top level?
Answer
In the situation you describe, you have to test the whole.
If you have just designed something, you want to know if it works at all. For this, always test the whole. The point of a complex system is that it is more than the sum of its parts. Reactions to parts will not predict reactions to the whole. In many situations, some parts won't even make sense without the whole (or at least without a large subset of other parts of the whole). Use an A/B test for it.
If you are starting a new redesign, you want to know how the existing design works, so you can tweak it. For this, test the parts. This assumes that you have a design, and you know how well it works (you have already tested the whole). Now you want to improve it. This is the right place to test each part, to find out how it contributes to the design working or not working.
But an A/B test is the wrong tool here. You should find some metric to judge the observed reactions to each element. This can rely on recorded usage data ("we designed an inbox notification reminding a user that he has a message to respond to, good metrics could be the ratio of users who go directly to their inbox after seeing the notification, or the mean time a message stays unopened in the inbox") or focused questions during user research ("We rearranged our data display, so we will ask the user how hard he feels it is to find data he wants").
Of course, you can end up testing a small element with an A/B test too. This is done when the only thing you redesigned was that small element, or alternatively, when you redesigned a few small elements independently from each other - but then you are technically running multiple tests of several small wholes in parallel, not a test of a connected whole. This still follows the principle I outlined above: test the whole to prove that a just-made redesign works, test parts to find out what to improve in the next redesign.
No comments:
Post a Comment