Short Version
If you run an A/B Test, and the results are that there's no difference in your key metric, do you keep the experimental version, or revert the site back to the control?
Story Version
Your client comes dashing into your office.
"I know what'll make this webpage POP! Change the product photo to a buff dude holding a beer, and put the call to action over his chest!"
"Umm...I don't think that really meets the needs of your users, and it doesn't fit with the theme of the..."
"My mother LOVES buff dudes!"
"Umm...ok, how about we A/B Test that, and see if it makes any difference."
So you run the silly A/B Test, and you're somewhat surprised to note that there's no difference. Would it make more sense to just implement the change request, or tell the client that it didn't actually help?
Answer
Assuming you're testing a change to an existing system, favor the control. It's what people are used to simply because it's already in place. If tests are inconclusive against users that aren't familiar with your site, at best the change won't help, at worst it briefly confuses current users.
You failed to reject the null hypothesis, stick to what you have.
If it's an A/B test of an unreleased feature...it's a judgement call. Do whatever fits best with the site or standard conventions. Since you didn't notice any difference, it's possible that both fit equally.
However, make sure you've run a proper test; if lots of your tests are inconclusive, it's possible you're testing the wrong changes (it's too little to matter) or you're not testing properly. It's possible that the feature you're A/B testing didn't matter very much, and if that's the case you shouldn't put the extra effort into making the change.
No comments:
Post a Comment