navigation - Has user testing found that the "three horizontal bars" for main menu on mobile is commonly understood?

"Three horizontal lines" aka "hamburger icon" seems to be becoming a convention for the "main menu" button, especially on mobile sites:

Has anyone done any user testing of this convention, or got any A/B testing results? (i.e. is there an increase in bounce rate compared to using the word "Menu"?)

I did some quick user testing with wireframes featuring this icon for the main menu. 3 out of 6 people got it and used it. The others did not.

One person in particular had a lot of trouble "getting back to the top" (i.e. restarting their navigation from the top level of the site) as they didn't know about the convention, or that the logo would link to the home page.

I guess it will be different for different demographics - users of Facebook on mobile probably know it.

Anyone got any findings they can share?

Answer

James Foster of Exis Web did an interesting A/B test on the hamburger icon:

Tests on mobile showed a difference, though not all that significant, when the icon was used with a border (so it looks like a button):

Perhaps more interestingly, the A/B test seemed to more clearly indicate that desktop users don't understand the icon:

I tested 4 variations of the content inside a blue button:

(Baseline) The word “MENU” The word “MORE” Hamburger icon + “MENU” Hamburger icon + “MORE” The results were far more significant.

Tests 2, 3, and 4 performed far worse (18%, 31%, and 43% worse respectively).

And interestingly, the icon + label performed worse than just the label (which is usually pretty easy to integrate into a desktop page/app). So it would appear for mobile the icon works serviceably, as long as it looks like a button, and on desktop it's best to not use it at all.

James Foster then conducted a second AB test, this time looking at how the hamburger menu tests when used in combination with the word MENU and with a button border and combinations thereof.

Based on this and my previous AB test, a flat hamburger icon may not be ideal on a responsive website (remember this is a website not an app). Using the word MENU (and making it look like a button) could be more helpful for visitors.

This does not mean that users do not understand the hamburger/sandwich – it could be that the word MENU draws more attention.

This also supports findings recently published by NNGroup

Clear labels help users make decisions faster: they give good information scent about what will come next. Labels should still be used for newer icons, such as the three-line menu icon (or “hamburger” icon)...

...Users are still unfamiliar with newer icons, including the three-line menu icon