Over the days we’ve been running the test, we had north of 300 people take it, and we think the numbers are a pretty good sample. Based on our analytics data, I could see that we had participants from 64 different countries, and it took the average user 2:56 to take the test.
So what did we learn from all that? Read on and find out:
Part one: Learned behaviors and conventions
Grids: The first three questions used a grid to discover users expected behaviours. When you overlay the grid over the different UIs you can see how the designs fit in with peoples perceptions.
Learned conventions tend to show through in this kind of area, but conventions are often an important standard for designers to follow (as you will see reflected in the results later on)
1) Where would you click to logout?
As you can see from the results, by far and away the majority of people expect to find the log out button on the top right hand side of the page. This is the most common location for a logout button across numerous websites (have a look for yourself), and so something people are very familiar with. If you want to put your logout button in a different location, you should make sure there is a good reason for doing so.
When overlaid, we can see both Gmail and Hotmail follow this convention, but for some reason, Yahoo! Mail locates their logout button in the middle of the page. What’s worse, it doesn’t render properly in the browser I’ve been using for this test – Safari 5.
Yahoo! Mail 0; Hotmail 1; Gmail 1
2) Where would you click to write a new email?
The results for this are fairly obvious – as you can see from the heatmap, more all less all users clicked in the top left hand corner to compose a new mail – this is the location for all three services we tested, and the most common location across more or less all webmail services.
Yahoo! Mail 1; Hotmail 2; Gmail 2
3)Where would you click to reply to an email?
Some interesting results for this one – there is really quite a spread all over the screen, which shows perhaps there aren’t quite so many learned behaviours for this. There is a concentration towards the top-middle and top left of the screen however. Lets overlay this over the different services and see how they compare:
Each of the services seems to have ever so slightly different locations for their buttons, and the users of each seem to have learnt these. I don’t see that one or the other is ahead in this test.
Personally though, I do like both the Hotmail and Gmail interface for having reply buttons at the bottom of messages, so you can easily reply after you’ve finished reading. What’s more, the Gmail message reply is the box you can see at the bottom of the messsage. Just put your cursor in there and you can begin typing. Pretty straight forward.
Yahoo! Mail 2; Hotmail 3; Gmail 3
4) Which interface do you prefer?
This is a bit of a popularity contest, but evidence shows all things being equal people have a better experience with nicer looking interfaces. Stands to reason. You can see the results here.
The numbers were:
- Yahoo! Mail: 20%
- Hotmail: 16%
- Gmail: 59%
The remaining 5% is people either skipping the question or clicked the side of the interface. Regardless, these numbers are pretty defenitive, and Gmail is well ahead.
Yahoo! Mail 2; Hotmail 3; Gmail 4
5)Do you prefer your email threaded or non-threaded?
Great results for this one, fairly overwhelming:
- Threaded: 73%
- Not Threaded: 12%
- Don’t know/care: 12%
At the present time, Gmail is the only provider that offers threded messages to all its users. Hotmail is currently rolling out threading on a limited basis, but by no means to all users (our test account doesn’t have threading), and Yahoo! Mail offers no threading at all. Results:
Yahoo! Mail 2; Hotmail 3; Gmail 5
6)Do you prefer to reply on a new page or the same page?
Another comprehensive result as you can see from the image below:
The numbers were as follows:
- Reply on a new page: 11%
- Reply on the same page: 81%
- Don’t know/care: 5%
Again, Gmail is the only provider of the three that enables you to reply on the same page – a behaviour prefered by 81% of users tested.
Yahoo! Mail 2; Hotmail 3; Gmail 6
7) Do you prefer banner ads or text ads?
I was slightly surprised by the results of this test – banner ads are evidently more popular than I thought, and although the results were comprehensively in favour of text ads, it seems there is life for banner ads yet. The results were:
- Banner ads: 17%
- Text ads: 68%
- Don’t know/care: 13%
Again though, seeing as Hotmail and Yahoo! Mail both use banner ads, and Gmail text ads, it looks like Gmail is the winner here.
Yahoo! Mail 2; Hotmail 3; Gmail 7
8) Do you prefer a site with or without a landing page?
I was surprised when testing Yahoo! Mail that they used a landing page when you log into your mail. This got me thinking as to whether or not this was the prefered behaviour of users. The results are:
- With landing page: 7%
- Without landing page: 82%
- Don’t know/care: 3%
Yahoo! Mail is the only one of the three that uses landing pages – and obviously this isn’t very popular with users. Score:
Yahoo! Mail 2; Hotmail 4; Gmail 8
Part two: Usability comparison
For the next series of questions, the success criteria are less than 10 seconds to complete a task – thus giving the user time to read instructions and to ensure a usable interface – and a success rate of over 75% – to make sure most users are completing the task satisfactorily.
9) Where would you click to change your settings? (Yahoo! Mail)
As you can see in the results, the success rate here is 75%, but what we also need to pay attention to is the average time to complete this test – in this case, 27.58 seconds.
This shows us that while users can indeed find the settings area, they are taking too long to do in. In this case Yahoo! Mail could try to make the text more prominent or more visible in order to make this area more usable.
Yahoo! Mail 2; Hotmail 4; Gmail 8
10) Where would you click to access your calendar? (Gmail)
These results are a bit of a fail for Gmail – it fails on both fronts. There is only a 69% success rate, and it took the average user just under 14 seconds to complete the task.
If they wanted to make the calendar more visible, they could try moving it’s location, or featuring it in more than one location. The fact that 16% of people clicked on the left hand side indicates that adding a calendar button in that area would be beneficial.
Yahoo! Mail 2; Hotmail 4; Gmail 8
11) How would you mark a message as spam? (Hotmail)
And at last we have a task that meets our criteria. 76% of users (or 87% if you count the ones marking the messages) successfully completed the task with an average time of 9.16 seconds. This could still be improved upon, but isn’t a bad result.
Yahoo! Mail 2; Hotmail 5; Gmail 8
12) Where would you click to access your calendar? (Yahoo! Mail)
As you can see here, a pretty poor result. It took an average of 17.73 seconds for people to click, and after that, there was still a success rate of just 54%.
Yahoo obviously needs to put a lot more consideration in to the placement of the calendar button. The ammount of people clicking in the upper left hand side tends to suggest that many people look to this area, so perhaps by offering a link there they would increase their success rate.
Yahoo! Mail 2; Hotmail 5; Gmail 8
13) How would you mark a message as spam? (Gmail)
As you can see from the results, there was an 85% success rate, and an average click time of 6.38 seconds. This is obviously well inside our success criteria, and a strong result for Gmail.
Yahoo! Mail 2; Hotmail 5; Gmail 9
14) Where would you click to change your settings? (Hotmail)
Another good result for Hotmail here – an 80% success rate, and an average click time of 7.54 seconds. I think this makes a lot of sense as this is a very common location for a settings/options link to be.
Yahoo! Mail 2; Hotmail 6; Gmail 9
15) How would you mark a message as spam? (Yahoo! Mail)
A much better showing for Yahoo! Mail in this case, an 87% success rate, and a click time of just 4.13 seconds. This really shows the value of following conventions – bear this in mind when designing.
Yahoo! Mail 3; Hotmail 6; Gmail 9
16) Where would you click to change your settings? (Gmail)
A 91% success rate, and an average click time of 6.01 seconds show that Gmails setting page location is working very well – these are very good numbers.
Yahoo! Mail 3; Hotmail 6; Gmail 10
17) Where would you click to access your calendar? (Hotmail)
And another success to round out our tests. 5.5 seconds average click time, and a 78% success rate. Quite a spread on this page though, so it shows people still aren’t quite sure where to look for calendars.
Yahoo! Mail 3; Hotmail 7; Gmail 10
Conclusion
This was a pretty interesting test, and created some interesting results. It shows the value of following conventions and observing user behaviours when developing or reworking your designs. If you’d like to browse through the whole test results, you can go here.
The final scores were: Yahoo! Mail 3; Hotmail 7; Gmail 10.
While Gmail was ultimately ahead in these tests, all three of the services have areas they could improve upon.
Hotmail was also a reasonably strong contender, and with updates which they are currently rolling out, the usability of their service should continue to improve.
Yahoo! Mail was lagging far behing in third place, and really needs to pick up its game if it wants to remain competitive. They are probably relying on people who have been using their service for many years, and I would be surprised if they are still signing up many users.