Abstract
Since at least the late 1990’s, many web development organizations have been asking the question “What is the industry standard response time for a web page?” As a result various attempts have been made to create such a standard, but none of these standards have been generally accepted – largely because every web user has a different personal feeling about whether a particular web page is responsive enough to be considered “acceptable”. This paper demonstrates why a blanket response time standard is not the best answer in terms of design criteria and explores the critical items that need to be considered in determining appropriate response time goals for websites.
Introduction
As an active member of the international Performance Testing community and as an experienced practitioner, I have seen questions like this one posed numerous times:
“I’m desperately trying to find out what the industry standard response times are. What are reasonable benchmarks that are acceptable for Web sites at the moment? Is 1.5 seconds a reasonable target????”
My answer to questions like this one always starts with “It depends on . . . ” My friend Joe Strazzere addressed this particular question quite well on a popular internet forum, as follows:
There are no industry standards. You must analyze your site in terms of who the customers are, what their needs are, where they are located, what their equipment and connection speed might be, etc., etc. I suspect 1.5 seconds would be a rather short interval for many situations. Do you really require that quick of a response?[1]
The bottom line is that what seems fast is different in different situations. So how do you determine how fast is fast enough for your application, and how do you convert that information into explicit, testable requirements? Those are the topics this article addresses.
Considerations Affecting Performance Expectations
Let’s start by discussing the leading factors that contribute to what we think fast is. I believe these considerations can be divided into three broad categories:
• user psychology
• system considerations
• usage considerations
None of these categories is any more or less important than the others. What’s critical is to balance these considerations, which we’ll explore individually here, when determining performance requirements.
Considerations Affecting Performance Expectations
Of the three categories, user psychology is the one most often overlooked – or maybe a better way to say this is that user psychology is often overridden by system and usage considerations. I submit that this is a mistake. User psychology plays an important role in perceived performance, which is the most critical part of evaluating performance.
Consider this example. I recently filled out my tax return online. It’s a pretty simple process: you navigate through a Web application that asks you questions to determine which pages are presented for you to enter data into. As I made a preliminary pass through my return, I was happy with the performance (response time) of the application. When I later went back to complete my return, I timed the page loads (because I almost always think about performance when I use the Internet). Most of the pages returned in less than 5 seconds, but some of the section summary pages took almost a minute! Why didn’t I notice the first time through that some pages were this slow? Why didn’t I get frustrated with this seemingly poor performance? I usually notice performance as being poor at between 5 and 8 seconds, and at about 15 seconds I’ll abandon a site or at least get frustrated. There’s no science behind those numbers; they’re just my personal tolerance levels. So what made me wait a minute for some pages without even noticing that it was slow? The answer is that when I requested a section summary page, an intermediate page came up that said:
“The information you have requested is being processed. This may take several minutes depending on the information you have provided. Please be patient.”
When I received that message, I went on to do something else for a minute. I went to get a drink, or checked one of my email accounts, or any of a million other things, and when I came back the page was there waiting for me. I was satisfied. If that message hadn’t been presented and I had found myself just sitting and waiting for the page to display, I would have become annoyed and eventually assumed that my request wasn’t submitted properly, that the server had gone down, or maybe that my Internet connection had been dropped.
So, getting back to the initial question of how fast is fast enough, from a user psychology perspective the answer is still “it depends.” It depends on several key factors that determine what is and isn’t acceptable performance.
The first factor is the response time that users have become accustomed to based on previous experience. This is most directly tied to the speed of their Internet connection. My mother, for example, has never experienced the Internet over anything other than a fuzzy phone line with a 56.6-kilobytes-per-second modem. I’m used to surfing via high-speed connections, so when I sign on at my mother’s house I’m frustrated. My mother thinks I’m silly: “Scott, I think you’re spoiled. That’s as fast as we can get here, and a lot faster than we used to get! You never were very patient!” She’s right – I’m not very patient, so I have a low tolerance for poor Web site performance, whereas she has a high tolerance.
Another factor is activity type. All users understand that it takes time to download an MP3 or MPEG video, and therefore have more tolerance if they’re aware that that’s the activity they’re performing. However, if users don’t know that they’re performing an activity like downloading a file and are just waiting for the next page to load, they’re likely to become frustrated before they realize that the performance is actually acceptable for the activity they’re performing.
This leads us to the factor of how user expectations have been set. If users know what to expect, as they do with the tax preparation system I use, they’re likely to be more tolerant of response times they might otherwise think of as slow. If you tell users that the system will be fast and then it isn’t, they won’t be happy. If you show users how fast it will be and then follow through with that level of performance, they’ll generally be pretty happy.
The last factor we should discuss here is what I call surfing intent. When users want to accomplish a specific task, they have less tolerance for poor performance than when they’re casually looking for information or doing research. For example, when I log on to the site I use to pay bills, I expect good performance. When I’m taking a break from work and searching for the newest technology gadgets, I have a lot of tolerance for poor performance.
So with all of these variables you can see why, as Joe Strazzere said, “There are no industry standards.” But if there are no industry standards, how do we know where to start or what to compare against? I’ll describe some rules of thumb later, when we get to the topic of collecting information about performance requirements.
