As recent online sales figures have shown increasing numbers of us are choosing the Internet for our regular purchases and to access services such as banking and utilities. Last year online sales totalled over £40billion in the UK alone making the development of a Web storefront something no serious retailer can afford to ignore.
As with their traditional bricks and mortar outlets in order to attract more customers and to give them an intense experience whilst accessing their website, many organisations have had to invest in a variety of techniques and content to garner their share of this huge revenue potential including a whole new way of thinking:
- Search engine optimisation to capture customers
- Ad-words to meet specific search requests
- Dynamic content and personalised content
- Flash and video content, including live video in some cases
- Integration with numerous web service providers for specific content services
- News, blog and social network feeds
- Hand-off to external payment processing organisations
However all this content has an impact on the complexity of the application and the infrastructure needed to support it. This increased complexity means the application is rarely in the same state for very long before a change is made either to content or to the code or to the infrastructure underlying it. Software and security updates get applied daily, any one of which could create a performance bottleneck, resulting in poor page load times on the customer’s desktop.
Page load time is extremely important for today’s online customer, with 2 seconds for a page to render being the limit for 47% of online shoppers. Speed also breeds loyalty, a scarce commodity in the online retailing world with 52% of online shoppers saying that quick page loading is important to their site loyalty according to a Forrester survey published in May 2009.
Dissatisfaction was attributed to a number of things, with the top two being due to the web site being too slow – 23% or taking too long to render due to site crashes or error messages, 17%.
Forrester go on to say that “seventy-nine percent of online shoppers who experience a dissatisfying visit are likely to no longer buy from that site. Also, 46% of dissatisfied online shoppers are more likely to develop a negative perception of the company, and 44% will tell their friends and family about the experience. With 87% of customers shopping in both online and retail channels, the impact of a bad online experience reaches beyond the Web”.
This background to current customer performance expectations needs to be treated very seriously, especially with the trend towards the use of mobile browsing devices such as smart phones and netbooks. Increasing numbers of customers have shopped via mobile devices, with a high proportion (27%) of them reporting that it is dissatisfying due to the mobile shopping experience being too slow.
As a consequence of this change in the level of importance attributed to their Web presence the need to embed performance testing as an element of the application development process has assumed critical importance.
It also follows that testing in general needs to evolve and become more disciplined, which to a certain extent it already has, as can be seen by the increasing influence of the Testing Maturity Model (TMMi). The application of the TMMi model to software quality tends to focus on functionality and meeting of the specification, with maturity of non-functional characteristics like performance lagging somewhat behind the curve.
Experimentus, one of the leading organisations involved in the TMMi foundation, offer a free TMMi survey through their website, whereby you can assess where you sit on the testing maturity curve. If you want to see how your organisation performs, visit http://experimentus.com/, the results are often quite a surprise.
The science behind testing is often mis-understood, resulting in a simplistic test approach that generates results with dubious value. Testing is a statistical process requiring discipline to ensure test results that can be used to forecast live performance with any degree of accuracy or certainty. Commonplace examples of bad science include:
- Not using a “live-like” test environment
- Using too small a sub-set of test data or test users
- Extrapolating results in a straight line
- Using record and playback tools without enhancing the resulting scripts
- Assuming a single test run will be sufficient
- Not running tests for long enough for volume issues to surface
- Not measuring from the end-user perspective
- Taking internal data centre measures as proxy for end-user experience
- Assuming network is same for all end-users
A number of strategies are available to tackle the performance testing aspects of application deployment:
- Do nothing.
- Acquire an open source tool and get developers to do as final phase.
- Buy a commercial tool and build a specialist team.
- Outsource, most likely to an offshore provider
- Testing as a service
Whilst the “Do Nothing” strategy has its place, usually for low risk applications, it is not a recommended strategy for any customer facing application where customer experience is important. The zero cost is more than offset by the risk of downstream issues, the impact on the company’s image, and the abdication of control.
Open source solutions offer a cheap entrance to performance testing, and will reduce the risk and cost of failure. The issue with open source is the levels of maintenance that is required of the scripts, coupled with the difficulty of retaining the necessary skills in-house to continue to operate the tools in the future.
Undoubtedly the best way to reduce the cost and risk of live performance issues is to invest in one of the commercially available tools, and to build a specialist team around the use of the tool. This approach ensures full control of the testing cycle, but the cost of software, servers and staff makes it something that is the preserve of only the largest organisations. Even with this approach, it is often necessary to top-up resources with subject matter experts who can tackle the more difficult application nuances.
Outsourcing the performance testing to one of the specialist outsourcers, if done correctly, can have a similar risk and impact profile to an in-house team. From a cost perspective, you would expect the overall costs to be lower, but experience suggests that outsourcers tend to over-resource their projects and require a significant amount of “service management” by the customer themselves. Overall this is a similar profile to the in-house team, but with a slight loss of control.
The concept of testing-as-a-service strikes a balance between all the above approaches. It is only used when it is needed, so no need for expensive licenses, servers and staff to be retained. The costs are very competitive, which when coupled with the improvement in risk and cost of performance issues, is comparable to the outsourced model. Since the solution itself sits outside the computing environment of the organisation, it is able to truly reflect the experience a customer would have in accessing and using the application.
TaaS is a relatively new concept but one which is rapidly gaining ground in the general software testing sector. This is being driven by the requirement for a quick and convenient method of testing the user journey both during the development phase and post-live as the application changes and evolves in response to market dynamics.
Traditional software testing methods, particularly the use of complex commercial tools in-house, do not offer the flexibility needed to keep pace with the changing development landscape, which can lead to corner cutting in order to meet the tight deadlines and budget constraints, with the risk of a major failure during a critical trading period.
In the recent past, day rates for testing have attracted a lot of attention, with offshore seeming very attractive based on the headline rate and the volume of testers available. The efficiency of such approaches is now starting to be questioned as offshore differentials start to be eroded, and experience shows that the day rate alone doesn’t guarantee the desired outcome of quality assured software.
TaaS offers a new “Third-Way” of approaching performance testing; using proven, marketing leading testing tools backed by qualified and experienced software testing consultants offers the best of all worlds. Applications can be tested using the most sophisticated methodologies available, recreating the end-to-end experience from the user perspective without the need for significant capex investment and fixed cost pricing means that projects can be delivered with much greater financial and technical control than is possible when using traditional outsourced services.
Loading ...