Written by Member
Software Testing Techniques
Introduction
In software engineering, performance testing is testing that is
performed, from one perspective, to determine how fast
some aspect of a system performs under a particular
workload. It can also serve to validate and verify other quality
attributes of the system, such as scalability, reliability and
resource usage. Performance testing is a subset of
Performance engineering, an emerging computer science
practice which strives to build performance into the design
and architecture of a system, prior to the onset of actual
coding effort.
Performance testing can serve different purposes. It can
demonstrate that the system meets performance criteria.
It can compare two systems to find which performs better.
Or it can measure what parts of the system or workload
cause the system to perform badly. In the diagnostic case,
software engineers use tools such as profilers to measure
what parts of a device or software contribute most to the
poor performance or to establish throughput levels
(and thresholds) for maintained acceptable response time.
It is critical to the cost performance of a new system, that
performance test efforts begin at the inception of the
development project and extend through to deployment.
The later a performance defect is detected, the higher the
cost of remediation. This is true in the case of functional
testing, but even more so with performance testing, due to
the end-to-end nature of its
scope.
In performance testing, it is often crucial (and often difficult
to arrange) for the test conditions to be similar to the
expected actual use. This is, however, not entirely possible
in actual practice. The reason is that production systems
have a random nature of the workload and while
the test workloads do their best to mimic what may happen
in the production environment, it is impossible to exactly
replicate this workload variability - except in the most
simple system.
Loosely-coupled architectural implementations (e.g.: SOA)
have created additional complexities with performance testing.
Enterprise services or assets (that share common infrastructure
or platform) require coordinated performance testing (with all
consumers creating production-like transaction volumes and
load on shared infrastructures or platforms) to truly replicate
production-like states. Due to the complexity and financial and
time requirements around this activity, some organizations now
employ tools that can monitor and create production-like conditions
(also referred as "noise") in their performance testing environments
(PTE) to understand capacity and resource requirements and
verify / validate quality attributes.
Risks Addressed Through Performance Testing
Speed-Related Risks
Speed-related risks are not confined to end-user satisfaction, although
that is what most people think of first. Speed is also a factor in certain
business- and data-related risks. Some of the most common speed-related
risks that performance testing can address include:
* Is the application fast enough to satisfy end users?
* Is the business able to process and utilize data collected by the
application before that data becomes outdated? (For example,
end-of-month reports are due within 24 hours of the close of
business on the last day of the month, but it takes the application
48 hours to process the data.)
* Is the application capable of presenting the most current information
(e.g., stock quotes) to its users?
* Is a Web Service responding within the maximum expected response
time before an error is thrown?
Scalability-Related Risk-Mitigation Strategies
The following strategies are valuable in mitigating scalability-related risks:
* Compare measured speeds under various loads. (Keep in mind that
the end user does not know or care how many other people are
using the application at the same time that he/she is.)
* Design load tests that replicate actual workload at both normal and
anticipated peak times.
* Conduct performance testing with data types, distributions, and volumes
similar to those used in business operations during actual production (e.g.,
number of products, orders in pending status, size of user base). You can
allow data to accumulate in databases and file servers, or additionally
create the data volume, before load test execution.
* Use performance test results to help stakeholders make informed
architecture and business decisions.
* Work with more meaningful performance tests that map to the
real-world requirements.
* When you find a scalability limit, incrementally reduce the load and retest
to help you identify a metric that can serve as a reliable indicator that the
application is approaching that limit in enough time for you to apply
countermeasures.
* Conduct performance tests beyond expected peak loads and observe
behavior by having representative users and stakeholders access the
application manually during and after the performance test.
Stability-Related Risks
Stability is a blanket term that encompasses such areas as reliability, uptime,
and recoverability. Although stability risks are commonly addressed with
high-load, endurance, and stress tests, stability issues are sometimes detected
during the most basic performance tests. Some common stability risks
addressed by means of performance testing include:
* Can the application run for long periods of time without data corruption,
slowdown, or servers needing to be rebooted?
* If the application does go down unexpectedly, what happens to partially
completed transactions?
* When the application comes back online after scheduled or unscheduled
downtime, will users still be able to see/do everything they expect?
* When the application comes back online after unscheduled downtime,
does it resume at the correct point? In particular, does it not attempt to
resume cancelled transactions?
* Can combinations of errors or repeated functional errors cause a
system crash?
* Are there any transactions that cause system-wide side effects?
* Can one leg of the load-balanced environment be taken down and
still provide uninterrupted service to users?
* Can the system be patched or updated without taking it down?
Stability-Related Risk-Mitigation Strategies
The following strategies are valuable in mitigating stability-related risks:
* Make time for realistic endurance tests.
* Conduct stress testing with the key scenarios. Work with key
performance indicators (network, disk, processor, memory)
and business indicators such as number of orders lost, user login
failures, and so on.
* Conduct stress testing with data feeds that replicate similar business
operations as in an actual production environment (e.g., number of
products, orders in pending status, size of user base). You can allow
data to accumulate in databases and file servers, or additionally create
the data volume, before stress test execution. This will allow you to
replicate critical errors such as database or application deadlocks and
other stress failure patterns.
* Take a server offline during a test and observe functional, performance,
and data-integrity behaviors of the remaining systems.
* Execute identical tests immediately before and after a system reboot.
Compare the results. You can use an identical approach for recycling
services or processes.
* Include error or exception cases in your performance test scenarios
(for example, users trying to log on with improper credentials).
* Apply a patch to the system during a performance test.
* Force a backup and/or virus definition update during a performance
test.
* Validate the functional accuracy of the application under various loads
by checking database entries created or validating content returned in
response to particular user requests.
Summary
Almost all application- and business-related risks can be addressed through
performance testing, including user satisfaction and the application’s ability
to achieve business goals.
Generally, the risks that performance testing addresses are categorized in
terms of speed, scalability, and stability. Speed is typically an end-user
concern, scalability is a business concern, and stability is a technical or
maintenance concern.
Identifying project-related risks and the associated mitigation strategies
where performance testing can be employed is almost universally viewed
as a valuable and time-saving practice.
