The
theory and promise of a 24/7 Web site are always talked about, the reality for
many company Web sites is that they need to be taken offline (at least
occasionally) to install new releases or upgrade software or hardware.
Occasionally you need to take your production site offline for an hour or two
during the night to install a new release. A variety of people are involved
with these releases and the pressure is high—everybody takes their turn working
through their piece of the process, knowing that offline time means lost
revenue and reduced customer service. In this race against the clock, quality
assurance is designated the last position—the final checkpoint before the site
is open to the public again.
In such a
scenario a quality assurance person brings to the table a critical career skill
that has nothing to do with the latest or coolest technology: the skill of
planning. It’s important to be organized in a profession that often senses
other people (such as the rest of the technical and management team) pacing and
waiting for QA to finish their job. Quality assurance needs to be clear with
the rest of the team about what needs to be done and the risks we take when
corners are cut because we are short on staff or short on time.
When the
magic hour approaches, the network staff begins by trying to close the site as
gently as possible for customers—waiting for people to sign off and hoping more
people aren’t logging in at three o’clock in the morning. Once the customer
count is as low as possible, a maintenance
page is posted stating that our site will be offline temporarily, and an email
will be sent when the site is available again. A connection outside of our firewall is used to verify that this
maintenance page is what is being shown to the "outside world by the
quality assurance guy." He enters his email address to receive
notification when the site is live again. Then, the network staff makes any
hardware or software upgrades as needed, followed by configuration changes.
Meanwhile,
our database administrators (DBAs) begin making modifications to the production
databases. Database tables may be added, changed, or deleted. Sometimes these
database changes can only happen when the DBAs can gain exclusive access to a
table. The only database changes that are made are those agreed upon in advance
by development, quality assurance, and the DBAs. These database changes are
well documented and executed according to the agreed-upon plan.
The
network staff continues by installing the new software. They follow release
notes that were prepared by quality assurance in advance. The developers review
configuration settings, ensure content is moved to production, and check that
the correct files are in place. Once the network, DBA, and developers are done,
quality assurance steps in. QA ensures that the release is ready, and that the
site is ready to go live again.
What does
quality assurance do at
Guidelines to check
the new release
Here are
the proposed steps that should be followed by the quality assurance team to
verify the correct release has been installed on each production server.
Log in
to Each Server using specifically created Login Page.
Generally
multiple production servers are used to support a Web site. A load-balancing
software is used that pushes each new customer session onto the server with the
lowest number of users. While this load balancer is an important element of our
production site, it is not helpful when quality assurance or development
needs/wants to log in to a particular production server. To work around this
issue, the developers should build a simplified version of our login page that
allows the quality assurance team to specify which production server they want
their session to be run on. For middle-of-the-night installations, this page
becomes an important tool as we use the simplified page to log in to each
server and ensure each server is ready to go.
Verify
Basics First
For each
server, some basic elements are checked that have nothing to do with the new
release. Always make sure you can access each server and log in using a
customer account that has been created ahead of time. Also check that basic
transactions relevant to our site can be performed (like searching for an item,
browsing the aisles, putting an item in the shopping cart, submitting an order,
etc.).
Verify
SSL is Works on Every Server
SSL
(secure sockets layer) is an imperative part of an e-commerce site, because it
renders any credit card number entry online reasonably safe. Check SSL to make
sure it’s working on each production server after every installation. There might
be secure and nonsecure pages in a web site. There are secure pages exposed
only to existing customers and nonsecure pages exposed to new customers. Check
both, and check each server.
Like any
assessment done by quality assurance, you have to assess risk, likelihood, and
impact. How big of a risk would it be to your site if SSL weren’t working? How
likely would it be for a customer to encounter the flaw? And, could you survive
the impact of that flaw on your production site? When you consider these points,
checking each secure page on each server is like maintaining your company’s own
life insurance policy; one that you wouldn’t want to skip payment on.
Use a
Checklist of Visual Features that Changed for the Release
While
most releases contain some functionality change that is not visually
noticeable, they may include some change to the user interface such as a newly
revamped page or a new icon/graphic. To prepare for your middle-of-the-night
releases, create a list of these quick visual checkpoints ahead of time. Log
into each production server and use these checkpoints to ensure each server has
the correct release in place. Remember, the entire release has already been
through the quality assurance process. This is the release process, not the
development process.
You can see how forcing a session on a particular server combined with knowing the visual changes of a release creates a reasonably fast, manual means of ensuring the new release is in place on each server. Logging in, checking the basics, checking SSL, and making a visual checkpoint of a few items have to be done on each server. Because you are working against the clock, log into each server rapidly so if a particular server is having an issue, you can call out that issue to the network staff immediately. Then go back through each server and check the basics, SSL, and visual checkpoints. Making sure the basic build is working before testing new features in depth is similar to the QA strategy used typically during the development process.
Verify
INI, Property Files, or Other Configuration Files
A
combination of property files and INI files are often used for configuration
settings that are specific to the environment. These files define where content
and graphics are being read from, session timeout values, error file locations,
etc. Since the system test environment is not configured the same as our
production environment, these configuration files must be carefully checked
with each release and for each server. You should have scripts in place that
the network staff uses which forces the files to be in sync from one server to
another, but still take the time to at least spot-check the configuration files
on each production server. Similar to the visual checkpoints, build a list of
any configuration file changes for the release in advance. This list is used by
the network staff who modify the files and by quality assurance, so everyone on
the team is reading from the same document.
Specific
Checkpoints for Specific scenarios
Sometimes
a change is made that impacts certain aspect of the site that are worth
checking in production. For example, a required field is added in online
registration that won’t work correctly unless the corresponding database
modification has been made. If needed, create accounts in advance for this type
of testing. If your database design is complicated or includes triggers, stored
procedures, etc., you might want to enter the first few orders or records and
check the data in the database before your site goes live.
Go
Live
Once
these checkpoints have been made and everyone on the team is confident the
build is ready, go live. Be ready to receive the user email notification
indicating that the site is now online, and then return to your test lab to
check how the site is being presented outside the firewall.
Steps to Verify the
Release is Working When the Site Is Live
Verify
the First One or More Records in the Database
If new
tables have been added to the database or major changes have been made, the
first few records in the production database are checked by the DBA and QA team
to ensure the data is accurate.
Verify
Any Third-Party Links
You might
have a couple of links that work with other companies and their Web sites. When
your site goes live, check to see if these links are working to ensure the
sites are working together as defined. Check them from both inside and outside
of our firewall.
Check
that Nightly or Weekly Jobs Are Ready for the Environment
Last on
the list should be checking of any nightly or weekly jobs that need to be run
in production have been modified and are in place. Typically, DBA will check to ensure data warehousing
jobs and other jobs are in place and configured correctly. Following the
release, the team should keep an extra eye open for any production issues. The
technical support team—which must be told in advance of each release—also waits
to hear if any issues arise. If so, the release team is immediately called into
action.
Conclusion
Preparing
a checklist in advance helps minimize time and confusion during the release
cycle. This is especially helpful if you have to complete the release cycle at
three or
