You are currently on IBM Systems Media’s archival website. Click here to view our new website.

MAINFRAME > Tips & Techniques > System Tuning

Quality Assurance: Transforming a Prototype into a Product

In my last article I discussed the first two stages of application testing: writing code, syntax checking, structure analysis and compiler code validation resulting in an error-free executable program or module; and verification of primary logic, basic functions, error handling, and interaction with external functions, subroutines and systems. When this process is complete, the program is essentially functional: It performs properly to the degree the programming staff can test.

But there’s a lot more to testing most programs beyond whether it performs its primary purpose; it has to handle all situations that users encounter. Testing basic business function is easy; the real challenge is to ensure or expand logic to handle exception, extraordinary or combinational situations that arise when the program handles a real-world workload, to catch and handle errors that rarely occur until hundreds or thousands of users are concurrently driving the code, to deal with multiple exceptions interacting, and to deal with insidious relationships of records or files populated in the millions or billions.

This is where quality assurance testing becomes vital and crucial, turning an automated business function into an effective tool for business agents. Because testers during this checkout phase need in-depth, comprehensive business knowledge and skills germane to the program(s) purpose, senior operational specialists are needed to design test cases and validate test results. Determination of correct results for a particular test case can often only be determined by highly-trained specialists because of their rarified and sophisticated nature.

A Case Study

Because it was so comprehensive, user-involved, user-led and interactive, the most successful quality assurance effort I’ve been part of was implemented in the most complex application development project I’ve experienced.

The application was an Accounts Receivable (A/R) online processing system that had countless variations including installment payments (cash, credit or check), credit offerings, date-driven late fees varying by state, partial versus full versus excess versus multiple payments, negotiated payment plans, state-determined taxes, special offers, discounts, bounced checks or declined credit payments, differing rules depending on first, further and last payments; the permutations and combinations led to virtually endless scenarios. The knowledge of how each situation or combination thereof resided in a select few individuals, and there were heretofore unforeseen situations that needed to be investigated.

With this as background, and with substantial testing and fixes already completed, the application was moved into production with minimal quality assurance testing. For a few days it appeared A/R had received a very useful new tool. The busy part of the year was nearing, and everyone was excited to have new capabilities make a frantic time more stable. Then calls started coming in, questioning invoices and payments. Even more disconcerting, most complaints proved valid. Further inspection revealed that every error was due to a combination of factors that hadn’t been considered during testing, and that the system was functioning as specified. The design had omitted innumerable exceptions or mix of factors. Even worse, the findings led to the discovery that even more rarified circumstances would occur and fail.

To call this a catastrophe was a massive understatement. While the percentage of flawed transactions was minor, the number of customers affected was large because of the approaching time of year. The additional customer service load was frightening and, as research continued, it was discovered that it would get even worse in the first quarter of next year, partly due to year-end processing and partly because the first quarter of the year was when most A/R activity occurred.

The new application was pulled from production and the previous system reinstalled to avert disaster. The fortunate aspect of this fiasco was that business volumes were still relatively low, and the old system was well understood by customer service. So while there were problems aplenty, things could have been worse, and the greatest blessing of all was that the debacle spurred an effort not just to re-test, but to dig deep and explore all possibilities. The problems that had reared their ugly heads turned out to be a blueprint on how to proceed.

A team was formed that took a different approach. It was no longer an exercise in checking the basic process. It became a challenge to see where they could break the system. The brightest and best senior A/R agents took an active leadership role, finding that technical discussions on outcomes of unplanned A/R conditions and coincidences both invigorated them and increased knowledge of their jobs and the system being tested. Underlings were brought onboard, trained in great depth, and challenged to identify situations that stumped their superiors, who in turn enjoyed assigning the underlings to document how to handle these situations—something never done before and that greatly improved the entire customer service function. Questions that would take days to resolve now had answers in minutes, and customer satisfaction improved.

All this information was passed back to IT via a problem management system, and was tracked through to complete resolution. IT and A/R Quality Assurance worked more closely than they ever had, holding both scheduled tracking meetings and frequent ad hoc meetings where they discussed the nature of problems, solutions, and how the resolution should function and be used. IT became conversant in A/R, and A/R gained insight into how IT worked and how technical issues could affect A/R. It took a few months, and there were detours and obstacles, but when the application went production, it exceeded expectations.

Testing Improves the Process

There’s the best and worst of it: first an insufficiently tested application—especially by end-users—failed and then a comprehensive testing program—organized and implemented to look at and verify every nook and cranny—resulted in a truly valuable business asset that was production-ready and improved both business efficiency and productivity. The quality assurance unit was formalized, spanning a gap between A/R and IT, improving relationships and understanding between the two; sharing a common goal can be very unifying.

This is not to say everything became rosy and convivial with a Cinderella ending; there were still problems with the move to production. But things went better than most major moves to production; nothing major went awry, and the problems that did arise were manageable and resolvable. Stress still existed between the departments; differing objectives, perspectives, personalities and cultures do that, but the bond that had formed remained as well.

Effective testing is impossible without the help of end-users and a quality assurance function in a department or business. Quality assurance can improve both IT and user skills, and can produce a far better business solution when business units work together, at the same time minimizing mistakes and enhancing usability. QA is how solutions meet expectations.

More in the Series

This series is based on my more than 35 years of work in mainframe IT, reviewing mistakes I’ve seen or participated in and helping others to avoid them. Find previous stories in the series on my author bio page.

Jim Schesvold can be reached at

Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.



2019 Solutions Edition

A Comprehensive Online Buyer's Guide to Solutions, Services and Education.


CICS TS Performance and Tuning: A Rich Tradition


DB2 Utilities Suite Improves Efficiency and Performance


Establishing an Infrastructure for Tuning a Distributed Network

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App Google Play Store
Mainframe News Sign Up Today! Past News Letters