Quality Assurance

Defect: From the producer’s viewpoint, a defect is a product requirement that has not been met, or a product attribute possessed by a product or a function performed by a product that is not in the statement of requirements that define the product. From the customer’s viewpoint, a defect is anything that causes customer dissatisfaction, whether in the statement of requirements or not.

Policy: Managerial desires and intents concerning either processes (intended objectives) or products (desired attributes).

Procedure: The step-by-step method followed to ensure that standards are met.

Process
1. The work effort that produces a product. This includes efforts of people and equipment guided by policies, standards, and procedures.

(2) A statement of purpose and an essential set of practices (activities) that address that purpose. A process or set of processes used by an organization or project to plan, manage, execute, monitor, control, and improve its software related activities.

Productivity: The ratio of the output of a process to the input, usually measured in the same units. It is frequently useful to compare the value added to a product by a process, to the value of the input resources required (using fair market values for both input and output).

Quality: Operationally, the word quality refers to products. A product is a quality product if it is defect free. To the producer, a product is a quality product if it meets or conforms to the statement of requirements that defines the product. This statement is usually shortened to: quality means meets requirements. To the customer, a product is a quality product if it meets the customer’s needs, regardless of whether the requirements were met. This is referred to as fit for use.

Quality – Producer View: The producer’s view of quality has these four characteristics: Doing the right thing, Doing it the right way, Doing it right the first time, and Doing it on time without exceeding cost.

Quality – Customer View: Meeting requirements is a producer’s view of quality. This is the view of the organization responsible for the project and processes, and the products and services acquired, developed, and maintained by those processes.

Standard: A requirement of a product or process. For example: 100 percent of the functionality must be tested.

Benchmarking: Benchmarking is the process of comparing and measuring an organization’s operations or its internal processes against those of a best-in-class performer from inside or outside its industry.

Process benchmarking is a systematic and continuous improvement process; a process of continually MEASURING and COMPARING an organization’s BUSINESS PROCESSES against business pro LEADERS anywhere in the world to gain INFORMATION which will help the organization take ACTION to IMPROVE its PERFORMANCE.

Key Points:

• Benchmarking is an increasingly popular improvement tool.
• Benchmarking concerns processes and practices.
• Benchmarking is a respected means of identifying processes that require major change.
• Benchmarking is done between consenting companies that may or may not be competitors.
• Benchmarking compares your process or practice with the target company’s best-in-class process or practice.
• The goal of benchmarking is to find “secrets of success” and then adapt and improve for your own application.

Benchmarking as it Relates to Continuous Improvement:

• Today’s competitive world does not allow time for gradual improvement in areas in which a company lags way behind.
• Benchmarking can tell a firm where it stands relative to best-in-class practices and processes, and which processes must be changed.
• Benchmarking provides a best-in-class model to be adopted, or even improved upon.
• Modern customers are better informed and demand the highest quality and lowest prices. Companies have a choice to either perform or go out of business.
• Benchmarking supports total quality by providing the best means for rapid, significant process/practice improvement.

Benchmarking Approach and Process:

• Obtain management commitment.
• Baseline your own processes.
• Identify your strong and weak process and document them.
• Select processes to be benchmarked.
• Form benchmarking teams.
• Research best-in-class.
• Select candidate best-in-class benchmarking partners.
• Form agreements with benchmarking partners.
• Collect data.
• Analyze data and establish the gap
• Plan action to close the gap/surpass.
• Implement change.
• Monitor.
• Update benchmarks; continue the cycle

Benchmarking Versus Re-Engineering: Process re-engineering should only be considered when it is impossible to use benchmarking.

• No known process available for benchmarking (rare).
• Best-in-class not willing to partner.
• Best-in-class inaccessible due to geography or expense

Role of Management in Benchmarking:

• For benchmarking to be productive, management must be committed to change.
• Management must provide the necessary funding.
• Management must allocate the appropriate personal.
• Information to be disclosed to benchmarking partners can only be cleared by management.
• Top level managers must themselves be directly involved in benchmarking activities.

Process Documentation:

• All people associated with the process should have a common understanding of it, and that can come only from documentation.
• A documented starting point is needed against which to measure performance improvement after benchmarking changes have been implemented.
• The organization will be dealing with people (the partners) who are not familiar with its processes. With an understanding of where the benchmarking organization is, the partner will be better able to help.

Settling for “OK-in-Class”: Reasons organizations too often choose benchmarking partners who are not best-in-class:

• The best-in-class is not interested in partnering.
• Research identified the wrong partner.
• The benchmarking company got lazy and picked a handy partner.

Obstacles to Successful Benchmarking:

• If a company is too internally focused it may not be aware that their processes are less efficient than best-in-class.
• An overly broad benchmarking objective can guarantee failure.
• Unrealistic timetables.
• Poor team composition–the people who use the process day in and day out, must be involved.
• Improper emphasis–A frequent cause of failure in benchmarking is that teams get bogged down in collecting endless data and put too much emphasis on the numbers.
• Insensitivity to partners–If you fail to observe protocol and common courtesy in all transactions, your organization runs the risk of being cut off.

BENCHMARKING CASE STUDY: XEROX (MANUFACTURING)

• Xerox’s success is the first one in the history of benchmarking.
• From a critical situation in 1972, Xerox became what we call today a “top benchmarking partner”.
• In 1979, Xerox starts benchmarking.
• In 1989, Xerox wins the Malcolm Baldrige National Quality Award.
• Product performance during the first 30 days of installation has increased 40%
• Manufacturing lead times have been reduced 50%.
• Manufacturing labor and material over-head rates have been improved by 31% and 46% respectively.
• Customer retention rate is 20% better than U.S. industry average
• A company wide performance measurement covering 240 key areas of product, service and business performance.
• The targets of world leaders.
• Tremendous gains in quality (78% defect reduction, increased reliability with 40% decrease in unscheduled maintenance, increased copy quality, 27% decrease in service response time).
• Significant reductions in labor and material over-heads.
• First company to offer three year product warranty.

There are many different steps that organizations follow in benchmarking. However, most baselining processes have these four steps:

1. Develop a clearly defined baseline in your organization: This means that all of the attributes involved in your baseline are defined. In our example of defects per lines of code, clearly defining what is meant by defect and a line of code would meet the objective of this step.
2. Identify the organizations you desire to baseline against: Many factors come into this decision, such as do you want to benchmark within your industry, do you want to benchmark what you believe are leading organizations, do you want to benchmark an organization that uses the same tools that are used in your organization, and do you want to benchmark against organizations with a similar culture.
3. Compare baseline calculations: Compare how your baseline is calculated versus the baseline calculation in the company you want to benchmark against. Benchmarking is only effective when you benchmark against an organization who has calculated their baseline using approximately the same approach that your organization used to calculate the baseline.
4. Identify the cause of baseline variance in the organization you benchmarked against: When you find a variance between the baseline calculation in your company and the baseline calculation in the organization you are benchmarking against, you need to identify the cause of variance. For example, if your organization was producing 20 defects per thousand lines of code, and you benchmarked against an organization that only had 10 defects per thousand lines of code you would want to identify the cause of the difference. If you cannot identify the cause of difference, there is little value in benchmarking. Let us assume that the company you benchmarked against had a different process for requirement definition than your organization. For example, assume they use JAD (joint application development) and you did not. Learning this, you may choose to adopt JAD in your organization as a means for reducing your developmental defect rates.

A less formal method for benchmarking is to visit other organizations. This will provide the quality professionals with these benefits:

1. The cost and effort to develop new and innovative quality approaches within IT is cost-prohibitive for most companies. Learn from others and don’t “reinvent the wheel”.
2. Comparing quality programs to those in other companies can identify gaps in current processes and lead to obtaining more effective quality practices.
3. Interfacing periodically with other quality individuals is good for professional development. Those colleagues will not exist internally unless the company is large.

Also See:

• Quality Baseline Study
• Conducting Objective and Subjecting Quality Baseline studies

Conducting Objective Baseline Studies: Objective baseline studies are ones which are viewed as factual and non argumentative. There are very few objective studies that can be conducted within information services. Objective means performed by counting, and what can be counted must be considered non argumentative. For example, if we wanted to know how many lines of executable code were in a program, and let us assume we can define what is meant by a line of executable code, then we could count those lines and have an objective baseline.

We need to note that what may appear as objective, may really be more subjective. For example, if we ask people to keep time records, and then record hours worked based on those times, we must make the assumption that the count is accurate. In most instances, the hours count is subjective because many will record their hours worked at the end of each month, and thus the hours count is a subjective measure and not an objective measure.

Objective measures are those, which can be accomplished by counting. Examples of objective baseline measures that can be used for baselines include:

• Project completed on schedule
• Lines of code
• Number of programs
• Number of people assigned to a projectNumber of abnormal terminations

Again, the exactness of the counting will actually determine whether the above measures are objective or subjective. It is important to recognize that there are very few objective measures, and thus we are forced to use subjective measures in measuring quality and productivity.

Conducting Subjective Baseline Studies: Subjective baseline studies will be the most commonly conducted studies in measuring quality and productivity. Subjective means that judgment is applied in making the measure. We noted in the discussion on objective measures that when the individual involved in recording time has the option of applying judgment, then the measure becomes subjective.

Baselines should be quantitative even if it is a subjective measure, but quantitatively subjective. For example, because quality conformance and nonconformance must be defined by people, we are looking for ways to put this information into a quantifiable format. This does not convey a lot of information, but it is indicative of a problem. However, if we develop a five-point scale for unresponsiveness, and ask your dissatisfied customer to complete that scale, we now have conveyed a lot more information. If our scale rates “1” as very poor service, and “5” as very good service, there is a great deal of difference between a “1” rating and a “3” rating for dissatisfaction.

Examples of products/services that can be measured subjectively for developing a baseline include:

• Customer satisfaction
• Effectiveness of standards/manuals
• Helpfulness of methodologies to solve problems
• Areas/activities causing the greatest impediments to quality/productivity
• Causes for missed schedules/over-budget conditions
• Understandability of training materials
• Value of tools
• Importance of activities/standards/methods/tools to individual activity

Baselines can be conducted for any one of the following three purposes:

• Planning: To determine where detailed investigation/survey should be undertaken.
• Internal analysis: To identify problems/areas for quality improvement. Once the problem/area has been identified, then no additional effort need be undertaken to formalize the results.
• Benchmarking: Comparison against external organizations.

The following are typical steps needed to perform a quality baseline study:

1. Identify products/services to be surveyed: This is the requirements phase of the baseline study. Studies should be directed at specific products or services. For example, computer programs as a product, and customer/user interaction as a service.
2. Define conformance and nonconformance: The individuals developing the survey instrument must have defined (at least on a preliminary basis) the expected conformance and nonconformance. Note that in many instances the survey will be used to help establish nonconformance, but the general areas of nonconformance will need to be identified in order to gather nonconformance data.
3. Identify survey population: The population of data/people to be surveyed needs to be identified. This is a critical step because the definition of nonconformance will vary significantly depending upon who is considered the population. For example, programming defects would look significantly different to the programmer than the end user of the program results. The programmer may only consider variance from specs a defect, but to the end user not meeting needs is a defect.
4. Identify size of population to be surveyed: This step is one that involves economics. It is always cheaper to look at fewer than more. The question that needs to be decided is how few can be examined to give valid results. Statistically, we rarely try to go below a sample size of twenty, but in surveying people we may be able to drop below this limit and still get valid results.
5. Develop survey instrument: A specific survey instrument must be developed to meet the survey objectives. Surveys need to be customized to the specific needs and vocabulary of the population being surveyed.
6. Conduct survey: The survey instruments should be completed by the surveyed population. Form a quality perspective; it is helpful to train the population on how to complete the survey questionnaire. This can be done through written instructions, but it is normally better to do it verbally. If the population group is small, they can be called together for a meeting, or have the survey instruments hand delivered and explained. Generally, the validity of the results will increase when extra time is spent to explain the intent and purpose of the survey.
7. Follow up on incomplete surveys: The survey should have a time limit for response. Normally this should not exceed one week. If the surveys have not been returned by that time, then the surveying group should follow up and attempt to get as many surveys completed as possible. Note that it is generally not realistic to expect every survey to be returned.
8. Accumulate and present survey results: The survey information should be accumulated, consolidated, and put into presentation format. Suggestions on how to do this follow in a later part of this quality initiative.
9. Take action and notify participants of that action: All surveys should result in some specific action. Even if that action is to do nothing, a decision should be made based on the survey results. That decision should be given to the survey participants. Note that whenever a survey is conducted, the participants expect some action. Not to inform the participants of the action will reduce cooperation in future surveys.

Edwards Deming outlined 14 quality principles, which must be used concurrently in order to achieve quality. While these principles were applied to industry, influencing government, schools, and hospitals, many are also applicable to achieving software quality from an information technology perspective. The following is a brief discuss of each point, followed by a description of how a quality assurance organization might apply each:

Point 1 – Create Constancy of Purpose: Most companies tend to dwell on their immediate problems without adequate attention to the future. According to Deming, “It is easy to stay bound up in the tangled knots of the problems of today, becoming ever more and more efficient in the future, but no company without a plan for the future will stay in business.” A constancy of purpose requires: innovation, e.g., long-term planning for it, investment in research and education, and continuous improvement of products and service. To apply this point, an information technology quality assurance organization can:

1. Develop a quality assurance plan that provides a long-range quality direction

2. Require software testers to develop and maintain comprehensive test plans for each project

3. Encourage quality analysts and testers to come up with new and innovative ideas to maximize quality

4. Strive to continuously improve quality processes

Point 2 – Adopt the New Philosophy: Quality must become the new religion. According to Deming, “The cost of living depends inversely on the goods and services that a given amount of money will buy, e.g., reliable service reduces costs. Delays and mistakes raise costs.” Consumers of goods and services end up paying for delays and mistakes, which reduces their standard of living. Tolerance of acceptability levels and defects in systems is the roadblock between quality and productivity. To apply this point, an information technology quality assurance organization can:

1. Educate the information technology organization on the need and value of quality

2. Promote the quality assurance department to the same level as any other department

3. Defuse the notion that quality assurance is negative and the “watch dogs”

4. Develop a risk management plan and not accept any anomalies outside the range of acceptable risk tolerance

Point 3 – Cease Dependence on Mass Inspection: The old way of thinking is to inspect bad quality out. A better approach is that inspection should be used to see how we are doing, and not be left to the final product, when it is difficult to determine where in the process a defect took place. Quality should be built in without the dependence on mass inspections. To apply this point, an information technology quality assurance organization can:

1. Promote and interject technical reviews, walkthroughs, and inspections as nondefensive techniques for achieving quality throughout the entire development cycle

2. Instill the need for the whole organization to be quality conscious and treat it as a tangible, measurable work product deliverable

3. Require statistical evidence of information technology quality

Point 4 – End the Practice of Awarding Business on Price Tag Alone: “Two or more suppliers for the same item will multiply the evils that are necessarily inherent and bad enough with any one supplier.” A buyer will serve his company best by developing a long-term relationship of loyalty and trust with a single vendor. Rather than using standards manuals by which vendors must qualify for business, a better approach is active involvement by the supplier’s management with Deming’s Fourteen Points. To apply this point, an information technology quality assurance organization can:

1. Require software quality and test vendors to provide statistical evidence of their quality

2. Pick the best vendor for each quality assurance tool, testing tool, or service, and develop a working relationship consistent with the quality plan

Point 5 – Improve Constantly and Forever the System of Production and Service: Improvement is not a one-time effort — management is obligated to improve quality continuously. “Putting out fires is not improvement. Finding a point out of control, finding the special cause and removing it, is only putting the process back to where it was in the first place. The obligation for improvement is a ceaseless process.” To apply this point, an information technology Quality Assurance organization can:

1. Constantly improve quality assurance and testing processes

2. Don’t rely on judgment

3. Use statistical techniques such as root cause and effect analysis to uncover the sources of problems and test analysis

Point 6 – Institute Training and Retraining: Often there is little or no training and workers do not know when they have done their jobs correctly. It is very difficult to erase improper training. Deming stresses that training should not end as long as performance is not in statistical control and there is something to be gained. To apply this point, an information technology quality assurance organization can:

1. Institute modern training aids and practices

2. Encourage quality staff to constantly increase their knowledge of quality and testing techniques by attending seminars and classes

3. Reward staff for creating new seminars and special interest groups

4. Use statistical techniques to determine when training is needed and completed

Point 7 – Institute Leadership: “There is no excuse to offer for putting people on a job that they know not how to do. Most so-called ‘goofing off’ — somebody seems to be lazy, doesn’t seem to care — that person is almost always in the wrong job, or has very poor management.” It is the responsibility of management to discover the inhibitors that prevent workers from taking pride in their jobs. From an information technology point of view, development often views the job of quality to be the QA department’s responsibility. QA should be very aggressive as quality leaders and point out that quality is everyone’s responsibility. To apply this point, an information technology quality assurance organization can:

1. For example, if a developer has an excessive number of defects discovered by QA testing, take the time to train him or her on how to unit test his or her code effectively.

2. Improve supervision, which is the responsibility of management

3. Allow the project leader to have more time to help people on the job

4. Use statistical methods to indicate where there are faults

Point 8 – Drive Out Fear: There is often no incentive for problem solving. Suggesting new ideas is too risky. People are afraid of losing their raises, promotions, or jobs. “Fear takes a horrible toll. Fear is all around, robbing people of their pride, hurting them, robbing them of a chance to contribute to the company. It is unbelievable what happens when you unloose fear.” A common problem is the fear of inspections. To apply this point, an information technology quality assurance organization can:

1. Promote the idea that quality is goodness and should be rewarded, and promote any new ideas to improve quality

2. Prior to a structured walkthrough, inspection, or JAD session, QA should make sure everyone understands the ground rules and promote an “egoless” environment

3. Periodically schedule a “Quality Day” in which quality improvement ideas are openly shared

Point 9 – Break Down Barriers Between Staff Areas: There are numerous problems when departments have different goals and do not work as a team to solve problems, set policies, or define new directions. “People can work superbly in their respective departments, but if their goals are in conflict, they can ruin the company. It is better to have teamwork, working for the company.” To apply this point, an information technology quality assurance organization can:

1. Quality assurance and other departments (particularly development) need to work closely together; QA should be viewed as the “good guys” trying to make the software products the best in the world

2. Quality assurance should point out that a defect discovered before production is one that won’t be discovered by the users

Point 10 – Eliminate Slogans, Exhortations, and Targets for the Workforce: “Slogans never helped anybody do a good job. They generate frustration and resentment.” Slogans such as “Zero Defects” or “Do it Right the First Time” are fine on the surface. The problem is that they are viewed as signals that management does not understand employees’ problems, or care. There is a common practice of setting goals without describing how they are going to be accomplished. To apply this point, an information technology quality assurance organization can:

1. Encourage management to avoid the use of slogans

2. Rather than generate slogans, the QA organization should develop and document quality standards, procedures, and processes which the rest of the organization can use to help them to maximize quality

Point 11 – Eliminate Numerical Goals: “Quotas or other work standards, such as measured day work or rates, impede quality perhaps more than any other single working condition. As work standards are generally used, they guarantee inefficiency and high costs.” A proper work standard would define what is and is not acceptable in terms of quality. To apply this point, an information technology quality assurance organization can:

1. Look not just at the numbers but look carefully at the quality standards

2. Avoid formally publicizing defects rates by individual or department

3. Work with the development organization to define quality standards and procedures to improve quality

4. When there are specific quality issues, the QA and Development manager need to address them informally

Point 12 – Remove Barriers to Pride of Workmanship: People are regarded as a commodity, to be used as needed. If not needed, they are returned to the market. Managers cope with many problems but tend to shy away from people problems. They often form “Quality Control Circles,” but this is often a way for a manager to pretend to be doing something about a problem. Management seldom invests employees with any authority, nor does it act upon their recommendations. To apply this point, an information technology quality assurance organization can:

1. Instill an image that quality is their deliverable and is a very valuable commodity

2. Delegate responsibility to the staff to seek out quality and do whatever it takes to accomplish it

Point 13 – Institute a Vigorous Program of Education and Retraining: People must acquire new knowledge and skills. Education and retraining is an investment in people, which is required for long-term planning. Education and training must fit people into new jobs and responsibilities. To apply this point, an information technology quality assurance organization can:

1. Encourage quality staff to constantly increase their knowledge of quality and testing techniques by attending seminars and classes

2. Reward staff for creating new seminars and special interest groups

3. Retrain individuals in new quality skills

Point 14 – Take Action to Accomplish the Transformation: Top management needs to push these thirteen points. Every employee, including managers, should acquire a precise idea of how to improve quality continually, but the initiative must come from top management. With this point Dr. Edwards Deming gave the Plan, Do, Check, Act Process that is commonly known as Deming Cycle. I’ll explain PDCA in my next post.

In Webster’s Dictionary, quality is defined as “the essential character of something, an inherent or distinguishing character, degree or grade of excellence.” If you look at the computer literature, you will see that there are two generally accepted meanings of quality. The first is that quality means “meeting requirements.” With this definition, to have a quality product, the requirements must be measurable, and the product’s requirements will either be met or not met. With this meaning, quality is a binary state, i.e., it is a quality product or it is not. The requirements may be very complete or they may be simple, but as long as they are measurable, it can be determined whether quality has or has not been met. This is the producer’s view of quality as meeting the producer’s requirements or specifications. Meeting the specifications becomes an end in itself.

Another definition of quality, the customer’s, is the one we will use. With this definition, the customer defines quality as to whether or not the product or service does what the customer needs. Another way of wording it is “fit for use.” There should also be a description of the purpose of the product, typically documented in a customer’s “requirements specification”. The requirements are the most important document, and the quality system revolves around it. In addition, quality attributes are described in the customer’s requirements specification. Examples include usability, the relative ease with which a user communicates with the application, portability, the capability of the system to be executed across a diverse range of hardware architectures, and reusability, the ability to transfer software components constructed in one software system into another.

While everyone is committed to quality, the following are some confusions shared by many individuals, which inhibit achieving a quality commitment:

• Quality requires a commitment, particularly from top management. Close cooperation of management and staff is required in order to make it happen.
• Many individuals believe that defect-free products and services are impossible, and accept certain levels of defects are normal and acceptable.
• Quality is frequently associated with cost, meaning that high quality equals high cost. This is a confusion between quality of design and quality of conformance.
• Quality demands requirement specifications in enough detail that the products produced can be quantitatively measured against those specifications. Many organizations are not capable or willing to expend the effort to produce specifications at the level of detail required.
• Technical personnel often believe that standards stifle their creativity, and thus do not abide by standards compliance. However, for quality to happen, well-defined standards and procedures must be followed.