Quality Assurance

Many methods are used to perform the process and product assurance functions. Audits are used to examine the conformance of a development process to procedures and of products to standards. Embedded SQA activities, which have the purpose of detecting and removing errors, take a variety of forms, including inspection and testing. Assessment is another method of process assurance. Analysis techniques, such as causal analysis, reliability prediction and statistical process control, help ensure both process and product conformance.

Process Assurance: Schulmeyer defines software quality assurance as “. . . the set of systematic activities providing the evidence of the ability of the software process to produce a software product that is fit for use.”

SQA oversight provides management with unbiased feedback on process compliance so process lapses can be addressed in a timely fashion. It provides management with an early warning of risks to product quality and can provide recommendations to address the situation.

It is essential that the software quality assurance personnel have a reporting path which is independent of the management responsible for the activities audited and the associated daily conflicts generated by schedule and budget. Independent oversight, through various methods, encourages adherence to the official process. Locating an appropriate level of management where SQA will have frequent access, active support, and be above the conflicts of interest may be a difficult but necessary step.

The methods typically used to accomplish process assurance include SQA audit and reporting, assessment and statistical process control analysis.

Product Assurance: Assurance that the product performs as specified is the role of product assurance. This includes “in process,” or embedded, product assurance, as well as some methods that involve independent oversight.

The purpose of embedded SQA processes is product quality assurance. The activities are part of the development life cycle which will “build-in” the desired product quality. This focus allows identification and elimination of defects as early in the life cycle as possible, thus reducing maintenance and test costs. Embedded SQA methods include formal inspection, reviews, and testing.

Independent oversight functions can also be a part of product assurance. An independent test function, or testing which is witnessed by an independent entity such as SQA, is one method of providing product assurance. Other options include tests witnessed by customers, expert review of test results, or audits of the product.

Methods and supporting technologies
Many methods are used to perform the process and product assurance functions. Audits are used to examine the conformance of a development process to procedures and of products to standards. Embedded SQA activities, which have the purpose of detecting and removing errors, take a variety of forms, including inspection and testing. Assessment is another method of process assurance. Analysis techniques, such as causal analysis, reliability prediction and statistical process control, help ensure both process and product conformance.

1. Audit: Auditing is a method used in both process and product assurance. Audits are embedded into the software life cycle, as well as being performed as part of SQA.

An SQA audit is performed to “determine the adherence to established standards and procedures.” Evaluation of the sufficiency or effectiveness of the procedures or standards is occasionally part of an SQA audit. This type of audit examines records, as opposed to products, according to a sampling process to determine if procedures are being followed correctly. Such an audit is often performed by an external auditor who is not part of the software project.

In contrast, an embedded audit examines products to determine if the software products conform to standards and if project status is accurate. An independent auditor may perform this function or evaluate the records of such an audit that was performed by the development process. For documents, the audit is often performed manually. For code, it may be done manually or by an automated tool.

2. Embedded SQA Error Detection Methods:

- Formal Inspection
- Reviews
- Walkthroughs
- Testing

– Formal Inspection: Formal inspection is an examination of the completed product of a particular stage of the development process (such as design or code), typically employing checklists, expert inspectors, and a trained inspection moderator. The objective is to identify defects in the product. There are many techniques of doing inspections, but many follow the methods developed by Michael Fagan over 20 years ago.

Certain projects which have an effectively performing inspection process report better than 80% defect detection rates.

- Reviews: Reviews are also applied as an alternative to formal inspections as an SQA method. Informal design and code review methods are difficult to quantify since they are generally done at the discretion of the product author, do not follow a detailed process and are not reported at the project level. Informal review is a valuable alternative if the more effective formal inspection is not used.

The term “review” is also used to refer to project meetings (e.g., a product design review) which emphasize resolving issues and which have a primary objective of assessing the value of the product.

- Walkthroughs: Walkthroughs are meetings in which the author of the product acts as presenter to proceed through the material in a stepwise manner. The objective is often raising and/or resolving design or implementation issues. Walkthroughs tend to be informal and lacking in “close procedural control.”

- Testing: Testing is a dynamic analysis technique that has the primary objective of error detection. Testing of software is performed on individual components during intermediate stages of development, subsystems following integration, and entire software systems. It involves execution of the software and evaluation of its behavior in response to a set of input against documented, required behavior.

3. Assessment: Assessment is determining the capability of a process through comparison with a standard. The exact methods used are dependent on the standard applied. Two standard assessment methods which are frequently employed are ISO 9000 and SEI SW-CMM. Malcolm Baldrige is another assessment standard, but is not used as often by software projects.

The Software Engineering Institute (SEI) at Carnegie Mellon University was established by Congress in 1984 to improve the practice of software engineering. A key product developed by the SEI to aid in this mission is the Software Capability Maturity Model. The SW-CMM is a model for software process improvement. The model establishes criteria describing the characteristics of a mature software organization and has staged software process maturity levels. There are 5 levels of process maturity, with level 1 being the lowest and level 5 being the highest. Within the maturity levels are groupings of software engineering topics called Key Process Areas (KPAs).

The ISO 9001 international standard was established to address quality requirements across diverse industries. As such, the requirements within the standard are written in a generic manner to accommodate the diversity of applications. The corresponding ISO 9000-3 document gives guidance for applying the standard to software. Note that, as of this writing, the ISO 9001 standard is under revision.

Use of assessments may involve individuals outside of the organization such as a CMM lead assessor or an ISO registrar, but many times the assessment is conducted using internal resources to identify areas for improvement or in preparation for a formal assessment. Assessment uses a combination of random auditing and interviewing to answer a list of questions which is tailored to fit the organization being assessed.

4. Analysis:

- Causal Analysis and Defect Prevention Processes
- Reliability Prediction
- Statistical Process Control

- Causal Analysis and Defect Prevention Processes: The purpose of these activities is to address the process weaknesses that allowed product defects to be inserted in order to prevent reoccurrence of similar types of defects. One method to accomplish this objective includes root cause analysis and process brainstorming. First the team of individuals, which may include developers and other analysts, determines the root cause of the defect insertion. If the cause is systemic and/or may be repeated, brainstorming for a remedy is performed to decrease the likelihood of reoccurrence of similar defects under similar circumstances. Ideas for process improvement are generated from the brainstorming session and passed on to a process management team. These activities may be performed at various stages of the software life cycle, but it is recommended that the elapsed time between defect discovery and this type of analysis be minimized.

- Reliability Prediction: The IEEE Standard Glossary of Software Engineering Terminology definition of software reliability is: “The ability of the software to perform its required function under stated conditions for a stated period of time.” The ability to predict the reliability of a software system would enable project management to better perform product assurance and assess readiness for release. Three bases used in estimating reliability are failure record, behavior for a random sample of input points, or quantity of actual and “seeded” faults detected during testing. However, these methods are imperfect; software reliability prediction is still a science under development. Furthermore, this technique requires an extensive error history database.

- Statistical Process Control: Statistical process control is the use of statistical methods to assure both process and product quality. These methods include Pareto analysis, Shewhart control charts, histograms, and scatter diagrams. This technique can be used to evaluate if a process is out of statistical control, thus indicating process defects and/or potential for increased product defects.