Quality Assurance

Formal quality assurance principles / fundamentals are based on a number of precepts that are not a good fit for the realities of software development today. Below are six precepts among the most prevalent-and erroneous-in the field today:

Assumption 1: Quality Requirements Dictate the Schedule

Fact For most software systems, market forces and competition dictate the schedule.

Traditional development model cannot keep up with the demand for consumer software products or the rapidly changing technology that supports them. Today’s rich development environment and ready consumer market has sparked the imagination of an enormous number of entrepreneurs. Consequently, this market is incredibly competitive and volatile. Product delivery schedules are often based on a first-to-market strategy. This strategy is well expressed in this 1997 quote from Roger Sherman, director of testing at Microsoft Corporation:

Schedule is often thought to be the enemy of quality, but at Microsoft it is considered to be part of the quality of the product.

Assumption 2: Quality = Reliability

This equation is interpreted as “zero defects is a requirement for a high-quality product.”

Fact: Reliability is only one component of the quality of a product.

The commercial software market is not willing to pay for a zero defect product or a 100% reliable product.

Users don’t care about faults that don’t ever become bugs, and users will forgive most bugs if they can work around them, especially if the features are great and if the price is right. For example, in many business network environments in 1994 and 1995, users religiously saved their work before trying to print it. The reason: About one in four print jobs submitted to a certain type of printer using a particular software printer driver would lock up the user’s workstation and result in the loss of any unsaved work. Even though many thousands of users were affected, the bug was tolerated for many months because the effects could be limited to simply rebooting the user’s workstation occasionally.

Safety critical and mission-critical applications are the notable exceptions to this fact. Consumers are willing to pay for reliability when the consequences of a failure are potentially lethal. However, the makers of these critical software systems are faced with the same market pressures from competition and constantly changing technology as the consumer software makers.

Assumption 3: Users Know What They Want

Fact: User expectations are vague and general, not detailed and feature-specific. This situation is especially true for business software products. This phenomenon has led to something that we call feature bloat

For example, if you asked several banking customers if they would like to be able to pay their bills online, many would say yes. But that response does not help the designer determine what type of bills customers will want to pay or how much they will use any particular type of payment feature. Consequently, in a well-funded development project, it is common to see every conceivable feature being implemented.

Assumption 4: The Requirements Will Be Correct

This fallacy assumes that designers can produce what the users want the first time, without actually building product or going through trial-and-error cycles.

Fact: Designers are commonly asked to design products using technology that is brand new and poorly understood. They are routinely asked to guess how the users will use the product, and they design the logic flow and interface based on those guesses.

Designers are people, and people evolve good designs through trial-and-error experimentation. Good requirements also evolve during development through trial-and-error experimentation. They are not written whole at the outset. A development process that does not allow sufficient time for design, test, and fix cycles will fail to produce the right product.

Assumption 5: Users Will Accept a Boring Product if the Features and Reliability Are Good.

Fact: To make an excellent product, we must consistently meet or exceed user expectations. For example, text-based Web browsers in cell phones have failed to captivate the consumer, even though they provide fast and efficient use of the slow data transmission rates inherent in the current cellular networks.

Software must be innovative in order to compete. The software leads the users to new accomplishments. Some examples of competitive innovations in the home consumer market include digital video editing, 3D animation, imaging, and video conferencing.

As corollaries of the preceding facts, the software must provide a competitive advantage to the business user and it must educate users.

Assumption 6: Product Maturity Is Required

Fact: Product maturity has little to do with the consumer’s buying decision. Price and availability are far more important considerations in most business scenarios.

The very mature premier high-end digital video creation software system has been supplanted by two new software editing systems that provide about 10 percent of the features it does, at 10 percent of the price. In addition, the new systems can be purchased and downloaded over the Internet, whereas the premier system cannot. We are also seeing this trend in large system software. The typical scenario involves dropping a massive, entrenched, expensive client/server system and replacing it with a lightweight, Web-based, database-driven application.

When analysis is performed on the current system, a frequent discovery is that the customers are paying for lots of features they are not using. Once the correct feature set has been determined, it can often be implemented quickly in a new Web-based application-where it can run very inexpensively.

Feature maturity is a far more important consideration than product maturity. As I have already pointed out, most consumers have realized that the latest release of software is not necessarily more reliable than the previous release. So product maturity is most often a myth. A mature product or system is typically overburdened by feature bloat.

For most software development organizations, ensuring quality means dealing with defects. Three generic ways to deal with defects include:

1) Defect Prevention
2) Defect Detection and removal
3) Defect Containment.

Defect prevention through error blocking or error source removal: These QA activities prevent certain types of faults from being injected into the software. Since errors are the missing or incorrect human actions that lead to the injection of faults into software systems, we can directly correct or block these actions, or remove the underlying causes for them. Therefore, defect prevention can be done in two generic ways:

- Eliminating certain error sources, such as eliminating ambiguities or correcting human misconceptions, which are the root causes for the errors.

- Faultprevention or blocking by directly correcting or blocking these missing or incorrect human actions. This group of techniques breaks the causal relation between error sources and faults through the use of certain tools and technologies, enforcement of certain process and product standards, etc.

Defect reduction through fault detection and removal: These QA alternatives detect and remove certain faults once they have been injected into the software systems. In fact, most traditional QA activities fall into this category. For example,

- Inspection directly detects and removes faults from the software code, design,
- Testing removes faults based on related failure observations during program execution etc.

Various other means, based on either static analyses or observations of dynamic executions, can be applied to reduce the number of faults in a software system.

We use defect detection and removal for the overall concept and activities related to what many people commonly call “debugging”.

When specific activities related to “debugging” are involved, we point the specifics out using more precisely defined terms, including,

- Specific activities related to defect discovery, including testing, inspection, etc.
- Specific follow-up activities after defect discovery, including defect diagnosis, analysis, fixing, and re-verification.

Defect containment through failure prevention and containment: These containment measures focus on the failures by either containing them to local areas so that there are no global failures observable to users, or limiting the damage caused by software system failures. Therefore, defect containment can be done in two generic ways:

- Some QA alternatives, such as the use of fault-tolerance techniques, break the causal relation between faults and failures so that local faults will not cause global failures, thus “tolerating” these local faults.

- A related extension to fault-tolerance is containment measures to avoid catastrophic consequences, such as death, personal injury, and severe property or environmental damages, in case of failures. For example, failure containment for real-time control software used in nuclear reactors may include concrete walls to encircle and contain radioactive material in case of reactor melt-down due to software failures, in order to prevent damage to environment and people’s health.

Various QA alternatives and related techniques can be used in a concerted effort to effectively and efficiently deal with defects and assure software quality. In the process of dealing with defects, various direct defect measurements and other indirect quality measurements (used as quality indicators) might be taken, often forming a multi-dimensional measurement space referred to as quality profile. These measurement results need to be analyzed using various models to provide quality assessment and feedback to the overall software development process.

By extension, quality engineering can also be viewed as defect management. In addition to the execution of the planned QA activities, quality engineering also includes:

- quality planning before specific QA activities are carried out,
- measurement, analysis, and feedback to monitor and control the QA activities

In this respect, much of quality planning can be viewed as estimation and planning for anticipated defects. Much of the feedback is provided in terms of various defect related quality assessments and predictions.

ISO-9126 offers a comprehensive framework to describe many attributes and properties we associate with quality. There is a strict hierarchy, where no sub-characteristics are shared among quality characteristics. However, certain product properties are linked to multiple quality characteristics or sub-characteristics. For example, various forms of redundancy affect both efficiency and maintainability.

Consequently, various alternative quality frameworks have been proposed to allow for more flexible relations among the different quality attributes or factors, and to facilitate a smooth transition from specific quality concerns to specific product properties and metrics.

ISO-9126 (ISO, 2001) provides a hierarchical framework for quality definition, organized into quality characteristics and sub-characteristics. There are six top-level quality characteristics, with each associated with its own exclusive (non-overlapping) sub-characteristics, as summarized below:

Functionality: A set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs. The sub-characteristics include:

- Suitability
- Accuracy
- Interoperability
- Security

Reliability: A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time. The sub-characteristics include:

- Maturity
- Fault tolerance
- Recoverability

Usability: A set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users. The sub characteristics include:

- Understandability
- Learnability
- Operability

Efficiency: A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used, under stated conditions. The sub-characteristics include:

- Time behavior
- Resource behavior

Maintainability: A set of attributes that bear on the effort needed to make specified modifications. The sub-characteristics include:

- Analyzability
- Changeability
- Stability
- Testability

Portability: A set of attributes that bear on the ability of software to be transferred from one environment to another. The sub-characteristics include:

- Adaptability
- Installability
- Conformance
- Replaceability

Disaster is any sudden or unplanned calamitous event that causes a significant disruption in information systems and /or telecommunications systems that affects the operation of an organization.

Disaster Recovery / Business Contingency These are commonly used terms to refer to the recovery of service following either a disaster or other actions which would disrupt business activity.

Recovery Plan is a documents used to define actions to be taken in the event of a disaster and reduce the number of decisions required during a stressful situation.

Business resumption is the process of restoring business activity to an acceptable level, and then to a normal level after an emergency event has disrupted normal operations.

Purpose and Scope: Many organizations now have installed very complicated on-line and diverse network systems. Organizations may have similar equipment and operating systems, they generally do not have the capacity to add a large number of users from another on-line environment to their systems even if the technical problems could be solved.

A trend is evolving to provide alternate sites near the central local systems where any additional equipment needed can be shipped in rapidly and critical on-line operations for the organization can be resumed in a reasonable time.

Redundancy in the communications network and a tie-in to the alternate site, or the ability to rapidly tie-in, is very important part of the disaster plan. Such type of site is called a cold backup site, as opposed to a hot backup site which contains all equipment necessary to start immediate operations.

DISASTER PLANNING PROCESS: In case of a disaster, a business should have a back-up for the following:

- Data file storage and retrieval
- Customer Services
- Communications and User Operations
- Hardware
- Software
- User operations
- Facilities for MIS and for users

A good and effective disaster recovery plan clearly identifies even the obvious details of how you will respond to disaster to ensure some of those details do not escape attention.

The aim of planning process should be:

- Assess existing vulnerabilities

- Implement disaster avoidance and prevention procedures

- Develop a comprehensive plan that will enable the organization to react appropriately and in a timely manner if disaster strikes

- Create an an awareness program to educate management and senior individuals who will be required to participate in the project

Preparing for a Disaster: This section contains the minimum steps necessary to prepare for a possible disaster and as preparation for implementing the recovery procedures. An important part of these procedures is ensuring that the off-site storage facility contains adequate and timely computer backup tapes and documentation for applications systems, operating systems, support packages, and operating procedures.

General Procedures: Responsibilities have been given for ensuring each of following actions have been taken and that any updating needed is continued.

- Maintaining and updating the disaster recovery plan.
- Ensuring that periodic scheduled rotation of backup media is being followed for the off-site storage facilities.
- Maintaining and periodically updating disaster recovery materials, specifically documentation and systems information, stored in the off-site areas.

Software Safeguards: Administrative software and data are secured by full backups each week and differential backups each weekday evening. The first backup of each month is retained for one year. Nightly differential backups are retained in Systems & Operations until the next full backup. A copy of the full backups is also stored in a safe deposit box. Backups are stored on 4mm DAT tapes and other compact media.

Recovery Procedures

Systems & Operations: In case of either a move to an alternate site, or a plan to continue operations at the main site, the following general steps must be taken:

- Determine the extent of the damage and if additional equipment and supplies are needed.

- Obtain approval for expenditure of funds to bring in any needed equipment and supplies.

- Notify local vendor marketing and/or service representatives if there is a need of immediate delivery of components to bring the computer systems to an operational level even in a degraded mode.

- If it is judged advisable, check with third-party vendors to see if a faster delivery schedule can be obtained.

- Notify vendor hardware support personnel that a priority should be placed on assistance to add and/or replace any additional components.

- Notify vendor systems support personnel that help is needed immediately to begin procedures to restore systems software

- Order any additional electrical cables needed from suppliers.

- Rush order any supplies, forms, or media that may be needed.

- In addition to the general steps listed at the beginning of this section, the following additional major tasks must be followed in use of the alternate site:

1. Notify officials that an alternate site will be needed for an alternate facility.

2. Coordinate moving of equipment and support personnel into the alternate site with appropriate personnel.

3. Bring the recovery materials from the off-site storage to the alternate site.

4. As soon as the hardware is up to specifications to run the operating system, load software and run necessary tests.

5. Determine the priorities of the client software that need to be available and load these packages in order. These priorities often are a factor of the time of the month and semester when the disaster occurs.

6. Prepare backup materials and return these to the off-site storage area.

7. Set up operations in the alternate site.

8. Coordinate client activities to ensure the most critical jobs are being supported as needed.

9. As production begins, ensure that periodic backup procedures are being followed and materials are being placed in off-site storage periodically.

10. Work out plans to ensure all critical support will be phased in.

11. Keep administration and clients informed of the status, progress, and problems.

12. Coordinate the longer range plans with the administration, the alternate site officials, and staff for time of continuing support and ultimately restoring the Systems & Operations section.

Managing Recovery Procedures: A disaster recovery plan must do more than identify alternatives. It should identify how all members of a business organization should respond in the event of a disaster. This includes the following:

- Develop a clear chain of command. It should be clear whom employees need to go to if they see a problem.

- Establish a clear sense of individual responsibility. All employees should be trained to look for signs of disaster and to respond immediately when they see these.

Documenting Procedures and Training Employees: Documentation of the disaster recovery plan needs to be readily available to all employees. It should be:

- written out and frequently revised as the business and threats change;
- available on the business network;
- referred to frequently in the training and ongoing education of all employees.

Training should help build employee awareness of what disasters might look like as they begin. Also, employees need to be aware of how, in addition to notifying someone else, they might immediately respond. Training should include:

- practice exercises;
- clearly written documents but frequent instruction in identifying who is to be notified and how to respond to a disaster.

Testing the Disaster Recovery Plan: A disaster recovery plan is only as good as its ability to be put into action. You may want to consider the following kinds of tests of your plan to maintain its effectiveness:

- An initial comprehensive test of responses to several kinds of disasters
- Tests that simulate specific disasters, affecting one part of the network
- Tests that simulate worst-case scenarios, when the entire network is affected

Common Review Roles

Author / Producer: individual responsible for the work product, and for correcting any defects.

Moderator / Leader / Facilitator: ensures that the review process is followed, and that the other reviewers perform their responsibilities throughout the review process.

Recorder / Scribe: records and classifies all the defects/ issues at the meeting, and assists the moderator / leader/ facilitator in preparing any reports / minutes.

Reviewer / Inspector: identifies defects/ issues in the work product.

Inspection Participants: The number of participants in an inspection range from a minimum of three to a maximum of six. The minimum number of participants is derived from the required roles to be assigned. Having more than six participants does not increase the number of defects uncovered, but does increase the cost of effectiveness. Plainly stated, having more than six participants increases the cost per defect discovered to unacceptable levels.

The participants in an inspection are all assigned at least one role. The roles in an inspection are:

- The Moderator
- The Recorder
- The Author or Producer
- The Reader
- Inspectors

Out of these roles, moderator, recorder and the author or producer roles are required roles.

Each participant in an inspection is an inspector. Participants can fill multiple roles with two exceptions. The author can be neither the moderator nor the recorder. Likewise the moderator should not serve as recorder. The author must be free to listen to the comments as they are given and provide clarification if required. The recorder is extremely busy documenting the potential problems and recording the other required data.

Moderator: The moderator is responsible for ensuring the item to be inspected has met the entry criteria for inspection readiness. If these criteria are met the moderator plans the inspection. This involves selecting the other participants, arranging for meeting rooms, ensuring the inspection data package is prepared and distributed, and ensuring sufficient time for participant preparation. During the preparation phase, the moderator acts as mentor and coach to the participants ensuring that they are prepared. The moderator facilitates the Inspection Meeting, maintaining the inspection focus. The moderator follows up with the author to ensure all potential problems have been properly handled. When these items have been correctly disposed, the moderator ensures all of the required data has been recorded and signs off that the inspection is complete.

Recorder: The recorder is responsible for logging the potential defects during the Inspection Meeting. This should be done as rapidly as possible, ensuring the essence of each comment is recorded without logging them verbatim. The recorder also writes down the other data collected during the meeting. It is important the other participants remember the recorder is not a secretary, but an active participant. The recorder, along with the other participants, is responsible for eliminating duplicate comments.

Author or Producer: The author or producer submits their portion of a work product for inspection. Additionally, they provide the rest of the reference material that makes up the inspection package distributed to each inspector. The author assists the moderator in planning the inspection and also acts as an inspector during the preparation phase. During the Inspection Meeting the author answers questions about the work product and provides clarification as needed. The author is responsible for determining the validity and disposition of all issues recorded during the meeting. After obtaining the moderator’s concurrence, the author modifies the work product.

Reader: The reader is the participant responsible for leading the inspection team through the work product during the Inspection Meeting. The moderator may be the reader to help control the pace of the meeting. Alternatively, the recorder may be the reader to ensure all comments get logged quickly. The reader does not actually read the work product, but usually uses paraphrasing or some other technique to get through the work product without confusion. The reader is also responsible for determining the most logical way to present the work product during the meeting.

Each participant, (as an inspector) is responsible for objectively examining the work product for potential defects. To aid the inspectors, checklists are provided to focus their efforts. The items on the checklists are derived from data from previous inspections on this type of work product. To increase the inspector’s efficiency, inspectors may be assigned responsibility for certain portions of the work product or assigned an emphasis role. In the former, the inspector is responsible for finding all defects in the portion of the work product assigned. In the latter, the inspector is responsible for finding specific types of problems in the entire work product. In either case, an inspector may record any potential defect discovered even if it is outside their area of responsibility.

All participants should be trained in the inspection process. Additionally, the moderator should receive training in facilitating meetings and leading inspections.

Facilitation skills are critical to the success of peer reviews.

The critical skills for a peer review meetings are building, clarifying, reacting, and process behaviors. A facilitator must learn to help everyone to work together effectively as a team, reach consensus, and “synergize”. A facilitator must learn to manage the different personality types and how to manage dysfunctional behavior right the first time.

The Inspection Process: There are six phases in the inspection process. They are

Inspection Planning
Inspection Overview – optional
Inspection Preparation
Inspection Meeting
Work Product Rework
Inspection Follow-up

A brief description of the activities in each phase is provided below. For the Inspection Planning phase, the moderator, in conjunction with the author, ensures the following activities occur:

- The work product is ready for inspection
- Inspectors are selected
- The inspection is scheduled and planned
- An inspection package is prepared and distributed
- The appropriate project quality assurance person is notified

The inspection package contains the material that the participants need to prepare for the inspection. The inspection package should include:

- Work product to be inspected
- Supporting documentation
- Checklists
- Inspection cover sheet to be completed

During planning, the inspectors’ preparation time is estimated. The recommended maximum inspection meeting time is two hours. An organization will determine their own preparation times based on the data collected during inspection meetings, but for an organization just starting there are guidelines. The time for participant preparation and the meeting time are about the same – for a two-hour meeting the preparation time is two hours per participant. Typical preparation rates for inspections are 4 – 8 pages per hour for documentation and 300 – 600 lines of code per hour. From these rates you can see that you don’t inspect a typical requirement specification in one meeting. The document must be “chunked” into logical pieces that can be completed within the planning guidelines.

The Inspection Overview is an optional phase. The moderator and author should determine the need for an Inspection Overview during the Inspection Planning phase. If the inspection under consideration is a normal review of a work product that is typically produced by the organization and the chosen inspectors are familiar with these types of work products and their background, an Inspection Overview is not required. However, if there are special circumstances or some unique aspect surrounding this inspection, the Inspection Overview is an opportunity for the moderator or author to provide the inspectors the information they need to ensure the success of the inspection. Examples of circumstances where an overview would be appropriate include a critical work product that will affect all downstream work products, unusual work product complexity, work product uses new or infrequently used technology, or the project is sufficiently small that inspectors must be drawn from outside the project. The Inspection Overview, if used, should not exceed the planned duration of the Inspection Meeting.

During the Inspection Preparation phase the inspectors individually get ready for the Inspection Meeting. Using the checklists and other reference material provided in the inspection package, they examine the work product ensuring that the item meets its requirements and adheres to the appropriate standards and conventions.

The checklists are crucial to the efficiency of inspections. The checklists allow the inspectors to look for defects typically found in the organization’s work products. These defects are determined historically from data collected during inspections and testing of similar work products. Checklists can be arranged by emphasis role. Emphasis roles allow the inspectors to focus on a specific aspect of the product under review and help further increase the efficiency of the inspection. Although any inspector can record any defect they find, they are only responsible for their assigned portion of the checklists. For example, errors made in requirement specifications are typically incorrect facts, omissions, inconsistencies, and ambiguities. Appropriate emphasis roles for a requirement document would be:

- Review for incorrect facts and ambiguities
- Review for omissions and inconsistencies (trace established)
- Review for testability

The moderator facilitates the Inspection Meeting. The reader leads the inspection team through the work product using an appropriate technique such as paragraph by paragraph, paraphrasing, or through scenarios. The recorder captures the defects identified during the inspection. The team, by consensus, classifies the captured defects by severity, type and cause. At the conclusion of the Inspection Meeting, the team determines the disposition of the work product by consensus.

It takes great skill on the part of the moderator to keep the meeting on track. It helps if the inspectors adhere to the following guidelines:

- Be prepared
- Review the product and not the author
- Raise issues, don’t resolve them
- Avoid discussions of style
- Avoid discussion about whether an issue raised is a problem

Because of the intensity of the inspection and because the participation of all inspectors is required for complete coverage of the work product, the Inspection Meeting should be canceled and rescheduled for any of the following reasons:

- Key inspector missing Lack of preparation by any inspector (preparation time recorded at start of meeting)Defect with large impact found (enough rework to dramatically change the work product)Ineffective inspection process (defect detection rate much less than expected).

During the Work Product Rework phase, the author resolves all potential defects. Resolution can take one of three forms. One, the defect in the work product is corrected. Two, the correction of the defect is deferred (i.e. a change request was initiated for correction at a later time). Three, a determination is made that the issue is not a defect, and the comment is retired. The effort expended during work product rework must be captured, reflecting time for fixing defects, disposing defects and rewriting material for re-inspection as part of the inspection data recorded on the inspection cover sheet. This data is used to gauge inspection effectiveness (cost per defect removed).

During Inspection Follow-up, the moderator examines the rework and the issue dispositions to ensure that all potential defects are disposed. Any material scheduled for re-inspection is re-inspected. Any remaining information on the inspection cover sheet is completed. The moderator signs and dates the inspection cover sheet to bring inspection to closure.

Review Metrics

• Size of work product
• Number of reviewers
• Total review effort
• Number of defects / issues
• Effort per size
• Effort per defect / issue
• Cost per defect / issue
• Defect / issue density
• Meeting rate
• Preparation rate

The main language of corporate management is money, so the concept of studying quality-related costs is essential. The quality guru, Joseph Juran has been advocating the analysis of quality-related costs since 1951. He give his theory of quality improvement which is commonly known as Juran Trilogy.

Quality costs: The costs that are associated with preventing, finding, and correcting defective work are Quality Costs. Normally, these costs are running at 20% – 30% of sales.

Many of these costs can be significantly reduced or completely avoided. One of the key functions of a Quality Analysis / Engineer is the reduction of the total cost of quality associated with a product / service.

Below are the main Quality Costs:

- Prevention Costs
- Appraisal Costs
- Failure Costs
- Internal Failure Costs
- External Failure Costs

Total Cost of Quality can be calculated as the sum of costs:

Prevention + Appraisal + Internal Failure + External Failure

Prevention Costs: Costs of activities that are specifically designed to prevent poor quality which include

- Coding errors
- Design errors
- Mistakes in the user manuals
- Dadly documented or unmaintainably complex code

Most of the prevention costs don’t fit within the Testing Group’s budget. This money is spent by the programming, design, and marketing staffs.

Appraisal Costs: Costs of activities designed to find quality problems, such as code inspections and any type of testing.

Design reviews are part of prevention and part appraisal. Please note the following two points:

1. To the degree that you’re looking for errors in the proposed design itself when you do the review, you’re doing an appraisal.

2. To the degree that you are looking for ways to strengthen the design, you are doing prevention.

Failure Costs: Costs that result from poor quality, such as the cost of fixing bugs and the cost of dealing with customer complaints.

Internal Failure Costs: Failure costs that arise before your company supplies its product to the customer. Along with costs of finding and fixing bugs are many internal failure costs borne by groups outside of Product Development. If a bug blocks someone in your company from doing his or her job, the costs of

- the wasted time
- the missed milestones
- and the overtime to get back onto schedule

are all internal failure costs.

The UI issues / bugs / defects – the ones that will be fixed later – can make it hard for these staff members to take accurate screen shots. Delays caused by these minor design flaws, or by bugs that block a packaging staff member from creating or printing special reports, can cause the company to miss its printer deadline.

External Failure Costs: External failure costs are much higher. The costs that arise after your company supplies the product to the customer such as

- Customer service costs
- Cost of patching a released product distributing the patch

It is much cheaper to fix problems before shipping the defective product to customers.

Examples of Prevention Costs:

- Fault-tolerant design
- Defensive programming
- Usability analysis
- Clear specification
- Staff training
- Requirements analysis
- Early prototyping
- Accurate internal documentation
- Evaluation of the reliability of development tools

Examples of Appraisal Costs:

- Training testers
- Beta testing
- Test automation
- Usability testing
- Design review
- Code inspection
- Glass box testing
- Black box testing

Examples of Internal Failure Costs:

- Wasted writer time
- Wasted marketer time
- Cost of late shipment
- Bug fixes
- Regression testing
- Wasted in-house user time
- Wasted tester time

Examples of External Failure:

- Lost sales
- Lost customer goodwill
- Discounts to resellers to encourage them to keep selling the product
- Warranty costs
- Liability costs
- Penalties
- Technical support calls
- Preparation of support answer books
- Investigation of customer complaints
- Refunds and recalls
- Coding / testing of interim bug fix releases
- Shipping of updated product
- All other costs imposed by law

There are three kinds of changes in the revision of CMMi Version 1.1 to CMMi Version 1.2:

- Wording changes in the practices, sub practices, and examples.
- Restructuring of some information into different chapters.
- There are about few major changes.

Major changes are:

1. CMMI for Development – name of model changed to CMMI for Development. Although name changed, but maintenance activities still remains the same. Maintenance also requires developing requirements, managing changes, planning and monitoring activities, ensuring quality etc.

2. Common features are eliminated.

3. More information is added relating to amplifications, GP (Generic Practices), examples, and hardware engineering.

4. Both the staged and continuous representations are now in one document.

5. Merge the supplier sourcing in the Supplier Agreement Management (SAM) process area. Also, Integrated Supplier Management (ISM) is removed and its concepts incorporated into SAM.

6. Advanced practices used in the continuous representation are eliminated. However, in some cases, base practices and advanced practices are combined into specific practices.

7. The numbering system for specific practices follows that of the staged representation, not that of the continuous representation.

8. Integrated Product and Process Development (IPPD) material is simplified and consolidated. IPPD is considered an “addition,” not reflected as separate process areas. Two Process areas are removed:

- Organizational Environment for Integration
- Integrated Teaming were removed

9. IPPD concepts like work environment material etc. are added to

- Organizational Process Definition (OPD)
- Integrated Project Management (IPM)

10. Guidelines for declaring a process area as “not applicable” is now clarified. Supplier Agreement Management is the only process area that can be considered “not applicable.”

11. The overview and glossary is now changed. New definitions are added to the glossary.