Quality Assurance

Conducting Objective Baseline Studies: Objective baseline studies are ones which are viewed as factual and non argumentative. There are very few objective studies that can be conducted within information services. Objective means performed by counting, and what can be counted must be considered non argumentative. For example, if we wanted to know how many lines of executable code were in a program, and let us assume we can define what is meant by a line of executable code, then we could count those lines and have an objective baseline.

We need to note that what may appear as objective, may really be more subjective. For example, if we ask people to keep time records, and then record hours worked based on those times, we must make the assumption that the count is accurate. In most instances, the hours count is subjective because many will record their hours worked at the end of each month, and thus the hours count is a subjective measure and not an objective measure.

Objective measures are those, which can be accomplished by counting. Examples of objective baseline measures that can be used for baselines include:

• Project completed on schedule
• Lines of code
• Number of programs
• Number of people assigned to a projectNumber of abnormal terminations

Again, the exactness of the counting will actually determine whether the above measures are objective or subjective. It is important to recognize that there are very few objective measures, and thus we are forced to use subjective measures in measuring quality and productivity.

Conducting Subjective Baseline Studies: Subjective baseline studies will be the most commonly conducted studies in measuring quality and productivity. Subjective means that judgment is applied in making the measure. We noted in the discussion on objective measures that when the individual involved in recording time has the option of applying judgment, then the measure becomes subjective.

Baselines should be quantitative even if it is a subjective measure, but quantitatively subjective. For example, because quality conformance and nonconformance must be defined by people, we are looking for ways to put this information into a quantifiable format. This does not convey a lot of information, but it is indicative of a problem. However, if we develop a five-point scale for unresponsiveness, and ask your dissatisfied customer to complete that scale, we now have conveyed a lot more information. If our scale rates “1” as very poor service, and “5” as very good service, there is a great deal of difference between a “1” rating and a “3” rating for dissatisfaction.

Examples of products/services that can be measured subjectively for developing a baseline include:

• Customer satisfaction
• Effectiveness of standards/manuals
• Helpfulness of methodologies to solve problems
• Areas/activities causing the greatest impediments to quality/productivity
• Causes for missed schedules/over-budget conditions
• Understandability of training materials
• Value of tools
• Importance of activities/standards/methods/tools to individual activity

Baselines can be conducted for any one of the following three purposes:

• Planning: To determine where detailed investigation/survey should be undertaken.
• Internal analysis: To identify problems/areas for quality improvement. Once the problem/area has been identified, then no additional effort need be undertaken to formalize the results.
• Benchmarking: Comparison against external organizations.

The following are typical steps needed to perform a quality baseline study:

1. Identify products/services to be surveyed: This is the requirements phase of the baseline study. Studies should be directed at specific products or services. For example, computer programs as a product, and customer/user interaction as a service.
2. Define conformance and nonconformance: The individuals developing the survey instrument must have defined (at least on a preliminary basis) the expected conformance and nonconformance. Note that in many instances the survey will be used to help establish nonconformance, but the general areas of nonconformance will need to be identified in order to gather nonconformance data.
3. Identify survey population: The population of data/people to be surveyed needs to be identified. This is a critical step because the definition of nonconformance will vary significantly depending upon who is considered the population. For example, programming defects would look significantly different to the programmer than the end user of the program results. The programmer may only consider variance from specs a defect, but to the end user not meeting needs is a defect.
4. Identify size of population to be surveyed: This step is one that involves economics. It is always cheaper to look at fewer than more. The question that needs to be decided is how few can be examined to give valid results. Statistically, we rarely try to go below a sample size of twenty, but in surveying people we may be able to drop below this limit and still get valid results.
5. Develop survey instrument: A specific survey instrument must be developed to meet the survey objectives. Surveys need to be customized to the specific needs and vocabulary of the population being surveyed.
6. Conduct survey: The survey instruments should be completed by the surveyed population. Form a quality perspective; it is helpful to train the population on how to complete the survey questionnaire. This can be done through written instructions, but it is normally better to do it verbally. If the population group is small, they can be called together for a meeting, or have the survey instruments hand delivered and explained. Generally, the validity of the results will increase when extra time is spent to explain the intent and purpose of the survey.
7. Follow up on incomplete surveys: The survey should have a time limit for response. Normally this should not exceed one week. If the surveys have not been returned by that time, then the surveying group should follow up and attempt to get as many surveys completed as possible. Note that it is generally not realistic to expect every survey to be returned.
8. Accumulate and present survey results: The survey information should be accumulated, consolidated, and put into presentation format. Suggestions on how to do this follow in a later part of this quality initiative.
9. Take action and notify participants of that action: All surveys should result in some specific action. Even if that action is to do nothing, a decision should be made based on the survey results. That decision should be given to the survey participants. Note that whenever a survey is conducted, the participants expect some action. Not to inform the participants of the action will reduce cooperation in future surveys.

Edwards Deming outlined 14 quality principles, which must be used concurrently in order to achieve quality. While these principles were applied to industry, influencing government, schools, and hospitals, many are also applicable to achieving software quality from an information technology perspective. The following is a brief discuss of each point, followed by a description of how a quality assurance organization might apply each:

Point 1 – Create Constancy of Purpose: Most companies tend to dwell on their immediate problems without adequate attention to the future. According to Deming, “It is easy to stay bound up in the tangled knots of the problems of today, becoming ever more and more efficient in the future, but no company without a plan for the future will stay in business.” A constancy of purpose requires: innovation, e.g., long-term planning for it, investment in research and education, and continuous improvement of products and service. To apply this point, an information technology quality assurance organization can:

1. Develop a quality assurance plan that provides a long-range quality direction

2. Require software testers to develop and maintain comprehensive test plans for each project

3. Encourage quality analysts and testers to come up with new and innovative ideas to maximize quality

4. Strive to continuously improve quality processes

Point 2 – Adopt the New Philosophy: Quality must become the new religion. According to Deming, “The cost of living depends inversely on the goods and services that a given amount of money will buy, e.g., reliable service reduces costs. Delays and mistakes raise costs.” Consumers of goods and services end up paying for delays and mistakes, which reduces their standard of living. Tolerance of acceptability levels and defects in systems is the roadblock between quality and productivity. To apply this point, an information technology quality assurance organization can:

1. Educate the information technology organization on the need and value of quality

2. Promote the quality assurance department to the same level as any other department

3. Defuse the notion that quality assurance is negative and the “watch dogs”

4. Develop a risk management plan and not accept any anomalies outside the range of acceptable risk tolerance

Point 3 – Cease Dependence on Mass Inspection: The old way of thinking is to inspect bad quality out. A better approach is that inspection should be used to see how we are doing, and not be left to the final product, when it is difficult to determine where in the process a defect took place. Quality should be built in without the dependence on mass inspections. To apply this point, an information technology quality assurance organization can:

1. Promote and interject technical reviews, walkthroughs, and inspections as nondefensive techniques for achieving quality throughout the entire development cycle

2. Instill the need for the whole organization to be quality conscious and treat it as a tangible, measurable work product deliverable

3. Require statistical evidence of information technology quality

Point 4 – End the Practice of Awarding Business on Price Tag Alone: “Two or more suppliers for the same item will multiply the evils that are necessarily inherent and bad enough with any one supplier.” A buyer will serve his company best by developing a long-term relationship of loyalty and trust with a single vendor. Rather than using standards manuals by which vendors must qualify for business, a better approach is active involvement by the supplier’s management with Deming’s Fourteen Points. To apply this point, an information technology quality assurance organization can:

1. Require software quality and test vendors to provide statistical evidence of their quality

2. Pick the best vendor for each quality assurance tool, testing tool, or service, and develop a working relationship consistent with the quality plan

Point 5 – Improve Constantly and Forever the System of Production and Service: Improvement is not a one-time effort — management is obligated to improve quality continuously. “Putting out fires is not improvement. Finding a point out of control, finding the special cause and removing it, is only putting the process back to where it was in the first place. The obligation for improvement is a ceaseless process.” To apply this point, an information technology Quality Assurance organization can:

1. Constantly improve quality assurance and testing processes

2. Don’t rely on judgment

3. Use statistical techniques such as root cause and effect analysis to uncover the sources of problems and test analysis

Point 6 – Institute Training and Retraining: Often there is little or no training and workers do not know when they have done their jobs correctly. It is very difficult to erase improper training. Deming stresses that training should not end as long as performance is not in statistical control and there is something to be gained. To apply this point, an information technology quality assurance organization can:

1. Institute modern training aids and practices

2. Encourage quality staff to constantly increase their knowledge of quality and testing techniques by attending seminars and classes

3. Reward staff for creating new seminars and special interest groups

4. Use statistical techniques to determine when training is needed and completed

Point 7 – Institute Leadership: “There is no excuse to offer for putting people on a job that they know not how to do. Most so-called ‘goofing off’ — somebody seems to be lazy, doesn’t seem to care — that person is almost always in the wrong job, or has very poor management.” It is the responsibility of management to discover the inhibitors that prevent workers from taking pride in their jobs. From an information technology point of view, development often views the job of quality to be the QA department’s responsibility. QA should be very aggressive as quality leaders and point out that quality is everyone’s responsibility. To apply this point, an information technology quality assurance organization can:

1. For example, if a developer has an excessive number of defects discovered by QA testing, take the time to train him or her on how to unit test his or her code effectively.

2. Improve supervision, which is the responsibility of management

3. Allow the project leader to have more time to help people on the job

4. Use statistical methods to indicate where there are faults

Point 8 – Drive Out Fear: There is often no incentive for problem solving. Suggesting new ideas is too risky. People are afraid of losing their raises, promotions, or jobs. “Fear takes a horrible toll. Fear is all around, robbing people of their pride, hurting them, robbing them of a chance to contribute to the company. It is unbelievable what happens when you unloose fear.” A common problem is the fear of inspections. To apply this point, an information technology quality assurance organization can:

1. Promote the idea that quality is goodness and should be rewarded, and promote any new ideas to improve quality

2. Prior to a structured walkthrough, inspection, or JAD session, QA should make sure everyone understands the ground rules and promote an “egoless” environment

3. Periodically schedule a “Quality Day” in which quality improvement ideas are openly shared

Point 9 – Break Down Barriers Between Staff Areas: There are numerous problems when departments have different goals and do not work as a team to solve problems, set policies, or define new directions. “People can work superbly in their respective departments, but if their goals are in conflict, they can ruin the company. It is better to have teamwork, working for the company.” To apply this point, an information technology quality assurance organization can:

1. Quality assurance and other departments (particularly development) need to work closely together; QA should be viewed as the “good guys” trying to make the software products the best in the world

2. Quality assurance should point out that a defect discovered before production is one that won’t be discovered by the users

Point 10 – Eliminate Slogans, Exhortations, and Targets for the Workforce: “Slogans never helped anybody do a good job. They generate frustration and resentment.” Slogans such as “Zero Defects” or “Do it Right the First Time” are fine on the surface. The problem is that they are viewed as signals that management does not understand employees’ problems, or care. There is a common practice of setting goals without describing how they are going to be accomplished. To apply this point, an information technology quality assurance organization can:

1. Encourage management to avoid the use of slogans

2. Rather than generate slogans, the QA organization should develop and document quality standards, procedures, and processes which the rest of the organization can use to help them to maximize quality

Point 11 – Eliminate Numerical Goals: “Quotas or other work standards, such as measured day work or rates, impede quality perhaps more than any other single working condition. As work standards are generally used, they guarantee inefficiency and high costs.” A proper work standard would define what is and is not acceptable in terms of quality. To apply this point, an information technology quality assurance organization can:

1. Look not just at the numbers but look carefully at the quality standards

2. Avoid formally publicizing defects rates by individual or department

3. Work with the development organization to define quality standards and procedures to improve quality

4. When there are specific quality issues, the QA and Development manager need to address them informally

Point 12 – Remove Barriers to Pride of Workmanship: People are regarded as a commodity, to be used as needed. If not needed, they are returned to the market. Managers cope with many problems but tend to shy away from people problems. They often form “Quality Control Circles,” but this is often a way for a manager to pretend to be doing something about a problem. Management seldom invests employees with any authority, nor does it act upon their recommendations. To apply this point, an information technology quality assurance organization can:

1. Instill an image that quality is their deliverable and is a very valuable commodity

2. Delegate responsibility to the staff to seek out quality and do whatever it takes to accomplish it

Point 13 – Institute a Vigorous Program of Education and Retraining: People must acquire new knowledge and skills. Education and retraining is an investment in people, which is required for long-term planning. Education and training must fit people into new jobs and responsibilities. To apply this point, an information technology quality assurance organization can:

1. Encourage quality staff to constantly increase their knowledge of quality and testing techniques by attending seminars and classes

2. Reward staff for creating new seminars and special interest groups

3. Retrain individuals in new quality skills

Point 14 – Take Action to Accomplish the Transformation: Top management needs to push these thirteen points. Every employee, including managers, should acquire a precise idea of how to improve quality continually, but the initiative must come from top management. With this point Dr. Edwards Deming gave the Plan, Do, Check, Act Process that is commonly known as Deming Cycle. I’ll explain PDCA in my next post.

In Webster’s Dictionary, quality is defined as “the essential character of something, an inherent or distinguishing character, degree or grade of excellence.” If you look at the computer literature, you will see that there are two generally accepted meanings of quality. The first is that quality means “meeting requirements.” With this definition, to have a quality product, the requirements must be measurable, and the product’s requirements will either be met or not met. With this meaning, quality is a binary state, i.e., it is a quality product or it is not. The requirements may be very complete or they may be simple, but as long as they are measurable, it can be determined whether quality has or has not been met. This is the producer’s view of quality as meeting the producer’s requirements or specifications. Meeting the specifications becomes an end in itself.

Another definition of quality, the customer’s, is the one we will use. With this definition, the customer defines quality as to whether or not the product or service does what the customer needs. Another way of wording it is “fit for use.” There should also be a description of the purpose of the product, typically documented in a customer’s “requirements specification”. The requirements are the most important document, and the quality system revolves around it. In addition, quality attributes are described in the customer’s requirements specification. Examples include usability, the relative ease with which a user communicates with the application, portability, the capability of the system to be executed across a diverse range of hardware architectures, and reusability, the ability to transfer software components constructed in one software system into another.

While everyone is committed to quality, the following are some confusions shared by many individuals, which inhibit achieving a quality commitment:

• Quality requires a commitment, particularly from top management. Close cooperation of management and staff is required in order to make it happen.
• Many individuals believe that defect-free products and services are impossible, and accept certain levels of defects are normal and acceptable.
• Quality is frequently associated with cost, meaning that high quality equals high cost. This is a confusion between quality of design and quality of conformance.
• Quality demands requirement specifications in enough detail that the products produced can be quantitatively measured against those specifications. Many organizations are not capable or willing to expend the effort to produce specifications at the level of detail required.
• Technical personnel often believe that standards stifle their creativity, and thus do not abide by standards compliance. However, for quality to happen, well-defined standards and procedures must be followed.

The revised ISO 9000:2000 series are based on eight quality management principles. These principles can be used by senior management as a framework to guide their organizations towards improved performance. The principles are derived from the collective experience and knowledge of the international experts who participate in ISO Technical Committee ISO/TC 176, Quality management and quality assurance, which is responsible for developing and maintaining the ISO 9000 standards.

Principle 1 – Customer Focus: Organizations depend on their customers and therefore should understand current and future customer needs, should meet customer requirements and strive to exceed customer expectations. Applying the principle of customer focus typically leads to:

• Researching and understanding customer needs and expectations
• Ensuring that the objectives of the organization are linked to customer needs and expectations
• Communicating customer needs and expectations throughout the organization
• Measuring customer satisfaction and acting on the results
• Systematically managing customer relationships
• Ensuring a balanced approach between satisfying customers and other interested parties (such as owners, employees, suppliers, shareholders, local communities and society as a whole).

Key Benefits:

• People will understand and be motivated towards the organization’s goals and objectives
• Activities are evaluated, aligned and implemented in a unified way
• Miscommunication between levels of an organization will be minimized

Principle 2 – Leadership: Leaders establish unity of purpose and direction of the organization. They should create and maintain the internal environment in which people can become fully involved in achieving the organization’s objectives. Applying the principle of leadership typically leads to:

• Considering the needs of all interested parties including customers, owners, employees, suppliers, financiers, local communities and society as a whole.
• Establishing a clear vision of the organization’s future.
• Setting challenging goals and targets.
• Creating and sustaining shared values, fairness and ethical role models at all levels of the organization.
• Establishing trust and eliminating fear.
• Providing people with the required resources, training and freedom to act with responsibility and accountability.
• Inspiring, encouraging and recognizing people’s contributions

Key Benefits:

• Increased revenue and market share obtained through flexible and fast responses to market opportunities.
• Increased effectiveness in the use of the organization’s resources to enhance customer satisfaction.
• Improved customer loyalty leading to repeat business

Principle 3 – Involvement of people: People at all levels are the essence of an organization and their full involvement enables their abilities to be used for the organization’s benefit. Applying the principle of Involvement of People typically leads to:

• People understanding the importance of their contribution and role in the organization.
• People identifying constraints to their performance.
• People accepting ownership of problems and their responsibility for solving them.
• People evaluating their performance against their personal goals and objectives.
• People actively seeking opportunities to enhance their competence, knowledge and experience.
• People freely sharing knowledge and experience.
• People openly discussing problems and issues

Key Benefits:

• Motivated, committed and involved people within the organization.
• Innovation and creativity in furthering the organization’s objectives.
• People being accountable for their own performance.
• People eager to participate in and contribute to continual improvement

Principle 4 – Process Approach: A desired result is achieved more efficiently when activities and related resources are managed as a process. Applying the principle of Process Approach typically leads to:

• Systematically defining the activities necessary to obtain a desired result.
• Establishing clear responsibility and accountability for managing key activities.
• Analyzing and measuring of the capability of key activities.
• Identifying the interfaces of key activities within and between the functions of the organization.
• Focusing on the factors such as resources, methods, and materials that will improve key activities of the organization.
• Evaluating risks, consequences and impacts of activities on customers, suppliers and other interested parties.

Key Benefits:

• Lower costs and shorter cycle times through effective use of resources.
• Improved, consistent and predictable results.
• Focused and prioritized improvement opportunities

Principle 5 – System Approach: Identifying, understanding and managing interrelated processes as a system contributes to the organization’s effectiveness and efficiency in achieving its objectives. Applying the principle of Systems Approach to management typically leads to:

• Structuring a system to achieve the organization’s objectives in the most effective and efficient way.
• Understanding the interdependencies between the processes of the system.
• Structured approaches that harmonize and integrate processes.
• Providing a better understanding of the roles and responsibilities necessary for achieving common objectives and thereby reducing cross-functional barriers.
• Understanding organizational capabilities and establishing resource constraints prior to action.
• Targeting and defining how specific activities within a system should operate.
• Continually improving the system through measurement and evaluation

Key Benefits:

• Integration and alignment of the processes that will best achieve the desired results.
• Ability to focus effort on the key processes.
• Providing confidence to interested parties as to the consistency, effectiveness and efficiency of the organization.

Principle 6 – Continual Improvement: Continual improvement of the organization’s overall performance should be a permanent objective of the organization. Applying the principle of Continual Improvement typically leads to:

• Employing a consistent organization-wide approach to continual improvement of the organization’s performance.
• Providing people with training in the methods and tools of continual improvement.
• Making continual improvement of products, processes and systems an objective for every individual in the organization.
• Establishing goals to guide, and measures to track, continual improvement.
• Recognizing and acknowledging improvements.

Key Benefits:

• Performance advantage through improved organizational capabilities.
• Alignment of improvement activities at all levels to an organization’s strategic intent.
• Flexibility to react quickly to opportunities

Principle 7 – Factual approach to decision making: Effective decisions are based on the analysis of data and information. Applying the principle of Factual Approach to decision making typically leads to:

• Ensuring that data and information are sufficiently accurate and reliable.
• Making data accessible to those who need it.
• Analyzing data and information using valid methods.
• Making decisions and taking action based on factual analysis, balanced with experience and intuition

Key benefits:

• Informed decisions.
• An increased ability to demonstrate the effectiveness of past decisions through reference to factual records.
• Increased ability to review, challenge and change opinions and decisions

Principle 8 – Mutually beneficial supplier relationships: An organization and its suppliers are interdependent and a mutually beneficial relationship enhances the ability of both to create value. Applying the principle of mutually beneficial supplier relationships typically leads to:

• Establishing relationships that balance short-term gains with long-term considerations.
• Pooling of expertise and resources with partners.
• Identifying and selecting key suppliers.
• Clear and open communication.
• Sharing information and future plans.
• Establishing joint development and improvement activities.
• Inspiring, encouraging and recognizing improvements and achievements by suppliers

Key benefits:

• Increased ability to create value for both parties.
• Flexibility and speed of joint responses to changing market or customer needs and expectations.
• Optimization of costs and resources.

An assessment is similar to a larger audit. It lasts several days and involve teams that are typically made up of three to five people.

An assessment checks the operational processes and practices against the requirements of an assessment model, analyzing, in interviews, not only the direct artifacts (guidelines, work instructions, process models and indirect artifacts (protocols, reports, filled-in templates) but also the actually “practiced” processes used in a project. Sometimes the assessment team may even “look over an expert’s shoulder” at their workplace to get a clear idea of applied work practices.

CMMI and SPICE are typical assessment or appraisal examples. Maturity and capability levels assigned to organizations or individual processes are determined. Strengths and weaknesses are pointed out as well as improvement potentials and identifying opportunities for synergies and lead competencies (especially where several organizational units or projects are compared).

Out of five parts of the ISO/IEC 15504 (SPICE) standard, three deal directly with assessments:

• Part 2 – Performing an assessment
• Part 3 – Guidance on performing an assessment
• Part 5 – An assessment model and indicator guidance

The purpose of these detailed descriptions is to ensure that different assessment results can be compared and that different assessors – i.e. the persons actually conducting the assessment – diverge as little as possible in their evaluation results.

Also See:

• What is an Audit

An audit is defined as a systematic and independent examination to evaluate products / services and development processes and determine compliance with standards, guidelines, specifications, and/or processes. Audits are based on objective criteria and documents that accomplish the following:

• Determine the design or content of a product solution
• Describe the product’s development process
• Specify the methods and techniques used to prove or measure compliance or non-compliance with standards and guidelines

Besides checking compliance with standards and guidelines, audits can also check on the effectiveness of an implementation and evaluate an artifact.

There are three types of audits:

• The system audit examines the quality management system or selected parts of it with regard to structure and workflow.
• The process audit analyzes processes with regard to human resources allocation, process monitoring and the order of individual process steps.
• A product audit evaluates products/parts relative to their compliance with specification and adherence to standards and guidelines.

An audit usually lasts one to two days and the audit team is typically made up of two to four people. All audits aim at revealing weaknesses, documenting them and proposing concrete and lasting improvements. Useful steps are as follows:

• Determine the need for an audit (necessity for internal process improvement or external customer requirement)
• Plan the audit (content, team, formal basis, resources, dates etc.)
• Perform the audit in accordance with the plan and determine the achieved results.
• Discuss strengths and weaknesses and decide whether or not improvements are necessary.
• Implement the improvement suggestions and ensure appropriate reporting and archiving.