Engineering processes shall include analysis, design, and coding.  Quality Assurance processes include verification and validation processes.  Products include documents and code. If the scope of the audit is limited; then planning will be within the defined limits.  During a limited audit, the auditor may verify or check only few processes or some set of work products. Planning and preparation phase is very important in a limited audit. The auditor/lead auditor must be very careful while identifying the processes or product.

First of all, the auditor must understand the business logic or objective of the software project and what end products are to be produced. The auditor must need to understand the contractual requirements for the deliverable software, documentation, project management, software engineering and quality assurance practices. After understanding these requirements, the auditor should review the software project plan, configuration management plan, quality assurance plan, risk tracker, staffing and training plan, development processes etc. to understand the processes or procedures being followed for the software development.

The review of status reports from the development team is helpful to understand the stage of completeness of products and may contain useful information from the audit point of view. After understanding the project and its current status, the auditor needs to define the critical areas that require the detailed attention. The source of this information may be status reports, minutes of meeting of the review meetings with client or management, non-conformance reports and results of previous audits.

The auditor / lead auditor must inform the auditee team about the intent / scope of audit including the date and time of audit.

Now, the next task for the auditor is to prepare a checklist. Basically, the checklists act as a road map during the actual audit. So, the checklists must include all checkpoints and questions to be asked from the auditee.

Generally, the checklist is already available in the Quality Management System. However, the checklist must be tailored for the specific project to be audited.

Regarding the checklists, my views are:

• General Audit – a comprehensive and less detailed checklist
• Limited Audit – a more detailed checklist in the specific area
• Internal Quality Audit – a detailed checklist in all the relevant areas

The quality of the audit is directly proportional to the quality of the checklist prepared. The checklists are very critical to the success of audit. These checklists are also very helpful for the auditee to understand the type of questions and records the auditor will examine.

Product Defects: Product Defects are the defects that are introduced and detected during the various stages of software development life cycle. While the defects get introduced during the various activities of the phase, the detection occurs during reviews and various types of testing efforts.

The various types of defects occuring during the SDLC activities can be categorized as follows:

- Functionality Errors
- External Interface Errors
- Performance Problems
- Environment Errors

Examples of SDLC defects:

Requirements: Performance Exceptions of the system not properly defined
Design: Program Specifications did not specify of number of decimal places
Design: Architecture was not perperly planned
Coding: Variable no initialized, Missing if condition

Process Defects: Process defects are caused during execution of project and usually detected during audits. While such defects get introduced during various project activities and get detected during various conducted during the project life cycle. All such defects are of the nature of deviations in process implementations of the organization or the project. Though it is difficult to draw a co-relation between process and product defects, however, the existence of process defects are indicative of larger number of product defects. Also, existence of such defects raise a question mark about the quality of the software product.

Examples of Process Defects:

- Risk Management Plan is not in place
- Change request impact on schedule and effort not reflected
- Project Management Plan not kept up to date with current status of the project
- Requirements tracking is incomplete
- Review Plan is not adequate
- Requirements gathering strategy not defined for the product
- Quality goals are not tracked
- Issue tracking mechanism is adhoc

SCM activities are constantly carried out during the development or major enhancement cycle of a product. Configuration items are identified, baselines are established and changes to baselined items are carried out in a formal way. The status accounting provides the information of the SCM activities that have been carried out. When the product the ready for release / delivery, it is necessary to establish that the software product has been build in accordance with the requirements including the approved CRs. In other words, we perform a formal review to verify that the product being delivered will work as advertised, promoted or in any way promised to the customers.

Configuration audits provide such a review. They are not the same as quality audits or product tests. However, they use the information available as an outcome of the quality audits and testing along with the configuration status accounting information, to provide assurance that what was required has been build. Configuration audits are typically performed at the time of delivery and major upgrades to the software.

By ensuring that only reviewed, verified and validated products are checked into the baseline and that only baselines are used to generate releases, the assurance is given that non-conforming products are not released. This is as per the ISO 9001 requirement. Further, by use of configuration audits prior to delivery, we ensure that the actual delivery made is of the correct product.

The audit team typically includes persons independent of the project’s configuration controller or other project team members. Configuration audits can be logically divided into two parts, the functional configuration audit and the physical configuration audit.

The objective of functional configuration audit is to verify that a configuration item is in accordance with its software requirements. The audit team will consist of 3-4 members comprising the customer representative, independent quality assurance members and configuration controller of other projects. The project manager, configuration controller and the testers of the project form the auditee team.

The project team presents the summary of the status of the requirements, the baselines, the CRs and Engineering Change Order and the Verification & Validation activities carried out. Any requirements that are not yet implemented or could not be tested are also presented as deviations to the FCA team.

The FCA team satisfies itself that the deviations will not reduce the effectiveness of the software that is being delivered. The FCA team also studies the CRs, check-in, check-out registers and the test reports to ensure that what has been developed and tested is complete with respect to the requirements. At the end of audit, the FCA team will identify all issues for which action is required.

The objective of the physical configuration audit is to ensure that items in the baseline are of the right version. The physical configuration audit typically follows the functional configuration audit and can be performed by the same audit team or a subset of the functional audit team. The main task is to ensure that naming, numbering and physical location of the configuration items is according to the laid out procedures and standards. If the functional audit is successful, the physical audit typically ends up identifying wrongly versioned configuration items or configuration items not properly checked-in. The issues raised by the physical configuration audit team also need to be closed through the appropriate actions.

Also see: Configuration Audit Checklist

The purpose of software quality assurance is to ensure that:

- all project activities are occurring according to plan
- all plans and project activities are consistent with defined processes
- all processes, activities, and software products comply with applicable standards
- software products satisfy the needs of the customer.

Ideally, a software quality assurance organization exists independent of your project. This group represents the interests of executive management and should be double-checking the activities and products associated with your project. This includes checking on your management artifacts as well as your execution of, and adherence to, defined planning and management processes.

When properly implemented, software quality assurance performs a critical support function by helping you detect and resolve problems at the earliest opportunity. Of course, you are detecting and resolving problems as a routine, daily part of your management activities. However, anything that software quality assurance finds is something that has either been undetected by you or something for which your resolution actions have not yet been effective. In either event, this is very important information for you to know.

In the absence of an external, independent quality assurance function, you will need to implement these actions internally on your project. Your approach should be similar to the approach used for software configuration management. That is, the areas you need to address include:

- Who has responsibility for software quality assurance?
- What process will they follow?
- What tools will they use?
- Do they need any additional training?
- Who will double-check their work?

As a project manager, if you decide to perform the software quality assurance activities yourself, by definition, there will be no protection at the management level of the project. No one will be performing the software quality assurance function on the management activities you perform or the plans and other project artifacts you produce. Considering the importance of successful management activities and products, not having a doublecheck process at your level is highly risky.

With regard to who will double-check the quality assurance process, this is the classic problem of who polices the police. Certainly not the software project manager. Anything you do to monitor software quality assurance internal to your project is essentially a routine part of your management tracking and oversight responsibilities. The oversight you do as a manager is the initial checking process, not the doublechecking process. One option is to bring in an external person occasionally to investigate the plans, procedures, and activities associated with software quality assurance.

Additionally, from one perspective, you can think of the software integration and system testing process as part of your double-check on the software quality assurance organization. In principle, effective software quality assurance will translate into comparatively fewer defects found during postdevelopment testing.

Checklist for Phase End Audit:

1. Are the NCRs of the previous Audits closed?

2. Have requirements been elicited, prioritized and validated?

3. Have the Client approved to the FSD / Design / relevant phase work products?

4. Is size and efforts were estimated after the FSD was approved?

5. Is the effort distribution mentioned in Project Metrics same as that mentioned in MS project Schedule?

6. Have the reviews been performed on the work products?

7. Was causal analysis performed for the phase, have suitable action items been identified?

8. Is a DP checklist prepared, as planned? Was it used in the phase? Has it been enhanced as needed?

9. Is the Functional Configuration Audit / Physical Configuration Audit conducted ?

10. Have all defects been closed?

11. Have the Traceability matrix updated?

12. Are all the planned metrics collected?

13. Are the sub-process such as preparation, review and rework monitored and controlled during the progress of this phase and corrective action if any, recorded in metrics sheets?

14. Are the inferences on the overall process performance based on the sub-process performance documented in metrics sheets?

15. Are the data available in the metrics sheet verified against the source?

16. Is the corrective and preventive action taken quantitatively within the phase?

17. Is correlation for variance data, predictions inferences appropriate for the process and sub-process variances observed in metrics sheets?

18. Have Decision Analysis & Resolution been applied in project as identified in the Project plan?

19. Is consistency maintained for the inference data and actual variance observed in metrics sheets?

20. Are changes to the schedule tracked & updated?

21. Are the changes in the schedule affects the project’s delivery date? If so, have the Client been informed the same?

22. Are support groups informed about schedule changes affecting them?

23. Are the support group commitments revised accordingly?

24. Is the Risk Mitigation Plan reviewed periodically?

25. Has the status of risk plan been updated?

26. Are all the work products been audited before delivery?

27. Have the Change Control Form generated for each change requested?

28. In case of major changes, has the approval for the same be obtained from respective authority before initiating work on the same?

29. Have the complaints from the client are recorded?

30. Are Client complaints tracked and closed?

31. Are all the team members recorded their efforts till date?

32. Are the Periodic reviews conducted in the planned frequency?

33. Have the effort on review & rework noted down?

34. Have the process database been updated with all the completed documents?

35. Is Master List of Project Documents updated?

36. Is the Access Control Library updated for changes?

37. Does the Requirement Tracebility Matrix contain the physical file name for source code mappings?

38. Is the total Number of requirements identified in Requirement Tracebility Matrix in sync with Functional Spcification Document?

39. Does Requirement Tracebility Matrix consist of bi-directional traceability mapping?

40. Is the build verification done before Testing?

Also See:

Work Product – FSD Audit Checklist