by JP Russell
Today’s organizations need to be agile and responsive to the changing requirements in private and public business sectors. Properly directed, internal audit program resources can help an organization stay focused and uncover new improvement opportunities.
Unfortunately, the effectiveness of many audit programs is limited to counting findings or other simplistic measures. Effective audit programs, however, should provide insight and support the organization’s objectives. But some organizations are not sure how to gauge the effectiveness of their audit program.
At first glance, gauging the effectiveness of your internal audit program may seem easy. If you have accomplished your objectives, the audit program is effective. But as you start to list the organization’s, department’s and audit program’s objectives, things start to get fuzzy. What methods will you use, and what measures do you need to monitor?
Note that while this column is about internal audit programs, the same techniques can be applied to any department or function simply by changing names and examples.
After a few minutes of scratching your head, you may be tempted to go back to counting audits conducted, nonconformities issued and closed corrective actions as your key performance indicators (KPI)—but please resist.
Effectiveness of the audit program should be of interest to audit managers and auditor team members: managers, because they should be making the right decisions to continually improve the internal audit function, and auditors because they will be asked for their input and agreement on the performance parameters by which they will be judged.
Effectiveness
Gauging the effectiveness of an internal audit program can be complicated. We want what we do to be effective, but sometimes we are not sure what effective means or how to determine it.
The dictionary definitions for effectiveness are too numerous and too vague for application in a technical field. In 1995, in After the Quality Audit, I explained that the effectiveness of a system or process is based on two components: process and product.1
The system or process is effective in two circumstances:
1. When it achieves the desired result that is consistent with organizational objectives (the product).
2. When the process is capable, efficient, and consistent with objectives (the process).
For example, adding three inspection steps to achieve the desired quality output may achieve the output objective (the product) but would make the process less efficient and perhaps less capable. There must be a proper balance between the two components (product and process). This means the end does not justify the means.
In 2000, ISO 9000 added a definition for the word “effectiveness to its vocabulary: the extent to which planned activities are realized and planned results achieved. In an elegant manner, it combined the two components’ processes (planned arrangements) and product (results achieved). Knowing what effective means is important because effective processes lead to an effective and successful organization.
As you can see, the number of audits conducted, or corrective action requests closed are not adequate KPIs of the effectiveness of the internal audit program. Counting audits addresses only the process and not the outcome (product).
Audit program objectives
Audit program performance indicators should be based on objectives that reflect the audit program mission and organizational objectives and goals. The organizational or function objectives and goals are the big picture of where you want to be one to five years from now.
Not all objectives are equally important to the organization or audit program. The objectives that avoid the greatest risks and identify the greatest opportunities for improvement are the most important.
For this column, I am going to lump objectives and goals into three groupings:
1. Alpha group: critical for the organization or function to operate. Top management wants to know if the organization adheres to all applicable standards to ensure critical licenses and certifications will be retained.
2. Bravo group: necessary for day-to-day management.
3. Charlie group: required for advancement and growth.
The Alpha-Bravo-Charlie order is not necessarily the order of importance of the individual organization’s objective or goal. Your organization probably already has groupings of some kind. Table 1 shows some examples of audit program objectives adapted from ISO 19011,2 clause 5.2.1.
You have auditing objectives, but because internal audits are a service for internal customers, you should also consider internal customer objectives when performing the service. For example, if you are a licensed barber or beautician, should you cut someone’s hair how you think best, or should you consider your customer’s objectives as you perform your professional duties?
If you are conducting second-party supplier audits and have good relationships with your suppliers, you may want to be aware of their objectives relevant to the product or service they provide.
For third-party certification audits, you may need to verify that the auditee organization has objectives that are promulgated throughout the organization, but you do not need to consider them as part of the audit purpose. It may be good business to do so, however.
Independent third-party auditors from governmental agencies need not be concerned with auditee organization objectives as long as the organization complies with statutory and regulatory requirements. However, some regulatory agencies believe that auditee objectives improve ongoing compliance, as well as effectiveness.
Strategies
Once you know the objectives, you, as the manager, or your management team can develop strategies to achieve the audit program objectives. The strategies will be based on the type of organization, the organizational culture, and resources. Some of the strategies may be simply to formalize what you are already doing.
If there was an objective to continually improve, some audit program strategies and tactics may include:
• Developing a process to collect complaints or feedback from audit program customers (auditee, audit program manager, stockholders, top management, function managers and supervisors). By collecting feedback, audit program management can learn what works and what doesn’t work, as well as identify customer needs.
• Adding value by reviewing department or area objectives as part of audit preparation and including them in the audit statement of purpose for that department or area when appropriate.
• Identifying and reporting completed corrective actions that improved or changed the system or process.
• Upgrading auditor competency for observing and reporting performance issues.
• Verifying claimed improvements by organization functions and reporting findings to top management.
• Implementing real-time audit reporting using mobile technology.
Another objective may be to maintain continuous compliance using fewer resources, with audit program strategies or tactics that could include:
• Reducing resources used to audit areas demonstrating continuous compliance by decreasing audit frequency or conducting mini audits supported by independently supported data (compliance indicators based on performance).
• Establishing a network of audit advisors for areas needing assistance to comply based on past results.
• Implementing a program to schedule audits based on changes in processes or key personnel to identify and prevent noncompliances by external auditors.
• Identifying situations in which outsourcing is a more cost-effective alternative than in-house oversight.
• Establishing and implementing an e-audit program.
We have discussed the first two steps (plan and do) of the success quadrangle (see Figure 1). Next, we need to establish performance measures that will result in the successful achievement of the objectives.
KPIs
Now that you have determined how to achieve objectives and goals, it is time to develop KPIs to ensure you stay on track. KPIs should be quantifiable measurements that are agreed to beforehand.
A KPI may be to achieve milestones during the implementation of a feedback program. Another one may be to maintain a 95% internal customer satisfaction rating for the audit program. A KPI for ongoing compliance could be to have no serious findings from external auditing organizations—just minor findings requiring remedial action with no system problems.
Process performance indicators may be the monitoring of internal complaints related to delivery of the service, redoing things or meeting agreed-on commitments.
Some desirable performance indicators, including the following examples, are less quantifiable:
• Do corrective action plans address the fundamental cause (not the symptom)?
• Do they contain real root causes?
• Were plans on time?
• Are solutions realistic (viable considering the environment)?
• Is the timetable for change reasonable?
For these less-quantifiable indicators, you might grade corrective action plans based on a marking scheme similar to those for essay tests, with a 100% grade matching all expectations.
This same technique could be used to improve the effectiveness of audit reports. For example:
• Do audit reports link findings to objectives or customer requirements?
• When possible, are findings quantified and analyzed?
• Is report terminology appropriate for the users of the report?
• Are attachments, examples, diagrams or images used to improve report effectiveness?
• Are unfamiliar terms defined?
Once you have established your performance indicators, be prepared to change them when objectives change.
Everyone involved in the audit program (auditors, staff, supervisors) should be focused on meeting or exceeding KPIs. You can post the KPIs in conference rooms, the lunchroom and on the company intranet or website to keep everyone informed of your progress.
Additional ideas for audit program strategies and performance indicators can be found in the latest edition of After the Quality Audit.3
Effective and efficient
An effective internal audit program is one that achieves its objectives via processes that are capable and efficient. It is about doing it right the first time and being lean. You will need output measures and process measures to verify the audit program is effective and efficient.
The audit function provides a valuable service for the organization. As with any service, it should be done right and professionally. Operate as if you were competing with other audit organizations and could lose the business. What value-added services does your audit program offer? What innovations are planned for future services? Can you demonstrate that you address your customer needs?
Many managers and executives have low expectations of audit programs. Many view audit programs as the cost of doing business to ensure compliance to regulations. Once audit program managers can demonstrate the effectiveness of how the audit program supports the organization’s objectives, managers will start to see that audit program verification services can add value beyond compliance to the law.
Auditors represent an independent set of eyes supporting the insight that is needed in our fast-paced world economy. We need to ensure we are putting solved problems behind us and are continually advancing our organizations to optimize their chances of success.
________________________________________
References
1. J.P. Russell, After the Quality Audit, ASQ Quality Press, 1995.
2. ANSI ISO/ASQ QE19011S, Guidelines for Management System Auditing—U.S. version with supplemental guidance added, ASQ, 2008.
3. J.P. Russell, After the Quality Audit: Closing the Loop on the Audit Process, ASQ Quality Press, 2000.
________________________________________
About the author
J.P. Russell is an ASQ Fellow and a voting member of the American National Standards Institute/ASQ Z1 committee. He is a member of the U.S. Technical Advisory Group to Technical Committee 176, the body responsible for the ISO 9000 standard series. Russell is the managing director of the internationally accredited Quality Web-Based Training Center for Education, www.jprlearning.com, an online auditing, standards, metrics, and quality tools training provider. A former RAB and IRCA lead auditor and an ASQ Certified Quality Auditor, Russell is the author of several ASQ Quality Press bestselling books, including Process Auditing Techniques; Internal Auditing Basics; ISO Lesson Guide 2015: Pocket Guide to ISO 9001:2015; and he is the editor of the ASQ Auditing Handbook.
This article first appeared in the ASQ Quality Progress Standards Outlook column June 2009. It has been repurposed for current review.