Game of Chance: Managing and reporting the risks associated with auditing

By J.P. Russell

 RISK is a popular term being added to the standards and procedures lexicon, a massive topic, and confusing to most of us. Risk is truly the bull in the china shop.

The term could refer to risks associated with auditing, verifying risk treatments, or risks observed while conducting an audit.

The ISO 19011:2011 standard states that management system auditors should understand risks associated with auditing. The standard includes audit program risk and auditing risk.1

ASQ’s 2004 CQA BoK included the topics of evaluating risks associated with:

  • Management and the organization as part of the audit purpose or objective
  • Managing the audit program
  • Collecting audit evidence

The 2012 CQA BoK expands and clarifies risk to include:

  • How the audit program affects an organization’s risk
  • How the audit organization’s risk can influence the number and frequency of audits performed
  • The use of risk management tools such as failure mode and effects analysis (FMEA), hazard analysis and critical control points (HACCP), critical to quality (CTQ) analysis, and health hazard analysis (HHA) 2

This topic can get even more complicated when we try to put risk in a box – similar to what is done with document control, corrective action programs, or purchasing controls.

 
What people fail to realize is that risk is the bull in the china shop – free to roam where it has access and can cause mayhem at any time. It can ruin product on the shelves and cause an organization to suffer loss of income. If it makes it to the street, it could injure people or destroy their property.

The china shop has different kinds of china, different departments, and sections but the bull is free to roam wherever he has access and can cause mayhem at any time. The bull can ruin product on the shelves, cause the business to suffer loss of income, and if the bull gets out into the street, he could injure other people and/or destroy their property.

What is it?

Risk is all around us; a part of every-day business.

Risk has been defined as the possibility of loss, injury, disadvantage, or destruction, or the product of the amount that may be lost and the probability of losing it.3

For example, in marbles, if I make a shot, I can add a marble to my bag. If I miss the shot, I could lose five marbles. So, what’s the risk? I think I have a 90% chance of making the shot or 10% of missing it. The product of the amount that may be lost (five marbles) and probability of losing it (10%) equals 0.5 or half a marble.

Based on this, I should go ahead and take the risk to add another marble to my bag. But, if my shot is risky and had only 30% of success, I risk 3.5 marbles instead of half a marble. I may decide to play it safe and not take the shot or opt for an approach that will obstruct my opponent.

In business school, I took a class that described risk as I did with the marbles except we were dealing with business ventures and the marbles were money. Due to a lack of statistics or adequate information, we brainstormed the probability of failure and estimated the resources – or revenue – that could be lost.

Risk has long been a term used by the insurance industry and lending institutions. The higher the probability of loss, the higher the premium or the higher the interest on a loan. Higher rates compensate for the higher probability of loss.

The same is true of an organization that decides on a threshold of 15% return on investment (ROI) for a venture of negligible or managed risk, yet it might be willing to make a risker investment if the ROI is 25%.

The standard writers from the International Organization for Standardization (ISO) have come up with a difference definition for risk: the effect of uncertainty on objectives.4

According to the first two notes in the standard, an effect is a deviation from the expected positive or negative, and objectives can have different aspects (such as financial, health and safety, and environmental goals) and can apply at different levels (such as strategic, organization-wide, project, product, and process).5 

One of the weaknesses of this definition is that organizations don’t have explicit objectives for every risk. It’s akin to quality being conformance to customer-stated requirements, except not all product or service requirements are specified.

The ISO definition of risk should be expanded to include the effect of uncertainty on objectives, including objectives that aren’t stated but are necessary for the organization to survive.

Risk management vs reporting

One reason the topic of risk is confusing is that people intermingle the need to manage risk with the need to monitor or report risk. ISO 31000 states that the risk management process involves:

  • Establishing context (scope and objectives),
  • Identifying risk
  • Analyzing risk
  • Evaluating risk
  • Treating risk 6

If you’re asked to manage risk, you need to follow the risk management process steps or a similar model. If you’re asked to monitor or report risk, you need to be able to recognize it or know it when you see it.

In many cases, auditors and others are asked to monitor or report what they observe relative to risk treatments or the context of risky processes or activities, with risk being the possibility of loss, injury, disadvantage, or destruction, or the effect of uncertainty on objectives.

If you’re asked to report risk, the reporting could occur during any of the risk assessment steps (identification, analysis, evaluation).7 The reporting may be based on intuitive assessment, such as a finding that could result in loss of license, certification, or a customer order. Auditors also may be directed to report that risk treatments are implemented and effective.

Auditing risk

An audit is a service performed by auditors that may be internal or external to the organization being audited. ISO 19011:2011 specifically lists audit performance risks that should be addressed. The first risk to address is in preparing the audit plan (clause 6.3.2.1).The audit team leader should be aware of the risk to the auditee organization created by the audit.

For example, the presence of audit team members could influence health, safety, environmental, or quality controls. A member of the team could be sick, get injured, pollute the environment, or interfere with an inspection. There may not be a specific audit objective that involves avoiding injury, but it is a risk that should be considered depending on the auditee site and requirements.

If you’re asked to analyze and evaluate risks associated with the audit, you may want to consider the aspects related to the product or service, as well as the causal factors such as people, equipment, environment, materials, methods, and measurements.

Auditors, for example, might contaminate a clean room, ruin a circuit board with a static spark, or void a calibration due to equipment damage. To identify potential risk, I would suggest you first consider the aspects that can create risk relative to the environment: the nature of the organization.

Any methods that will be used to mitigate or treat risk should be discussed in the audit’s opening meeting and included in the audit plan. You can include managed risk in the audit plan under managed risk or another suitable title. I’ve done this in the past, but I may have used titles such as special requirements, special topics, or issues.

One of the biggest risks when conducting an audit is the risk associated with sampling.  Samples may not be representative of the population from which they are selected, and thus, any conclusions based on the sample would be wrong.

I won’t explore consumer or producer risk in this column, but sampling error is an identified aspect of risk that must be addressed. External auditors are more likely than internal auditors to experience sampling error, but it’s important for all auditors to be vigilant and attentive listeners for any indication a sample may be skewed.

Perhaps the auditee changed processes 45 days ago, the form I have is just for special orders, or the records you selected are for a service that is no longer provided. In these cases, you’re being asked to monitor and report risk, but they wouldn’t necessarily be included in the audit plan because it could identify the exact samples you will be reviewing.

Do not delay

As part of the performance, any evidence collected that suggests an immediate and significant risk – effect uncertainty on objectives – to the auditee should be reported without delay to the auditee and, as appropriate, to the audit client. 9

Reporting something that suggests an immediate and significant risk is subjective which is why auditors are frequently asked to use their judgment. If the risk doesn’t have a significant impact on the organization, the auditee will let you know. This type of activity isn’t managing risk; it’s reporting risk.

The 2012 CQA BoK indicates that audit results may be classified by the level of risk. This may be as simple as reporting results as major or minor, rating nonconformities on a scale of 1-10, or comparing them relative to the business bottom line or budget. In this case, auditors are being asked to assess their observations and report audit findings based on relative risk.

Auditing for risk

Some audit programs or objectives include risk. Organizations may conduct risk audits while they conduct compliance audits. At other times, the identification of risky processes or activities beyond conformity or compliance to requirements is added to the purpose of the audit.

Conducting a risk audit may involve collecting evidence to verify known risk is being controlled and that risk treatment plans are effective. The objective or purpose of the audit would be to start with a list of identified and treated risks and then verify the controls are effective – similar to an auditor verifying that corrective actions have been implemented and are effective.

Risk treatments must be verified in the short and long term, and when there are changes to processes related or linked to identified risk. Some standards, such as ISO 22000 for food safety, have plans to treat/mitigate significant hazards and risk. Auditors can verify or validate those plans.

Process audits are an effective approach for identifying new risk. Auditors conducting process audits are more familiar with the process and would be able to spot and identify processes or events that could be a significant risk to the organization. System and product audits, however, do not exclude auditors from identifying processes or events that could be significant risk.

Auditors are not charged with conducting a formal risk management assessment; they are merely making observations that there might be an aspect of risk that needs formal evaluation. Later, however, an auditor may be assigned to a team that conducts a formal risk management analysis.

For example, an auditor may observe that the ink is smearing on a product label with return instructions. This may be a performance issue in which the product is not being returned in an efficient manner or it could be a potential risk to the organization if product is put in landfill in-lieu of proper instructions.

As I mentioned at the outset, risk is a complicated topic – complicated enough that it’s difficult to cover in just one column. So, look for a continuation of this discussion in next month’s Standards Outlook. QP

REFERENCES

  1. International Organization for Standardization, ISO 19011:2011 – Guidelines for auditing management systems.
  2. ASQ “Quality Auditor Certification–BoK,“ http://prdweb.asq.org/certification/control/quality-auditor/bok.
  3. Merriam-Webster, “Risk,” https://www.merriam-webster.com/dictionary/risk.
    4. International Organization for Standardization, ISO Guide 73:2009 – Risk management – vocabulary, clause 1.1.
  4. Ibid.
  5. International Organization for Standardization, ISO 31000:2009 – Risk management – Principles and guidelines, clause 5.1.
  6. Ibid, clause 5.4.
  7. International Organization for Standardization, ISO 19011:2011 – Guidelines for auditing management systems.
  8. Ibid, clause 6.4.4.

 

About the author J.P. Russell is an ASQ Fellow and a voting member of the American National Standards Institute/ASQ Z1 committee. He is a member of the U.S. Technical Advisory Group to Technical Committee 176, the body responsible for the ISO 9000 standard series. Russell is the managing director of the internationally accredited Quality Web-Based Training Center for Education, www.jprlearning.com, an online auditing, standards, metrics, and quality tools training provider.

A former RAB and IRCA lead auditor and an ASQ Certified Quality Auditor, Russell is the author of several ASQ Quality Press bestselling books including Process Auditing Techniques; Internal Auditing Basics; ISO Lesson Guide 2015: Pocket Guide to ISO 9001:2015; and the editor of the ASQ Auditing Handbook.

 

Note: This is Part I of a two-part article. The article first part appeared in the ASQ QP Magazine Standards Outlook column August 2012, pp 52-54. The information is extremely relevant to the ISO 9001:2015 quality principle of risk-based thinking. Other value-added articles such as this may be accessed by navigating to www.jprlearning.com, The J.P. Russell Library.