Measuring SMS Effectiveness
In aviation, a Safety Management System (SMS) is more than a compliance exercise—it’s a living framework that should evolve, improve, and inform decision making at every level. But how do you know if it’s working? Measuring SMS effectiveness is essential for maintaining regulatory compliance, fostering a strong safety culture, and, most importantly, protecting people, operations, and operational assets.
In this SMS Quick Take, we explore practical Safety Performance Indicators (SPI) and qualitative metrics that align with the four components of SMS outlined in Title 14, Code of Federal Regulations (14 CFR) part 5, Safety Management Systems; Federal Aviation Administration (FAA) Advisory Circular (AC) 120-92D, Safety Management Systems for Aviation Service Providers; and International Civil Aviation Organization (ICAO) Doc 9859, Safety Management Manual (SMM).
Why Measuring SMS Effectiveness Matters
A well-structured SMS should—
Proactively identify hazards before they lead to incidents,
Mitigate risk through effective controls and corrective actions,
Adapt and improve through feedback and performance monitoring, and
Engage the workforce in meaningful safety activities.
Without defined effectiveness metrics, organizations risk—
Complacency—Assuming the system is working without evidence.
Inefficiency—Wasting resources on ineffective safety initiatives.
Non-compliance—Falling short of FAA and ICAO expectations.
By tracking the right metrics, you can identify trends, uncover blind spots, and make data‑driven decisions that enhance safety performance.
What the FAA and ICAO Say About Measurement
FAA 14 CFR part 5 requires organizations to monitor, analyze, and assess SMS performance. AC 120-92D further recommends using both quantitative metrics and qualitative methods to evaluate risk controls, safety objectives, and cultural maturity.
ICAO Doc 9859 defines SPIs as measurable data points used to monitor and assess safety performance. ICAO encourages the use of SPIs in combination with Safety Performance Targets (SPT) (in other words, is your intended target for safety performance over a given period in line with your safety objectives?).
These requirements tie directly to the four SMS components:
Four Dimensions of SMS Performance
Aligned with both FAA and ICAO guidance, SMS performance can be evaluated across these four dimensions:
Compliance—Are regulatory and internal SMS requirements being met?
Risk-based Performance—Are safety outcomes and risk profiles improving?
Participation and Engagement—Are employees actively participating in hazard reporting and safety initiatives?
Safety Culture—Are safety values and behaviors embedded across the organization?
These dimensions reinforce the Safety Assurance and Safety Promotion components in part 5 and reflect ICAO’s emphasis on performance-based oversight in ICAO’s Annex 19 to the Convention on International Civil Aviation, Safety Management, and Doc 9859.
SMS SPIs That Align With Part 5 and ICAO Guidance
1. Hazard Reporting Rate
What it tells you: Frequency of reported safety concerns.
Why it matters: A healthy reporting rate often reflects trust and open participation—key traits of a positive safety culture (FAA AC 120-92D 3.5.7, p. 3-50; ICAO DOC 9859 9.3.4.6, p. 157 & 9.4.4.4, p. 166).
How to act: Watch how reporting changes over time, adjust for the size of your operation, and consider how safety outreach efforts may be influencing the numbers.
2. Average Time to Close Safety Reports
What it tells you: Efficiency of your safety issue resolution process.
Why it matters: While not specified in FAA or ICAO regulation, long resolution times may indicate resource gaps or unclear responsibilities. Timely closure is a recommended best practice.
How to act: Track by severity and complexity and set target timelines to help prioritize.
3. Repeat or Recurrent Events
What it tells you: Effectiveness of corrective actions.
Why it matters: Indicates whether risk mitigations are working as intended. 14 CFR § 5.73(a)(3) requires organizations to evaluate the effectiveness of safety risk controls as part of their safety performance assessments and AC 120-92D provides guidance to meet this requirement.
How to act: Use root cause analysis and monitor closure effectiveness through followup audits or checks.
4. Safety Training Completion Rate and Currency
What it tells you: Workforce readiness and safety competency.
Why it matters: Safety Promotion relies on training and communication (14 CFR § 5.91)
How to act: Track training by job role and type, as well as plan refreshers based on the level of risk involved.
5. Audit Findings and Closure Rate
What it tells you: Responsiveness to internal oversight and external audits.
Why it matters: Timely closure shows accountability and a commitment to continuous improvement.
How to act: Highlight systemic findings and big-picture issues, track their recurrence, and measure closure timelines.
6. Safety Communication Engagement
What it tells you: Reach and clarity of your safety messages.
Why it matters: Clear, consistent communication helps reinforce safety priorities, shape behavior, and support a positive safety culture.
How to act: Track things like how many people open safety emails or newsletters, attend briefings, or scan QR codes linked to safety content. Adapt based on feedback.
7. Voluntary Safety Participation
What it tells you: How invested employees are in improving safety beyond their required duties.
Why it matters: Participation in safety initiatives often correlates with increased reporting and risk awareness.
How to act: Track involvement, reward contributions, and link participation to safety outcomes.
Go Deeper: Qualitative Indicators of Safety Maturity
SPIs provide a performance snapshot—but qualitative tools reveal the why behind performance trends.
Use tools like—
Safety culture surveys (for example, National Aeronautics and Space Administration’s (NASA) Aviation Safety Reporting System (ASRS) instruments),
Data from FAA Aviation Safety Action Program (ASAP) and Flight Operational Quality Assurance (FOQA) programs,
Focus groups and interviews to uncover trust and reporting barriers, and
Observational safety assessments during field audits or safety walkarounds.
These are especially useful for small or low-activity operators with limited data volume.
Practical Tips for SMS Measurement
Map SPIs to SMS components—Align each indicator with Safety Risk Management, Safety Assurance, or Safety Promotion.
Automate tracking—Use SMS software to simplify tracking and reporting.
Review regularly—Use monthly and quarterly trends to drive action.
Share results—Use dashboards or briefings to share results, celebrate progress, and build transparency and buy-in.
Final Thoughts
An effective SMS isn’t measured by documentation volume, but rather by its ability to improve decision making and reduce risk. When you combine thoughtful SPIs with frontline insights and leadership accountability, you strengthen the feedback loops that make your SMS meaningful, measurable, and sustainable. Measuring performance is not just a compliance exercise—it’s how you build a safer, more resilient operation.
Need help with SMS adoption?
PAI Consulting can help you measure your SMS effectiveness.
Contact us today for a consultation and let our expert Senior Aviation Analysts help your aviation SMS program succeed.
Contact us for a consultationDo you have questions or concerns about SMS? Share them with us!
PAI Consulting | Helping you succeed. Learn more.
Any opinions expressed in this article are those of the author and not the opinion of PAI Consulting. In addition, this article may contain links to third-party websites. PAI Consulting does not endorse or make any representations about them, or any information, software, or other products or materials found there, or any results that may be obtained from using them.
We welcome thoughtful and respectful discussion.
To keep this space safe and productive, please follow these guidelines:
Be respectful. Personal attacks, name-calling, and abusive language will not be tolerated.
Stay on topic. Keep comments relevant to the content of the post.
No spam or self-promotion. Links and promotions that are not relevant to the discussion will be removed.
Use appropriate language. This is a professional environment—please avoid profanity or offensive language.
Protect privacy. Don’t share personal information—yours or anyone else’s.
You may reply to, like, or flag other comments.
Comments are moderated and may be edited or removed at our discretion.
By commenting, you agree to abide by this policy.