Ani Matrai-Monk – Sr. Consultant This article originally appeared in The Data & Measurement Issue of Performance Matters magazine. Download a digital copy of the magazine to read more on this topic.
As one of the largest U.S. commercial passenger airlines with an extensive network of domestic and international routes, Delta Air Lines supports 15,000 pilots flying a massive fleet of multiple aircraft types, each with numerous configurations, making their pilot training program the largest and most complex in the world.
Pilot training at Delta poses a unique challenge: data saturation. Across the organization, terabytes of data are collected daily and flowing in constantly from individual flights, irregular events, equipment use, maintenance operations, employee and business performance, and training programs. Bringing together huge amounts of different types of data in a meaningful way, correlating the data, and understanding implications on business and training performance is easier said than done.
That’s why Delta’s Flight Operations, Pilot Training team partnered with TiER1 to better understand the performance and effectiveness of their training programs across key performance indicators. Together we sought to answer this seemingly simple question: “How is our training program performing?” The team’s executive stakeholders and decision-makers needed a comprehensive view of Delta’s extensive pilot training footprint to compare the many components within its vast curricula. Together we defined the team’s goals:
We then worked together to build a single-source solution that demonstrates and visualizes pilot training program effectiveness and provides real-time, role-based actionable insights. Keep reading to learn more about the team’s winning approach.
Delta, like many organizations, has a lot of data that are collected and managed by siloed groups in various departments. Those groups analyze and report on the data to their functional leaders, and that is often where the story ends.
The hard truth is, no single data source can answer a complex question. In the parable of the wise men meeting an elephant in the middle of a dark night, when each one examined just one portion of the elephant—the trunk, the tail, the tusk—independently of each other, they arrived at the wrong conclusion, thinking it was a garden hose, a broom, or a very large spear. The key is to combine all findings to understand the full picture: the elephant. In Delta’s case, the “elephant” is the entire pilot training footprint.
Many organizations use simple post-training surveys to determine the success of their training programs. Unfortunately, the data are not reliable to determine training effectiveness. Learners may be happy with the training program and provide positive feedback, but this doesn’t reflect their actual performance in post-learning evaluation (grades). And if learning doesn’t transfer to tangible business performance (e.g., pilot flying skills or safe aircraft operation), the training program is ineffective, even with stellar post-learning evaluation and learner feedback. (For an example of learning evaluation done well, visit here.)
Historically, data—pieces of the “elephant”—have been siloed at Delta across various data management teams. To build a shared collective understanding of the available data generated, we supported the team’s identification of data owners and channels of communication across these siloed working groups. By creating collaborative networks with these executive decision-makers, the team is empowered to design solutions that address the needs of the business in a meaningful way.
When building a database for analysis, actionable insights are only as good as the supporting data. To see the entire “elephant,” you need reliable information to guide you toward the correct actions. The consequences of faulty decisions based on bad data can be extremely costly and dangerous in the airline industry; data integrity is critical for Delta.
We considered the following steps to build a reliable database to inform actionable insights:
After identifying and correlating the right data sources, our next step was to build a user-friendly, single-source solution that effectively visualized the findings and provided the insights for making decisions about necessary changes to the training footprint. We recommend the following steps to anyone designing a similar solution:
When designing Delta’s dashboard solutions, we applied three key principles: simplify, visualize, personalize. Using these strategies helped us create a user experience that made it easy to access use-case-specific information quickly and efficiently, while also allowing users to filter or modify their views and search results based on a specific issue or focus area.
We introduced the pilot training dashboard solution through a carefully crafted adoption plan. The dashboards were first shared during the kickoff meeting at the start of Delta’s recurring training design process where all key stakeholders and designers were present. Using the dashboards, the team demonstrated the results of the previous nine-month training cycle and its impact on learners, training results, and business performance. The dashboards surfaced practical insights with immediate applicability for how to move forward with training design for the upcoming training cycle. We answered questions about the data and the dashboard itself. Each stakeholder also received access to a persona-specific dashboard so they could discover and explore the tool on their own.
In the following weeks, each “persona group” (users with the same dashboard view) received basic data literacy training plus targeted dashboard user training on how to use the tool, access different views, and filter results to dig deeper. Time was carved out for step-by-step practice of the most common daily uses for researching data and insights, answering questions along the way. We also provided a detailed user guide for reference and set up a support line to address questions and issues.
The reception of the solution was extremely positive and was promoted by top executives across the Fight Operation training organization, and it’s still being used by the organization today. With the number of role-based users continuing to expand, the training team continues to add new personas for the dashboard. Building these persona-based pilot training dashboards provided an exciting opportunity for both Delta and TiER1 to solve for the problems of data saturation by bringing together people and translating complex data to easily consumable and meaningful insights, accessible anywhere, anytime.