A Winning Approach for Navigating Complex Data

As one of the largest U.S. commercial passenger airlines with an extensive network of domestic and international routes, Delta Air Lines supports 15,000 pilots flying a massive fleet of multiple aircraft types, each with numerous configurations, making their pilot training program the largest and most complex in the world.

Pilot training at Delta poses a unique challenge: data saturation. Across the organization, terabytes of data are collected daily and flowing in constantly from individual flights, irregular events, equipment use, maintenance operations, employee and business performance, and training programs. Bringing together huge amounts of different types of data in a meaningful way, correlating the data, and understanding implications on business and training performance is easier said than done.

That’s why Delta’s Flight Operations, Pilot Training team partnered with TiER1 to better understand the performance and effectiveness of their training programs across key performance indicators. Together we sought to answer this seemingly simple question: “How is our training program performing?” The team’s executive stakeholders and decision-makers needed a comprehensive view of Delta’s extensive pilot training footprint to compare the many components within its vast curricula. Together we defined the team’s goals:

• • Identify cost-effective training approaches
  • Troubleshoot and pinpoint issues in real time
  • Remedy issues in real time (or prevent issues before they even happen)
  • Turn data into actionable insights

We then worked together to build a single-source solution that demonstrates and visualizes pilot training program effectiveness and provides real-time, role-based actionable insights. Keep reading to learn more about the team’s winning approach.

Making sense of the data

Delta, like many organizations, has a lot of data that are collected and managed by siloed groups in various departments. Those groups analyze and report on the data to their functional leaders, and that is often where the story ends.

The hard truth is, no single data source can answer a complex question. In the parable of the wise men meeting an elephant in the middle of a dark night, when each one examined just one portion of the elephant—the trunk, the tail, the tusk—independently of each other, they arrived at the wrong conclusion, thinking it was a garden hose, a broom, or a very large spear. The key is to combine all findings to understand the full picture: the elephant. In Delta’s case, the “elephant” is the entire pilot training footprint.

Many organizations use simple post-training surveys to determine the success of their training programs. Unfortunately, the data are not reliable to determine training effectiveness. Learners may be happy with the training program and provide positive feedback, but this doesn’t reflect their actual performance in post-learning evaluation (grades). And if learning doesn’t transfer to tangible business performance (e.g., pilot flying skills or safe aircraft operation), the training program is ineffective, even with stellar post-learning evaluation and learner feedback. (For an example of learning evaluation done well, visit here.)

Collaboration

Historically, data—pieces of the “elephant”—have been siloed at Delta across various data management teams. To build a shared collective understanding of the available data generated, we supported the team’s identification of data owners and channels of communication across these siloed working groups. By creating collaborative networks with these executive decision-makers, the team is empowered to design solutions that address the needs of the business in a meaningful way.

Data integrity

When building a database for analysis, actionable insights are only as good as the supporting data. To see the entire “elephant,” you need reliable information to guide you toward the correct actions. The consequences of faulty decisions based on bad data can be extremely costly and dangerous in the airline industry; data integrity is critical for Delta.

We considered the following steps to build a reliable database to inform actionable insights:

1. 1. Identify all available data sources. What data are currently collected? Who owns which data? Where are the data stored? In what application and format?
   2. Select and use the relevant data and discard the rest. Which data are relevant to the question we are trying to answer? In Delta’s case, flight performance, flight safety, and pilot errors are directly correlated to learning and are necessary to understand training effectiveness.
   3. Identify any missing data showing up as gaps in the big picture. To fully understand Delta’s training effectiveness, we collected feedback from training instructors. Qualitative data from instructors helped answer questions regarding specific issues during training with training equipment (such as simulators and flight training devices) that often led to training delays and additional costs.
   4. Determine data quality: Garbage in, garbage out. How reliable are your data? How “clean” are they? Are they comprehensive enough to provide a meaningful insight? For example, post-training surveys returned by only a few participants are unreliable in determining how the program is perceived by the whole learning group.
   5. Include qualitative data to answer the “why.” Numerical data inform us of performance gaps and potential opportunities for improvement (the “what”). Yet they cannot show how we got here and how we move forward (the “why”). Qualitative data is crucial when searching for actionable insights. When used carefully with reliable quantitative data, qualitative data can help reveal the reasons behind the performance issues highlighted by the numbers.

Bringing data to life

After identifying and correlating the right data sources, our next step was to build a user-friendly, single-source solution that effectively visualized the findings and provided the insights for making decisions about necessary changes to the training footprint. We recommend the following steps to anyone designing a similar solution:

1. 1. Select the right tool. Combining multiple data sources with different formats that are hosted in various data warehouses or applications requires the right tool that can bring together all your data under one roof and can offer the necessary capabilities for data collection, mining, and visualization to process and present the data in a meaningful way. Based on Delta’s needs, we considered tools with capabilities such as robust survey design (e.g., branching, filtering, advanced text analysis), advanced visualization (e.g., highly configurable dashboards capable of combining multiple data sources), and 24/7 access to up-to-the-minute live data on any device.
   2. Consider your audience. Executive stakeholders and decision-makers are focused on the business. They are not necessarily data experts. The focus should be on translating raw data into information, and then the information into digestible, intuitive, and actionable insights for stakeholders and decision-makers. This translation is a key step to help bridge the gap for non-data experts when it comes to understanding what the data mean, how they affect the operation, and what actions should be taken.

When designing Delta’s dashboard solutions, we applied three key principles: simplify, visualize, personalize. Using these strategies helped us create a user experience that made it easy to access use-case-specific information quickly and efficiently, while also allowing users to filter or modify their views and search results based on a specific issue or focus area.

• • Simplify. The goal is to convey information in clear, easy-to-understand, digestible chunks. Focus on the most important pieces. Show what needs to be seen first. Do not fall into the trap of trying to squeeze everything on the visible screen area. Move from the general to the more detailed. Once the most important (or top level) information is surfaced, allow users to dig deeper and drill down to the details.
  • Visualize. A picture is worth a thousand words. Intentionally use color coding to help draw the eye to the most important information. Highlight the “big picture” or overall results using large font and diagrams. Apply consistent standards for page organization so that no matter which dashboard the user is viewing, they can always see the same type of information in the same areas of the page.
  • Personalize. Users need to access the information they need in the shortest time possible. Identify unique use cases—the different ways that users need to see the data in their distinct roles. Build personas around each use case. Design persona-specific dashboards showing different views of the same data to surface relevant information based on real needs. Apply multiple filters to allow users to focus on specific information; filters for Delta included fleet types, specific training programs, target pilot functions, tenure, and age groups.

Rolling out the solution

We introduced the pilot training dashboard solution through a carefully crafted adoption plan. The dashboards were first shared during the kickoff meeting at the start of Delta’s recurring training design process where all key stakeholders and designers were present. Using the dashboards, the team demonstrated the results of the previous nine-month training cycle and its impact on learners, training results, and business performance. The dashboards surfaced practical insights with immediate applicability for how to move forward with training design for the upcoming training cycle. We answered questions about the data and the dashboard itself. Each stakeholder also received access to a persona-specific dashboard so they could discover and explore the tool on their own.

In the following weeks, each “persona group” (users with the same dashboard view) received basic data literacy training plus targeted dashboard user training on how to use the tool, access different views, and filter results to dig deeper. Time was carved out for step-by-step practice of the most common daily uses for researching data and insights, answering questions along the way. We also provided a detailed user guide for reference and set up a support line to address questions and issues.

The reception of the solution was extremely positive and was promoted by top executives across the Fight Operation training organization, and it’s still being used by the organization today. With the number of role-based users continuing to expand, the training team continues to add new personas for the dashboard. Building these persona-based pilot training dashboards provided an exciting opportunity for both Delta and TiER1 to solve for the problems of data saturation by bringing together people and translating complex data to easily consumable and meaningful insights, accessible anywhere, anytime.