Let’s start with the key question: what is extended reality (XR)? Although the exact definition remains a matter of debate, it broadly refers to the wider industry encompassing augmented (AR), virtual (VR) and mixed reality (MR) and while it is not an entirely new concept, we have seen an explosion of interest in XR training and simulation within high-risk industries. What better use than within SAR and HEMS training, which needs to meet the myriad of clinical, aviation, and maintenance priorities?

Simulating reality

The use of simulation in training is nothing new. Crude versions of medical simulation mannequins can be traced back for centuries and remain, albeit with increasingly acutely realistic models, and flight simulation training has existed in various forms for as long as people have been taking to the air.

The different branches of XR are well developed and used in consumer and industrial environments:

• A typical usage of AR employs smart glasses or smartphones with passthrough vision, allowing data to be overlayed onto the real world
• VR uses headsets that completely immerse the learner in a virtual and interactable environment using specialized controllers, often with haptic feedback
• MR usually refers to combining a virtual reality headset with the physical environment, such as using VR goggles while driving an actual steering wheel and pedals

The first virtual reality machines date back more than a half century. However, in the last decade, significant advancements in VR technology and drastically lower prices have allowed its use to become relatively popular with consumers, including within computer gaming. The current push to eliminate the need for robust and expensive computers through standalone VR headsets like the Oculus Quest 2 has significantly reduced the cost of entry. Although the gaming industry is one of the most public-facing driving forces behind VR development, we as educators have benefitted as much as the consumers. The sudden decrease in price and the explosion of developers has opened an entire industry of XR training at a lower price point than ever before.

Meeting clinical demands

Over recent years, VR has gone from a fringe idea in healthcare education to a hot topic among medical simulationists. Current research into the use of VR for clinician training, published in the Journal of Medical Internet Research in March 2020, shows knowledge retention that meets or exceeds traditional physical simulation at cost savings of as much as 83% over mannequin-based training. Turning any empty meeting room into a fully functional simulation training center can significantly reduce overhead and allow for various scenarios to be conducted that would otherwise be unrealistic due to the surrounding environment.

Ryan Ribeira, CEO of SimX, a VR training company based in California, US, believes that the pandemic has driven a lot of the recent interest in virtual training. The SimX software platform allows multiple users to be present in the same virtual scenario while physically being thousands of miles apart. This ability to conduct high-fidelity simulations without the need for in-person meetings or travel has an obvious appeal when many organizations are struggling to protect their employees from exposure and navigate travel restrictions.

When explicitly asked about the use of SimX in HEMS, Ribeira stated: "The power of VR is that, in theory, it lets you simulate and track almost anything. This is especially true in HEMS, where we can create an immersive scenario requiring the crew to continue caring for a patient while also dealing with a mechanical failure. Outside of VR, this would be difficult, expensive, or impossible to simulate."

CAE, a Canada-based company that creates aviation and healthcare simulation equipment, has taken a different approach to implement XR in clinical training. Using the Microsoft HoloLens, CAE has designed a series of augmented reality programs that project 3D animations on their physical simulation manikins. When paired with their high-fidelity ultrasound trainer Vimedix, the HoloLens shows the user internal organs in their correct location. Following voice commands, the software can elevate the organs to float above the body where the learner can manipulate the projection. While this application requires the use of specific CAE mannequins, it highlights the possibility of AR projections that can be custom made to extend the working life of ageing patient simulators reducing the cost of repairs and the necessity of replacement.

Early adopters of XR in flight training

The use of XR can also provide improved training or reduce the cost of current training within air medical and SAR operations, including pilot training. Luckily, we don’t have to look far for examples.

The United States Air Force (USAF) has been using MR cockpits – combining VR headsets with realistic physical controls – to train pilots for several years. According to Task & Purpose, in 2018, the USAF program, Pilot Training Next, found that the use of virtual reality cut pilot training time in half. A year later, the UK Royal Air Force (RAF) partnered with USAF Pilot Training Next to provide VR training to their recruits, citing the reduction in training time to help combat the military pilot shortage. At the time, Colonel Paddy Logan, Assistant Director for Flying Training at the RAF, commented: “We don’t have the capacity to experiment this way, so having this partnership and people here to learn what the [USAF] is doing is invaluable.”

When focusing on rotor-wing operations, the USAF has also been an early adopter of XR. The Air Force reported reducing pilot training time by 35% while being able to add an additional 15 hours of training. The program, nicknamed Project Da Vinci, included 23.5 hours of VR flight training and resulted in students being able to ‘hover, taxi, and perform various other helicopter maneuvers unassisted by their instructor pilots on their very first flight’ in an actual aircraft.

Outside of the military, US-based training company, FlightSafety International, markets its Mixed Reality Flight program as an economical way to augment pilot training and offer virtual cockpits for both fixed-wing and rotor-wing aircraft. With options geared for both government and civilian operators, the MR offerings are capable of familiarizing pilots in new airframes, training basic procedures, and practicing emergency procedures. Depending on the customer's needs, the hardware can be as simple as a laptop and basic spring-centering controls or as advanced as a full-motion platform and type-specific controls, and all options employ a VR headset to provide the necessary visuals of the cockpit and the aircraft's surroundings.

These examples are only a tiny sample of the many XR pilot training applications available. It is crucial to recognize that XR is an additional tool that can augment education and reduce time on the aircraft by preparing for real-world experiences in a controlled environment. XR is not, and likely never will be, a total replacement for instructor-led training or time handling an aircraft. However, in April 2021, the European Union Aviation Safety Agency granted the first certificate for a VR-based Flight Simulation Training Device, particularly considering addressing key risk areas in rotorcraft operations, where statistics show around 20 per cent of accidents occur during training flights, and opens the door for rigorously tested and effective XR training opportunities.

XR in maintenance training

The use of XR in training maintenance professionals may well bypass its use in healthcare simulation. The technology lends itself well to recreating any airframe and part in precise detail for VR scenarios. An AR heads-up display can scroll real-time instructions and crucial data to a maintainer's smart glasses while completing an inspection or repair. Like clinical and aviation training, the technology is already used elsewhere in the aviation industry, so a precedent exists for its use in HEMS.

Airbus, which makes more than half of the global HEMS fleet, is potentially the most significant proponent of VR maintenance training in the aviation industry. The use of XR headsets at Airbus spans back to 2016, when a video demonstrating AR glasses projecting a 3D rendering of an aircraft interior was publicized. XR is used across several divisions within the company. In 2019, Airbus emphasized a 75-per-cent reduction in verification and validation times compared to traditional digital mock-up and computer-based modelling methods. A press release explicitly stated that ‘virtual reality is truly helping us with our real daily tasks while reducing time and costs’.

The USAF has turned to XR for enhancing maintenance training and efficiency. AR headsets or smart glasses allow for virtual overlays to be projected into the mechanic's actual environment, allowing them to work on an aircraft with necessary information readily available. The cameras in the smart glasses allow for recording the activities for playback. In some instances, a live video feed can be shared to enable a remote third party to provide real-time instruction to the wearer.

XR is already in use throughout the healthcare and aviation industries. The use of smart glasses to improve maintenance and warehouse safety will likely become commonplace within the next decade. With the emerging technology and current market for hiring custom content developers, we will likely see a surge in XR use within HEMS training and operations. The ability to create entire virtual environments has endless potential for clinical, aviation, and maintenance training.