Background

Virtual reality (VR) is used by pilots to hone their skills, with current technology most commonly integrating VR headsets or multi-projected environments – sometimes in combination with physical environments or props – to generate realistic images, sounds and other sensations that simulate a user’s physical presence in a virtual or imaginary environment. VR systems that include transmission of vibrations and other sensations to the user through a game controller or other devices are known as haptic systems. This tactile information is generally known as force feedback video gaming and training applications.

Augmented reality (AR) systems are also sometimes considered to be a form of VR – one that layers virtual information over a live camera feed into a headset or through a smartphone or tablet device, giving the user the ability to view three-dimensional images. AR – which is related to two largely synonymous terms, mixed reality and computer-mediated reality – is an interactive experience of a real-world environment, where the objects that reside in the real world are ‘augmented’ by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. The overlaid sensory information can be constructive (i.e. additive to the natural environment) or destructive (i.e. masking of the natural environment) and is seamlessly interwoven with the physical world to such a degree that it is perceived as an immersive aspect of the real environment.

In this way, AR alters one’s ongoing perception of a real-world environment, whereas VR completely replaces the user’s real-world environment with a simulated one. The application of VR and AR to the air medical world comes in very useful for pilots, medics and mechanics.

Aviation application

When it comes to the cost of training for pilots, headsets that make use of VR are but a few thousand dollars and are thus infinitely more accessible to organisations with limited training budgets than FFS are. They also offer the possibility of training in remote locations – something that is of particular use to the military.

FFS have made use of VR for some time, but new uses of the technology are bringing more possibilities to the foreground. Toll Helicopters in Australia, for instance, has joined forces with Seeing Machines to launch an eye-tracking technology trial as part of Toll’s AW139 FFS at the ACE Training Centre. The trial is based on Seeing Machines’ Crew Training System prototype, which incorporates modern eye-tracking technology. The system supports aviation training instructors, pilots and crewmen with evidence-based data to provide detailed insights into pilot scanning techniques and situational awareness.

“The unique technology provides instructors [with] an objective assessment of what their trainees are looking at (such as speed tape, height, glide slope, heading, flight mode annunciator) at any given point in time during a flying sequence. The information can be used to quickly identify scan breakdowns, missed information, crew resource management (CRM) attention distribution, and standard operating procedure / Flight Crew Operating Manual (FCOM) adherence or deviation,” said Toll Group spokesperson Diane Tremain.

Darryl Humphreys, AW139 Standards Manager and Flight Examiner, described the tool as a ‘valuable asset’ for instructors and pilots. “For instructors, the ability to see in real time where a student’s eyes are tracking is remarkable,” he said. “You can observe, analyse and interpret the pilot’s situational awareness. In addition, having the eye tracking auto-recorded for replay during training debriefs offers pilots a comprehensive review tool and the ability to self-remediate areas for improvement.”

Japan Airlines (JAL) is one of several that is making use of enhanced technology to train flight crew. According to a Microsoft spokesperson, JAL is using the Microsoft HoloLens to train both pilots and mechanics. For the pilots, who had until recently been using videos and printouts of cockpit panel instruments and switches, the HoloLens means that trainees can convert ‘intellectual memory into muscle memory’, according to Koji Hayamizu, Senior Director of Planning for JAL’s Products & Service Administration Department.

Aircrew application

While pilots have been using simulators for years, and maintenance crew are starting to make inroads into its application, elsewhere in the air rescue sector, hoist crew are seeing more products come online that are being developed to aid in their skills development.

Off Planet Simulation is an offshoot of the Becker Group in Australia, which provides military flight training. In 2018, Off Planet Simulation was established with the task of developing training devices to meet specific needs, one of which was air crew training. Designed to meet the requirements of military and civilian SAR / EMS operators, the VR simulation device provides the ability to integrate rear crew procedural training and operational rehearsal in a virtual environment within a traditional static trainer.

Priority1 Air Rescue has two SAR Tactical Training Academies in the US and France. The company states: “Increasingly, effective mission training is not only accomplished by focusing solely on live flight training, it is now significantly enhanced by employing blended programs utilising synthetic aircrew training and use of virtual simulation.” It offers several courses, one of which is a virtual helicopter hoist SAR course using synthetic training devices.

The ACE Training Centre, run by Toll in Australia, offers training to operational and clinical crews from across the country, as well as internationally. The system was originally designed for military use, and Toll has taken the base line operating system and made it bespoke for their EMS requirements.

Colin Gunn told AirMed&Rescue more about how VR is used in the Centre: “The VR component of our training programmes utilises the Complete Aircrew Training System (CATS), which provides a high-fidelity and immersive VR-based training environment for aircrew and air medical crew. Traditionally, operators and trainers are reliant on utilising a live aircraft for training to maintain currencies and proficiencies. Through VR simulation, ACE is achieving a majority of these currencies and proficiencies, with a level of reliability and realism that cannot be achieved safely using a live aircraft. For example, the aircrew can practise winching a simulated intubated patient in significantly degraded environmental conditions that cannot be achieved or simulated in the live aircraft.” 

This May, Canada’s Bluedrop Training & Simulation Inc. secured an investment from Boeing to develop a next-generation Special Mission Aviator Ramp Trainer (SMART) for the V-22 Osprey. The program was funded under the Innovation, Science and Economic Development Canada (ISED), Investment Framework Transaction program by Boeing. The development phase is expected to take between 12 and 18 months to build a full-sized VR ramp trainer with a high-fidelity immersive environment, and enhanced specific mission capabilities including ramp operation, hoisting systems, and various mission critical procedural training capabilities.

Bluedrop has also been tasked with developing hoist mission training systems (HMTS) to the Royal Canadian Air Force 19 Wing Comox for the CH-149 Cormorant. The VR simulator provides high-fidelity cable behaviour that responds to flight dynamics and operator inputs with accurate turbulent flow zone rendering and complex rescue co-ordination scenarios. “Our HMTS is proving to be the standard for rear-crew training. We are so happy to be contributing to SAR operations training in Canada. Canadian SAR capabilities are a national asset and priority. Bluedrop values being able to improve operational readiness and mission effectiveness through improved training of these skilled operators in some of the harshest rescue environments,” said Jean-Claude Siew, Vice-President of Technology & Simulation.

Medical application

In the US, the Tele-Critical Care Unit at Naval Medical Center San Diego is going one step further and using AR-capable helmets for their field medics. Personnel in San Diego can provide real-time guidance to providers in the field using this technology.

AirMed&Rescue spoke to Captain Konrad Davis, Director, Navy Tele-Critical Care and Acting Director of the Virtual Medical Center in San Diego, which is going to start a trial by the end of this year in which the use of AR by field medics will be examined.

Applications for the US Navy could be far-reaching, particularly onboard ships where there are no doctors or nurses onboard, but those who have medical training could be talked through complex procedures by a trauma surgeon on land

Applications for the US Navy could be far-reaching, particularly onboard ships where there are no doctors or nurses onboard, but those who have medical training could be talked through complex procedures by a trauma surgeon on land, for example. “The tools, technology and processes to extend specialist expertise into the operational environment is what’s being trialled,” explained Captain Davis. “The Army and Air Force have been granted funding to explore the impact of AR as part of critical decision-making processes.”

A trial using a reperfuse cadaver model will hopefully be completed by the end of 2020, the results of which will establish the future direction of the technology in the military. In addition, the US Army has a contract with Microsoft to develop a field-ready version of the firm’s HoloLens technology.

The funding of such trials is key, and probably why the military is going to be applying this technology before the civilian HEMS/rescue sector, although there is no doubt it will be of use to both. “The military personnel are in an isolated environment, and so need the additional assistance from specialists,” said Captain Davis. “They may not have the tools or skills necessary to respond and AR can make a real difference to the medical resources available.” The same could apply to HEMS operators that carry paramedics instead of doctors.

A challenge that must be overcome if the technology is to be successfully applied, though, is bandwidth, said Davis.

Maintenance application

For the JAL mechanics, meanwhile, Hayamizu pointed out that VR means they can ‘study and be trained just as if they were working on the actual engine’. No more waiting for an appointment or available aircraft. A spokesperson for Microsoft told AirMed&Rescue: “With HoloLens, customers can flexibly train employees at a lower cost and with high-quality results. Employees can learn by doing, with hands-on and interactive instructions, allowing them to learn new skills faster and with fewer errors.”

Helicopter manufacturer Leonardo, meanwhile, has developed its HeliLink digital product support engineering assistant, which provides a remote video-call support for maintenance technicians with AR. The company said that this ‘results in quick assistance for trouble shooting, improving helicopter fleet reliability and management’.

United Technologies Research Centre is working with Pratt & Whitney’s customer training division to invest in VR engine maintenance training for mechanics, with testing underway that allows personnel to go ‘inside’ a Geared turbofan (GTF) engine to examine certain parts and view a running engine in motion.

Current developments

NLR, based in the Netherlands, is working in its X-lab (‘X’ standing for eXperimentation and X-reality, which refers to a wide range of simulation technology) to create a mixed-reality training platform using VR and AR. The company points out that AR / VR technology allows for more flexible training than traditional Full Flight Simulators (FFS) do.

“The trend I observe is that training and simulations are becoming increasingly cheaper, more flexible and simpler to use,” noted Roy Arents, one of the creative minds behind X-Lab. Harrie Bohnen, Manager of NLR’s Training, Simulation and Operator Performance department, explained that the company does not actually produce the training course that aviators or maintenance personnel would use, but instead ‘brings together applied research, unique knowledge and expertise of aviation and new virtual technologies so as to develop new, innovative training concepts and simulations together with the user’. NLR is working with the Royal Netherlands Air Force, KLM Royal Dutch Airlines, the European Aviation Safety Agency and the US Federal Aviation Administration, which it says helps to promote the exchange of ideas between civilian and military operations.

NLR is working with the Royal Netherlands Air Force, KLM Royal Dutch Airlines, the European Aviation Safety Agency and the US Federal Aviation Administration, which it says helps to promote the exchange of ideas between civilian and military operations

Anneke Nabben, one of the training specialists from NLR, teamed up with KLM on a project involving maintenance personnel using Microsoft’s HoloLens. She explained: “We wanted to devise something that would give students a greater insight into how one of the aircraft systems works. To do this, we focused the training more on cultivating understanding than on knowledge of facts. We came up with tasks that encouraged students to work with each other. We also got the instructor to impart the subject matter interactively to make it easier for the students to absorb. Technically, we supported this collaboration by interconnecting multiple HoloLens goggles to allow everybody to see the aircraft from his or her own perspective.”

Reaching new heights

The pace at which AR technology is developing suggests that it won’t be too long before there is an application for all staff members involved in SAR and airborne special missions. With many budgets becoming increasingly tight for operators, the ability to choose a more cost-efficient way of training all staff on one device, which can run multiple programs, is doubtless an attractive option.

This article originally appeared in the November 2019 issue of AirMed&Rescue