The INTREPID project intends to increase the data available to FRs dramatically, in the form of sensor data, positioning information or alerts and recommendations. This wealth of information, comes with its own challenge though: how to display and share it quickly and efficiently, in a context where every minute counts and lives are at stake?

INTREPID has chosen eXtended Reality (XR) as an innovative way to display information for increased situational awareness. XR covers three potential modes: Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR).

Virtual Reality (VR): the full immersive solution

VR is well known by now, thanks to the existence of various levels of consumer-grade products, from the Google Cardboard to UHD Pimax helmets. It immerses the users in a fully virtual world, and allows them to interact with it. 

The INTREPID platform, through its Digital Mock-Up module, gathers not only geographical data and existing 3D models, but also updated 3D representations of the area, extracted from real-time data acquired by INTREPID drones and robots (Environment Mapping Module) as well as data from on-site sensors and cameras. This allows for a life-size, real-time representation of the disaster area. 

Figure 1: VR view includes existing geospatial information and models, and real-time data from UxV and sensors.

When in operation, however, VR cuts the user from his or her surroundings. It can thus have an impact on the workflow in the command centre. An extensive use can also lead to physical uneasiness (nausea, motion sickness). In training though, with an adapted curriculum, VR would allow the user  to experience being there, without actually having to set up a mock incident. Even better, the virtual world is not limited to a representation of its physical counterpart. It can make invisible threats visible, and display additional information from the INTREPID system right where needed. 

Augmented Reality (AR): the best of both worlds

AR consists in adding virtual elements provided by the system to the real world (seen either through a camera, or through the very eyes of the user). 

It comes in all shapes and sizes, from the more mundane (playing Pokemon Go on a smartphone or trying on virtual glasses on a website) to the most serious, life-saving use cases (surgery assistance or assistance to First Responders). With such a wide array of possibilities, capabilities strongly depend on the device used, the context and the functionalities of the system as a whole. 

In the INTREPID project, solutions based on dedicated wearable devices (AR glasses) are favoured during the analysis phase, since they offer advanced display capabilities, and are able to closely intertwine visual metaphors with the real world without forcing the user to shift their gaze or use their hands. 

Such devices would give the First Responders a global, rich understanding of their surroundings through: INTREPID sensor data (for instance, gas concentrations); data from other actors (for instance positions of the other team members); or even alerts and recommendations from the INTREPID Intelligence Amplification components (for instance, highlighting the path to the nearest victim, identifying a target area, providing alerts about nearby CBRN threats…). FRs immediately receive all the relevant information, and only the relevant information. 

However, to obtain a seamless, efficient and unobtrusive integration of both worlds (real and virtual), positions and orientations should be determined precisely, not only for the situational data, but also for the AR device. Highly detectable “points of interest” can be used to track environmental features and extract knowledge about the position and orientation of the device. With INTREPID, though, the platform could be deployed anywhere and the presence of points of interest cannot be guaranteed. The AR mode would then rely on the system’s own real time indoor/outdoor positioning capabilities. 

Figure 2: Hololens-wearing user receives navigation assistance (up) and alerts (rights) when exploring a building

Mixed Reality (MR): the whole situation in the palm of the hand

MR (Mixed Reality) can be considered as an advanced version of Augmented Reality. It also consists in bringing virtual elements into the real world. Yet, instead of displaying them as passive additions to the reality, the MR rendering takes into account the physical elements. It also allows the user to interact finely with the virtual components of the scene, blurring the limit between real and virtual. 

This is embodied in INTREPID’s Holographic Common Operational Picture (Holographic COP). In this concept, a virtual representation of the disaster area is built and displayed for the connected users through AR devices, like a mock-up before their eyes. Like its physical, static cardboard counterpart, this scale model displays the 3D representations of the area, but its MR nature comes with unparalleled features, such as: navigation and manipulation at the snap of a finger, real-time situational information such as the position of actors or threats, regular updates throughout the crisis or the ability to go back and forth in time for an after-action review.

Figure 3: Hololens-wearing user viewing the COP when handling incidents on a cruise boat 

Advanced visualisation components based on eXtended Reality come hand in hand with advanced situational awareness and decision-support capabilities. The vast amount of information, acquired or generated, allows a detailed, up-to-date and rich rendering of the situation. Conversely, such an amount of data requires visualisation options that go beyond a 2D computer screen. The XR modes considered in INTREPID offer multiple possibilities, and the various workshops organised in the scope of the project will help narrow down the better and most relevant uses to harness the system capabilities. 

Post-disaster rescue operations are hazardous by nature, carried out in very uncertain, unknown environments, all the while relying solely on the perception of First Responders. Gathering and sharing information is done at a high risk.

Extended reality for early rescue operations:  the INTREPID approach

The INTREPID approach for extended reality

Both man-made hazards (anthropic) and natural hazards (earthquakes, floods or volcanic eruptions)[1] have unfortunately become part of our daily lives, and the work of first responders in emergency situations has become increasingly important.

Early detection, prompt identification, control and/or neutralization of hazardous substances and an accurate delimitation of the risk area are fundamental premises to optimize the effectiveness of first responders, thus avoiding loss of lives and mitigating damage to the environment. From this perspective, the use of technology aimed at first responders in emergency situations can play a fundamental role, helping to detect, reduce and mitigate the risks and dangers that may appear in a disaster.

The development of technological solutions to support first responders in emergency situations has given rise to a line of action within the European Union Research and Innovation Programme Horizon 2020, where a list of projects have been funded to support such technological development.

The INTREPID project is one of this project aimed at developing technology for first responders. It has been funded by the European Union within its H2020 Research and Innovation Programme. The Spanish School of Rescue and Detection with Dogs (ESDP) is part of the INTREPID consortium, whose main challenges are to develop technologies aimed at improving safety and efficiency in the detection of risks, guaranteeing a rapid, dynamic and safe response by the first intervention teams and, of course, by the canine teams (K9) as first responders.

One of the possible risks that can appear in a disaster and that is not always immediately perceptible are chemical risks. For this reason, chemical detectors are one of INTREPID's key strengths, and our work consists of supporting to make them as efficient as possible.

HOW CAN CHEMICAL DETECTORS HELP K9 TEAMS? 

For a right assessment of hazards in the area, the following aspects are of great importance and can be faced using technology: On the one hand, the recognition and evaluation of the area. For this purpose, it is important to determine the chemical detectors, which must be used BEFORE the first responders´ entrance, both guides and dogs. Drones and robots, technologies that are also included in the INTREPID project and which, equipped with these detectors, will play an essential role in assessing the danger, securing the intervention area and thus guaranteeing the safety of the first responders, rescue dogs and victims.

On the other hand, after the recognition and evaluation of the area, the detectors worn by the first responders´ uniforms respond to different purposes which can be described as follows:

The continuous assessment of the safety margins that need to be maintained while the rescue work is being carried out.

Continuous control and correction, giving the possibility of warning of possible changes or the appearance of new leaks that could endanger the lives of the first responders involved.

Detection of new substances that did not appear in previous readings and assessment.

In order to achieve what is called "Operational Hygiene"[2], risk elimination or reduction mesures must be established throughout the rescue manoeuvre. 

We not only need to know what we are dealing with, but also some additional information as: the concentration of chemicals, the size of particles, the exposure time, the combination of pollutants or the detection of new hazards. The detection of these aspects using technological solutions is a significant challenge which provides considerable added value to the project's results.

Technology is setting a new horizon in disaster work, helping rescue teams to achieve better results in less time and increasing the safety and efficiency of first responders and rescue dogs.

Ana Aldea Reyes | Marta Burgos González | Susana Izquierdo Funcia

Escuela Española de Salvamento y Detección con Perros /  @ESDP_esp 

[1]Salvago B. La amenaza riesgos, emergencias y catástrofes. La función de inteligencia en la gestión de emergencias y catástrofes, págs. 23-50. 2019.

[2] Baraza X, Castejón E, Guardino X. Higiene industrial. I- Introducción a la higiene industrial. Metodología de actuación. Higiene operativa.2016

Threats and dangers come to humans in many different ways. Some of them can be perceived with the naked eye, making us aware of their existence and allowing us to take measures to protect ourselves. However, other risks and dangers are hidden, causing damage without being seen or perceived by us, which makes more difficult to take protective measures.

Chemical detectors, searching for the silent hazard

Articles

The INTREPID approach for extended reality

Chemical detectors, searching for the silent hazard

Let's keep in touch

Newsletter