SOCM, or Symbiotic Operation Control Module, is the way to create virtual robots able to integrate their different capabilities in order to carry out tasks that either cannot be done by a single robot or because it will be more efficient to do it by combining several functionalities of several robots. INTREPID’s entity responsible for this module is CERTH. 

As an example of this symbiosis, one may imagine drones (UAV), which are not able to open a door. In order for a drone to scan an area that is block by a closed door, a terrain robot (UGV) should assist and open the door. SOCM will handle the sequencing of the different mini tasks, once the drone has been given the mission to scan a closed area. SOCM will ask the UGV to come in to help an unblock drone’s way by opening the door. By combining their functionalities, the drone can enter the premises and do the scanning. 

In more detail, the overall procedures that will take place are the followings:  

The INMOS system will receive a request to locate a place and then the IAM module will advise that it cannot do that with a single drone, and that it needs to combine the functionalities of the drone and the UGV.  

To do so, IAM will send a mission to the SOCM saying that it wants a virtual robot combining a drone and a UGV to do this mission.  

Therefore, SOCM will handle the way the 2 robots (drone and UGV) will collaborate to fulfil the task at hand.  

Symbiotic Operation Control Module (SOCM) is used for:

Optimization for Symbiotic (Collaborative) Scan Missions. Symbiotic scan area mission using more than one UxVs. The system gives ”scan mission” providing info about the area that has to be scanned and the set of UxVs that will do the scan. However, it does not define which agent will scan which sub-area. Here it comes SOCM which takes into account the battery level of the agents, their speed and their location and splits the given area to sub-areas and assigns each one to each agent for scanning aiming at the minimization of the total scan time.

Collaborative UAV charging on top of UGV agents. In this case SOCM gets the “charging mission” along with the UAV (that has to be charged) the UGV (that will charge the UAV) and the coordinates of a location in which both agents have to go. In this scenario SOCM orchestrates the movement of the agents since it sends messages to both, tracks their position and gives command at the right time.

The SOCM is based on the intrinsic capabilities of the robots (e.g., I am a UAV, I can't open a door by myself) but also constrained on other objectives such as doing the final task faster in time or wasting as little energy as possible.   

For instance, if the commanding officer want to launch a task, the IAM will try to understand the best agents possible for the work. At this time, it will be decided if SOCM or Mission Controller (MC) will need to take the lead. Therefore, the SOCM divides the instructions for each robot in order to complete the task in the best way possible based on the aforementioned criteria... 

Right now, SOCM is using a combination of rule-based criteria and deep learning techniques. INTREPID’s aim is to integrate rules into the training process of the deep learning algorithms built to support SOCM.

During Pilot 3, the next and last Pilot, 2 SOCM missions should be tested: symbiotic scan area and collaboration UAV-UGV charging.

The Symbiotic Operation Control Module, the strength of the collective.

An article on the The Symbiotic Operation Control Module, or how to combine robots functionalities.

The Symbiotic Operation Control Module, or how to combine robots functionalities.

SOCM, or how to combine robots functionalities.

INTREPID integrates 10 major innovative technological components, each offering a set of capabilities tailored for critical operations in perilous environments.

Let's take a look today at its central component, the INTREPID Mobile System, aka. the INMOS (INTREPID Mobile System).

The INMOS is a portable operational crisis management system that is operated on-the-field to provide improved situational awareness to all the involved First Responders, thanks to a multidimensional view of both the crisis site and the ongoing situation, and to offer advanced capabilities to coordinate and command the operations. 

With INMOS, first responders can exploit and orchestrate all the platform capabilities to rapidly explore the crisis site and build a map of damages, threats and victims. It is based on the Crimson (http://crimson.eu) platform, the European leading hypervision and operation management solution adopted by numerous fire brigades and resulting from several previous EC projects. 

Decision support and intelligence amplification 

INMOS interface screenshot showing Pilot 2 BIM.

INMOS enables to orchestrate the exploration operations using its intuitive user interface to command the INTREPID cybernetic assistants, a set of autonomous drones and robots made to measure, to explore disaster areas and gather information about the situation. These cybernetic assistants can work in a symbiotic way to improve their exploration capacities. For instance, UGVs can open doors or call elevators for UAVs to reach areas they would not have been able to reach otherwise. 

The collected information is provided in the real-time to the INMOS that updates the 3D digital twin of the site accordingly. 

The INMOS provides a 2D/3D/4D interactive representation of the Digital Twin of the site aggregating the multi-source, multi-format datasets available about the site (maps, BIM models, aerial imagery, documentation...) or collected in real-time by field sensors, First Responders, or cyber-assistants.  

This massive set of information is optimized in real-time by the INMOS to enable the sharing of a user-legible and interactive a Common Operational Picture among all participants including updated 3D geometry and videos from the site, position and situation of the teams and cyber-assistants, position and status of first responders, threats, incidents, victims… 

INMOS integrates advanced Decision Support capabilities powered by an Intelligence Amplification Module (IAM). This module exploits all the information available in the Digital Twin about the site, the situation and the ongoing operations, to propose in INMOS a number of recommended actions and guidelines based on rule-based doctrines predefined by the operational experts.  

The whole INTREPID platform may include several instances of interoperating INMOS deployable at will, totally autonomous and secured. The integrated, ground-to-C2 approach implies equipment of not only command centers, but also First Responders in the field, for 360° information gathering and communication. To meet this need, the INMOS is available for desktop and mobile devices, on handheld devices as well as mixed reality headset enabling hands-free operation and 3D COP visualization anywhere in the field.

INMOS interface screenshot showing inside Pilot 2 BIM.

The INTREPID project is user-centric by design. In this context, a pilot has been set up from Y1, the second one was in October 2022, and the third one, will be in Madrid, in June 2023. It's in preparation to test these functionalities, requested by the end-users, in a very operational context. The main challenge for INMOS lies in its ability to display the information and offer intuitive functions to properly command and orchestrate operations without causing mental overloading. 

INTREPID, (INtelligent Toolkit for Reconnaissance and assessmEnt in Perilous InciDents), is a ground-breaking platform for First Responders to improve and accelerate the exploration and assessment of disaster areas.

An article explaining INMOS, the plateform that centralised all the information, created for the INTREPID project.

INMOS, the operational crisis management system

INMOS - an operational crisis management system.

The INTREPID project is not only a European project, it is also a collaborative project. Several companies work together to enrich the system. INMOS is the sum of many modules. Among them, the EAM, or Environmental Assessment Module, is considered the eyes of the INTREPID system. 

Indeed, the purpose of the EAM is to assess the environment. It detects, recognises and locates crucial elements of the area, in order to help the End-Users (EU) to make important decisions. 

To do this, the EAM collects information from different types of sensors (RGB camera, LiDAR, ...), placed on robotic agents (UAV, UGV) and first responders (FR). Then, through Artificial Intelligence and deep learning networks, it processes each sensor's data to extract features and perform detection.

The EAM detecting fire on a car during Pilot 2 test.

The EAM is part of a whole, and in order to function properly, it needs other modules. Indeed, to geolocate the data collected by the various sensors, the RTPM (Real Time Positioning Module) is indispensable. It is the combination of these two modules that allows geolocation detection. 

On the other hand, other modules use EAM to improve their productivity: 

PPM (Path Planning Module) takes detection into account, 

EMM (Environment Mapping Module) uses it to perform the 3D mapping function, 

IAM (Intelligence Amplification Module) uses it to produce the recommendation and mission, linked to detection. 

By disaster we mean a situation where there may be fire, heat, rubble, all of these conditions are factors that make the environment what we call "harsh". 

This makes detection much more difficult. But, as the information comes from different sources (it is a multi-modal input), the module can handle some sensor limitations, if a sensor fails to detect an object, the EAM can still ensure the detection by exploiting the other sensors. For example, having an infrared camera sensor and a LiDAR can help when the RGB camera is not very effective due to poor lighting or smoke. 

The deep learning methods used by the model to fuse the data from these sensors, also address this issue. 

The EAM detecting several items such as a backpack, or a dog.

INTREPID is often tested at different stages. For the moment, the Alpha version of the EAM is already available and has been tested in the first pilot, where it has been integrated into the overall system and has shown its ability to detect victims. 

Our consortium partner CERTH, responsible for the EAM, tested the beta version in the second pilot in Marseille. 

Extending the detection capabilities of EAM with a list of objects of interest, defined with the EU (backpack, hazardous materials, ...). 

Research on multimodal detection of 3D objects, by merging 3D LiDAR data and RGB data. 

Research on the tracking of non-rigid objects (smoke, fire).  

The EAM is the Environmental Assessment Module, the assessment tool for the INTREPID project.

An article explaining the ins and outs of the Environment Assessment Module in the INTREPID project.

The Environment Assessment Module, the eyes of the INTREPID system.

The UGV, Unmanned Ground Vehicle, is an autonomous and intelligent robot developed within the INTREPID project. A robot with a weight of 40 kilograms plus 10 kilograms of cargo. 

It’s developed by our partner Robotnik. UGV is a completely new development, for the company, it is a concept that they had not deployed until now. It’s a complicated improvement process, both in terms of design and technology. 

For the first simulation in real life, in November 2021, the developed software was demonstrated. The same software that is running on the INTREPID’s UGV can also work on an existing robot with six wheels, or the walker UGV that is under construction.   

The robot will be equipped with an arm. The actual one, is a commercial arm, from a brand called Kinova. The arm needs to be integrated with the platform. It includes the functionality to pick up from the ground, unknown objects that the robot had not seen before. It’s working with an integrated camera, which scans the object to take, then it looked for the best way to pick it up. It recognized the shape, over the type of object, and then the software will analyse the object to see what the best way is to grab it. The arm has a load capacity are 1,3 kilos.

In navigation mode, the robot should move at 3 meters per second, which is 10 kilometres per hour.   

The whole system should move autonomously around, about 2 hours continuously, and 10h if it’s stopped, i.e. without the need for human intervention. But a teleoperation will also be possible, if the automatic navigation system fails or if the operator prefers to teleoperate. 

The six wheels allow the robot to overcome obstacles such as rubble.

The idea is that the payload is interchangeable, so that it is easy, for example, to interchange between the arm and the drone fence, for example. The same principle is used with the rest of the sensors. 

The following sensors will be added to the robot: navigation sensors, environment identification (3D LIDAR, 3D cameras), gas sensors to measure the concentration of different gases such as CO2 or carbon monoxide. The communications system developed within the project will also be installed. And also RTPM, the localization module being developed by our partner FOI. 

The real challenge is to develop a walking robot, because Robotnik is specialized in robots with wheels.

It will have a standing navigation mode: the legs will be standing, so the speed will be reduced, it’ll be about 1 meter per second maximum, which is about 3 kilometres per hour. But it will have the ability to move over obstacles like for example stairs, climbing curbs, climbing stones etc.

For the third pilot the first version of the walking robot, the quadruped, should be ready. 

The UGV, Unmanned Ground Vehicle are the autonomous and fully equipped robots developed by INTREPID project.

The article is exlpaining how we are developping specific UGV (Unmanned Ground Vehicule) in the INTREPID project

The unmanned ground vehicule, a super equipped robot

The RTPM or Real Time Positioning Module, is a family of different devices that allows the first responders to know their own position in real time, as well as the position of their team members. Our partner FOI is in charge of this module. 

The RTPM is designed to handle different situations and use complementary sensors technologies. Since each type of sensor works better in some conditions than in others, we want to combine several of them to make something that works more robustly and accurately over time.   

The RTPM requires no pre-installed components in the environment. If you have the luxury of putting markers or magnets or reference beacons in the environment beforehand, first responders could position themselves extremely accurately, but in the INTREPID use case first responders (FRs) arrive on the scene where nothing is installed, and they have to act fast, using only what they bring. 

Complementary sensors increase robustness 

Global navigation satellite system (GNSS) sensor

The foot-mounted sensors measure step lengths very accurately and sense even very subtle changes in direction of the feet. Integration of such data over time gives an estimate of the position. A challenge of such inertial measurement units (IMUs) in general is that the error can grow quite quickly over time. However, in RTPM we exploit the fact that during a walking or running cycle a foot stands still for even a short moment of time, and we can reduce the drift quite significantly and can maintain reliable estimates for several minutes.

An inherent limitation of the foot-mounted sensors is that they have no perception of the environment and thus no ability to recover once an error occurs, say, when a foot slips or the first responder moves in an unusual manner. Therefore, FOI combine this type of sensor with cameras-based simultaneous localization and mapping (SLAM) techniques, to increase robustness to changing conditions. In situations where camera-based techniques have problems, such as in scenes with low contrast and few features (landmarks), the foot-mounted sensors can maintain a good enough position estimate until the camera finds something to “lock on” to.

The helmet-mounted RTPM version sports cameras in the near-infrared and the thermal infrared domains. Thermal cameras give the capability to navigate in hot and smoke-filled environments. In addition to position information, the cameras track the orientation of the wearer’s head in real-time, so that team members can relay important information to each other by knowing where they are currently looking. The precision of the helmet-mounted camera is usually below 1% (1 meter if the user walks 100 meters).

Helmet-mounted RTPM with thermal stereo cameras.

The RTPM’s Ultra-wideband (UWB) radio-based ranging system measures distances between team members. This provides yet another means of positioning and further increases the robustness. While camera-based based systems work better in buildings with lots of walls and objects for cameras to look at, the precision and range of the UWB sensing go up in less cluttered environment and can help maintain position information when other systems struggle.

The laser scanner is on the robot (UGV) for extra precision. It measures distances very accurately and has longer effective range than the camera system.

The GNSS is commonly available and really useful as soon as first responders are outdoors, to take care of their basic position needs and to provide reference points to the other sensors. A typical example is when first responders move in and out of buildings. Then the GNSS helps tie everything together in the same reference system.

The sensors complement each other and they can be combined in different configurations depending on requirements concerning accuracy, size, weight, etc. The current main task is to find the best way to fuse data from the sensors to make separate components work together as an integrated unit.

The real-time positioning module includes several devices for determining the position of agents.

An article on the Real Time Positionning Module, and all the localisation solutions developped during the INTREPID project.

The Real Time Positionning Module, the combined positioning solution for first responders.

RTPM - the combined positioning solution for first responders.

PPM, or Path Planning Module, is the module which takes care of calculating and creating the path on the INMOS between two designated points in the area, for a mobile object. TUM is in charge of this module. 

A path can be built for several goals: 

Optimize the path to take the safest path.  

Optimize the path so the agent would be able to investigate the whole area in the quickest time.  

For now, TUM is focusing on building the shortest and safest path, as it’s the primary goal in disaster situation, for first responders, drones (UAV), and robots (UGV).  

In order to plan a path, an environment is needed. For the INTREPID project, we are using indoor and outdoor environments. The digital environment representation can be created from a given building model, in our case, we are using the BIM (Building Information Modeling). The BIM is a detailed 3D model of a building. This environment format is graph based, so we are trying to find out location where the notes are set, then we use an algorithm to create a path in this graph.  

Pilot 2 Building Information Modeling screenshot

As already mentioned, the BIM is the main source in which the path will be calculated. But added to the environment the PPM needs to include specific information for the different mobile objects. For instance, robots can’t access windows (at least for now), when drones can, and first responders might use it. So, this needs to be taken into account to create the right path for the right agent. This information is defined in advance, thanks to collaborative work between technical partners and first responders. 

Screenshot of the path planning recommandation on INMOS.

But in order to be the most accurate, and safe possible it also relies on updated information, as in disaster scenario there could be unavailable areas, or destroyed areas on the building. 

To collect all these “fields” information, the PPM can count on the others module, such as the IAM, the EAM and of course the EMM. They are gathering information from the field, from how the field looks. Therefore, the PPM gathered all these information, and calculate the best path depending on the demand. The PPM proposes multiple paths to the IAM and the IAM chooses the right one for its recommendation.  

It's the integration of a large amount of semantic information which can be processed by the Path Planning Module in order to calculate the shortest, safest, accessible paths through the building, as well as path planning around dynamic obstacles which occur during the disaster scenario. 

An essentiel module of the INTREPID project : the Path Planning Module

This article explained how we developped the path planning in order to be closer to the end users needs.

The Path Planning Module, or how to calculate the best path for field agents.

PPM - how to calculate the best path for field agents.

In a few words, EMM, or Environment Mapping Module, creates 3D maps of outdoor and indoor environments using sensors mounted on UxVs. Miguel A. Vega Torres from the Chair of Computational Modeling and Simulation at the Technical University of Munich (TUM) is in charge of this module in INTREPID. 

Since global positioning systems (GPS) are usually not reliable in indoor environments, one usually relies on other sensors on the robot to know its position.  

The EMM leverages prior information, for example, 3D building information models (BIM Models) or 2D floor plans to localize the robot accurately in indoor and outdoor environments.  

With accurate position information, the EMM can then create an updated map of the environment, which is also aligned with the prior information.   

The EMM is a back-end module, which means that it is something that the users don't see but that works behind the scenes and provides content that end users will eventually see on the interface.   

EMM receives information from sensors on the robots. For example, what is received from the LiDAR sensor over the unmanned ground vehicle (UGV) is a bunch of points created after a sensor scan in one position.  

EMM collects this information and tries to match it with the prior map, for instance, with the BIM model. In this way, EMM can track the robot's position and orientation while it moves through the environment.  

Difficulties occur when there are many deviations between the real environment and the model. In the BIM model, there are usually only walls, doors, and windows; however, there is no exact position of the furniture or all the elements in the environment, which creates discrepancies between what the robot sees and what there is in the reference model.  

All the elements that aren't present in the model make the alignment difficult. Not only furniture but also dynamic elements like moving people and objects that aren't present in the model will affect the localization accuracy.   

Moreover, in the future, the idea is to be able to map the current state of the environment, which may be very different from the as-planned status of the building.  

With an updated map, we can not only keep track of the robot's trajectory but also, for example, localize the victims accurately. More importantly, we can give an overview to the first responders of the current situation, increasing situational awareness and enabling fast decision-making. 

Aligned 3D map created by the EMM during pilot 2.

The two main motivational aspects behind the EMM in INTREPID are: 

Rapidly increase situational awareness in disaster areas without risking the lives of first responders. 

The accurate, up-to-date 3D map will support decision-making and autonomous robotic tasks. 

To resume, the EMM has three main goals:  

Use prior reliable information to improve the localization and mapping accuracy in challenging indoor and outdoor environments.  

Create an aligned updated 3D map with the correct georeferentiation so that it can be displayed in a general map with semantic information at the building and city level. 

Detect changes between the acquired data and the reference map. 

With such a pipeline, the end goal of the EMM in the INTREPID project is to gather different sources of information from different robots and put them in a single georeferenced map.  

Moreover, since the EMM also calculates the transformation from the global coordinate system to the local BIM coordinate system, it enables the exploitation of semantic BIM models for high-level autonomous robotic tasks, like autonomous navigation and recurrent inspection. 

In INTREPID, the EMM, together with the Path Planning Module (PPM) and, in general, TUM, as part of the consortium, also contributes to creating 3D BIM models from architectural floor plans for the pilots. The pilots are the scenarios where the different capabilities of the numerous INTREPID modules are validated. 

While for Pilot 1, the environment was a large metro station in Stockholm, for Pilot 2, it was a multi-story building designed for firefighters' training purposes in Marseille. 

Both models were created with a high level of detail, considering indoor spaces, and with all the needed semantics for the other modules, e.g., for the Path Planning Module.

1st INTREPID pilot – BIM model of the subway station

BIM model for Pilot 1.

BIM Pilot 2

BIM model for Pilot 2.

The future focus of the EMM can be summarized in the following three points. 

If the robot moves fast, the degradation of the position accuracy can be pretty significant. Therefore, we would like to exploit also of information from the inertial measurement unit (IMU) sensors on the robots to improve localization accuracy. 

Once an aligned map is successfully created, this should also allow the update of the current map, not only considering the new elements (positive differences) but also the removed elements (negative differences). 

So far, the main focus has been placed on LiDAR information. However, in the future, we also would like to consider camera information to improve pose estimation (localization) and give realistic color to the final aligned maps. 

The EMM identified new objects in the 3D map during Pilot 2.

Pilot 2 BIM in the 3D map

Pilot 2 Bim in the 3D map.

If you are curious about the functionality of EMM, you can contact Miguel, the person in charge directly via LinkedIn or via email.

The Environment Mapping Module (EMM) is creating georeferenced aligned 3D updated maps of outdoor and indoor environments with sensors on board Unmanned Vehicles (UVs).

An article on the functionning of the Environment Mapping Module on the INTREPID project.

The Environment Mapping Module creates a coherent, updated map.

Modestly, we can say that the IAM (the Intelligence Amplification Module) is the brain of the INTREPID system.  

Indeed, it’s a module that has the ability to collect all the information coming from other modules of the system, like the paths, the available agents, the available operations within the particular crisis situations. Once all that is taking into account, it does propose to First Responders, what is the estimated best course of action at the moment with the best possible outcome. 

Based to the recommendations of IAM, and in conjunction with their own experience and what’s going on in the scene, the First Responders will decide what course of action they are going to take. 

In the INTREPID Project, this module is developed by our partner CERTH. 

There are 2 flows of input to the IAM:  

One is the knowledge of the first responders of what’s happens in any particular crisis situation, codified in a machine-understandable doctrine module. It's a pre-coded knowledge of First Responders' protocols, courses of action and things to consider within a crisis scene. 

The other flow is the situational context, what’s happening right now in the scene, (what agents are available, where are robot and drones, if they are available, what are the optimal paths for each agent, etc.). 

The two together allowed a clear and specific understanding of the situation, and serve to provide appropriate proposals. 

The proposals are sent to the End Users through the INTREPID interface and from this interface, they will choose the course of action they want to take at that time and the interface will send to the mission controller the exact mission to be undertaken, to the appropriate agent, whether it be the first responder or the UxV.

A screenshot of the recommandations display on INMOS interface.

CERTH is currently working on several projects: 

Fusing expert knowledge with learning. CERTH's research directive is oriented  towards integrating machine learning with the existing expert-based system, in a neuro-symbolic approach. Combining explicit and implicit knowledge, i.e. machine learning understanding of crisis scenes while at the same time adhering to explicit First Responders’ protocols is expected to elevate the system’s decision-making and notification process. 

Decision explainability. This is an important aspect, which allows the IAM to gain the trust of the first responders, because by explaining to them how the project makes its decisions, they will see the logic and reasoning, and by seeing that it potentially coincides with their own process, they will be able to trust it more.   

Integrate more information. Expanding the whole system to more variables that can go into the modules. Indeed the IAM is the last part of this workflow, it will eventually receive more information, so it has to be extended and be able to accommodate this information.   

Pilot focus. A work is done on the process of integrating all the features that the users have identified that they needed from the system and from IAM for pilot 3, based on their feedback from the first and second pilot.  

One essential module of the INTREPID project is the IAM (the Intelligence Amplification Module).

An article explaining the role of the IAM or the Intelligence Amplification Module, the brain of INTREPID system.

IAM or the Intelligence Amplification Module, the brain of INTREPID system.

The INTREPID project intends to increase the data available to FRs dramatically, in the form of sensor data, positioning information or alerts and recommendations. This wealth of information, comes with its own challenge though: how to display and share it quickly and efficiently, in a context where every minute counts and lives are at stake?

INTREPID has chosen eXtended Reality (XR) as an innovative way to display information for increased situational awareness. XR covers three potential modes: Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR).

Virtual Reality (VR): the full immersive solution

VR is well known by now, thanks to the existence of various levels of consumer-grade products, from the Google Cardboard to UHD Pimax helmets. It immerses the users in a fully virtual world, and allows them to interact with it. 

The INTREPID platform, through its Digital Mock-Up module, gathers not only geographical data and existing 3D models, but also updated 3D representations of the area, extracted from real-time data acquired by INTREPID drones and robots (Environment Mapping Module) as well as data from on-site sensors and cameras. This allows for a life-size, real-time representation of the disaster area. 

Figure 1: VR view includes existing geospatial information and models, and real-time data from UxV and sensors.

When in operation, however, VR cuts the user from his or her surroundings. It can thus have an impact on the workflow in the command centre. An extensive use can also lead to physical uneasiness (nausea, motion sickness). In training though, with an adapted curriculum, VR would allow the user  to experience being there, without actually having to set up a mock incident. Even better, the virtual world is not limited to a representation of its physical counterpart. It can make invisible threats visible, and display additional information from the INTREPID system right where needed. 

Augmented Reality (AR): the best of both worlds

AR consists in adding virtual elements provided by the system to the real world (seen either through a camera, or through the very eyes of the user). 

It comes in all shapes and sizes, from the more mundane (playing Pokemon Go on a smartphone or trying on virtual glasses on a website) to the most serious, life-saving use cases (surgery assistance or assistance to First Responders). With such a wide array of possibilities, capabilities strongly depend on the device used, the context and the functionalities of the system as a whole. 

In the INTREPID project, solutions based on dedicated wearable devices (AR glasses) are favoured during the analysis phase, since they offer advanced display capabilities, and are able to closely intertwine visual metaphors with the real world without forcing the user to shift their gaze or use their hands. 

Such devices would give the First Responders a global, rich understanding of their surroundings through: INTREPID sensor data (for instance, gas concentrations); data from other actors (for instance positions of the other team members); or even alerts and recommendations from the INTREPID Intelligence Amplification components (for instance, highlighting the path to the nearest victim, identifying a target area, providing alerts about nearby CBRN threats…). FRs immediately receive all the relevant information, and only the relevant information. 

However, to obtain a seamless, efficient and unobtrusive integration of both worlds (real and virtual), positions and orientations should be determined precisely, not only for the situational data, but also for the AR device. Highly detectable “points of interest” can be used to track environmental features and extract knowledge about the position and orientation of the device. With INTREPID, though, the platform could be deployed anywhere and the presence of points of interest cannot be guaranteed. The AR mode would then rely on the system’s own real time indoor/outdoor positioning capabilities. 

Figure 2: Hololens-wearing user receives navigation assistance (up) and alerts (rights) when exploring a building

Mixed Reality (MR): the whole situation in the palm of the hand

MR (Mixed Reality) can be considered as an advanced version of Augmented Reality. It also consists in bringing virtual elements into the real world. Yet, instead of displaying them as passive additions to the reality, the MR rendering takes into account the physical elements. It also allows the user to interact finely with the virtual components of the scene, blurring the limit between real and virtual. 

This is embodied in INTREPID’s Holographic Common Operational Picture (Holographic COP). In this concept, a virtual representation of the disaster area is built and displayed for the connected users through AR devices, like a mock-up before their eyes. Like its physical, static cardboard counterpart, this scale model displays the 3D representations of the area, but its MR nature comes with unparalleled features, such as: navigation and manipulation at the snap of a finger, real-time situational information such as the position of actors or threats, regular updates throughout the crisis or the ability to go back and forth in time for an after-action review.

Figure 3: Hololens-wearing user viewing the COP when handling incidents on a cruise boat 

Advanced visualisation components based on eXtended Reality come hand in hand with advanced situational awareness and decision-support capabilities. The vast amount of information, acquired or generated, allows a detailed, up-to-date and rich rendering of the situation. Conversely, such an amount of data requires visualisation options that go beyond a 2D computer screen. The XR modes considered in INTREPID offer multiple possibilities, and the various workshops organised in the scope of the project will help narrow down the better and most relevant uses to harness the system capabilities. 

Post-disaster rescue operations are hazardous by nature, carried out in very uncertain, unknown environments, all the while relying solely on the perception of First Responders. Gathering and sharing information is done at a high risk.

Extended reality for early rescue operations:  the INTREPID approach

An article on extended reality for early rescue operations:  the INTREPID approach

The INTREPID approach for extended reality

Both man-made hazards (anthropic) and natural hazards (earthquakes, floods or volcanic eruptions)[1] have unfortunately become part of our daily lives, and the work of first responders in emergency situations has become increasingly important.

Early detection, prompt identification, control and/or neutralization of hazardous substances and an accurate delimitation of the risk area are fundamental premises to optimize the effectiveness of first responders, thus avoiding loss of lives and mitigating damage to the environment. From this perspective, the use of technology aimed at first responders in emergency situations can play a fundamental role, helping to detect, reduce and mitigate the risks and dangers that may appear in a disaster.

The development of technological solutions to support first responders in emergency situations has given rise to a line of action within the European Union Research and Innovation Programme Horizon 2020, where a list of projects have been funded to support such technological development.

The INTREPID project is one of this project aimed at developing technology for first responders. It has been funded by the European Union within its H2020 Research and Innovation Programme. The Spanish School of Rescue and Detection with Dogs (ESDP) is part of the INTREPID consortium, whose main challenges are to develop technologies aimed at improving safety and efficiency in the detection of risks, guaranteeing a rapid, dynamic and safe response by the first intervention teams and, of course, by the canine teams (K9) as first responders.

One of the possible risks that can appear in a disaster and that is not always immediately perceptible are chemical risks. For this reason, chemical detectors are one of INTREPID's key strengths, and our work consists of supporting to make them as efficient as possible.

HOW CAN CHEMICAL DETECTORS HELP K9 TEAMS? 

For a right assessment of hazards in the area, the following aspects are of great importance and can be faced using technology: On the one hand, the recognition and evaluation of the area. For this purpose, it is important to determine the chemical detectors, which must be used BEFORE the first responders´ entrance, both guides and dogs. Drones and robots, technologies that are also included in the INTREPID project and which, equipped with these detectors, will play an essential role in assessing the danger, securing the intervention area and thus guaranteeing the safety of the first responders, rescue dogs and victims.

On the other hand, after the recognition and evaluation of the area, the detectors worn by the first responders´ uniforms respond to different purposes which can be described as follows:

The continuous assessment of the safety margins that need to be maintained while the rescue work is being carried out.

Continuous control and correction, giving the possibility of warning of possible changes or the appearance of new leaks that could endanger the lives of the first responders involved.

Detection of new substances that did not appear in previous readings and assessment.

In order to achieve what is called "Operational Hygiene"[2], risk elimination or reduction mesures must be established throughout the rescue manoeuvre. 

We not only need to know what we are dealing with, but also some additional information as: the concentration of chemicals, the size of particles, the exposure time, the combination of pollutants or the detection of new hazards. The detection of these aspects using technological solutions is a significant challenge which provides considerable added value to the project's results.

Technology is setting a new horizon in disaster work, helping rescue teams to achieve better results in less time and increasing the safety and efficiency of first responders and rescue dogs.

Ana Aldea Reyes | Marta Burgos González | Susana Izquierdo Funcia

Escuela Española de Salvamento y Detección con Perros /  @ESDP_esp 

[1]Salvago B. La amenaza riesgos, emergencias y catástrofes. La función de inteligencia en la gestión de emergencias y catástrofes, págs. 23-50. 2019.

An article on how to detect dangers that can't be seen with Faster project example

Articles

SOCM, or how to combine robots functionalities.

INMOS - an operational crisis management system.

EAM - the eyes of the INTREPID system.

UGV - super equipped robot.

RTPM - the combined positioning solution for first responders.

PPM - how to calculate the best path for field agents.

EMM - creating a coherent, updated map.

IAM - the brain of the INTREPID system.

The INTREPID approach for extended reality

Chemical detectors, searching for the silent hazard

Let's keep in touch

Newsletter