INTREPID consists of many components, such as agents, modules, mobile phones, tablets and others that must exchange messages with each other. The agents are the First Responders, the drones, the UGVs and modules are the EAM (Environment Assessment Module), IAM (Intelligence Assessment Module), INMOS (INTREPID Mobile System) and others. Therefore the network is a very critical piece to the INTREPID project puzzle, it is the modern Mercury, the messenger of messages between INTREPID agents and modules. So it was designed to be stable, robust, scalable, reliable, and cheap to implement with off-the-shelf products.

Figure 1: The devices on the field, in an abandoned hospital, in Los Molinos.

Α large disaster area could not be covered with a single antenna, (see Figure 6), so the solution of a network with many antennas that communicate with each other and act as a grid in the area of the disaster was chosen, this category of networks is called mesh.

In the case of an incident, the first responders are exploring the area and put the network devices in order to extend the coverage area of INTREPID network.

So, since we will use many antennas, how will they be connected?

There are various types of network architectures. The simplest one is the star where all the connectivity devices are connected through a central one (Figure 2). However, this approach is not the optimal one for INTREPID due to the way of network deployment (the First Responders have to know in advance where to put the central antenna, which is not possible since they are not aware of the incident area in advance) and the fact that a mal function of the main antenna will make the network not operable.

Thus, in INTREPID the mesh network approach has been chosen as the most efficient one. A mesh network is a group of connectivity devices, such as Wi-Fi routers and access points that act as a single network (Figure 2), so there are multiple sources of connectivity on the field instead of just a single router.

It is a communications network in which there are at least two pathways to each node. If one of the paths fails, the other is still available.

Figure 2: In a mesh network, there is more than one path to each node. In contrast, a star network relies on a central node to always be up and running.

Is this something big/heavy? Is it easy to deploy it?

The network consists of a central router, two switches (optional), and 13 range extenders (Figure 3). Since it is modular, more range extenders can be easily added depending on the needs of the pilot/incident area.

The central router and the two switches are placed and powered by the First Responders trucks while the range extenders are placed at the disaster-area by First Responders.

Figure 3: The network equipment, network base on the left, range extender with power bank on the right

First Responders as they move in the field, leave the range extenders along with their power bank (their power bank lasts 3.5 hours) which automatically connect to the rest of the network. This way the area that the network can cover is increased while at the same time its reliability is increase as well as more transmission paths are created for the messages. It should be noted that each range extender/power bank pair is very light since it weights ~ 400-500 gr making it very easy carrying and installing them.

Also, INTREPID network is able to connect to the internet through a landline or mobile network. The network consists of a central controller station, two smaller stations and 13 range extenders. All stations can be connected to each other wirelessly or by cable, also range extenders are connected wirelessly to the network. All stations and range extenders, as soon as they are supplied with power, automatically create a wireless net communication network between them, so that if an intermediate device is interrupted, the others will not be interrupted.

Which devices are connected to the INTREPID network?

The UAVs (drones) and the UGVs are equipped with sensors that capture and send information (RGB images, depth images, air quality, point clouds) which are send through INTREPID network to the INTREPID modules for further processing. Furthermore, INTREPID network supports the communications between First Responders and Command post

In general, INTREPID network can support any device that can connect to a 2.4Ghz or 5Ghz WiFi network, such devices are cell phones, tablets, UAVs, UGVs, laptops, push to talk devices and various sensors.

This network has been tested in a real life test which took place at the sanatorio de la Marina, in los Molinos, in March 2023. First, the central router was placed on the police truck and then as the First Responders searched for victims placed in various points the range extenders to increase their radius of action. The final placement is shown in Figure 6, in red are the range extenders in the outdoor area, in blue are the range extenders on the ground floor, in yellow are the range extenders on the first floor and in green are the range extenders on the second floor. As the pilot area had many obstacles and expanded on several square meters, the use of the 2.4Ghz band was chosen. In many cases there is simultaneous transmission from different agents RGB-D (Normal video with depth information) image and RGB (Normal video), Figure 4, image without any difficulty. As can be seen in the photographs, the building is long and narrow, Figure 5, and with three range extenders per floor (spreaded in the corridors) and one between the floors on the stairs, Figure 6, the whole building was covered.

Figure 4: The RGB video on the left and the corresponding depth video on the right.

Figure 6: The arrangement of range extenders in the building and outside area.

The range extenders outside and the network base served the command post (EAM, IAM, INMOS and other INTREPID modules), the first aid tent and the police truck. The range extenders inside the building were spread by the First Responders as they proceeded to scan the building for victims of the disaster. Thus, the network was gradually created as defined by the project that the rescue teams needed.

Want access to exclusive INTREPID content? 💌 Subscribe to our newsletter! 💌

The connection between all INTREPID modules.

An article explaining the network deployed and developed in the framework of INTREPID

How the INTREPID system network works

The Network, the "Hermes" of INTREPID system

SOCM, or Symbiotic Operation Control Module, is the way to create virtual robots able to integrate their different capabilities in order to carry out tasks that either cannot be done by a single robot or because it will be more efficient to do it by combining several functionalities of several robots. INTREPID’s entity responsible for this module is CERTH. 

As an example of this symbiosis, one may imagine drones (UAV), which are not able to open a door. In order for a drone to scan an area that is block by a closed door, a terrain robot (UGV) should assist and open the door. SOCM will handle the sequencing of the different mini tasks, once the drone has been given the mission to scan a closed area. SOCM will ask the UGV to come in to help an unblock drone’s way by opening the door. By combining their functionalities, the drone can enter the premises and do the scanning. 

In more detail, the overall procedures that will take place are the followings:  

The INMOS system will receive a request to locate a place and then the IAM module will advise that it cannot do that with a single drone, and that it needs to combine the functionalities of the drone and the UGV.  

To do so, IAM will send a mission to the SOCM saying that it wants a virtual robot combining a drone and a UGV to do this mission.  

Therefore, SOCM will handle the way the 2 robots (drone and UGV) will collaborate to fullfil the task at hand.  

Symbiotic Operation Control Module (SOCM) is used for:

Optimization for Symbiotic (Collaborative) Scan Missions. Symbiotic scan area mission using more than one UxVs. The system gives ”scan mission” providing info about the area that has to be scanned and the set of UxVs that will do the scan. However, it does not define which agent will scan which sub-area. Here it comes SOCM which takes into account the battery level of the agents, their speed and their location and splits the given area to sub-areas and assigns each one to each agent for scanning aiming at the minimization of the total scan time.

Collaborative UAV charging on top of UGV agents. In this case SOCM gets the “charging mission” along with the UAV (that has to be charged) the UGV (that will charge the UAV) and the coordinates of a location in which both agents have to go. In this scenario SOCM orchestrates the movement of the agents since it sends messages to both, tracks their position and gives command at the right time.

The SOCM is based on the intrinsic capabilities of the robots (e.g., I am a UAV, I can't open a door by myself) but also constrained on other objectives such as doing the final task faster in time or wasting as little energy as possible.   

For instance, if the commanding officer want to launch a task, the IAM will try to understand the best agents possible for the work. At this time, it will be decided if SOCM or Mission Controller (MC) will need to take the lead. Therefore, the SOCM divides the instructions for each robot in order to complete the task in the best way possible based on the aforementioned criteria... 

Right now, SOCM is using a combination of rule-based criteria and deep learning techniques. INTREPID’s aim is to integrate rules into the training process of the deep learning algorithms built to support SOCM.

During Pilot 3, the next and last Pilot, 2 SOCM missions should be tested: symbiotic scan area and collaboration UAV-UGV charging.

The Symbiotic Operation Control Module, the strength of the collective.

An article on the The Symbiotic Operation Control Module, or how to combine robots' functionalities.

The Symbiotic Operation Control Module, or how to combine robots' functionalities.

SOCM, or how to combine robots' functionalities.

An UAV, Unmanned Aerial Vehicle, is a fully equipped and autonomous drone, INTREPID project is developing. 

In the INTREPID consortium, ALX society is in charge of developing them. Its main specificities are the ability to fly indoors and outdoors, to have a 2D and 3D sensing. It has a relatively accurate indoor location. The machine makes a 3D analysis in real time of its environment to better locate itself in its environment. It’s really interesting when there is no GPS. Also, if it's not possible to estimate the position GPS of the moving vehicle, it has the ability to estimate displacement based on the 2D image. A software is also developed to interpret what the 3D camera sees. 

Precision landing: on the rover (7-8cm margin). The development of the landing algorithm is based on a QR code, ultrasound system, or colored LED system. 

Autonomous exploration: unmapped and referenced environment. The UAV will reconstruct in its internet memory a map of the building as it moves inside. It will share the map with all the drones, and will be able to navigate more efficiently with this map. The drone will calculate the optimized route based on what it has recorded.   

Storage: it can manage difficulty in terms of bandwidth, it passes through the communication system developed in the project. If the bandwidth decreases, a management software will transfer video in a compressed way to not overload the bandwidth.   

Speed: indoor it can fly at 4 KM/H to be able to scan, and outdoor it can go up to 60KM/H, but for scanning, the limit speed is 20 KM/H. It can continuously fly for 22 minutes. 

The UAV is equipped with 15 sensors: for obstacle and positioning. It’s designed for the INTREPID project. The UAV is also involved in the symbiotic operation, developed by CERTH.  

The UAV, Unmanned Aerial Vehicle, is a fully equipped and autonomous drone, INTREPID project is developing.

An article on the functionalities of the Unmanned Aerial Vehicle, in the INTREPID project.

The Unmanned Aerial Vehicle, a drone with 15 sensors.

The INTREPID project is not only a European project, it is also a collaborative project. Several companies work together to enrich the system. INMOS is the sum of many modules. Among them, the EAM, or Environmental Assessment Module, is considered the eyes of the INTREPID system. 

Indeed, the purpose of the EAM is to assess the environment. It detects, recognises and locates crucial elements of the area, in order to help the End-Users (EU) to make important decisions. 

To do this, the EAM collects information from different types of sensors (RGB camera, LiDAR, ...), placed on robotic agents (UAV, UGV) and first responders (FR). Then, through Artificial Intelligence and deep learning networks, it processes each sensor's data to extract features and perform detection.

The EAM detecting fire on a car during Pilot 2 test.

The EAM is part of a whole, and in order to function properly, it needs other modules. Indeed, to geolocate the data collected by the various sensors, the RTPM (Real Time Positioning Module) is indispensable. It is the combination of these two modules that allows geolocation detection. 

On the other hand, other modules use EAM to improve their productivity: 

PPM (Path Planning Module) takes detection into account, 

EMM (Environment Mapping Module) uses it to perform the 3D mapping function, 

IAM (Intelligence Amplification Module) uses it to produce the recommendation and mission, linked to detection. 

By disaster we mean a situation where there may be fire, heat, rubble, all of these conditions are factors that make the environment what we call "harsh". 

This makes detection much more difficult. But, as the information comes from different sources (it is a multi-modal input), the module can handle some sensor limitations, if a sensor fails to detect an object, the EAM can still ensure the detection by exploiting the other sensors. For example, having an infrared camera sensor and a LiDAR can help when the RGB camera is not very effective due to poor lighting or smoke. 

The deep learning methods used by the model to fuse the data from these sensors, also address this issue. 

The EAM detecting several items such as a backpack, or a dog.

INTREPID is often tested at different stages. For the moment, the Alpha version of the EAM is already available and has been tested in the first pilot, where it has been integrated into the overall system and has shown its ability to detect victims. 

Our consortium partner CERTH, responsible for the EAM, tested the beta version in the second pilot in Marseille. 

Extending the detection capabilities of EAM with a list of objects of interest, defined with the EU (backpack, hazardous materials, ...). 

Research on multimodal detection of 3D objects, by merging 3D LiDAR data and RGB data. 

Research on the tracking of non-rigid objects (smoke, fire).  

The EAM is the Environmental Assessment Module, the assessment tool for the INTREPID project.

An article explaining the ins and outs of the Environment Assessment Module in the INTREPID project.

The Environment Assessment Module, the eyes of the INTREPID system.

The UGV, Unmanned Ground Vehicle, is an autonomous and intelligent robot developed within the INTREPID project. A robot with a weight of 40 kilograms plus 10 kilograms of cargo. 

It’s developed by our partner Robotnik. UGV is a completely new development, for the company, it is a concept that they had not deployed until now. It’s a complicated improvement process, both in terms of design and technology. 

For the first simulation in real life, in November 2021, the developed software was demonstrated. The same software that is running on the INTREPID’s UGV can also work on an existing robot with six wheels, or the walker UGV that is under construction.   

The robot will be equipped with an arm. The actual one, is a commercial arm, from a brand called Kinova. The arm needs to be integrated with the platform. It includes the functionality to pick up from the ground, unknown objects that the robot had not seen before. It’s working with an integrated camera, which scans the object to take, then it looked for the best way to pick it up. It recognized the shape, over the type of object, and then the software will analyse the object to see what the best way is to grab it. The arm has a load capacity are 1,3 kilos.

In navigation mode, the robot should move at 3 meters per second, which is 10 kilometres per hour.   

The whole system should move autonomously around, about 2 hours continuously, and 10h if it’s stopped, i.e. without the need for human intervention. But a teleoperation will also be possible, if the automatic navigation system fails or if the operator prefers to teleoperate. 

The six wheels allow the robot to overcome obstacles such as rubble.

The idea is that the payload is interchangeable, so that it is easy, for example, to interchange between the arm and the drone fence, for example. The same principle is used with the rest of the sensors. 

The following sensors will be added to the robot: navigation sensors, environment identification (3D LIDAR, 3D cameras), gas sensors to measure the concentration of different gases such as CO2 or carbon monoxide. The communications system developed within the project will also be installed. And also RTPM, the localization module being developed by our partner FOI. 

The real challenge is to develop a walking robot, because Robotnik is specialized in robots with wheels.

It will have a standing navigation mode: the legs will be standing, so the speed will be reduced, it’ll be about 1 meter per second maximum, which is about 3 kilometres per hour. But it will have the ability to move over obstacles like for example stairs, climbing curbs, climbing stones etc.

For the third Pilot the first version of the walking robot, the quadruped, should be ready. 

The UGV, Unmanned Ground Vehicle are the autonomous and fully equipped robots developed by INTREPID project.

The article is exlpaining how we are developping specific UGV (Unmanned Ground Vehicule) in the INTREPID project

The unmanned ground vehicule, a super equipped robot

PPM, or Path Planning Module, is the module which takes care of calculating and creating the path on the INMOS between two designated points in the area, for a mobile object. TUM is in charge of this module. 

A path can be built for several goals: 

Optimize the path to take the safest path.  

Optimize the path so the agent would be able to investigate the whole area in the quickest time.  

For now, TUM is focusing on building the shortest and safest path, as it’s the primary goal in disaster situation, for first responders, drones (UAV), and robots (UGV).  

In order to plan a path, an environment is needed. For the INTREPID project, we are using indoor and outdoor environments. The digital environment representation can be created from a given building model, in our case, we are using the BIM (Building Information Modeling). The BIM is a detailed 3D model of a building. This environment format is graph based, so we are trying to find out location where the notes are set, then we use an algorithm to create a path in this graph.  

Pilot 2 Building Information Modeling screenshot

As already mentioned, the BIM is the main source in which the path will be calculated. But added to the environment the PPM needs to include specific information for the different mobile objects. For instance, robots can’t access windows (at least for now), when drones can, and first responders might use it. So, this needs to be taken into account to create the right path for the right agent. This information is defined in advance, thanks to collaborative work between technical partners and first responders. 

Screenshot of the path planning recommandation on INMOS.

But in order to be the most accurate, and safe possible it also relies on updated information, as in disaster scenario there could be unavailable areas, or destroyed areas on the building. 

To collect all these “fields” information, the PPM can count on the others module, such as the IAM, the EAM and of course the EMM. They are gathering information from the field, from how the field looks. Therefore, the PPM gathered all these information, and calculate the best path depending on the demand. The PPM proposes multiple paths to the IAM and the IAM chooses the right one for its recommendation.  

It's the integration of a large amount of semantic information which can be processed by the Path Planning Module in order to calculate the shortest, safest, accessible paths through the building, as well as path planning around dynamic obstacles which occur during the disaster scenario. 

An essentiel module of the INTREPID project : the Path Planning Module

This article explained how we developped the path planning in order to be closer to the end users needs.

The Path Planning Module, or how to calculate the best path for field agents.

PPM - how to calculate the best path for field agents.

In a few words, EMM, or Environment Mapping Module, creates 3D maps of outdoor and indoor environments using sensors mounted on UxVs. Miguel A. Vega Torres from the Chair of Computational Modeling and Simulation at the Technical University of Munich (TUM) is in charge of this module in INTREPID. 

Since global positioning systems (GPS) are usually not reliable in indoor environments, one usually relies on other sensors on the robot to know its position.  

The EMM leverages prior information, for example, 3D building information models (BIM Models) or 2D floor plans to localize the robot accurately in indoor and outdoor environments.  

With accurate position information, the EMM can then create an updated map of the environment, which is also aligned with the prior information.   

The EMM is a back-end module, which means that it is something that the users don't see but that works behind the scenes and provides content that end users will eventually see on the interface.   

EMM receives information from sensors on the robots. For example, what is received from the LiDAR sensor over the unmanned ground vehicle (UGV) is a bunch of points created after a sensor scan in one position.  

EMM collects this information and tries to match it with the prior map, for instance, with the BIM model. In this way, EMM can track the robot's position and orientation while it moves through the environment.  

Difficulties occur when there are many deviations between the real environment and the model. In the BIM model, there are usually only walls, doors, and windows; however, there is no exact position of the furniture or all the elements in the environment, which creates discrepancies between what the robot sees and what there is in the reference model.  

All the elements that aren't present in the model make the alignment difficult. Not only furniture but also dynamic elements like moving people and objects that aren't present in the model will affect the localization accuracy.   

Moreover, in the future, the idea is to be able to map the current state of the environment, which may be very different from the as-planned status of the building.  

With an updated map, we can not only keep track of the robot's trajectory but also, for example, localize the victims accurately. More importantly, we can give an overview to the first responders of the current situation, increasing situational awareness and enabling fast decision-making. 

Aligned 3D map created by the EMM during pilot 2.

The two main motivational aspects behind the EMM in INTREPID are: 

Rapidly increase situational awareness in disaster areas without risking the lives of first responders. 

The accurate, up-to-date 3D map will support decision-making and autonomous robotic tasks. 

To resume, the EMM has three main goals:  

Use prior reliable information to improve the localization and mapping accuracy in challenging indoor and outdoor environments.  

Create an aligned updated 3D map with the correct georeferentiation so that it can be displayed in a general map with semantic information at the building and city level. 

Detect changes between the acquired data and the reference map. 

With such a pipeline, the end goal of the EMM in the INTREPID project is to gather different sources of information from different robots and put them in a single georeferenced map.  

Moreover, since the EMM also calculates the transformation from the global coordinate system to the local BIM coordinate system, it enables the exploitation of semantic BIM models for high-level autonomous robotic tasks, like autonomous navigation and recurrent inspection. 

In INTREPID, the EMM, together with the Path Planning Module (PPM) and, in general, TUM, as part of the consortium, also contributes to creating 3D BIM models from architectural floor plans for the pilots. The pilots are the scenarios where the different capabilities of the numerous INTREPID modules are validated. 

While for Pilot 1, the environment was a large metro station in Stockholm, for Pilot 2, it was a multi-story building designed for firefighters' training purposes in Marseille. 

Both models were created with a high level of detail, considering indoor spaces, and with all the needed semantics for the other modules, e.g., for the Path Planning Module.

1st INTREPID pilot – BIM model of the subway station

BIM model for Pilot 1.

BIM Pilot 2

BIM model for Pilot 2.

The future focus of the EMM can be summarized in the following three points. 

If the robot moves fast, the degradation of the position accuracy can be pretty significant. Therefore, we would like to exploit also of information from the inertial measurement unit (IMU) sensors on the robots to improve localization accuracy. 

Once an aligned map is successfully created, this should also allow the update of the current map, not only considering the new elements (positive differences) but also the removed elements (negative differences). 

So far, the main focus has been placed on LiDAR information. However, in the future, we also would like to consider camera information to improve pose estimation (localization) and give realistic color to the final aligned maps. 

The EMM identified new objects in the 3D map during Pilot 2.

Pilot 2 BIM in the 3D map

Pilot 2 Bim in the 3D map.

If you are curious about the functionality of EMM, you can contact Miguel, the person in charge directly via LinkedIn or via email.

The Environment Mapping Module (EMM) is creating georeferenced aligned 3D updated maps of outdoor and indoor environments with sensors on board Unmanned Vehicles (UVs).

An article on the functionning of the Environment Mapping Module on the INTREPID project.

The Environment Mapping Module creates a coherent, updated map.

INTREPID integrates 10 major innovative technological components, each offering a set of capabilities tailored for critical operations in perilous environments.

Let's take a look today at its central component, the INTREPID Mobile System, aka. the INMOS (INTREPID Mobile System).

The INMOS is a portable operational crisis management system that is operated on-the-field to provide improved situational awareness to all the involved First Responders, thanks to a multidimensional view of both the crisis site and the ongoing situation, and to offer advanced capabilities to coordinate and command the operations. 

With INMOS, first responders can exploit and orchestrate all the platform capabilities to rapidly explore the crisis site and build a map of damages, threats and victims. It is based on the Crimson (http://crimson.eu) platform, the European leading hypervision and operation management solution adopted by numerous fire brigades and resulting from several previous EC projects. 

Decision support and intelligence amplification 

INMOS interface screenshot showing Pilot 2 BIM.

INMOS enables to orchestrate the exploration operations using its intuitive user interface to command the INTREPID cybernetic assistants, a set of autonomous drones and robots made to measure, to explore disaster areas and gather information about the situation. These cybernetic assistants can work in a symbiotic way to improve their exploration capacities. For instance, UGVs can open doors or call elevators for UAVs to reach areas they would not have been able to reach otherwise. 

The collected information is provided in the real-time to the INMOS that updates the 3D digital twin of the site accordingly. 

The INMOS provides a 2D/3D/4D interactive representation of the Digital Twin of the site aggregating the multi-source, multi-format datasets available about the site (maps, BIM models, aerial imagery, documentation...) or collected in real-time by field sensors, First Responders, or cyber-assistants.  

This massive set of information is optimized in real-time by the INMOS to enable the sharing of a user-legible and interactive a Common Operational Picture among all participants including updated 3D geometry and videos from the site, position and situation of the teams and cyber-assistants, position and status of first responders, threats, incidents, victims… 

INMOS integrates advanced Decision Support capabilities powered by an Intelligence Amplification Module (IAM). This module exploits all the information available in the Digital Twin about the site, the situation and the ongoing operations, to propose in INMOS a number of recommended actions and guidelines based on rule-based doctrines predefined by the operational experts.  

The whole INTREPID platform may include several instances of interoperating INMOS deployable at will, totally autonomous and secured. The integrated, ground-to-C2 approach implies equipment of not only command centers, but also First Responders in the field, for 360° information gathering and communication. To meet this need, the INMOS is available for desktop and mobile devices, on handheld devices as well as mixed reality headset enabling hands-free operation and 3D COP visualization anywhere in the field.

INMOS interface screenshot showing inside Pilot 2 BIM.

The INTREPID project is user-centric by design. In this context, a pilot has been set up from Y1, the second one was in October 2022, and the third one, will be in Madrid, in June 2023. It's in preparation to test these functionalities, requested by the end-users, in a very operational context. The main challenge for INMOS lies in its ability to display the information and offer intuitive functions to properly command and orchestrate operations without causing mental overloading. 

INTREPID, (INtelligent Toolkit for Reconnaissance and assessmEnt in Perilous InciDents), is a ground-breaking platform for First Responders to improve and accelerate the exploration and assessment of disaster areas.

An article explaining INMOS, the plateform that centralised all the information, created for the INTREPID project.

INMOS, the operational crisis management system

INMOS - an operational crisis management system.

The RTPM or Real Time Positioning Module, is a family of different devices that allows the first responders to know their own position in real time, as well as the position of their team members. Our partner FOI is in charge of this module. 

The RTPM is designed to handle different situations and use complementary sensors technologies. Since each type of sensor works better in some conditions than in others, we want to combine several of them to make something that works more robustly and accurately over time.   

The RTPM requires no pre-installed components in the environment. If you have the luxury of putting markers or magnets or reference beacons in the environment beforehand, first responders could position themselves extremely accurately, but in the INTREPID use case first responders (FRs) arrive on the scene where nothing is installed, and they have to act fast, using only what they bring. 

Complementary sensors increase robustness 

Global navigation satellite system (GNSS) sensor

The foot-mounted sensors measure step lengths very accurately and sense even very subtle changes in direction of the feet. Integration of such data over time gives an estimate of the position. A challenge of such inertial measurement units (IMUs) in general is that the error can grow quite quickly over time. However, in RTPM we exploit the fact that during a walking or running cycle a foot stands still for even a short moment of time, and we can reduce the drift quite significantly and can maintain reliable estimates for several minutes.

An inherent limitation of the foot-mounted sensors is that they have no perception of the environment and thus no ability to recover once an error occurs, say, when a foot slips or the first responder moves in an unusual manner. Therefore, FOI combine this type of sensor with cameras-based simultaneous localization and mapping (SLAM) techniques, to increase robustness to changing conditions. In situations where camera-based techniques have problems, such as in scenes with low contrast and few features (landmarks), the foot-mounted sensors can maintain a good enough position estimate until the camera finds something to “lock on” to.

The helmet-mounted RTPM version sports cameras in the near-infrared and the thermal infrared domains. Thermal cameras give the capability to navigate in hot and smoke-filled environments. In addition to position information, the cameras track the orientation of the wearer’s head in real-time, so that team members can relay important information to each other by knowing where they are currently looking. The precision of the helmet-mounted camera is usually below 1% (1 meter if the user walks 100 meters).

Helmet-mounted RTPM with thermal stereo cameras.

The RTPM’s Ultra-wideband (UWB) radio-based ranging system measures distances between team members. This provides yet another means of positioning and further increases the robustness. While camera-based based systems work better in buildings with lots of walls and objects for cameras to look at, the precision and range of the UWB sensing go up in less cluttered environment and can help maintain position information when other systems struggle.

The laser scanner is on the robot (UGV) for extra precision. It measures distances very accurately and has longer effective range than the camera system.

The GNSS is commonly available and really useful as soon as first responders are outdoors, to take care of their basic position needs and to provide reference points to the other sensors. A typical example is when first responders move in and out of buildings. Then the GNSS helps tie everything together in the same reference system.

The sensors complement each other and they can be combined in different configurations depending on requirements concerning accuracy, size, weight, etc. The current main task is to find the best way to fuse data from the sensors to make separate components work together as an integrated unit.

The real-time positioning module includes several devices for determining the position of agents.

An article on the Real Time Positioning Module, and all the localisation solutions developped during the INTREPID project.

The Real Time Positioning Module, the combined positioning solution for first responders.

RTPM - the combined positioning solution for first responders.

Modestly, we can say that the IAM (the Intelligence Amplification Module) is the brain of the INTREPID system.  

Indeed, it’s a module that has the ability to collect all the information coming from other modules of the system, like the paths, the available agents, the available operations within the particular crisis situations. Once all that is taking into account, it does propose to First Responders, what is the estimated best course of action at the moment with the best possible outcome. 

Based to the recommendations of IAM, and in conjunction with their own experience and what’s going on in the scene, the First Responders will decide what course of action they are going to take. 

In the INTREPID Project, this module is developed by our partner CERTH. 

There are 2 flows of input to the IAM:  

One is the knowledge of the first responders of what’s happens in any particular crisis situation, codified in a machine-understandable doctrine module. It's a pre-coded knowledge of First Responders' protocols, courses of action and things to consider within a crisis scene. 

The other flow is the situational context, what’s happening right now in the scene, (what agents are available, where are robot and drones, if they are available, what are the optimal paths for each agent, etc.). 

The two together allowed a clear and specific understanding of the situation, and serve to provide appropriate proposals. 

The proposals are sent to the End Users through the INTREPID interface and from this interface, they will choose the course of action they want to take at that time and the interface will send to the mission controller the exact mission to be undertaken, to the appropriate agent, whether it be the first responder or the UxV.

A screenshot of the recommandations display on INMOS interface.

CERTH is currently working on several projects: 

Fusing expert knowledge with learning. CERTH's research directive is oriented  towards integrating machine learning with the existing expert-based system, in a neuro-symbolic approach. Combining explicit and implicit knowledge, i.e. machine learning understanding of crisis scenes while at the same time adhering to explicit First Responders’ protocols is expected to elevate the system’s decision-making and notification process. 

Decision explainability. This is an important aspect, which allows the IAM to gain the trust of the first responders, because by explaining to them how the project makes its decisions, they will see the logic and reasoning, and by seeing that it potentially coincides with their own process, they will be able to trust it more.   

Integrate more information. Expanding the whole system to more variables that can go into the modules. Indeed the IAM is the last part of this workflow, it will eventually receive more information, so it has to be extended and be able to accommodate this information.   

Pilot focus. A work is done on the process of integrating all the features that the users have identified that they needed from the system and from IAM for pilot 3, based on their feedback from the first and second pilot.  

One essential module of the INTREPID project is the IAM (the Intelligence Amplification Module).

An article explaining the role of the IAM or the Intelligence Amplification Module, the brain of INTREPID system.

IAM or the Intelligence Amplification Module, the brain of INTREPID system.

The INTREPID project intends to increase the data available to FRs dramatically, in the form of sensor data, positioning information or alerts and recommendations. This wealth of information, comes with its own challenge though: how to display and share it quickly and efficiently, in a context where every minute counts and lives are at stake?

INTREPID has chosen eXtended Reality (XR) as an innovative way to display information for increased situational awareness. XR covers three potential modes: Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR).

Virtual Reality (VR): the full immersive solution

Extended reality for early rescue operations:  the INTREPID approach

Articles

The Network, the "Hermes" of INTREPID system

SOCM, or how to combine robots' functionalities.

The UAV, a drone with 15 sensors.

EAM - the eyes of the INTREPID system.

UGV - super equipped robot.

PPM - how to calculate the best path for field agents.

EMM - creating a coherent, updated map.

INMOS - an operational crisis management system.

RTPM - the combined positioning solution for first responders.

IAM - the brain of the INTREPID system.

Let's keep in touch

Newsletter