In the dynamic landscape of public and utility infrastructure sectors, challenges such as aging infrastructure, workforce shortages, escalating regulatory requirements, and costly operational procedures are becoming increasingly apparent in recent years. Traditional methods that rely on manual inspections, fragmented data management, and limited visibility have created barriers to the optimal functioning of these crucial infrastructures. The strategic alliance between Kudan and STS Group seeks to address these challenges head-on, bringing together Kudan’s SLAM-based mobile mapping technology and digital asset platform solution offering with STS Group’s extensive industry experience in public infrastructure to guide the asset management in these sectors to next level.

“We are excited to partner with STS Group and leverage their experience in the energy industry and large-scale project as main contractor to further enhance and expand the scope of our SLAM technology and turnkey End-to-End Digital Asset Platform Solution,” said Daiu Ko, CEO of Kudan. “This cooperation will help us to deliver our state-of-art asset digitization solutions faster and provide them for Municipalities, Energy Companies, Utilities and Transmission System Operators enabling them to achieve significant productivity increase, faster design and a highly effective Operation and Maintenance against their critical assets.”

“STS Group has been a leading group of companies in the Hungarian energy market for 20 years, designing and constructing power grids and power plants, renewable energy power plants, operation and maintenance.” said Tamás Gyepes, Founder and Group CEO of STS Group. “Digitalisation in this industry is opening an increasingly important space for design and operation & maintenance. We observe a very strong connection in its development with the use of Kudan’s technology and solution, which open up a new business opportunity in the Europe electricity and utility market. Digitalization could lead to significant technological advances and efficiency improvements in the future energy sector, and the collaboration between Kudan and STS Group would play a crucial role in contributing to these advancements.”

The partnership between Kudan and STS Group represents a powerful alliance that promises innovation, efficiency, and excellence in the realm of digital asset management. As both companies embark on this critical business together, we are committed to working closely and transforming the asset operation and management for the municipality and utility infrastructure sectors in Europe.

About STS Group.
Founded in 2002, STS Group is an internationally recognized Engineering, Procurement, and Construction (EPC) company with over 15 years of experience. Specializing in High Voltage infrastructure projects, Renewable Energy initiatives, and Energy Asset Operation and Maintenance, STS Group is committed to delivering high-quality solutions across the energy sector. Learn more at https://stsgroup.hu/.

About Kudan Inc.
Kudan is a deep tech research and development company specializing in algorithms for artificial perception (AP). As a complement to artificial intelligence (AI), AP functions allow machines to develop autonomy. Currently, Kudan is using its high-level technical innovation to explore business areas based on its own milestone models established for deep tech which provide wide-ranging impact on several major industrial fields.
For more information, please refer to Kudan’s website at https://www.kudan.io/.

■Company Details
Name: Kudan Inc.
Securities Code: 4425 (TSE Growth)
Representative: CEO Daiu Ko

■For more details, please contact us from here.

The post Kudan Announces Strategic Partnership with STS Group to Jointly Expand Digital Asset Platform Solution Business towards European Municipality and Utility Infrastructure Sectors first appeared on Kudan global.

There have been considerable demands for a packaged solution of hardware and our 3D-Lidar SLAM software so that customers can easily get point clouds using our SLAM software. Even though there are existing mobile mapping solutions in the market, some segments of the geospatial industry are looking for a Dev Kit type of solution with better price-to-performance, increased flexibility of parameter configuration, and easier operation for essential functions. We observed that this requirement is prominent in the academic and research sector, and in order to respond to this market demand, we launched our own mobile mapping dev kit.

The most noticeable advantages of our dev kit are the following.

1. Excellent return on investment/price for performance: Targets half of the prices of alternative solutions without compromising the quality of point clouds
2. Flexibility: Supports different parameter settings for data collection and point cloud generation to meet the needs of various application use cases.
3. Simplicity: Less than an hour to start scanning after unboxing for handheld, with all essential functionalities supported.

You can find more information about the dev kit on our website, so please visit and send us an inquiry for more details.

• Handheld: https://www.kudan.io/mapping_dev_kit/handheld_version/
• Vehicle mount: https://www.kudan.io/mapping_dev_kit/vehicle_mount_version/

We are continuing to offer our software to enhance existing mobile mapping solutions that rely on INS or other SLAM solutions to enhance their performance.

About Kudan Inc.
Kudan is a deep tech research and development company specializing in algorithms for artificial perception (AP). As a complement to artificial intelligence (AI), AP functions allow machines to develop autonomy. Currently, Kudan is using its high-level technical innovation to explore business areas based on its own milestone models established for deep tech which provide wide-ranging impact on several major industrial fields.
For more information, please refer to Kudan’s website at https://www.kudan.io/.

■Company Details
Name: Kudan Inc.
Securities Code: 4425 (TSE Growth)
Representative: CEO Daiu Ko

■For more details, please contact us from here.

The post Kudan launched its affordable mobile mapping dev kit for vehicle and handheld first appeared on Kudan global.

This time, our demo showcases the result in one of the most typical environments and setups.

• Recorded in an urban area (station square of a large train station in Japan) with some buildings around and roofed areas, which prevents good GNSS signals
• Recorded with a vehicle-base sensor setup with a tilted 3D-Lidar with INS

Here is the demo video – Kudan 3D-Lidar SLAM in Action: Vehicle Mobile Mapping in an Urban Area

There’s nothing much to say about this video since it’s quite straightforward and the video explains itself. However, the main benefits you get from KdLidar are …

1. Up to <1cm accuracy in various environments: Tested and proven with multiple geospatial partners to perform with up to <1cm accuracy both indoors and outdoors
2. Flexible sensor choices: Various sensors (3D Lidar, IMU, and INS) tested and supported
3. Faster time-to-market: Much faster time-to-market than internal development or other SLAM software (We have a team of 30 SLAM engineers)

Please stay tuned for further updates!

■For more details, please contact us from here.

The post Kudan 3D-Lidar SLAM (KdLidar) in Action: Vehicle-Based Mapping in an Urban area first appeared on Kudan global.

We share the basic concept of SLAM with cloud/ 5G network and its example in autonomous mobile robots applications in our previous article. In this second part, we are going to share examples of visual positioning, AR cloud, and autonomous driving applications.

We believe this will help you understand various use cases where SLAM and 5G/Cloud can play a role together.

Use-case examples in visual positioning

We have seen an increasing demand for use cases in the visual positioning of people or machines in an indoor setting together with AMR.

One example is understanding the positions of operators in industrial facilities such as a warehouse or a power plant so that the user can provide a warning to operators when they get close to a hazardous area. The user can also use this information to improve the productivity of operators.

In these applications, the cost of the hardware they wear needs to be very affordable as the solution requires as many units as the number of operators. Generally, more than 100 units are required in a warehouse.

In such a scenario, offloading SLAM processing onto the cloud is the key to minimizing hardware costs. Figure 1 illustrates a suggested architecture to achieve this.

Figure 1: High-level architecture of SLAM on cloud example of operator positioning in a warehouse.

A stereo camera can be used to create a map instead of a mono camera, resulting in more stable performance. However, each positioning/ tracking device mounted can be a mono camera mounted on the helmet to minimize the hardware cost.

This application doesn’t need 5G necessarily, but 5G ensures mission-critical connectivity and low latency for these operators.

Another noteworthy example of visual positioning is forklift position tracking.

Many companies deploy a fleet of manual forklifts and want to understand how efficiently they operate and improve overall productivity. The key to achieving this is recognizing the position of the forklifts.

Since multiple forklifts would be deployed, the hardware costs need to be a bare minimum, and 2D or 3D lidars aren’t an option. We can create a map with one forklift by scanning the area and then using this map to get the positions of forklifts in the area using their stereo camera or even a mono camera, depending on the required accuracy. This architecture is further painted in figure 2.

Figure 2: High-level architecture of SLAM on cloud example of forklift position tracking

This scenario doesn’t demand the position of forklifts at every camera image frame (such as 30 fps) but does require something around 1 FPS. So the system could only send 1 pair of images (from the left and right lenses) every second to understand the overview of forklift operations.

As you may have already understood, this application doesn’t need 5G, but 5G would make communication more reliable.

AR Cloud use case example of SLAM and 5G/Cloud

Augmented Reality (AR) Cloud implies an AR application using a map stored on the cloud and also using the position of the device on that map. This is another common usage of SLAM.

However, it is hard to meet the computation and memory requirements for augmented reality applications [1].

For example, a person holds his smartphone and looks around with it. The smartphone understands where it is and which direction the person is facing.

So an AR cloud app can show the direction to a specific location based on this or overlay ads on the actual scenery on the screen. Figure 3 shows the simple architecture of this application.

Figure 3: High-level architecture of SLAM on cloud example of AR cloud.

Autonomous driving using SLAM and 5G/Cloud

SLAM on the cloud provides another level of scalability and flexibility to autonomous driving applications.

One of the main challenges of autonomous driving at scale is maintaining up-to-date maps on each of the vehicles. A straightforward solution is to keep a master map on the cloud and let each vehicle consume the necessary part of this map as they drive.

Now the problem gets simpler, as the task is to maintain that single map. So how can we keep this single map up-to-date?

Figure 4: High-level architecture of SLAM on cloud example in autonomous driving.

One approach is using sensor data from each vehicle and updating the single map whenever needed using this data. Thus each vehicle not only consumes the map data but also acts as a mapping agent, as seen in figure 4.

The system could adjust how what portion of the map each vehicle downloads based on the vehicle speed and the network speed. As you already might have guessed, 5G can help download the partial map with low latency and stable connectivity to each vehicle.

Final words

This article showed how cloud and 5G communication technology could help SLAM be widely adopted across multiple use cases.

The examples we showed prove that this crossover of technologies adds more flexibility to how the SLAM can be used. Of course, the use cases we’ve listed here may not be all, and you may have your specific use case requirements for the business problems you have at hand.

Say hi, and we’d be happy to help you transform your business through our SLAM solutions!

References

[1] Jiao, J., Yun, P. and Liu, M. (2017). A Cloud-Based Visual SLAM Framework for Low-Cost Agents. 471–484. [PDF]

The post Endless Possibilities with SLAM and 5G/Cloud Technology Together (Part 2) first appeared on Kudan global.

3D-SLAM is of the instrumental technologies across several use-cases involving robotics, mapping, drones, autonomous vehicles, and AR/VR. Not second to any, the cloud and 5G communications are disrupting the technology industry with multiple use cases.

What possibilities can be unlocked if we can combine these two disruptive technologies in use cases?

At Kudan, we have already delivered multiple projects in the crossover: SLAM on the cloud and SLAM using 5G communications. This article will further explore and paint a clear picture of some of the problems that can be solved by the 3D-SLAM and 5G/cloud technologies.

As a first step, let’s understand how these technologies can be combined in the use cases.

Understanding the crossover of SLAM and Cloud technology

Kudan Visual SLAM can run 5–10 times faster than other 3D SLAM algorithms (eg: ORB-SLAM2 on some Arm-based processors). Still, as it requires iteration of optimization using a large number of 3D point information, 3D-SLAM can be heavy and process-intensive for tight processing budget applications.

As a result, not all applications can afford the processing hardware that is suitable for 3D SLAM [1]. Often this acts as a blocker for adopting 3D SLAM for use cases. The ability to offload the SLAM process elsewhere can help adopt the technology into many more use cases.

Then cloud comes into play here. We can offload the SLAM process to the cloud. Let us explain through an example: for robotic applications, the robots can send the images from their sensors to the cloud. The SLAM process can now run on the cloud and send the information only regarding the poses back to the robot. The robot can use this information for its own control and motion.

This is truly disruptive and SLAM on the cloud already has significant potential to expand the usage of SLAM to multiple use cases. The figure 1 below does a great job of explaining the architecture in the usage of cloud alongside SLAM.

Figure 1: High-level architecture of Visual SLAM on the cloud.

Deteriorating accuracy and role of 5G communication technology

The problem we faced for some SLAM-on-cloud projects was the latency and robust connectivity that wasn’t up to the mark when sending the image stream to the cloud and receiving the required information back.

For use cases requiring continuous pose information at 10Hz or more, this is crucial. And as a result, we experienced deteriorated accuracy and an increased risk of collision in the robots. We needed a way to communicate better between the robots and the cloud.

That’s when 5G technologies becomes relevant with SLAM. It provides higher speeds, superior reliability, and negligible latency. In our context, image sequences from multiple robots can be sent through 5G technology without significant latency. It enabled the application to use low-cost, low-power hardware for edge devices while still benefiting from 3D SLAM features.

You now probably have a good understanding of the architecture and the usage of both cloud and 5G technology for SLAM. Let’s now visit a use-case example that leverage these technologies.

Use-case example: Autonomous Mobile Robots (AMRs)

Visual SLAM functionality can be added to an existing autonomous mobile robot (AMR) to make its localization performance more robust and stable.

Many AMRs use 2D-Lidar SLAM for localization. Though it shows acceptable performance for SLAM-friendly environments, it struggles in environments where the scenery is constantly changing or occasional outdoor operations such as between factories.

Visual SLAM has a clear advantage when fused with its 2D counterpart in these scenarios. Figure 2 illustrates how the Visual SLAM can be used with the cloud service.

Figure 2: High-level architecture of Visual SLAM on cloud example in robotics applications.

As shown in the architecture above, you can obtain the pose from Visual SLAM through the cloud and fuse it with 2D-Lidar SLAM pose to get more redundancy.

The frequency of sending images can be flexible depending on the purpose of Visual SLAM and the availability of 5G.

In instances where there is a good 5G network available, you may send an image stream at 30 frames per second (FPS). When not available, you may send an image every second (1 FPS) and use it as an aiding approach on top of 2D-Lidar SLAM.

Another option is to dynamically change between Visual SLAM and 2D Lidar SLAM; for instance, when the robot is outdoor, it sends images more frequently so that it can rely solely on Visual SLAM, and when indoors, it sends at a lower frequency indoors to limit network usage.

As you can see, you suddenly have many options to enhance the system’s overall performance.

Final words

Did you really think that’s all the use case examples of SLAM with 5G/Cloud? There’s plenty more. But let’s stop here for now as we already introduced many new ideas around 3D SLAM.

Many more practical examples will be discussed in more detail in part 2 of this article which includes examples from Visual positioning and autonomous driving. So you can have an even more detailed understanding of the applications of SLAM. Stay tuned!

Meanwhile, feel free to say hi, and we’d be happy to help you transform your business through our SLAM solutions!

References

[1] Kamburugamuve, S., He, H. & Fox, G. and Crandall, D. (2016). Cloud-based Parallel Implementation of SLAM for Mobile Robots. [PDF]

The post Endless Possibilities with SLAM and 5G/Cloud Technology Together (Part 1) first appeared on Kudan global.

Do you want to implement 3D-Lidar SLAM for your use case successfully?

Have you heard of the 3D-Lidar SLAM but are unsure how to maximize its performance?

Do you find tweaking the 3D-Lidar SLAM parameters complex and confusing?

If you answered yes to any of these questions, you would find a lot of value in this article. Once you’ve decided to use 3D-Lidar SLAM for your business use case, you will need to tick off these three checkboxes:

1. Pick an appropriate 3D-Lidar sensor unit
2. Decide on a proper 3D-Lidar SLAM approach
3. Choose the relevant parameters for the SLAM system

If you are unsure of the first two checkboxes, we’ve written about the 3D-Lidar SLAM approach and how you can select the appropriate sensor for your use case. We’ll leave the links at the end of this article for you to delve deeper if you haven’t already.

In this article, we’ll share some descriptions, typical values, and high-level guidelines on the 4 main parameters, which commonly are included in many 3D-Lidar SLAM approaches. We’ll also talk about how these parameters may impact the overall performance of the system.

Voxel sizes and why they matter

Smaller voxel size improves robustness and accuracy but has a price to pay

For a SLAM system, we treat the 3D space as a group of small 3D spaces known as voxels. Voxel size indicates how granular or coarse you want to dissect the 3D space. In other words, what dimension should a voxel be?

In theory, a smaller voxel size means improved robustness and accuracy even though it saturates at some point. This is because when the voxel size is small, the SLAM system dissects the 3D space into many smaller 3D spaces and thus has been able to extract more points for tracking and mapping. This ability to capture more points allows for more points to track against when moving, resulting in greater robustness and accuracy.

However, when the system has more points to process for tracking and mapping, the system is inevitably slower, introducing the processing time vs. accuracy trade-off.

So when should you lower the voxel size? We recommend using a smaller voxel size if you are in a small indoor place near because the points concentrate in a small 3D space and it’s not possible to have meaningful number of points if voxel size is large. Also, we recommend to do so for mapping applications when real-time processing isn’t required.

Here’s what typical voxel sizes in meters may look like:

Figure 2: Typical values for voxel size (in meter)

As a thumb of the rule, we would start from 1.0m for outdoor robotics, 0.5m for large indoor robotics, and 0.3m for small indoor (e.g., office space) robotics. As you may have noticed, the parameters are relatively low for mapping to gain more accuracy in sacrifice of processing speed in general.

However, in the case that your SLAM system doesn’t track well with the suggested voxel size, we would recommend trying a smaller voxel size. Conversely, if your SLAM system cannot catch up with real-time feed, try a larger voxel size to reduce the number of points used.

The key is understanding the impact of the voxel size parameter to inform the decision on changing the value as required, which you do now.

Understand the maximum distance for matching the current frame to the map

This parameter indicates how far you want to use an existing keyframe in the Iterative Closest Point [1] process to determine the current pose instead of creating a new one.

In other words, this parameter determines how frequently you want to create a new keyframe. The longer the distance you set, the less frequent the system creates a new keyframe.

Some typical values (in meters) are given in the image below:

Figure 3: Typical values for maximum distance for matching frames to map (in meters)

For indoor applications, we set smaller values since we’d like to have more frequent keyframes than our outdoor use cases to detect movements more precisely.

Pay attention to the voxel size when deciding on the minimum matched points to track

The parameter “minimum number of matched points to track” sets the threshold to decide whether or not the tracking has failed.

When the number of matched points between the keyframes and the current frame is below this threshold value, the SLAM system is said to be ‘lost’.

Intuitively, you can say that the smaller the number of matched points that are required to track, the easier it is for the SLAM system to continue tracking without getting lost. However, setting a too small number introduces false tracking and accumulates larger drift.

It’s essential to consider the voxel size you have already set in order to select the appropriate value for this parameter. If you’ve set a larger voxel size, you’d have fewer points for matching; the minimum matched points needs to be set lower.

The number below is the example for our SLAM system which uses absolute values, but it could be relative numbers or percentages as well.

Figure 4: Typical values for the minimum number of points to track

If the environment has rich structures, set a higher value while reducing it when there are fewer objects around the lidar (e.g., a fairly open field with several trees). Thus, indoor environments typically have higher values than outdoor environments.

Setting the minimum number of match points to relocalize

The parameter “minimum number of points to relocalize” sets a threshold to decide if the initial position on the map is identified.

When the number of matched points between the keyframes and the current frame is above the given threshold value, the SLAM system considers it as being “relocalized.” Here’s an example from our SLAM system. Again we’re using absolute values here, but it could be relative numbers or percentages as well.

Figure 5: Values for “minimum number of points to relocalize”

When you set this parameter to a smaller value, the SLAM finds it easier to relocalize. However, selecting a too-small value introduces false relocalization [2].

A rule of thumb is to set it to 1500 and depending on how the SLAM behaves (eg. more loop closures performed or higher experience of false relocalization) you can reduce or increase it accordingly.

Final words

At Kudan, through our blog, we’ve aimed to distill the information and present it as simply as possible. Here are some of our previous articles on 3D Lidar SLAM:

• 3D lidar SLAM: The Basics
• How to Select the Best 3D Lidar for SLAM

In this article, we’ve presented the goal of tuning 3D-Lidar SLAM parameters, general guidelines, typical values, and their impact on the system’s performance. The list of parameters to tweak and additional tricks to improve SLAM accuracy and robustness expands wider than what we’ve been able to cover here. If you’re interested to learn more or if you have a question in mind, we invite you to say hi to us for your specific needs around the technology.

References

[1] Chen, Yang & Medioni, Gerard. (1992) “Object modeling by registration of multiple range images,” Image and Vision Computing, vol. 10, no. 3, pp. 145–155.[PDF].
[2] Vysotska, Olga & Stachniss, Cyrill. (2017). Relocalization under Substantial Appearance Changes using Hashing [PDF]

■For more details, please contact us from here.

The post How to Tune 3D-Lidar SLAM Parameters first appeared on Kudan global.

Long narrow corridors in office buildings and industrial facilities are quite common environments for robotics and mapping applications. However, they are one of the more challenging environments for Lidar SLAM due to repetitive structural appearances and limited GNSS signals (You can learn how 3D Lidar SLAM works in this blog post). This time, we are showcasing Kudan Lidar SLAM (KdLidar) taking on this challenging environment with ease in this challenging environment using a very simple setup.

Kudan Lidar SLAM works robustly in these challenging but common environments without any external sensors

This scan was done only with an Ouster lidar (OS1-32) using its built-in IMU as a handheld scanner shown in the picture below.

Kudan Lidar SLAM in action: In long narrow corridors in an office building

(The data credit: UCS in Korea)

As you may know, Kudan Lidar SLAM will detect and perform loop closure to optimize the point cloud map, and it indeed detected some loops during this scan but the effects are barely noticeable due to the very limited drift generated during the scan. This is a good indicator that Kudan Lidar SLAM is managing this challenging environment with ease.

This type of environment is challenging yet quite common across various applications.

• Mapping and tracking for autonomous service robots operating within offices and commercial buildings
• Mapping of office buildings for inspection, maintenance, and facilities management
• Progress monitoring and documentation of construction worksites

Here are some of the details of the environment, and demo parameters.

• Size of the area: Each corridor runs 50m – 60m length
• Sensor: Ouster OS1-32 3D lidar: Only lidar, without further sensor fusion (however, we can utilize other sensors if needed)
• The SLAM video is generated in real-time for progress monitoring and the high-density point cloud map is generated during post-processing

Please feel free to reach out to us if you operate within these types of environments or have other challenging environments that you need a solution for. See the difference commercial-grade SLAM can make as part of your solution. We are happy to solve the problems together.

About Kudan Inc.
Kudan (Tokyo Stock Exchange securities code: 4425) is a deep tech research and development company specializing in algorithms for artificial perception (AP). As a complement to artificial intelligence (AI), AP functions allow machines to develop autonomy. Currently, Kudan is using its high-level technical innovation to explore business areas based on its own milestone models established for deep tech which provide wide-ranging impact on several major industrial fields.
For more information, please refer to Kudan’s website at https://www.kudan.io/.

■Company Details
Name: Kudan Inc.
Securities Code: 4425
Representative: CEO Daiu Ko

■For more details, please contact us from here.

The post Kudan 3D-Lidar SLAM (KdLidar) in action：Long narrow corridors first appeared on Kudan global.

This data was collected by a handheld device with a 3D-lidar. As some of you already know, underground environments are one of the more challenging environments for conventional geospatial equipment due to the unavailability of GNSS and complex structure.

The video shows how KdLidar works in this environment to collect data, and create a beautiful point cloud.

Try to guess what sensor we used for this work while you are watching this video!

Here is the video:
Kudan Lidar SLAM: In an underground cave

Many existing solutions only allow you to see the output once you store the data and process it offline, and you cannot see how the scanning work is going during the capturing process or immediately after it. This can potentially lead to significant loss of productivity. As you can see, the user is able to have a good understanding of how the scanning is going in real-time (as the point cloud is being generated), and then create a denser and more crisp point cloud in post-processing.

One thing to note, is that this data was collected only using an Ouster OS0-32, without using its internal (or an external IMU, and obviously without GNSS). Lidar SLAM without IMU in a handheld configuration is another major challenge due to continuous natural human motions, which can cause ghosting, blurred points and trajectory drift.

Our partner, who collected this data, was quite satisfied with the final result for their mapping purpose. However, if you want consistently good-looking results, with higher accuracy, we can boost the performance with more tuning and sensor fusion of IMU.

Here are some of the details of the environment, and demo parameters.

• Size of the area: 100m x 70m (or 330 ft x 230 ft)
• Sensor: Ouster OS0-32 3D lidar: Only lidar, without further sensor fusion (however, we can utilize other sensors if needed)
• The map was generated after successful loop closure

We hope you are enjoying this “Kudan SLAM in action” series as you get to see Kudan SLAM’s robustness, accuracy, reliability and flexibility in real environments.

About Kudan Inc.
Kudan (Tokyo Stock Exchange securities code: 4425) is a deep tech research and development company specializing in algorithms for artificial perception (AP). As a complement to artificial intelligence (AI), AP functions allow machines to develop autonomy. Currently, Kudan is using its high-level technical innovation to explore business areas based on its own milestone models established for deep tech which provide wide-ranging impact on several major industrial fields.
For more information, please refer to Kudan’s website at https://www.kudan.io/.

■Company Details
Name: Kudan Inc.
Securities Code: 4425
Representative: CEO Daiu Ko

■For more details, please contact us from here.

The post Kudan 3D-Lidar SLAM (KdLidar) in action: In a subterranean cave for geospatial applications first appeared on Kudan global.

In my last article, we looked at feature-based visual SLAM (or indirect visual SLAM), which utilizes a set of keyframes and feature points to construct the world around the sensor(s). This approach initially enabled visual SLAM to run in real-time on consumer-grade computers and mobile devices, but with increasing CPU processing and camera performance with lower noise, the desire for a denser point cloud representation of the world started to become tangible through Direct Photogrammetric SLAM (or Direct SLAM). A denser point cloud would enable a higher-accuracy 3D reconstruction of the world, more robust tracking especially in featureless environments, and changing scenery (from weather and lighting). In this article, we will specifically take a look at the evolution of direct SLAM methods over the last decade, and some interesting trends that have come out of that.

Direct SLAM started with the idea of using all the pixels from camera frame to camera frame to resolve the world around the sensor(s), relying on principles from photogrammetry. Instead of extracting feature points from the image and keeping track of those feature points in 3D space, direct methods look at some constrained aspects of a pixel (color, brightness, intensity gradient), and track the movement of those pixels from frame to frame. This approach changes the problem being solved from one of minimizing geometric reprojection errors, as in the case of indirect SLAM, to minimizing photometric errors.

The direct visual SLAM solutions we will review are from a monocular (single camera) perspective. Having a stereo camera system will simplify some of the calculations needed to derive depth while providing an accurate scale to the map without extensive calibration.
In the interest of brevity, I’ve linked to some explanations of fundamental concepts that come into play for visual SLAM:

• Structure from motion (SfM)
• Motion estimation
• Optical flow

While these ideas help in the deeper understanding of some of the mechanics, we’ll save them for another day.

Dense tracking and mapping (DTAM, 2011)

The DTAM approach was one of the first real-time direct visual SLAM implementations, but it relied heavily on the GPU to make this happen. Grossly simplified, DTAM starts by taking multiple stereo baselines for every pixel until the first keyframe is acquired and an initial depth map with stereo measurements is created. Using this initial map, the camera motion between frames is tracked by comparing the image against the model view generated from the map. With each successive image frame, depth information is estimated for each pixel and optimized by minimizing the total depth energy.
The result is a model with depth information for every pixel, as well as an estimate of camera pose. Since it is tracking every pixel, DTAM produces a much denser depth map, appears to be much more robust in featureless environments, and is better suited for dealing with varying focus and motion blur.
The following clips compare DTAM against Parallel Tracking and Mapping: PTAM, a classic feature-based visual SLAM method.

With rapid motion, you can see tracking deteriorate as the virtual object placed in the scene jumps around as the tracked feature points try to keep up with the shifting scene (right pane). DTAM on the other hand is fairly stable throughout the sequence since it is tracking the entire scene and not just the detected feature points.

Since indirect SLAM relies on detecting sharp features, as the scene’s focus changes, the tracked features disappear and tracking fails. This can occur in systems that have cameras that have variable/auto focus, and when the images blur due to motion.

In this instance, you can see the benefits of having a denser map, where an accurate 3D reconstruction of the scene becomes possible.

Source video: https://www.youtube.com/watch?v=Df9WhgibCQA

Visual odometry and SLAM

We will start seeing more references to visual odometry (VO) as we move forward, and I want to keep everyone on the same page in terms of terminology. As described in previous articles, visual SLAM is the process of localizing (understanding the current location and pose), and mapping the environment at the same time, using visual sensors. An important technique introduced by indirect visual SLAM (more specifically by Parallel Tracking and Mapping – PTAM), was parallelizing the tracking, mapping, and optimization tasks on to separate threads, where one thread is tracking, while the others build and optimize the map. For the purposes of this discussion, VO can be considered as focusing on the localization part of SLAM. The VO process will provide inputs that the machine uses to build a map. However, it will need additional functions for map consistency and optimization.

Large Scale Direct SLAM (LSD-SLAM, 2014)

Building on earlier work on the utilization of semi-dense depth maps for visual odometry, Jakob Engel (et al.), proposed the idea of Large Scale Direct SLAM. Instead of using all available pixels, LSD-SLAM looks at high-gradient regions of the scene (particularly edges) and analyzes the pixels within those regions. The idea being that there was very little to track between frames in low gradient or uniform pixel areas to estimate depth. For single cameras, the algorithm uses pixels from keyframes as the baseline for stereo depth calculations.
The following image highlights the regions that have high intensity gradients, which show up as lines or edges, unlike indirect SLAM which typically detects corners and blobs as features.

Image from Engel’s 2013 paper on “Semi-dense visual odometry for monocular camera

To complement the visual odometry into a SLAM solution, a pose-graph and its optimization was introduced, as well as loop closure to ensure map consistency with scale. In the following clip, you can see a semi-dense map being created, and loop closure in action with LSD-SLAM. You can see the map snap together as it connects the ends together when the camera returns to a location it previously mapped.

Source video: https://www.youtube.com/watch?v=GnuQzP3gty4

With the move towards a semi-dense map, LSD-SLAM was able to move computing back onto the CPU, and thus onto general computing devices including high-end mobile devices. Variations and development upon the original work can be found here: https://vision.in.tum.de/research/vslam/lsdslam

Semi-direct Visual Odometry (SVO / SVO2, 2014 / 2016)

In the same year as LSD-SLAM, Forster (et al.) continued to extend visual odometry with the introduction of “Semi-direct visual odometry (SVO)”. SVO takes a step further into using sparser maps with a direct method, but also blurs the line between indirect and direct SLAM. Unlike other direct methods, SVO extracts feature points from keyframes, but uses the direct method to perform frame-to-frame motion estimation on the tracked features. In addition, SVO performs bundle adjustment to optimize the structure and pose. Extracted 2D features have their depth estimated using a probabilistic depth-filter, which becomes a 3D feature that is added to the map once it crosses a given certainty threshold.
The advantages of SVO are that it operates near constant time, and can run at relatively high framerates, with good positional accuracy under fast and variable motion. However, without loop closure or global map optimization SVO provides only the tracking component of SLAM.

Source video: https://www.youtube.com/watch?v=2YnIMfw6bJY

Direct Sparse Odometry (DSO, 2016)

After introducing LSD-SLAM, Engel (et al.) took the next leap in direct SLAM with direct sparse odometry (DSO) – a direct method with a sparse map. Unlike SVO, DSO does not perform feature-point extraction and relies on the direct photometric method. However, instead of using the entire camera frame, DSO splits the image into regions and then samples pixels from regions with some intensity gradients for tracking. This ensures that these tracked points are spread across the image.
The result of these variations is an elegant direct VO solution. Similar to SVO, the initial implementation wasn’t a complete SLAM solution due to the lack of global map optimization, including loop closure, but the resulting maps had relatively small drift. As you can see in the following clip, the map is slightly misaligned (double vision garbage bins at the end of the clip) without loop closure and global map optimization.

The following clip shows the differences between DSO, LSD-SLAM, and ORB-SLAM (feature-based) in tracking performance, and unoptimized mapping (no loop closure).

You can see LSD-SLAM lose tracking midway through the video, and the ORB-SLAM map suffers from scale drift, which would have been corrected upon loop closure. But it is worth noting that even without loop closure DSO generates a fairly accurate map.

Source video: https://www.youtube.com/watch?v=C6-xwSOOdqQ

There is continuing work on improving DSO with the inclusion of loop closure and other camera configurations. However, DSO continues to be a leading solution for direct SLAM. The research and extensions of DSO can be found here: https://vision.in.tum.de/research/vslam/dso

Final Words

While the underlying sensor and the camera stayed the same from feature-based indirect SLAM to direct SLAM, we saw how the shift in methodology inspired these diverse problem-solving approaches. We’ve seen the maps go from mostly sparse with indirect SLAM to becoming dense, semi-dense, and then sparse again with the latest algorithms. At the same time, computing requirements have dropped from a high-end computer to a high-end mobile device. It’s important to keep in mind what problem is being solved with any particular SLAM solution, its constraints, and whether its capabilities are best suited for the expected operating environment.

For further reading:

1. Newcombe, S. Lovegrove, A. Davison, “DTAM: Dense tracking and mapping in real-time,”  (PDF)
2. Engel, J. Sturm, D. Cremers, “Semi-dense visual odometry for a monocular camera, “ (PDF)
3. Engel, T. Schops, D. Cremers, “LSD-SLAM: Large-scale direct monocular SLAM,” (PDF)
4. Forster, M. Pizzoli, D. Scaramuzza, “SVO: Fast semi-direct monocular visual odometry,” (PDF)
5. Forster, Z. Zhang, M. Gassner, M. Werlberger, D. Scaramuzza, “SVO: Semi-direct visual odometry for monocular and multi-camera systems,” (PDF)
6. Engel, V. Koltun, D. Cremers, “Direct Sparse Odometry,” (PDF)

The post Direct Visual SLAM first appeared on Kudan global.