【虹科方案】 aiSim 3DGS 自動駕駛仿真方案 – 高保真場景重建 × 數位孿生 × 智能測試閉環

Hongke's latest articles

HongKe

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

From "reconstruction" to "usability": How aiSim 3DGS closes the loop on automated driving simulation scenarios?

In the rapid development of automated driving technology, high precision and high fidelity simulation scene construction is the core key.3D Gaussian Splatting (3DGS)With efficient rendering and realistic scene restoration capabilities, 3DGS has gradually become the focus of 3D reconstruction and simulation. However, in practical applications, how to efficiently transform heterogeneous data from multiple sources into usable 3DGS scenes and ensure their consistency with the real environment is still a pain point in the industry.

Addressing the challenge of landing 3DGS for automated driving simulations.aiSim SolutionA complete closed-loop process from raw data standardization to high-fidelity simulation validation has been constructed: through the aiData Toolchain Realize multi-source data concordance; ensure high realism of the scene by combining algorithms; and GGSR Renderer It achieves a "high efficiency + realistic" rendering closed loop, and can freely configure extreme environments such as rainstorms and nighttime, simulate multimodal sensors, and virtual traffic flow to cover various extreme test conditions of autonomous driving.

3DGS Underlying Technology Analysis

3DGS is a technology based on 3D Gaussian Distribution The core concept of the scene representation is to transform each object in the scene into multiple 3D Gaussian points. Each Gaussian point acts as a "data capsule" containing key information such as position, covariance matrix, and opacity, which is used to accurately depict the geometric contours and lighting characteristics of the scene.

Discretely distributed Gaussian points (left) → three-dimensional world composed of multiple Gaussian points (right)

From a process perspective, 3DGS first utilizes the SfM (Structure from Motion) Data preprocessing. Through multi-view image analysis, the technology corrects the camera position and restores its internal and external parameters, generating sparse point clouds that serve as the basis for subsequent scene construction. Based on these point clouds, the system initializes a set of 3D Gaussian points with initial values for position, covariance matrix, and opacity.

3DGS Process Schematic

During the training phase, 3DGS will continue to optimize the position, shape and transparency of the Gaussian points with innovativeAdaptive Density Control Strategy. Gaussian points with low contribution to the scene are automatically removed after each reverse propagation, and keypoints are split or cloned according to detail requirements to balance efficiency and detailed representation.

In contrast to the traditional NeRF (Neural Radiance Fields) Compared to 3DGS, NeRF has a significant advantage in generating realistic continuous 3D scenes and mapping spatial coordinates to colors and densities, but it is computationally intensive, and the training of a single scene often requires a lot of computational power and time, especially at high-resolution outputs.

In addition, NeRF has limited editability and any scene editing requires retraining of the entire process, whereas 3DGS is a new and more efficient way of editing withApparent ModelingThe 3D Gaussian points capture the details of the scene, enable highly accurate reconstruction, and support the development of a new, more accurate rendering of the scene, while avoiding the training overhead of a highly loaded neural network.Instant RenderingThe

Comparison of NeRF and 3DGS Realization Processes

However, 3DGS is not perfect. In extremely complex 3D scenes, a large number of Gaussian points may be required to visualize every detail, resulting in a higher computational and memory burden. Currently, 3DGS is mostly used for static reconstruction, so how to efficiently handle this?Dynamic SceneAccurately tracking the shape and trajectory of an object remains a scientific and engineering challenge.

3DGS Total Process Solution based on aiSim

Raw Data Entry and Standardization

Starting from multi-source sensor data, real road images, point clouds, and positional data are captured by cameras, LiDAR, and in-vehicle motion sensors. In the face of heterogeneity in format, accuracy and timestamps, theaiData Toolchain Standardized conversion of third-party data allows point clouds, images, and calibration information to work in unison to ensure accurate execution of follow-up processes.

3D Scene Preprocessing

incorporate 3D auto annotation, 2D semantic segmentation, camera position optimization Three parts:

3D auto annotation:In the aiData toolchain, image, point cloud, and millimeter wave features are fused by multimodal data and Super MS2N algorithm to accurately identify 3D targets and generate annotation frames. "The Non-Causal Tracking (NCT) module optimizes the trajectory across frames, corrects errors, and finally outputs high-precision GT data to semantically associate 3DGS scenes.
2D semantic segmentation:Semantic segmentation of image data and output of annotations to assist in 3D detail optimization.
Camera Position Optimization:Calibrate and optimize sensor positions to ensure accurate spatial coordinates for 3D reconstruction.

3DGS Scene Reconstruction

Based on high quality data after cleaning, theaiSim initiates neural network reconstruction process: Combining the geometric generalization capabilities of NeRF with the real-time rendering features of 3DGS to build the T-S transmodal structureIn this work, the depth, normal, appearance, and other supervision signals learned from NeRF are migrated to the Gaussian parameter optimization of 3DGS through the multimodal synergy training (LiDAR depth constraints are introduced). Finally, the discrete point clouds and images are transformed into continuous 3D Gaussian scenes, realizing the efficient mapping of "real scene → digital derivative".

Schematic diagram of scene reconstruction model

The T-S structure serves as a key bridge to enable the depth and appearance signals learned by NeRF to be smoothly injected into 3DGS, which, combined with the LiDAR depth constraints, further enhances the geometric accuracy so that the Gaussian point locations and the covariance matrices can be optimized to highly match the real-world scenarios.

Through this process, discrete point clouds and image data are transformed into a continuous and realistic 3D Gaussian scene, laying a reliable foundation for subsequent scene editing and simulation.

To verify the accuracy of the reconstructionaiSim Introduces DEVIANT and Mask2Former Dual Algorithm CalibrationDEVIANT focuses on geometric accuracy and simulates monocular 3D inspection logic to detect whether the depth, position and size of vehicles and pedestrians in the scene are the same, avoiding target drift and deformation.

Validation of 3D target inspection based on DEVIANT algorithm

The results show that the model is able to successfully recognize reconstructed vehicles without significant domain bias; the non-detection at long range is due to model range limitation.

Mask2Former Focuspixel consistencyIn order to verify the texture and boundary consistency, the reconstructed rendering is compared with the real image by semantic segmentation, and the local features are extracted to verify the texture and boundary consistency.

Measure the difference between synthesized and real data based on Mask2Former.

The comprehensive results show that aiSim's 3DGS reconstruction is highly compatible with the real environment in terms of geometry, texture, and semantics, realizing "both the spirit and the shape", and providing a credible simulation basis for the autonomous driving test.

Scene Editing and Simulation Closures

aiSim scene editing tool Provides powerful customization capabilities: Flexibility to add virtual traffic flow on top of the 3DGS base scene to set vehicle routes, speeds and densities to simulate city or highway scenes; also supports extreme climates such as rainstorms, snowstorms, night lighting, etc. to enhance test fidelity.

Deploying multi-mode sensors can simulate the behavior of different sensors in various environments, which can comprehensively test the fusion capability of autonomous driving on multi-source data and expand the application value of a single scene.

GGSR (General Gaussian Splatting Renderer) The high-fidelity rendering core is deeply optimized for wide-angle lens distortion, ensuring high consistency and clarity at large FOVs and reducing artifacts. It also supports arbitrary camera distortion models to accurately simulate color, brightness, contrast, and distortion corrections so that the simulated data more closely matches the real sensor output. Based on the shared ray-Gaussian interaction logic, it can accurately simulate laser reflection and collision behaviors, realizing a complete closed loop from data acquisition to simulation verification.

Render Pipeline Overview

Value Closure of aiSim 3DGS Solution

Hi-Tech aiSim 3DGS Solution Builds "The Future of 3DGS" with Full-Process Technological InnovationData standardization - Scene high fidelity - Simulation full coverageThe value loop of "3DGS" will be closed, realizing the transformation of 3DGS from technical potential to engineering practice.

In terms of pain point solution, aiData toolchain coordinates multi-source data to overcome the fragmentation of 3DGS input; T-S architecture combines the strengths of NeRF and 3DGS with LiDAR depth constraints to achieve accurate geometry and appearance reconstruction; DEVIANT and Mask2Former double verification ensures that the reconstructed scene is consistent with the real environment; and GGSR Renderer combines high efficiency and high fidelity to satisfy the stringent requirements of autonomous driving simulation. The GGSR renderer combines high efficiency and high fidelity to meet the demanding needs of autonomous driving simulation.

In terms of application value, the solution enables efficient mapping from real-world scenes to digital tachyons and supports flexible configurations of extreme weather, virtual traffic flow, and multimodal sensors, allowing a single scene to be extended to a variety of test scenarios.
these "Data-Scene-Test Closed Loop Capability It not only reduces the reliance on real-world testing, but also provides a highly reliable simulation environment for the iteration of autonomous driving algorithms.

Enterprise Cloud IT Solutions

Optical Inspection

Automotive Electronics

Biomedicine

Vuzix Industrial AR

Test Measurement

Industrial Internet of Things

Visual Inspection

Industrial Measurement

Autopilot