3D Perception — Fine-tuning & Distillation
📝 Detailed writeup coming soon — key information is shown below.
3D Perception — Model Fine-tuning & Distillation Internship
Overview
This internship focused on accelerating 3D reconstruction models for real-time robot arm planning. State-of-the-art reconstruction models (MAST3R, VGGT) can produce high-quality point clouds from images end-to-end, but their inference speed does not meet the requirements of interactive robot planning loops. The goal was to reduce latency while maintaining output quality.
MAST3R Fine-tuning
- Applied low-resolution ROPE (Rotary Position Embedding) fine-tuning to MAST3R to improve inference speed at reduced input resolutions
- Tuned relative position encoding to maintain spatial accuracy under lower resolution inputs
- Benchmarked the trade-off between point cloud quality and inference speed across different resolution settings
VGGT Distillation
- Designed a distillation loss function to transfer knowledge from the full VGGT model to a lightweight student network
- Trained and evaluated the distilled model on indoor robot arm planning scenarios
- Targeted improvement in image-to-point-cloud inference throughput
Quantitative results (speed vs. quality curves) and visual comparisons will be added here.
📝 详细内容即将补充，以下为主要信息。
3D 感知——端到端重建模型微调与蒸馏 实习
项目概述
本实习面向机器人手臂规划的实时 3D 重建加速问题。主流重建模型（MAST3R、VGGT）能从图像端到端生成高质量点云，但推理速度不满足交互式机器人规划的实时性要求。目标是在保持输出质量的前提下降低推理延迟。
MAST3R 微调
- 对 MAST3R 进行低分辨率 ROPE（旋转位置编码）微调，提升低分辨率输入下的推理速度
- 调整相对位置编码以在较低输入分辨率下保持空间精度
- 对不同分辨率设置下点云质量与推理速度的权衡进行基准测试
VGGT 蒸馏
- 设计蒸馏 loss 函数，将 VGGT 全量模型的知识迁移到轻量学生网络
- 在室内机械臂规划场景下训练和评估蒸馏模型
- 目标：提升图像到点云的推理吞吐量
速度与质量的定量对比曲线和可视化结果将在后续补充。