Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation

Seungyeon Yoo*,1, Gawon Lee*,1, Seungwoo Jung*,1, Inkyu Jang2, H. Jin Kim1
1 Seoul National University, 2 UC Berkeley
* Equal contribution
arXiv

We enable safe visual navigation in dynamic environments by distilling a privileged state-based CBF teacher into an RGB-only student safety filter.

Abstract

RGB-only end-to-end visual navigation policies remain vulnerable to collisions in real-world dynamic environments, motivating a dedicated safety layer. Existing visual Control Barrier Function (CBF) approaches seek to provide safety from RGB observations, but often rely on real-time rendering or explicit scene reconstruction and are primarily designed for static scenes, limiting their practicality for onboard deployment. We propose a teacher-student visual distillation framework that transfers the safety behavior of a privileged CBF teacher to an RGB-only student filter for dynamic environments. The student maps a short RGB history, robot velocity, and a nominal control action directly to a safe action, while the teacher uses ground-truth robot and obstacle states in a real-to-sim dynamic Gaussian Splatting environment. To reduce the teacher-student information gap, the teacher constructs safety constraints only from obstacles observable within the student’s RGB history. It also accounts for obstacle-velocity uncertainty to improve robustness to motion variations, while action augmentation exposes the student to diverse safe and unsafe nominal actions to better capture the safety boundary. At deployment, the student requires only RGB observations and robot velocity, without explicit 3D reconstruction or online rendering. Experiments show that the proposed method outperforms visual CBF baselines and improves the safety of RGB-based navigation policies under dynamic obstacle motion.

Video

Training pipeline

We propose a distillation framework in which an RGB-only student safety filter learns to map RGB observations, robot velocity, and a nominal action to a safe action. The student is supervised by a privileged CBF teacher with access to ground-truth states in a real-to-sim dynamic Gaussian Splatting simulation.

Overview

Training pipeline.

Comparison of the action filter boundaries of the teacher and student.

The student distills not only safe-action labels, but also the teacher's intervention structure.

Hardware Experiments

Real-World Navigation Results (Sim-to-Real Transfer)

Our RGB-only safety filter improves safety across multiple visual navigation policies.


Safety Analysis Under Aggressive Nominal Inputs

Our filter provides braking and steering interventions even under unsafe nominal commands.

Lobby environment.

Library environment.

BibTeX


    @misc{yoo2026distillingprivilegedcontrolbarrier,
      title={Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation}, 
      author={Seungyeon Yoo and Gawon Lee and Seungwoo Jung and Inkyu Jang and H. Jin Kim},
      year={2026},
      eprint={2609.36520},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.36520}, 
    }