September 29, 2023

Mac Schwager
Associate Professor
Stanford University
Department of Aeronautics and Astronautics
Learn more about Mac: https://web.stanford.edu/~schwager/

New developments in computer vision and deep learning have led to the rise of neural environment representations: 3D maps that are stored as deep networks that spatially register occupancy, color, texture, and other physical properties. These environment models can generate photo-realistic synthetic images from unseen view points, and can store 3D information in exquisite detail. In this talk, I investigate the questions: How can robots use neural environment representations for perception, motion planning, manipulation, and simulation? I will present recent work from my lab in navigating a robot through a neural radiance field map of an environment while preserving safety guarantees. I will talk about realtime NeRF training, where we produce a neural map online in a SLAM-like fashion. I will also discuss open-vocabulary semantic navigation in a neural map, where we find or avoid objects specified at runtime. I will present the concept of dynamics-augmented neural objects, which are assets captured from RGB images whose motion (including contact) can be simulated in a differentiable physics engine. I will show how such models can be used in real-to-sim transfer and robot manipulation planning scenarios. I will conclude with future opportunities and challenges in integrating neural environment representations into the robot autonomy stack.