Program

Abstract

Equipping AI and robotic systems with the ability to understand human behavior is essential for enabling them to assist people across a wide range of everyday applications. This need is more pressing than ever: the heaviest consumers of such knowledge are no longer perception systems alone, but robot policies that must learn to act in the physical world. Yet the high-quality 3D human motion data required to learn this knowledge remains extremely scarce.

In this talk, I will present our lab's efforts to scale and enrich 3D human motion data by capturing everyday movements and natural human-object interactions, with the ultimate goal of teaching robots to move like humans.

I will first introduce our multi-year effort in building multi-camera capture systems, from Panoptic Studio to ParaHome, and most recently OmniRoboHome, a new system designed to capture human-object interactions in natural home environments. Next, I will present a complementary direction: learning everyday interactions and affordances from generative image and video models, which offer a scalable source of human behavior priors that capture systems alone cannot reach. Finally, I will discuss the missing pieces that vision and image models cannot provide, including physics, contact, and the gap across diverse embodiments, together with our recent efforts to fill them.

Bio

Hanbyul Joo is an associate professor at Seoul National University (SNU) in the Department of Computer Science and Engineering. Before joining SNU, Hanbyul was a Research Scientist at Facebook AI Research (FAIR), Menlo Park. Hanbyul received his PhD from the Robotics Institute at Carnegie Mellon University. Hanbyul is a recipient of the Best Student Paper Award at CVPR 2018.

Abstract

The ability to build an actionable understanding of the environment of a robot is crucial for autonomy and prerequisite to a large variety of applications, ranging from home, service, care and consumer robots to autonomous vehicles, augmented reality, and disaster response. Notably, much of this depends on long-term operation in human-centric domains that are complex, widely diverse, and highly dynamic. This talk presents an autonomy pipeline to address these challenges. At the core, I argue that time, and thus memory, dynamics, and adaptation, should be an integral component of AI systems. I will introduce methods to capture the present and past of complex and dynamic scenes through symbolic abstractions, which further facilitate predicting future scene outcomes. I will show how robots as embodied agents can leverage our actionable scene representations and predictions to complete tasks such as actively gathering data that helps them improve their scene models and perception capabilities, and how all these tools can combine for robots to fully autonomously learn over time.

The presented methods are demonstrated on-board fully autonomous aerial and ground robots, run in real-time on the limited hardware available, and are released as open-source software.

Bio

Lukas Rosenberger Schmid (formerly Schmid) is a Tenure-Track Professor of Machine Intelligence at UTN. Before that, he was a Research Scientist and Postdoctoral Fellow at the SPARK Lab led by Prof. Luca Carlone at MIT, and a Postdoctoral Researcher at the Autonomous Systems Lab (ASL) led by Prof. Roland Siegwart at ETH Zürich. He earned his PhD in 2022 from ASL at ETHZ, where he also was a visiting researcher at the Microsoft Spatial AI Lab led by Prof. Marc Pollefeys. His work has been recognized by several honors, including RSS Pioneers 2025, NOKOV New Generation Star at IROS 2025, the RSS Outstanding Systems Paper Award 2024, two ETH Medals for outstanding PhD and M.Sc. Theses, the Willi Studer Prize for the best graduate of the year at ETHZ, the first place in the 2024 Hilti SLAM challenge, and a Swiss National Science Foundation (SNSF) Postdoc Fellowship.

His research focuses on embodied spatio-temporal AI for human-centric robot autonomy. This includes research on scene representations and abstraction, on detection, prediction, and understanding of moving and changing entities, active perception and information gathering, as well as lifelong learning for continuous adaptation to the robot environment, embodiment, task, and human preference.