Vision, Language, and Multimodal Human Instructions
for Interactive Intelligent Vehicles

Physical AI companies collect enormous volumes of data, but identifying the moments that actually improve models remains a major bottleneck. We present Nomadic, an agentic data intelligence platform that combines VLMs with targeted evidence gathering to discover, validate, and explain complex events in robotics and autonomous driving datasets. By grounding model predictions in visual and motion evidence, Nomadic improves event accuracy, reduces manual review, and enables scalable curation of high-value training data for physical AI.

Bio: Varun Krishnan is Cofounder / CTO at Nomadic AI. He was previously a Research Scientist on Lyft's Autonomous Navigation Team.

Large Vision-Language Models have created new opportunities for transportation scene understanding by allowing researchers and practitioners to query complex traffic scenes using natural language. However, transportation perception is not simply an image-understanding problem. Safety-relevant reasoning often depends on contextual information not fully captured by a single RGB frame, including temporal motion, 3D spatial structure, road-user interactions, environmental conditions, and visibility constraints.

This talk presents recent work on context-grounded LVLMs for transportation through two complementary directions. The first is 3D spatial grounding, where 2D visual inputs are augmented with LiDAR-, stereo-, or tracking-derived information so that LVLMs can reason about depth, object position, motion, and vehicle-to-vehicle interaction — moving beyond visual description toward spatially informed traffic-scene reasoning critical for safety questions such as lane-change feasibility, vulnerable-road-user presence, and interaction risk. The second is concept grounding, where LVLMs are encouraged to reason through human-interpretable evidence rather than directly predicting a final label, making outputs more interpretable, auditable, and robust for safety-critical transportation applications.

One of the central challenges in developing robust vision–language models (VLMs) for real-world autonomy is their ability to recognize, interpret, and reason about rare and hazardous situations—so-called edge cases. Unlike routine traffic patterns, which are well-represented in large-scale datasets, these scenarios occur infrequently, are highly diverse, and often involve subtle contextual cues that challenge both object detection and semantic understanding.

EdgeScenes is a new dataset under development at the WISE Lab that directly targets this limitation. It systematically captures and organizes rare road situations using a structured, ontology-driven taxonomy of hazardous conditions spanning infrastructure anomalies, abnormal road-user behavior, foreign objects, environmental extremes, and complex interactions. The dataset is constructed from crowdsourced video footage and annotated with a rich multimodal schema covering over 300 fine-grained hazardous conditions, along with temporal extents and contextual tags. As such, it establishes a testbed specifically designed for evaluating VLMs on out-of-distribution, safety-critical driving scenarios.

This talk will present the motivation, design principles, and annotation methodology behind EdgeScenes, with particular emphasis on how taxonomy-based labeling enables systematic identification of edge cases and gaps. I will discuss key insights gathered during dataset construction, including patterns in real-world hazard emergence and challenges in visual–semantic grounding. Finally, I will report early experimental results using frontier VLMs, showing that while current models can detect a broad range of hazards beyond closed-vocabulary vision systems, they still exhibit severe limitations, showing both high false positive and false negative rates in recognizing road hazards. These findings point to an urgent need for progress in improving the reliability of VLMs to support trustworthy multimodal perception in autonomous systems.

Krzysztof Czarnecki is a Professor of Electrical and Computer Engineering and a University Research Chair at the University of Waterloo, where he leads the Waterloo Intelligent Systems Engineering (WISE) Laboratory. He currently also serves as an Associate Director of the Waterloo Centre for Automotive Research (WatCAR). His research focuses on assuring the safety of AI systems and driving behavior. In 2018, he co-led the development of the first autonomous vehicle tested on public roads in Canada. He has made significant contributions to automotive AI and software safety standards, including SAE J3164 and ISO 8800. Before joining the University of Waterloo, he worked at DaimlerChrysler Research in Germany (1995–2002), where he advanced software development practices and technologies for enterprise, automotive, and aerospace sectors. His work has been recognized with numerous awards, including the Premier's Research Excellence Award (2004) and the British Computing Society’s Upper Canada Award for Outstanding Contributions to the IT Industry (2008). He has also received twelve Best Paper Awards, three ACM Distinguished Paper Awards, and five Most Influential Paper Awards.