doScenes Instructed Driving Challenge

The doScenes Instructed Driving Challenge evaluates whether natural language instructions can improve trajectory prediction for autonomous driving. Traditional trajectory prediction methods on datasets such as nuScenes rely on historical motion and scene context, yet human drivers often know their intended maneuver before executing it. The doScenes dataset augments nuScenes with human-authored natural language instructions describing upcoming driving maneuvers, enabling models to incorporate this privileged information.

Trajectory prediction, the task of forecasting future vehicle positions from observed history and environmental context, is fundamental to autonomous driving. Large-scale real-world benchmarks such as nuScenes have driven rapid progress by providing multi-modal sensor data, high-definition maps, and annotations across 1,000 driving scenes collected from Boston and Singapore.

Despite this progress, a central assumption remains largely unchanged: future motion is predicted using only observed history and scene context. This is appropriate for predicting other agents, but it is an artificial constraint when predicting the ego vehicle, where the intended maneuver may be known before execution. Natural language instructions such as "turn left at the next intersection" expose this intent and can reduce uncertainty in otherwise ambiguous scenarios.

This challenge is built on doScenes, an instruction-augmented extension of nuScenes. All instructions are human-authored, with multiple annotators providing diverse phrasings for the same maneuver. This variation challenges models to condition on meaning rather than surface form.

Task Definition

For the Full Multimodal track, participants are given:

• Visual scene input from permitted nuScenes cameras
• Historical trajectory information (past 2 seconds)
• High-definition map data
• Natural language instruction describing the intended maneuver

Each sample provides 2 seconds of ground-truth history. Participants must generate one ego-vehicle trajectory for the next 6 seconds, sampled at 2 Hz for 12 future timesteps. Leaderboard evaluation uses single-shot prediction per sample; sliding-window evaluation is not used for official scoring.

Evaluation is open-loop: models receive ground-truth history at inference time, and predicted trajectories are not fed back into the model for rollout or closed-loop scoring. The key research question is whether models can effectively leverage the language instruction to improve prediction accuracy compared to baselines that use the same non-language inputs.

Evaluation Metrics

Submissions are evaluated using standard trajectory prediction metrics:

• Average Displacement Error (ADE): Mean L2 distance between predicted and ground-truth positions across all 12 predicted timesteps.
• Final Displacement Error (FDE): L2 distance between predicted and ground-truth positions at the final timestep, 6 seconds into the future.
• Instruction Conditioning Gain (ΔADE): Improvement in ADE relative to the history-only baseline, measuring the benefit of incorporating language instructions.

ADE and FDE are computed per predicted trajectory, then averaged across all samples in the evaluated dataset. Each model must submit exactly one predicted trajectory per sample.

The Instruction Conditioning Gain is computed as:

ΔADE = ADE_baseline − ADE_instruction (1)

For official leaderboard reporting, the history-only baseline is defined per track: it uses the same non-language inputs and identical evaluation protocol except that no language instruction is provided. Participants must report ADE for the instruction-conditioned model, ADE for the baseline, and ΔADE.

Evaluation Tracks

Participants may submit to one or more tracks:

1. Full Multimodal: Uses cameras, 2 seconds of history, map data, and language instruction.
2. Language + History: Uses 2 seconds of history and language instruction only; visual input is not allowed.
3. Ablation: Participants must report both with-language and without-language results under the same protocol to demonstrate the instruction benefit.

Goals and Expected Outcomes

The doScenes Instructed Driving Challenge aims to:

1. Establish instruction-conditioned prediction as a research paradigm: Demonstrate that language instructions provide actionable information for trajectory prediction.
2. Benchmark vision-language models for driving: Evaluate how well current VLM architectures can ground natural language instructions in driving behavior.
3. Quantify the instruction benefit: Measure the gap between history-only and instruction-conditioned prediction, providing motivation for human-in-the-loop autonomy systems.
4. Build community: Foster collaboration between the computer vision, NLP, and autonomous driving research communities.

Leaderboard

Test-set leaderboard for instruction-conditioned trajectory prediction. Baselines are listed first.

Full Multimodal

Language + History

Ablation

Leaderboard scores use single-shot, open-loop evaluation with one 6-second trajectory per sample. ADE reports ade_6s, and FDE reports the final 6-second displacement error. VERIFIED instruction-conditioned rows were locally evaluated on the 127 nuScenes v1.0-test scenes with non-empty doScenes instructions. The full nuScenes v1.0-test split contains approximately 150 scenes, but 23 current doScenes test scenes have blank instruction fields and are excluded from instruction-conditioned leaderboard scoring. For submitted files with all 150 scene rows, evaluation filters to the 127 language-available scenes. SELF-REPORTED rows come from submitted papers where the local CSV could not be verified for that track, and DIAGNOSTIC rows required non-official duplicate handling. For teams with multiple files, the displayed row uses the organizer-selected best available submission for that applied track, except Ablation uses matched with-language and without-language files. Non-verified results are provisional and will undergo further organizer review before being treated as final verified leaderboard results.

Dataset & Data Splits

doScenes extends nuScenes with human-authored instruction annotations. Public challenge development uses the nuScenes v1.0-trainval split, while website leaderboard evaluation uses the nuScenes v1.0-test split. Instructions are written by multiple annotators to capture natural variation in phrasing.

The nuScenes v1.0-test split contains approximately 150 scenes. In the current doScenes test annotations, 127 of those scenes have non-empty language instructions and form the official instruction-conditioned leaderboard evaluation set. An instruction field may be blank when no instruction interaction is needed to cause the observed action, such as waiting at a red light or continuing with traffic. The 23 blank-instruction scenes remain part of the nuScenes test split, but are not used for instruction-conditioned leaderboard scoring.

Instructions cover a variety of maneuvers including:

• Lane changes (left/right)
• Turns at intersections
• Proceeding straight
• Stopping and yielding
• Speed adjustments

The diversity of annotators ensures that models must handle natural variation in instruction phrasing, rather than memorizing specific templates. For leaderboard evaluation, each test scene corresponds to one trajectory prediction from the first scene segment, matched using the scene's scene_token.

Dataset Statistics

Annotation Statistics

doScenes provides controlled evaluation of instruction-conditioned trajectory prediction across grounded and non-grounded language. Dynamic-referential instructions are grounded in moving agents or temporal interactions, such as following the car ahead. Static-referential instructions are grounded in scene elements such as lanes, signs, or intersections. Mixed instructions combine both dynamic and static references, while non-referential instructions describe motion without explicitly grounding to scene entities, such as continuing straight or slowing down.

Annotation statistics describe doScenes instruction annotations and may not exactly match the rounded nuScenes split counts shown above.

Dataset Splits and Evaluation Protocol

The doScenes benchmark uses the standard nuScenes dataset splits as the basis for training, validation, and website evaluation.

Dataloader Update

The doScenes repository includes an updated dataloader that provides the correct split for history, anchor, and future trajectory points. For training and validation, participants may switch the dataloader version to v1.0-trainval. For evaluation, the challenge uses the first segment of each scene from the test set.

The dataloader returns a scene_token for each scene. Use this value as the sample_token in submission.csv, since each evaluation scene corresponds to a single trajectory.

Official Evaluation Protocol

• Each sample uses 2 seconds of ground-truth history and a 6-second prediction horizon sampled at 2 Hz.
• Leaderboard evaluation uses single-shot, open-loop prediction per sample.
• Instruction-conditioned leaderboard evaluation uses the first segment of each language-available doScenes test scene from the nuScenes v1.0-test split.
• Sliding-window evaluation is not used for official scoring.
• Models must generate exactly one trajectory per evaluation sample, matched by sample_token.
• The current official instruction-conditioned evaluation set contains 127 test scenes with non-empty doScenes instructions.

Fair Evaluation Requirements

To ensure fair evaluation:

• Participants may use the nuScenes v1.0-trainval split for training and validation.
• The nuScenes v1.0-test split may be used only to generate website evaluation submissions, not for training, validation, model selection, or tuning.
• Ground-truth future trajectories from the nuScenes v1.0-test split must not be used for training, validation, tuning, model selection, or submission debugging.
• Test-set ground truth is not publicly available and must not be inferred, reconstructed, or used through any external source.
• Participants may use non-egocentric nuScenes data when it is allowed by the selected track, available at or before the prediction time, and does not reveal future information.
• Predictions must be generated for the nuScenes v1.0-test split and submitted for website evaluation.
• Teams are expected to submit reproducible code that adheres to this protocol.

Required Inputs

All required inputs for prediction, including historical trajectories, sensor data, maps, and language instructions, can be obtained by combining the standard nuScenes dataset with the doScenes annotation files provided in the repository. Participants may use egocentric or non-egocentric nuScenes inputs when allowed by the selected track, provided the inputs are available at or before the prediction time and do not leak future information.

Submission Instructions

Ready to submit? Use the official submission form.

Submission CSV Format

Prediction files must be named submission.csv. Each evaluation sample must have exactly one corresponding prediction row, and predictions are matched using sample_token.

The required CSV header is:

sample_token,instruction,x1,y1,x2,y2,...,x12,y12

The sample_token uniquely identifies each evaluation instance. When using the updated doScenes dataloader, use the returned scene_token as the sample_token, since each evaluation scene corresponds to a single trajectory.

Coordinates (x_t, y_t) represent the predicted 2D ego-vehicle position at each future timestep. The prediction horizon is 6 seconds at 2 Hz, for 12 total timesteps. All coordinates must be expressed in meters in the ego vehicle coordinate frame at prediction time t = 0, where the x-axis points forward and the y-axis points left.

Submission Checklist

• submission.csv with one row per evaluation sample and columns sample_token,x1,y1,x2,y2,...,x12,y12.
• ADE for the instruction-conditioned model, ADE for the baseline, and ΔADE.
• Public code repository link
• Paper PDF link
• Submitted papers are encouraged to follow the CVPR paper format

Workshop Participation

The doScenes challenge runs across two workshops:

• DriveX Workshop at CVPR 2026 — June 3–4, Denver, CO. Leaderboard freeze: May 20, 2026.
• VL-IIV Workshop at IEEE IV 2026 — June 22, Detroit, MI. Leaderboard freeze: June 15, 2026.

Participation in either workshop is not required to complete the challenge, but teams are encouraged to attend if possible. Details about online participation will be posted once available.

Key Dates

FAQ

Ethical Considerations

The doScenes dataset is built upon nuScenes, which uses public road data. All videos are anonymized with no identifiable faces or license plates. Language annotations are collected consensually and anonymously for research purposes.

Participants should consider the following questions:

• What should a system do when an instruction conflicts with traffic laws?
• Should a model defer to scene context when language is underspecified?
• How robust is the system to misleading or adversarial instructions?

Language must not override safety constraints. Models developed for this challenge should prioritize regulatory compliance, interpretability of instruction influence, and robustness to adversarial phrasing.

Concluding Remarks

The doScenes Instructed Driving Challenge introduces a trajectory prediction paradigm that leverages natural language descriptions of intended maneuvers to improve forecasting accuracy. By augmenting the established nuScenes benchmark with human-authored instructions, the challenge enables research into instruction-conditioned prediction models that can outperform history-only baselines.

Contact

Resources

• Dataset and project page: https://github.com/rossgreer/doScenes

@inproceedings{roy2025doscenes,
      title={doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation},
      author={Parthib Roy and Srinivasa Perisetla and Shashank Shriram and Harsha Krishnaswamy and Aryan Keskar and Ross Greer},
      booktitle={2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC)},
      pages={1651--1658},
      year={2025},
      organization={IEEE},
      url={https://arxiv.org/abs/2412.05893},
}

@inproceedings{martinezsanchez2026naturallanguage,
  title={Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models},
  author={Angel Martinez-Sanchez and Parthib Roy and Ross Greer},
  booktitle={2026 IEEE Intelligent Vehicles Symposium (IV)},
  year={2026},
  note={To appear},
  url={https://arxiv.org/abs/2602.04184}
}

@inproceedings{greer2026visionlanguagenovelrepresentations,
      title={Vision and language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning},
      author={Ross Greer and Maitrayee Keskar and Angel Martinez-Sanchez and Parthib Roy and Shashank Shriram and Mohan Trivedi},
      booktitle={Enhanced Safety of Vehicles Conference},
      year={2026},
      note={To appear},
      url={https://arxiv.org/abs/2602.07680}
}

@inproceedings{greer2026lookinglisteninginsideoutside,
      title={Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making},
      author={Ross Greer and Laura Fleig and Maitrayee Keskar and Erika Maquiling and Giovanni Tapia Lopez and Angel Martinez-Sanchez and Parthib Roy and Jake Rattigan and Mira Sur and Alejandra Vidrio and Thomas Marcotte and Mohan Trivedi},
      booktitle={Enhanced Safety of Vehicles Conference},
      year={2026},
      note={To appear},
      url={https://arxiv.org/abs/2602.07668}
}