AccidentBench logo

Human–AI Gaps in Understanding Safety-Critical Traffic Scenes

Shangding Gu1,7, Xiaohan Wang2, Donghao Ying3, Haoyu Zhao4, Runing Yang5, Boyi Li6, Xianyang Liu7, Qin Liu8, Sheng Liu9, Ming Jin5, Serena Yeung2, Muhao Chen8, Weinan Zhang7, Xia Hu7, Jun Wang4, Marco Pavone6,10, Costas Spanos1, Dawn Song1
1UC Berkeley EECS 2Stanford AI Lab 3UC Berkeley IEOR 4University College London 5Virginia Tech 6NVIDIA 7Safe AI and Robot Learning Lab 8UC Davis 9KTH 10Stanford Aeronautics & Astronautics
Under review
1,628
real-world accident videos
5,766
human-annotated questions
17,069
QA pairs at 3 nested resolutions
56.11%
human mean (Exact)
42.05%
best model (Exact)

Abstract

AI systems have demonstrated remarkable capabilities in mathematical reasoning, programming and the game of Go. Whether current multimodal models can match human understanding of dynamic physical events remains unclear. Here we show that leading multimodal models fall short of human performance in interpreting real-world traffic accidents. We develop AccidentBench, comprising 1,628 videos and 5,766 human-annotated questions spanning temporal, spatial and intent reasoning. Three nested answer levels vary the required precision while keeping each video and question fixed. On a shared 600-question evaluation set at the finest resolution, the strongest evaluated model achieves 42.05% accuracy, compared with a mean of 56.11% across three independent human evaluators. Even on the questions answered correctly by all three evaluators, the best-performing model fails in approximately one third of cases. Model accuracy declines as answer precision increases, including after correction for differences in chance performance. A video-free control shows above-chance accuracy in frontier models, consistent with possible prior exposure to benchmark content during training. These findings reveal human–AI gaps in interpreting safety-critical traffic scenes and highlight the need to assess whether model predictions reflect the temporal, spatial and behavioural evidence required by a question. These results further suggest that precise understanding and reasoning about dynamic physical events remain a key bottleneck in advancing multimodal AI towards human-level performance.

Real-World Traffic Accident Scenes

The Benchmark

AccidentBench is a human-centered benchmark built on real-world traffic accident videos drawn from published academic datasets. The videos span intersections, highways, freeways, rural and urban roads, tunnels, bridges and parking lots; snow, rain, fog and night-time conditions; and road users including cars, trucks, buses, motorcycles, bicycles and pedestrians, seen from both ego-centric and third-person viewpoints. Every question and reference answer is written by human annotators β€” none are generated by language models.

Six representative accident scenes, each shown as four frames in temporal order

Representative traffic accident scenes. (a) Intersection, (b) urban road, (c) nighttime, (d) rural road, (e) snow-covered road, (f) freeway. Each panel shows four frames from one video in temporal order.

Examples of temporal, spatial and intent questions

Three reasoning types. Temporal reasoning evaluates event sequences and motion over time; spatial reasoning evaluates relative positions and orientations among traffic participants; intent reasoning evaluates goal-directed behaviour and decision-making in dynamic scenes.

Nested answer resolutions: Coarse β†’ Fine β†’ Exact

Each question is posed at three levels of required precision while the video and question stay fixed β€” only the answer space is refined. For quantitative questions (counting, ordering, most intent tasks) the three levels quantize the same underlying integer: Coarse uses three width-4 intervals (e.g. [0, 3], [4, 7], [8, 11]), Fine uses six width-2 intervals, and Exact requires the precise integer from 0 to 11. For spatial questions, Coarse asks for a lateral relation (left/right), Fine adds an explicit uncertainty option, and Exact requires resolving both axes, {front, back} Γ— {left, right}. Because the answer spaces are nested, every Exact answer maps to a Fine and a Coarse answer, so we can measure both how accuracy changes with precision and whether a model's answers stay mutually consistent.

Since the three resolutions have different chance rates, we report chance-corrected accuracy ac = 100 (a βˆ’ c) / (100 βˆ’ c) alongside raw accuracy a, where c is the uniform-guessing rate on the evaluated items. ac = 0 is chance and 100 is perfect.

Human reference. Dataset construction and human evaluation were performed by separate groups: four annotators wrote the questions and reference answers, two reviewers ran quality checks, and three independent evaluators answered a 600-question Exact-resolution evaluation set without seeing the reference answers. All participants are licensed drivers.

Humans vs. Models on the Shared Evaluation Set

Human 2
58.33
Human 1
58.00
Human mean
56.11
Human 3
52.00
GPT-5.6-terra
42.05
Gemini 3.6 Flash
41.71
Claude Sonnet 5
27.18
Chance
16.91

Exact-resolution accuracy (%) on the 600-question set answered by all three independent human evaluators.

SubsetMetricnChance Claude Sonnet 5Gemini 3.6 FlashGPT-5.6-terra Human 1Human 2Human 3Human Mean
All itemsAcc. (%)60016.9127.1841.7142.0558.0058.3352.0056.11
ac6000.0012.3629.8530.2649.4549.8542.2347.18
All 3 humans correctAcc. (%)18819.2738.8363.3065.43100.00100.00100.00100.00
ac1880.0024.2354.5457.18100.00100.00100.00100.00

The 188-question subset is selected by human correctness, so it does not give an unbiased estimate of the overall gap; it does establish that model errors persist on questions that all three independent evaluators resolve correctly.

πŸ† Leaderboard

Accuracy (%) and chance-corrected accuracy ac across Exact, Fine and Coarse answer resolutions, sorted by Exact accuracy. Cell shading runs from lowest (red) to highest (green) within each column across models. Human evaluators answer Exact-resolution questions directly; their Fine and Coarse scores are obtained by projecting Exact responses through the nested answer space.

ModelSize ExactFineCoarse
Acc.acAcc.acAcc.ac
πŸ‘₯ Human, majority of three πŸ₯‡-57.67+49.066.83+56.976.83+62.8
GPT-5.6-terra (reasoning=high) -41.67+29.751.33+36.861.67+38.4
Gemini 3.6 Flash -41.37+29.350.43+35.666.32+45.9
GPT 5 -37.33+24.548.34+32.454.86+26.6
Gemini 2.5 Pro-29.76+15.436.46+16.854.56+26.1
Claude 3.5-28.71+14.135.35+15.448.59+16.4
Gemini 2.5 Flash (thinking)-28.67+14.137.52+18.250.00+18.7
Claude Sonnet 5-27.69+12.838.29+19.857.95+32.4
GPT 4o-24.41+8.936.99+17.545.92+12.1
Gemini 2.5 Flash (no thinking)-24.34+8.836.70+17.251.40+21.0
InternVL2.526B23.78+8.235.11+15.152.55+22.9
InternVL38B23.55+7.933.78+13.349.55+18.0
Qwen3 VL8B22.89+7.132.45+11.647.67+14.9
Qwen2.5 VL7B22.89+7.129.89+8.240.67+3.5
InternVL2.58B22.67+6.834.66+14.550.11+18.9
Qwen2.5 VL32B22.66+6.828.55+6.543.22+7.7
LLaVA Video7B19.78+3.325.67+2.731.44βˆ’11.5
InternVL2.54B19.56+3.133.89+13.544.89+10.4
Gemini 1.5 Pro-18.76+2.133.89+13.546.00+12.2
LLaVA Next32B16.22βˆ’0.920.00βˆ’4.731.25βˆ’11.8
LLaVA OneVision7B13.67βˆ’4.016.67βˆ’9.129.78βˆ’14.2
Uniform chance-17.070.023.020.037.750.0

GPT-5.6-terra, Gemini 3.6 Flash and Claude Sonnet 5 are evaluated with a standardized visual input budget (16 frames). Nested consistency β€” whether a model's Exact, Fine and Coarse answers are mutually compatible β€” ranges from 49.5% to 63.3% across evaluated models, showing structured cross-resolution behaviour even when answers are wrong.

Video-free control (GPT-5.4)

Input conditionCoarseFineExact
Uniform random guessing37.7523.0217.07
Question + options52.3132.8224.44
Question + options + video55.2144.6231.28

Without any video frames GPT-5.4 is above chance at every resolution. Adding video raises accuracy by only 2.90, 11.80 and 6.84 points, and Exact accuracy stays below one third.

Performance by video length

ModelShortMediumLong
Acc.acAcc.acAcc.ac
Human (majority)76.31+68.064.00+50.851.78+35.7
GPT-5.6-terra60.67+46.948.44+29.636.44+15.3
Gemini 3.6 Flash62.55+49.453.09+35.932.67+10.2
Claude Sonnet 548.22+30.038.27+15.730.22+7.0
Uniform chance26.000.026.800.025.000.0

Scores pool the three answer resolutions; models use the same 16-frame budget across lengths.

Comparison with Existing Benchmarks

DatasetSafetyTrafficAnnotationReal-worldMain scenario# VideosAvg. dur. (s)# QAFormat
MMWorldβœ—βœ—Humanβœ“General1,9101076,627Multiple-choice
MVBenchβœ—βœ—Human & LLMβœ“General4,000164,000Multiple-choice
LongVideoBenchβœ—βœ—Humanβœ“General3,7634736,678Multiple-choice
DriveBenchβœ“βœ“Human & LLMβœ“General drivingβœ—βœ—19,200Multiple-choice
DriveLMβœ“βœ“Humanβœ“General drivingβœ—βœ—15,480Open-ended
nuScenes-QAβœ—βœ“Humanβœ“General drivingβœ—βœ—83,337Open-ended
RoadSocialβœ“βœ“LLMβœ“General driving13.2k35.6260kOpen-ended
EchoTrafficβœ“βœ“LLMβœ“General driving29,8653.76149,325Open-ended
AVD2βœ“βœ“LLMβœ—General driving8,39963-Open-ended
AccidentBench (ours)βœ“βœ“Humanβœ“Traffic accident1,62850.7 (median 5.0)17,069Multiple-choice

AccidentBench is not the largest traffic-video benchmark. Its contribution is a different set of design choices: an exclusive focus on safety-critical accident scenes, fully human-authored questions and answers, verifiable multiple-choice formats, and a nested answer-resolution design for controlled analysis of answer precision.

BibTeX

@misc{gu2026accidentbench,
  title  = {Human--AI Gaps in Understanding Safety-Critical Traffic Scenes},
  author = {Gu, Shangding and Wang, Xiaohan and Ying, Donghao and Zhao, Haoyu and Yang, Runing and
            Li, Boyi and Liu, Xianyang and Liu, Qin and Liu, Sheng and Jin, Ming and Yeung, Serena and
            Chen, Muhao and Zhang, Weinan and Hu, Xia and Wang, Jun and Pavone, Marco and
            Spanos, Costas and Song, Dawn},
  year   = {2026},
  note   = {Under review},
  url    = {http://accident-bench.site}
}