Summary (Overview)

• New benchmark for multimodal context learning: CLBench-V organizes evaluation into three hierarchical capabilities—context grounding (L0), new information application (L1), and new knowledge learning (L2)—making it diagnostic of where failures occur (visual access vs. reasoning vs. knowledge acquisition).
• Diverse task composition: 3,443 instances drawn from 10+ sources, including converted public benchmarks (e.g., ReasonMap, CourtSI, MIRBench) and two newly constructed domain-specific tasks (financial ROE analysis and medical paper conclusion inference).
• Current models far from saturation: The best overall score across six recent multimodal models is only 0.2847 (InternVL3.5-30B-A3B). No single model dominates all levels; Qwen3.5-Plus leads on L1 (0.2954) while InternVL3.5-30B-A3B leads on L0 (0.3080) and L2 (0.3536).
• Failure taxonomy identified: Six failure modes are observed—evidence missing, evidence misbinding, context misuse, prior override, incomplete induction, and format failure.
• Judge reliability and complexity analysis: LLM judge choice can shift reported scores; input length and image count show weak or model-dependent correlation with performance, suggesting that usable context limits matter more than raw length.

---

Introduction and Theoretical Foundation

The paper argues that real-world tasks often require models to learn from task-specific context rather than relying solely on pre-trained parametric knowledge. While existing context-learning benchmarks (e.g., CL-Bench, Bai et al., 2024, 2025) have driven progress, they are restricted to unimodal textual scenarios. In practice, contexts are inherently multimodal: scientific figures, financial tables, maps, web pages, and sports scenes.

The authors identify two key challenges for extending context-learning evaluation to multimodal settings:

1. Diagnostic localization: Prior benchmarks reveal failures (ignoring context, misusing it) but do not localize which aspect of context use breaks down. The paper therefore organizes CLBench-V around a three-level hierarchy that separates grounding failures, information misuse, and knowledge-learning failures.

2. Annotation cost: High-quality context-learning tasks are expensive to annotate manually. The authors introduce automated construction and filtering procedures for new domain-specific tasks.

The hierarchy is defined by what the supplied context contributes:

• L0 Context Grounding: locate, read, and combine evidence from multimodal context (e.g., visual search, counting, spatial localization). No new rule system or domain theory is required.
• L1 New Information Application: use newly provided facts, values, or cases while relying on already-known reasoning skills. Tests whether models correctly bind context-specific values and avoid overriding with prior knowledge.
• L2 New Knowledge Learning: acquire new rules, empirical patterns, or scientific conclusions from the context and apply them. Requires synthesizing context-defined knowledge and transferring it to answer generation.

The levels are cumulative (failures in grounding can prevent information use) but analytically distinct—they expose different failure modes.

The paper further motivates the need for such a benchmark by noting that long-context studies show evidence use degrades with input length, evidence position, and number of visual items (Liu et al., 2024a; Song et al., 2024; Wang et al., 2024c). The authors distinguish context capacity from context learning: models may access long inputs yet still fail to internalize context-defined rules or resolve cross-modal conflicts.

---

Methodology

Benchmark Construction

CLBench-V is built from two complementary sources:

1. Data Integration (converted public benchmarks): Candidate datasets are converted into a unified input-output format after applying four curation criteria:

   • Visual dependency: images must provide task-critical evidence; text-only shortcuts should not suffice.
   • Low irrelevant context: minimize unrelated content to avoid collapsing into pure needle-search.
   • Multi-hop preference: prefer samples requiring compositional reasoning across multiple evidence pieces.
   • Scope exclusion: no video or quasi-video tasks (dense contiguous multi-frame reasoning).

   After conversion, rejection sampling with Qwen3.5-Plus removes shortcut-solvable instances, followed by manual review. The final integrated sources include: Pix2Fact, MMLongBench, PRISMM-Bench, ReasonMap, Insight-O3, MIRBench, ZeroBench, CourtSI-Bench, BrowseComp-V3, and CL-Bench.

2. Data Construction (newly built tasks): Two domain-specific tasks are constructed to cover gaps:

   • Financial Report ROE Analysis: Collect recent financial reports, convert PDFs to Markdown-style multimodal documents, and ask models to compute ROE using the DuPont decomposition: $$ROE = \text{Net Profit Margin} \times \text{Asset Turnover} \times \text{Equity Multiplier}. \tag{1}$$ Models must identify and combine net income, revenue, total assets, and shareholders’ equity from the converted report. Evaluation uses exact match or mean absolute error.
   • Paper Conclusion Inference: Collect recent medical papers, create two versions per paper: a reference-construction version (includes Results) to generate reference findings, and an evaluation version (only before Results). Models predict the scientific findings supported by the paper’s results. The paper-level score is: $$s_{\text{paper}} = \max\left(0, \frac{N_{\text{hit}} - N_{\text{extra}}}{N_{\text{ref}}}\right), \tag{2}$$ where $N_{\text{ref}}$ is the number of reference findings, $N_{\text{hit}}$ is the number of correct model predictions, and $N_{\text{extra}}$ is the number of unsupported extra findings. This penalizes over-prediction.

Evaluation Protocol

All tasks are evaluated under a unified framework with standardized inputs, metadata, and answer extraction. Evaluation is task-specific:

• Multiple-choice: option extraction + exact matching.
• Numeric: normalized exact match, tolerance-based match, or mean absolute error.
• Route planning: topology judge (verifies endpoints, lines, transfer continuity).
• Open-ended: answer extraction + rubric-based or semantic LLM judging (reserved for tasks where semantic equivalence is necessary).

Models Evaluated

Six multimodal models: InternVL3.5-30B-A3B, Qwen3.6-27B, Qwen3.5-Plus, Doubao-Seed-2.0-Lite, Kimi-K2.6, and GPT-5.4. Qwen3.6-27B is used as the main judge for judge-based tasks.

---

Empirical Validation / Results

Main Results by Level

Table 2 (reproduced below) shows the aggregated performance:

Table 2: Main results aggregated by capability level. Bold = best; underlined = second-best. MIRBench assigned to L0; PRISMM-Bench to L1.

Key observations:

• The best overall score is only 0.2847, indicating multimodal context learning remains far from saturated.
• InternVL3.5-30B-A3B excels on L0 (grounding) and L2 (knowledge) but struggles on L1 (information application).
• Qwen3.5-Plus leads on L1, showing better use of newly supplied information.
• GPT-5.4 performs poorly on L2 (0.0694), suggesting weak context-defined knowledge acquisition despite moderate L1 performance.

Results by Dataset

Table 3 presents dataset-level results (abbreviated columns):

Table 3: Dataset-level results (selected columns). Bold = best; underlined = second-best. InternVL3.5-30B-A3B scores 0.0000 on Financial Report ROE due to context length limit.

Key observations:

• No model dominates across all datasets.
• Qwen3.6-27B leads on Insight-O3, Pix2Fact, and CL-Bench-Table.
• Kimi-K2.6 leads on all ReasonMap subsets (route planning).
• InternVL3.5-30B-A3B leads on MIRBench and Paper Conclusion.
• GPT-5.4 leads on BrowseComp-V3, CourtSI, and Financial Report ROE.
• Several tasks remain very difficult across models: Pix2Fact, BrowseComp-V3, CL-Bench-Table, CourtSI.

Judge Reliability

Table 4 shows score variation across LLM judges on semantically evaluated tasks:

Table 4: Comparison of LLM judges. Bold = best; underlined = second-best.

The non-negligible variation confirms that judge choice affects reported performance, motivating transparent reporting and further calibration.

Context Complexity Analysis

For Qwen3.6-27B, token length is essentially uncorrelated with score (Pearson $r = -0.0042$, Spearman $\rho = 0.0500$), and image count is weakly negative ($r = -0.0756$, $\rho = -0.0739$). For InternVL3.5-30B-A3B, both measures are negative under Pearson but weakly positive under Spearman, indicating that a small tail of extremely long inputs (financial reports exceeding context limit) drives the negative correlation. Excluding financial reports, the token-score correlation turns positive for both models. Image count shows model-dependent effects: InternVL improves with more images ($\rho = 0.2317$ after exclusion), while Qwen3.6 does not ($\rho = -0.0402$).

Failure Taxonomy

Six failure types are identified through manual inspection:

1. Evidence missing: model does not attend to the relevant visual region.
2. Evidence misbinding: evidence attached to wrong entity, figure, page, or option.
3. Context misuse: relevant information read but applied incorrectly.
4. Prior override: model relies on parametric knowledge despite contradictory context.
5. Incomplete induction: local facts noticed but broader context-defined rule or conclusion not inferred.
6. Format failure: answer violates expected output schema.

---

Theoretical and Practical Implications

Theoretical Implications

• Hierarchical capability decomposition: The three-level hierarchy (grounding → information application → knowledge learning) provides a principled framework for diagnosing where multimodal context learning fails. This goes beyond simple aggregate scores and enables fine-grained analysis of model weaknesses.
• Context learning vs. context capacity: The paper empirically demonstrates that accepting long multimodal inputs does not guarantee robust context learning. Models may ground evidence well yet fail to apply or learn from it, and vice versa.
• Domain-specific knowledge acquisition: The newly constructed paper conclusion inference task requires models to synthesize experimental motivation, methods, and multimodal evidence to predict findings—a challenging form of learning that current models handle poorly (best L2 score 0.3536).

Practical Implications

• Benchmark for model development: CLBench-V provides a standardized, multi-task evaluation suite for multimodal context learning. The low overall scores (best 0.2847) indicate that significant improvement is needed before models can reliably use multimodal contexts in real-world deployments.
• Diagnostic tool: The level-wise and dataset-level breakdowns help developers pinpoint specific weaknesses (e.g., a model strong on grounding but weak on information application may need better attention to document-specific values).
• Judge sensitivity: The variation across LLM judges (Table 4) highlights the importance of choosing and reporting judge models carefully, especially for open-ended multimodal tasks.
• Automated construction pipeline: The paper demonstrates that automated data construction and filtering can produce high-quality context-learning tasks at scale, reducing the cost of benchmark expansion.

---

Conclusion

The paper introduces CLBench-V, a benchmark for multimodal context learning that organizes tasks into three hierarchical capabilities: context grounding (L0), new information application (L1), and new knowledge learning (L2). The benchmark combines converted public benchmarks with newly constructed tasks from medical papers and financial reports, totaling 3,443 instances across diverse domains.

Experimental evaluation of six recent multimodal models reveals that multimodal context learning remains far from saturated, with the best overall score only 0.2847. Models exhibit contrasting capability profiles: InternVL3.5-30B-A3B excels on grounding and knowledge learning, while Qwen3.5-Plus leads on information application. The paper further analyzes judge reliability, context complexity, and failure modes, showing that input length alone does not explain failures and that judge choice can affect reported scores.

The authors acknowledge limitations: heterogeneous data sources, small subsets, LLM judge biases, and incomplete scoring of intermediate reasoning steps. They hope CLBench-V provides a useful diagnostic framework for building multimodal systems that can not only see context but also use and learn from it. The code is publicly available.