Summary (Overview)

• FORGE is a new fine-grained multimodal benchmark for manufacturing scenarios, integrating real-world 2D images and 3D point clouds with detailed domain semantics (e.g., exact model numbers like M10–M20 nuts).
• The benchmark evaluates 18 state-of-the-art MLLMs across three tasks: Workpiece Verification (WORKVERI), Structural Surface Inspection (SURFINSP), and Assembly Verification (ASSYVERI).
• Key finding: Visual grounding is not the primary bottleneck; instead, insufficient domain-specific knowledge and morphology understanding limit MLLM performance in manufacturing.
• Models perform well on coarse-grained workpiece-level tasks but struggle significantly with fine-grained model-number-level discrimination and microscopic surface defect analysis.
• Beyond evaluation, FORGE serves as a training resource: supervised fine-tuning of a compact 3B-parameter model yields up to 90.8% relative improvement on held-out manufacturing scenarios, approaching the performance of models 78× larger.

Introduction and Theoretical Foundation

The manufacturing sector generates massive heterogeneous data and increasingly relies on data-driven decision-making and human-machine collaboration. Traditional vision models function as perception modules (object localization, anomaly detection) within pipelined architectures, but they lack reasoning and autonomous control capabilities. Multimodal Large Language Models (MLLMs) offer a promising path to bridge low-level perception and high-level planning.

However, existing evaluations fail to reflect real-world manufacturing demands due to three fundamental challenges:

1. Data Scarcity Gap: Current datasets are limited in scale/diversity, with many relying on simulated or CAD-based data.
2. Lack of Fine-Grained Domain Semantics: Existing datasets treat workpieces as generic visual subjects without explicit domain semantics (e.g., model numbers).
3. Absence of Comprehensive Evaluation Frameworks: No systematic benchmarks assess reasoning, understanding, and decision-making in manufacturing scenarios.

The paper poses the core question: Can MLLMs understand, explain, and execute decisions for tasks inherently characteristic of the manufacturing domain?

Comparison with existing benchmarks:

Methodology

Dataset Curation

• 3D Point Cloud Subset: High-fidelity geometric data covering 14 workpiece categories across 90 distinct models.
• Image Subset: ~3,000 images capturing four manufacturing scenarios (e.g., expansion screw assemblies), including normal and abnormal samples.

Data Processing:

• 2D images: Ground-truth labels via automated contour/coordinate extraction + manual refinement.
• 3D point clouds: Batch samples synthesized by stitching 4–5 individual point clouds with random orientations; defects (Crack, Deformation, Dent, Cut) simulated using morphology-based algorithms.
• Multi-view projection strategy: All 3D point clouds rendered as three-view (3V) images (front, side, top orthogonal projections) to remain compatible with standard visual inputs of general MLLMs.

Three Evaluation Tasks

1. WORKVERI (Material Sorting): Identify workpieces that do not belong to the current batch given explicit specifications. Three scenarios: pneumatic connectors (PCS), cup head screws (CHS), nuts (NUTS).

2. SURFINSP (Quality Inspection): Two-step task: (1) defect detection, (2) defect type classification (Crack, Cut, Deformation, Dent, or Good). Covers 14 manufacturing components.

3. ASSYVERI (Assembly Recognition): Reason over complex assembly rules to identify workpieces failing assembly specifications. Four scenarios: metal expansion screws (MES), plastic expansion screws (PES), CNC fixtures (CNC), and screw-washer-nut compatibility (SWN).

Evaluation Settings

Three progressively informative settings:

• Zero-Shot: Test image + task query only.
• Reference-Conditioned (Ref-Cond): Adds reference images of correct/normal assemblies.
• In-Context Demonstration (ICD): Adds complete solved examples as multi-turn dialogue pairs.

Error scenarios categorized into two difficulty levels:

• Different workpiece: Coarse-grained discrepancies (wrong types, missing components).
• Different Model Number: Fine-grained inconsistencies (subtle model variations like screw length or thread pitch).

Metric: Exact-match accuracy on multiple-choice questions (MCQs). Random-chance baseline is the weighted average of random guessing.

Empirical Validation / Results

Main Benchmark Results (Table 3 excerpt)

Key Findings

A. Semantics > Morphology: Models perform better on macroscopic part discrimination (WORKVERI, ASSYVERI) than microscopic surface analysis (SURFINSP), which yielded the lowest performance despite simpler objectives.

B. Domain knowledge is the bottleneck: Ref-Cond strategies did not consistently improve performance (sometimes degrading it), while ICD with complete reasoning demonstrations universally improved results—indicating MLLMs lack deep understanding of task logic, not simple visual references.

C. 3D spatial confusion: For three-view modality, performance declined with Ref-Cond and ICD, suggesting additional examples induced spatial confusion rather than helpful guidance.

D. Model-number-level tasks are harder: MLLMs consistently outperform on workpiece-level tasks compared to model-number-level tasks across all models.

Bottleneck Analysis

A. Visual grounding is NOT the bottleneck (Table 4):

Near-ceiling results on L→C grounding confirm failures cannot be attributed to poor visual localization.

B. Fine-grained part identification is a domain-knowledge bottleneck (Table 5): In missing-part detection, top models achieve 74.9–90.7% accuracy on images (random baseline: 23.3%). However, a systematic failure on flat washer detection (23.3–60.0% on images) reveals insufficient fine-grained knowledge of functional/morphological differences between part variants—not perceptual failure.

C. Visual projection is necessary (Table 6): Serializing raw 3D coordinates as text tables yields near-random performance (~20% baseline) on SURFINSP, confirming multi-view visual rendering is the most effective interface for general MLLMs.

Qualitative Error Analysis

• Material property misjudgment: Models incorrectly infer material composition from visual textures and over-rely on these erroneous properties in reasoning.
• Emerging capabilities: Models show latent potential in assessing service conditions (e.g., detecting "heavy wear" or "chipping"), suggesting potential support for Predictive Maintenance (PdM).

Fine-tuning Results

Supervised fine-tuning of Qwen2.5-VL-3B-Instruct on FORGE annotations:

• WORKVERI three-view: 90.8% relative improvement, matching Qwen3-VL-235B (54.4%), a model 78× larger.
• ASSYVERI image: 27.1% relative gain, surpassing all reference models except Gemini-3-Flash and GPT-5.2.

Gains measured on held-out product categories confirm FORGE annotations encode transferable manufacturing knowledge.

Theoretical and Practical Implications

• Benchmark contribution: FORGE is the first large-scale fine-grained manufacturing dataset integrating aligned 2D images and 3D point clouds with model-number-level annotations, filling a critical gap in MLLM evaluation for manufacturing.
• Bottleneck identification: Counter to conventional assumptions, visual grounding is not the limiting factor—insufficient domain-specific knowledge and morphology understanding are the key gaps. This redirects future research toward domain-knowledge injection rather than perception improvements.
• Training resource: The dataset provides a practical pathway for domain adaptation, demonstrating that compact models can approach frontier-scale performance with modest amounts of structured annotations.
• Industry relevance: The three tasks (material sorting, quality inspection, assembly verification) map directly to critical manufacturing automation pillars, making the benchmark reflective of real-world production demands.
• Emerging capabilities: Qualitative analysis reveals MLLMs are developing latent abilities (material recognition, wear assessment) that could support advanced applications like Predictive Maintenance.

Conclusion

FORGE establishes a comprehensive fine-grained multimodal benchmark for manufacturing, evaluating 18 state-of-the-art MLLMs across three tasks. Key takeaways:

1. Current MLLMs handle macroscopic part recognition but fall short on fine-grained reasoning and microscopic surface analysis.
2. Visual grounding is not the primary bottleneck—insufficient manufacturing domain knowledge and morphology understanding are the key gaps.
3. FORGE serves as an actionable training resource: domain-specific fine-tuning enables compact models to approach frontier-scale performance on held-out scenarios.

Future directions include closing the domain knowledge gap in manufacturing intelligence, improving fine-grained morphology understanding, and leveraging FORGE's structured annotations for broader domain adaptation in MLLMs.