##Summary (Overview)

• Standardized representation-level evaluation: TRL-BENCH introduces a unified protocol that evaluates heterogeneous tabular encoders (from generic text to tabular specialists) by freezing their exported row-, column-, or table-level embeddings and probing them with shared lightweight heads, enabling direct cross-paradigm comparison without end-to-end fine-tuning.
• Three complementary benchmark suites: TRL-CTBENCH (13 column/table tasks: schema, joinability, unionability, grounding), TRL-RBENCH (row prediction across 50 curated tables with 123 targets + 16 record-linkage datasets), and TRL-DLTE (compositional data-lake table enrichment from 47,772 tables, testing how well embeddings compose end-to-end).
• Key empirical findings: (1) Transfer is capability-specific—generic text encoders lead on surface-text tasks, while tabular specialists win where pretraining aligns with task demands. (2) Row-level signal is not single-faceted—within-table prediction and cross-table linkage favor different training regimes. (3) Compositional fit matters—capability-matched hybrids consistently outperform single-encoder reuse in DLTE, and per-stage marginals are informative but do not determine optimal pipelines.
• Curated assets released: 50 OpenML tables with 123 hand-verified targets, 16 explicit row-pair linkage rewrites, 20 column/table datasets standardized for representation-level evaluation, and a 47,772-table enrichment lake built from 1,379 parent tables.
• Open gaps identified: A specialization gap (no universal representation), a transfer-scope gap (intra-table vs. inter-table trade-offs), and a composition gap (granularity interactions) that single-paradigm evaluations cannot isolate.

Introduction and Theoretical Foundation

Tables are fundamental for structured data, and recent work has produced strong row-, column-, and table-level encoders. Many of these are intended as reusable components: tables can be encoded once and their embeddings indexed and reused across tasks and large multi-table corpora (e.g., data lakes) where per-task fine-tuning is impractical. However, existing evaluations are mostly conducted inside task-specific end-to-end pipelines, making it impossible to directly compare models from different training paradigms because a strong result may come from the wrapped predictor, training budget, or task-specific adaptation rather than from the encoder itself.

The paper frames the central comparability question: under one shared evaluation protocol over exported representations, how do heterogeneous tabular encoders actually differ? TRL-BENCH is designed to isolate reusable representation quality by:

• Running each model once through its supported wrapper to export row-, column-, or table-level embeddings.
• Using shared lightweight downstream modules (training-free, learned, or query-conditioned) to evaluate those embeddings across tasks.
• Treating retrieval, schema alignment, linkage, prediction, and grounding as atomic capabilities that serve as reusable building blocks for downstream tabular systems in the encode-once, reuse-many setting.

The theoretical foundation draws on two established properties of good representations from the representation-learning literature: recoverability under simple, capacity-limited readouts (the probing tradition) and transferability across many downstream tasks. A reusable tabular representation is good to the extent that a single exported embedding satisfies both.

Methodology

Standardized Representation-Level Protocol

For a table $T$ with columns $C(T) = (c_1, \dots, c_M)$ and rows $R(T) = (r_1, \dots, r_N)$, a tabular encoder $f_\theta$ may expose:

$$E_{\text{col}}(T) = (e_{\text{col}}^1, \dots, e_{\text{col}}^M), \quad E_{\text{row}}(T) = (e_{\text{row}}^1, \dots, e_{\text{row}}^N), \quad e_{\text{tbl}}(T)$$

For each task, the relevant exported encoder output(s) $e$ are fed into a downstream module $r$ (or $r_\psi$ with learned parameters) that maps to the task output. Three module types are used:

• Training-free modules $r(e)$: operate directly on embedding geometry (e.g., cosine ranking for schema matching, $k$-means for column clustering).
• Learned modules $r_\psi(e)$: lightweight supervised probes (linear head or one-hidden-layer MLP with hidden size 256) trained on exported embeddings with Adam under standardized settings.
• Query-conditioned modules $r_\psi(q, e)$: additionally consume a frozen text-query embedding $q = f_{\text{text}}(\text{query})$ (e.g., dual-projection head for table retrieval).

For supervised probes, both a linear head and an MLP are trained; the arithmetic average of the two is the canonical score. For table-level tasks, multiple aggregations (CLS, COL-MEAN, TOK-MEAN) are tried per model and the strongest is reported.

Benchmark Suites

TRL-CTBENCH (Column/Table Level): 13 tasks across four families:

• Schema understanding: column type prediction ($F_1$), column clustering (NMI), column relation prediction ($F_1$).
• Joinability: join search (MAP), column overlap (nRMSE), table-level join classification ($F_1$).
• Unionability: union search (MAP), schema matching (R@GT), union classification ($F_1$), union regression (nRMSE), table subset ($F_1$).
• Grounding: table QA (Accuracy), table retrieval (MRR). Query-conditioned tasks.

TRL-RBENCH (Row Level):

• Row prediction: 50 OpenML tables (123 curated targets: 77 classification, 46 regression). Each table has 2–3 targets. The encoder sees only observed columns $X$, produces one target-agnostic embedding per row, reused to predict each target $y_k \in Y = \{y_1, \dots, y_K\}$ ($K \geq 2$) with a lightweight probe.
• Record linkage: 16 datasets from DeepMatcher (8 clean, 4 dirty) and WDC Products (4 size variants). Split into Clean Linkage (DM-C) and Robust Linkage (DM-D + WDC). Paired exported row embeddings via concatenation to a lightweight supervised probe.

TRL-DLTE (Compositional Data-Lake Table Enrichment):

• Starts from a complete parent table. A block of rows and a block of columns are removed, leaving a seed query subtable. The removed rows form the union target (same schema, additional rows), and the removed columns form the join target (same rows, additional attributes).
• The system must recover both targets from a shared retrieval lake (47,772 tables: 11,032 targets + 36,740 CKAN distractors) via three stages:
  1. Stage 1 (Table Retrieval): Retrieve candidate tables using table embeddings.
  2. Stage 2 (Column Alignment): Align columns and predict union/join/none using column embeddings.
  3. Stage 3 (Row Matching & Merge): Match rows and merge content using row embeddings.
• Primary end-to-end metric: $UJ\text{-}H$, the per-query harmonic mean of union recall $R_{\text{union}}$ (fraction of removed-row-block cells recovered in seed columns) and join recall $R_{\text{join}}$ (fraction of removed-column-block cells recovered for seed rows):

$$UJ\text{-}H = \frac{2 R_{\text{union}} R_{\text{join}}}{R_{\text{union}} + R_{\text{join}}}$$

Pipelines can use a single multi-granular model or combine different specialists across stages.

Compared Models

20 models spanning:

• Generic Text: BERT, GTE
• Tabular-Pretrained: Table-Text (TaBERT, TAPAS, TAPEX), Table-Structure (TABBIE, TURL, TUTA), Column-Centric (Starmie, TabSketchFM)
• Transfer-Based: BERT, GTE, TABBIE, TUTA (also used as row encoders)
• Prior-Based: TabICL, TabPFN
• Target-Table Learners: VIME, SCARF, DAE, TabBinning, SAINT, SubTab, TabTransformer, TransTab

Empirical Validation / Results

Column- and Table-Level Results (TRL-CTBENCH)

Table 2 (partial reproduction below) shows results across 13 tasks for 10 models that natively expose column/table embeddings.

Key findings:

• Generic-text rankings (NR, lower is better) worsen from Schema through Grounding (BERT: 0.000 → 0.048 → 0.260 → 0.397), consistent with surface-text signal dominance.
• Tabular specialists win where pretraining aligns with task: Starmie (contrastive column-centric) leads Union Search and Schema Matching; Table-Text models lead Grounding; TURL leads Table QA.

Row-Level Results (TRL-RBENCH)

Table 3 summarizes row-level transfer.

Key findings:

• Prediction and linkage decouple: Prior-based TabICL leads prediction (AUROC 0.816, Macro-$F_1$ 0.671, SGM 0.505). Linkage leaders are transfer-based (BERT on Clean, GTE on Robust). Target-table SSL methods are competitive on prediction but weak on linkage.
• Intra- vs. inter-table transfer: Target-table SSL methods (trained per-table) fit locally → competitive on prediction, weak on linkage. Transfer-based encoders (shared model) produce comparable row spaces → lead linkage but mid-pack on prediction.
• Combining both axes: TransTab (cross-table contrastive + per-table SSL) is second on Robust Linkage; TabICL (meta-pretrained prior + target-table adaptation) leads prediction and ranks 5th on Robust Linkage.

Compositional Results (TRL-DLTE)

Table 4 shows per-stage model frequencies in the top-50 pipelines (out of 1,120) by dev-selected $UJ\text{-}H$.

Key findings:

• Capability-matched hybrids beat single-encoder reuse: Best hybrid TUTA/GTE/GTE achieves $UJ\text{-}H=0.229$, 0.090 above best monolithic BERT/BERT/BERT (0.139). Dev/test rankings correlate strongly (Spearman $\rho=0.96$, top-50 overlap 42/50).
• Compositional fit matters: Per-stage marginal leaders assemble to Starmie/TABBIE/TransTab at 0.134 $UJ\text{-}H$, well below the test rank-1 pipeline (Starmie/GTE/GTE, 0.253). Marginals are lossy selection rules. Atomic retrieval strength (GTE leads Table Retrieval MRR 0.476) decouples from compositional Stage-1 utility (GTE ranks 3rd in Stage-1 marginal).
• Shared identity-resolution capability: DLTE Stage-3 row-model rankings agree with RBench Robust Linkage NR at Spearman $|\rho|=0.80$ ($p=6.3\times10^{-4}$), identifying a consistent cross-table identity-resolution capability of frozen row embeddings.

Theoretical and Practical Implications

• Theoretical: The results challenge the notion of a universal tabular representation. Instead, encoder quality is capability-specific—different pretraining objectives excel at different atomic tasks. The specialization, transfer-scope, and composition gaps suggest that tabular representation learning has not yet converged on a unified account, and that evaluations must be multi-granular and multi-task.
• Practical: For practitioners deploying tabular encoders in encode-once, reuse-many settings (e.g., data lakes), the benchmark provides actionable guidance:
  • No single best encoder: Choose based on the target task family (surface-text vs. structural, intra-table vs. cross-table).
  • Hybrid pipelines outperform monoliths: Combining a strong table retriever (e.g., Starmie) with a strong column aligner (e.g., GTE) and a strong row matcher (e.g., GTE or TransTab) yields better end-to-end enrichment than any single multi-granular model.
  • Atomic capability rankings are necessary but not sufficient for pipeline composition—compositional fit (non-additive compatibility across stages) must be measured directly.
• Benchmarking community: TRL-BENCH sets a new standard for cross-paradigm comparison by isolating representation quality from task-specific wrappers. It complements end-to-end benchmarks (e.g., TabArena, LakeBench) by focusing on the reusable artifact (embeddings) rather than the full pipeline.

Conclusion

TRL-BENCH reframes tabular encoder evaluation around the artifact many downstream systems actually reuse: exported embeddings. Under a single representation-level protocol, heterogeneous encoders become comparable without conflating their embeddings with task-specific wrappers, retraining budgets, or adaptation.

The benchmark’s three suites (TRL-CTBENCH, TRL-RBENCH, TRL-DLTE) cover column/table transfer, row transfer, and compositional enrichment over 16 tasks and 87 datasets.