Summary (Overview)

• This paper identifies that LLM-derived text embeddings, when projected onto the vocabulary space via Logit Lens, are dominated by high-frequency but semantically uninformative tokens, leading to suboptimal zero-shot performance.
• Using Logit Spectroscopy on a reverse-engineered "average token," the authors discover that the "edge spectrum" (subspaces corresponding to the largest and smallest singular values of the unembedding matrix) is responsible for encoding these frequent tokens.
• They introduce EmbedFilter, a simple linear transformation that filters out the edge spectrum by retaining only the bulk of the singular vectors, thereby suppressing frequent token influence and enhancing semantic representations.
• EmbedFilter provides up to +14.1% improvement on MTEB across multiple LLMs under zero-shot settings, and naturally enables dimensionality reduction (e.g., to 1/8 of original size) without loss of performance.
• Extensive experiments show EmbedFilter outperforms existing calibration methods (e.g., whitening) while requiring no training or calibration data.

Introduction and Theoretical Foundation

The paper addresses the persistent gap in LLMs' ability to serve as off-the-shelf zero-shot embedding models. While LLMs show strong zero-shot capabilities on many tasks, their text embeddings underperform on benchmarks like MTEB.

Core Observation: When applying Logit Lens (projecting hidden states to the vocabulary space) to raw LLM text embeddings, the top-decoded tokens are consistently high-frequency but uninformative tokens (e.g., "the", ",", "a"), regardless of input semantics. This phenomenon holds across different LLM families (Qwen, Llama, Mistral, see Figure 1 in the paper).

Theoretical Insight: The authors link this to the well-known anisotropy problem – embeddings lie in a narrow cone. They hypothesize that the centroid of this cone corresponds to an "average token" derived from the training corpus. The unembedding matrix $\boldsymbol{W}_U$ encodes a subspace that projects embeddings toward this commonality, overshadowing semantic features.

Key Tools:

• Logit Lens: Maps embeddings to vocabulary probabilities via $\mathrm{Softmax}(\boldsymbol{h}\boldsymbol{W}_U^\top)$.
• Logit Spectroscopy: Uses SVD of $\boldsymbol{W}_U = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^\top$ and a filter $\boldsymbol{\Psi}_i = \boldsymbol{I} - \boldsymbol{V}_{[i]}\boldsymbol{V}_{[i]}^\top$ to measure the contribution of each singular subspace.

Methodology

Step 1: Reverse-Engineer the Average Token. Using the unembedding matrix and empirical token frequencies $\hat{\boldsymbol{p}}$ (from RedPajama corpus), the "average token" representation is:

$$\hat{\boldsymbol{h}} = \log(\hat{\boldsymbol{p}})\, \boldsymbol{W}_U^+$$

where $\boldsymbol{W}_U^+$ is the Moore-Penrose pseudo-inverse. The bias term $\boldsymbol{b}$ is omitted as it does not affect spectral properties.

Step 2: Identify Edge Spectrum via Logit Spectroscopy. For each singular dimension $i$, apply $\boldsymbol{\Psi}_i$ to $\hat{\boldsymbol{h}}$ and measure the cumulative logit shift for the top $k$ frequent tokens $V^+$:

$$\Delta_\pi(i) = \frac{\sum_{j\in V^+}\big(\tilde{w}_j^{(i)} - \hat{w}_j\big)}{\sum_{j\in V^+}\hat{w}_j}$$

Figure 2 (paper) shows $\Delta_\pi$ is significantly larger at the extremes of the spectrum (largest and smallest singular values), confirming the edge subspace encodes frequent tokens.

Step 3: EmbedFilter Formulation. Construct $\boldsymbol{\Phi}_\tau$ by keeping only the mid-range ("bulk") singular vectors:

$$\boldsymbol{\Phi}_\tau = \boldsymbol{V}_{[l_\tau: r_\tau]} \boldsymbol{V}_{[l_\tau: r_\tau]}^\top$$

where $\tau$ is a filtering ratio, and $l_\tau, r_\tau$ define the retained columns. Refined embeddings:

$$\tilde{\boldsymbol{e}}_i = \boldsymbol{e}_i \boldsymbol{\Phi}_\tau^\top$$

Dimensionality Reduction: Because $\boldsymbol{V}$ is orthogonal, distance is preserved:

$$\| \boldsymbol{x}\boldsymbol{\Phi}_\tau^\top - \boldsymbol{y}\boldsymbol{\Phi}_\tau^\top \|_2 = \| \boldsymbol{x}\boldsymbol{V}_{[l_\tau:r_\tau]} - \boldsymbol{y}\boldsymbol{V}_{[l_\tau:r_\tau]} \|_2$$

Thus the embedding dimension can be reduced to $d/\tau$ (number of retained columns) without any similarity metric loss.

Empirical Validation / Results

Experiments are conducted on the MTEB benchmark (STS, Classification, Clustering, PairClassification, Reranking, Retrieval) using three LLM backbones: Qwen2.5-0.5B, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3.

Main Results (Table 1):

EmbFilter consistently improves across all setups, with gains up to +14.1%. Even at $\tau = 8$ (dimension reduced to 1/8), performance remains competitive or superior.

Ablation Studies (Table 5):

• Truncation (taking first half dims) and random dimension selection both underperform vanilla baseline, proving improvement is not due to simple dimension reduction.
• Filtering only the secondary (small singular values) subspace gives improvements (+3.12), but the full EmbedFilter (both edges removed) achieves the best result.
• The optimal strategy (filtering dimensions with highest $\Delta_\pi$) yields nearly identical performance to EmbedFilter, confirming its effectiveness without task-specific calibration.

Comparison with Whitening (Table 6): EmbedFilter outperforms BERT-whitening (55% improvement vs 5.9%) despite requiring no calibration data.

Dimensionality Reduction (Table 4): With Llama and $\tau=8$ (512 dimensions), EmbFilter achieves an average of 56.61 on MTEB, surpassing strong baselines like SimCSE (53.54) and coCondenser (55.48) while using far fewer dimensions.

Theoretical and Practical Implications

• Theoretical: The work provides a mechanistic interpretation of LLM embedding anisotropy: the unembedding matrix encodes an "edge spectrum" subspace that biases representations toward frequent, uninformative tokens. This explains the suboptimal zero-shot performance and suggests that the unembedding matrix can serve as a "feature lens" for analyzing embedding spaces.
• Practical: EmbedFilter is a lightweight, training-free post-processing step that can be applied to any LLM. It improves performance across multiple backbones and prompt strategies (PromptEOL, ECHO, MetaEOL, GenEOL). The built-in dimensionality reduction reduces index storage and speeds up retrieval by a factor of $\tau$, making LLM-based embeddings viable for large-scale applications.
• Broader impact: The insights suggest that future text embedding training should explicitly account for and suppress high-frequency token biases, potentially by designing loss functions that penalize alignment with edge spectrum directions.

Conclusion

The authors discover that the unembedding matrix of LLMs encodes a subspace (edge spectrum) responsible for projecting text embeddings toward high-frequency tokens, which limits their zero-shot semantic representation ability. They propose EmbedFilter, a simple linear transformation that filters out this subspace, yielding substantial improvements on MTEB (up to +14.1%) across multiple model families without training. EmbedFilter also enables lossless dimensionality reduction, improving efficiency. The work provides a mechanistic understanding of LLM embedding shortcomings and offers a practical, principled fix. Future work may explore optimal asymmetric filtering strategies and deeper integration into embedding training pipelines.