# LLM2Vec-Gen: Generative Embeddings from Large Language Models > LLM2Vec-Gen introduces self-supervised generative embeddings that encode an LLM's potential response, achieving state-of-the-art performance and inheriting safety and reasoning capabilities from the frozen backbone. - **Source:** [arXiv](https://arxiv.org/abs/2603.10913) - **Published:** 2026-03-13 - **Permalink:** https://picx.dev/p/KPCD70 - **Whiteboard:** https://picx.dev/p/KPCD70/image ## Summary # LLM2Vec-Gen: Generative Embeddings from Large Language Models - Summary ## Summary (Overview) * **Novel Paradigm**: Proposes a self-supervised framework where text embeddings represent the LLM's *potential response* to an input, rather than encoding the input's semantics, to bridge the "input-output gap" in embedding tasks. * **Efficient Training**: Adds trainable "thought" and "compression" special tokens to a frozen LLM's vocabulary. Training uses a dual objective of reconstructing the LLM's own response and aligning with an unsupervised teacher embedding, requiring only unlabeled queries. * **State-of-the-Art Performance**: Achieves new self-supervised SOTA on the MTEB benchmark, improving over the best unsupervised teacher by up to 9.3% and closing over 60% of the gap to supervised methods. * **Transferred Capabilities**: Demonstrates significant improvements in safety (up to 43.2% reduction in harmful content retrieval on AdvBench-IR) and reasoning-intensive retrieval (up to 29.3% improvement on BRIGHT) by inheriting these aligned capabilities from the frozen LLM backbone. * **Interpretable Embeddings**: The learned compression token embeddings are interpretable and can be decoded back into text via the LLM, revealing the semantic content they capture (e.g., safe refusals or answer concepts). ## Introduction and Theoretical Foundation Text embeddings are fundamental for NLP applications like semantic search and retrieval. While LLM-based encoders have advanced the state-of-the-art, they largely follow an **input-centric paradigm**, where the model encodes the semantic content of the input text itself. This paradigm struggles with the core requirement of embedding tasks: mapping **diverse inputs** (e.g., different queries about the same topic) to **similar outputs** in the embedding space. Bridging this "input-output gap" typically requires large-scale, curated paired data and contrastive learning. This work proposes a paradigm shift: instead of encoding the input, an embedding model should encode the **LLM's potential response** to that input (see Figure 1). The motivation is twofold: 1. It directly addresses the input-output gap, as semantically distinct inputs that elicit similar LLM responses (e.g., different harmful queries that all receive a refusal) will naturally be mapped closer together. 2. It allows the transfer of LLM capabilities like **safety alignment** and **reasoning** into the embedding space, as these capabilities manifest in the model's *response*, not its input. The theoretical foundation connects to **Joint Embedding Predictive Architectures (JEPA)**, which advocate learning by predicting in representation space. LLM2Vec-Gen's alignment objective shares this motivation, predicting a target representation (the teacher's embedding of the response) rather than reconstructing raw tokens. ## Methodology LLM2Vec-Gen is a self-supervised framework that distills a frozen LLM's potential response into a fixed set of latent suffix embeddings. **Core Components and Process** (see Figure 2): 1. **Data Generation**: For a corpus of unlabeled queries $\{q_i\}$, generate a response $r_i$ for each using the frozen LLM $M$. 2. **Teacher Embedding**: Use an unsupervised teacher encoder $E$ (e.g., LLM2Vec) to create a target embedding for the response: $e_i = E(r_i)$. 3. **Special Tokens**: Add two types of trainable tokens to $M$'s vocabulary: * $m$ **thought tokens** $(t_1, ..., t_m)$: Act as an intermediate computational buffer. * $n$ **compression tokens** $(c_1, ..., c_n)$: Designed to capture the semantic content of the response. These tokens are appended to the input query: $x_i = q_i \oplus t_{1:m} \oplus c_{1:n}$. 4. **Forward Pass & Embedding Formation**: Process $x_i$ through the frozen LLM to get the last-layer hidden states of the compression tokens: $[h^1_i, ..., h^n_i] = \text{LLM}(x_i)$. These are projected via lightweight MLPs: * $[p^1_i, ..., p^n_i] = \text{MLP}_{\text{recon}}([h^1_i, ..., h^n_i])$ (for reconstruction) * $\hat{e}_i = \text{Pool}(\text{MLP}_{\text{align}}([p^1_i, ..., p^n_i]))$ (for alignment) **Training Objectives**: * **Reconstruction Loss ($\mathcal{L}_{\text{recon}}$)**: Forces the compressed representation $p_i$ to preserve the content of $r_i$. The projected tokens $p_i$ are used as a soft prompt in a second forward pass, and the LLM is trained to reconstruct $r_i$ via next-token prediction: $$ \mathcal{L}_{\text{recon}} = -\sum_{j=1}^{|r_i|} \log P_{\text{LLM}}(r_{i,j} \mid p^1_i, ..., p^n_i, r_{i,