# LatentPress: Context Compression Beyond Text and Vision

> LatentPress writes conversational history into continuous memory tokens read directly by a frozen decoder, achieving 7.7x compression with accuracy exceeding raw context while reading 5-9x faster.

- **Source:** [arXiv](https://arxiv.org/abs/2609.01507)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/7antf0
- **Whiteboard:** https://picx.dev/p/7antf0/image

## Summary

## Summary (Overview)

- **LatentPress** introduces a novel context compression interface that writes conversational histories and long documents into **continuous memory tokens** that a frozen decoder reads directly through its input-embedding interface, with **no text reconstruction at inference time**.
- The system trains only a **small reader-matched adapter** (4.2M–26.2M parameters, ~0.1% of decoder parameters) while keeping the entire downstream decoder frozen.
- On **LongMemEval**, LatentPress achieves **0.504 accuracy at 7.70× compression** versus 0.490 for uncompressed evidence, significantly outperforming text summaries (0.184) and OCR-based compression (0.426→0.312).
- On **LongBench-QA**, in-domain writers match or exceed raw-context reading at 4–8× compression, with writing taking **43 ms per conversation** (roughly 10–22× faster than alternatives).
- Reading is **5–9× faster** than raw context or cached OCR, validating soft tokens as a practical machine-facing context interface beyond text and vision.

## Introduction and Theoretical Foundation

### Background and Motivation

Long-running assistants and agents accumulate more history than they can afford to reread. A deployment trace may hold instructions, dialogue, plans, tool calls, observations, and environment feedback, yet a later decision often depends on only a small part of it. The paper argues that while text is convenient for people and interoperable across systems, **a model need not require its stored or compressed context to be human-readable**.

### Theoretical Basis

The paper separates context use into two operations:
- **WRITE**: maps text to a compact state
- **READ**: supplies that state to a frozen decoder for downstream QA

This abstraction covers both conversational histories and long documents without attempting to replace retrieval, reflection, update policies, or conflict resolution in a complete memory system.

### Positioning vs. Prior Work

The key distinction from prior compression methods (Gist, Auto-Compressor, ICAE, xRAG) is that LatentPress:
1. Leaves the downstream decoder **entirely frozen**
2. Trains only a **small reader-matched adapter** (~0.1% of decoder parameters)
3. Consumes vectors **directly at the decoder's input-embedding layer**
4. Supports **variable-length, role-based compression rates**

## Methodology

### Direct-Read Soft Context Interface

Let a context $x = (x_1, \ldots, x_T)$ be a sequence of segments (dialogue turns or document chunks). A frozen decoder $f_\theta$ answers a question $q$ from a compact representation. A small trainable writer maps $x$ to a short sequence of continuous vectors $m$:

$$
m = \mathrm{WRITE}_\phi(x; \pi), \quad y = f_\theta([m; \mathrm{emb}(q)]).\tag{1}
$$

For each position $i$, the writer fuses the literal input embedding $E_i$ with a context-aware abstraction $c_i$:

$$
h_i = H(E_i, c_i).\tag{2}
$$

### Compression Rate Scheduling

The rule $\pi = (k_1, \ldots, k_T)$ determines how many neighboring token positions are pooled into each soft token:

- **Uniform pooling**: $k_i = k$ for every segment (used for documents)
- **Role-based schedule**: $k_{user} = 1$ and $k_{assistant} \in \{8, 16, 32\}$, so user turns retain raw token embeddings while assistant turns are encoded and pooled

### Training Objectives

For generic representation learning, the loss combines reconstruction and forward-KL distillation:

$$
\mathcal{L}(\phi) = \mathcal{L}_{\mathrm{rec}} + \lambda \mathcal{L}_{\mathrm{fkl}},\tag{3}
$$

where for a target sequence $y = (y_1, \ldots, y_N)$:

$$
\mathcal{L}_{\mathrm{rec}} = -\frac{1}{N} \sum_{t=1}^{N} \log p_{\mathrm{comp}, t}(y_t),\tag{4}
$$

$$
\mathcal{L}_{\mathrm{fkl}} = \frac{1}{N} \sum_{t=1}^{N} \mathrm{KL}(p_{\mathrm{full}, t} \parallel p_{\mathrm{comp}, t}).\tag{5}
$$

Here $p_{\mathrm{full}, t}$ and $p_{\mathrm{comp}, t}$ are the frozen decoder's teacher-forced next-token distributions given the full and compressed context, respectively. The forward-KL term distills full-context behavior into the writer (with $\lambda = 1.0$).

### Writer Sizes by Backbone

| Backbone | Writer Parameters |
|----------|------------------|
| Qwen2.5-7B | 12.849M |
| Qwen3-8B | 16.781M |
| Qwen3-1.7B | 4.196M |
| Qwen2.5-14B | 26.220M |

## Empirical Validation / Results

### LongMemEval: Conversational Memory

The writer is trained on 2,000 UltraChat conversations (text only, no QA labels) and evaluated zero-shot on 500 held-out LongMemEval oracle-evidence questions.

**Table 2: Zero-shot LongMemEval results (Qwen2.5-7B reader)**

| Method | Compression | Overall | user-fact |
|--------|-------------|---------|-----------|
| uncompressed evidence | 1.0× | 0.490 | 0.946 |
| LatentPress, $k_a=8$ | 4.62× | 0.476±0.014 | 0.938±0.007 |
| LatentPress, $k_a=16$ | 6.27× | 0.478±0.020 | 0.891±0.015 |
| LatentPress, $k_a=32$ | 7.70× | 0.504±0.024 | 0.938±0.010 |
| ICAE | 4.12× | 0.452±0.017 | 0.548±0.019 |
| ICAE | 8.96× | 0.318±0.022 | 0.381±0.023 |
| DeepSeek-OCR | 2.33× | 0.426 | 0.797 |
| DeepSeek-OCR | 9.34× | 0.312 | 0.594 |
| text summary | 12.06× | 0.184 | 0.297 |

**Cross-backbone generalization** (Table 3): LatentPress beats uniform pooling by +0.34 to +0.45 across all three readers (Qwen2.5-7B, Qwen3-8B, Qwen3-1.7B), and outperforms DeepSeek-OCR on Qwen2.5-7B (0.504 vs. 0.426) and Qwen3-1.7B (0.434 vs. 0.264).

### LongBench-QA: Long-Document QA

**Table 4: In-domain LatentPress on LongBench-QA English (official overall score in %)**

| Reader | Setting | Overall |
|--------|---------|---------|
| Qwen2.5-7B | raw context (1×) | 43.80 |
| Qwen2.5-7B | in-domain f4 | **49.06**±2.30 |
| Qwen2.5-7B | in-domain f8 | 43.77±2.83 |
| Qwen2.5-7B | in-domain f16 | 37.78±3.46 |
| Qwen3-8B | raw context (1×) | 30.80 |
| Qwen3-8B | in-domain f4 | **39.62**±2.31 |
| Qwen3-8B | in-domain f8 | 36.93±2.82 |
| Qwen3-8B | in-domain f16 | 26.12±3.33 |
| Qwen2.5-14B | raw context (1×) | 47.93 |
| Qwen2.5-14B | in-domain f4 | **57.99**±2.35 |
| Qwen2.5-14B | in-domain f8 | 52.18±2.82 |
| Qwen2.5-14B | in-domain f16 | 40.30±3.51 |

Key findings:
- In-domain adaptation **exceeds uncompressed baselines at 4×** on all three readers (e.g., Qwen2.5-14B: 57.99 vs. 47.93)
- **16× compression degrades** below raw baselines, exposing the cost of losing verbatim detail
- Cross-domain transfer (from LongMemEval-derived QA) partially succeeds, with 4× being the preferred rate

### Efficiency Results

**Write cost** (Qwen3-8B, bfloat16, H100 GPU, batch of 8):
- LatentPress: **43 ms/conversation**
- DeepSeek-OCR: 844–1056 ms (~22× longer)
- Text summarization: 407–645 ms (9–15× longer)
- ICAE: 350–700 ms (8–15× longer)

**Read cost** (Table 5, warm-loaded inference, 30 LongBench-QA examples):

| Reader | Raw context | LatentPress f8 | Cached OCR b640 |
|--------|-------------|----------------|-----------------|
| Qwen2.5-7B | 2.44s | **0.49s** | 2.71s |
| Qwen2.5-14B | 4.14s | **0.49s** | 4.34s |
| Qwen3-8B | 3.97s | **0.43s** | 4.03s |

LatentPress is **5.0–9.2× faster** than raw inference and **5.5–9.4× faster** than the cached OCR route.

## Theoretical and Practical Implications

### Key Insights

1. **More tokens are not automatically better**: The uncompressed oracle evidence on LongMemEval reaches only 0.490, while LatentPress achieves 0.504 at 7.70× compression—demonstrating that aggressive compression can actually improve accuracy by filtering noise.

2. **Role-based compression is effective**: Keeping short user turns lossless while pooling assistant turns preserves answer-bearing facts, suggesting that **input structure contains valuable information about information density**.

3. **The interface is reader-agnostic**: LatentPress generalizes across three Qwen backbones spanning two model families and a 4.7× range in scale, with only a small adapter trained per reader.

4. **Continuous vectors as a machine-facing interface**: The results establish soft tokens as a practical alternative to text and vision for context storage, with significant implications for memory systems, agent architectures, and long-document processing.

### Practical Applications

- **Conversational memory systems**: Complementary to systems like MemGPT, Memory-Bank, and Generative Agents—LatentPress provides the representation layer while those handle retrieval and reflection
- **Long-document QA**: Enables faster inference with comparable or better accuracy at mild compression rates
- **Deployment efficiency**: Near-real-time writing (43 ms) and 5–9× faster reading make it suitable for latency-sensitive applications

## Conclusion

LatentPress demonstrates that compressed context need not be stored as text or reconstructed from images before a language model can use it. The system satisfies four practical criteria:
1. **Accuracy**: Compressed readers preserve or improve accuracy on both benchmarks
2. **Write cost**: Encoded tokens generated in near-real time (43 ms)
3. **Read cost**: 5–9× faster than raw context or cached OCR
4. **Trainable footprint**: Only a small adapter (~0.1% of decoder parameters)

### Future Directions

- **Dynamic compression**: Learning a policy to choose compression rates per segment via reinforcement learning against downstream answer reward
- **Learned token-wise fusion**: Implementing a learned importance-weighted fusion $H$ of literal and contextual features
- **Additional readers and modalities**: Extending to non-text contexts such as tool, multimodal, or embodied traces
- **Integration with retrieval**: Pairing LatentPress with a retriever for full memory systems (the current LongMemEval setup uses oracle evidence sessions)

---

_Markdown view of https://picx.dev/p/7antf0, served by PicX — AI-generated visual whiteboard summaries of research papers._