Summary (Overview)

• LatentPress introduces a novel context compression interface that writes conversational histories and long documents into continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference time.
• The system trains only a small reader-matched adapter (4.2M–26.2M parameters, ~0.1% of decoder parameters) while keeping the entire downstream decoder frozen.
• On LongMemEval, LatentPress achieves 0.504 accuracy at 7.70× compression versus 0.490 for uncompressed evidence, significantly outperforming text summaries (0.184) and OCR-based compression (0.426→0.312).
• On LongBench-QA, in-domain writers match or exceed raw-context reading at 4–8× compression, with writing taking 43 ms per conversation (roughly 10–22× faster than alternatives).
• Reading is 5–9× faster than raw context or cached OCR, validating soft tokens as a practical machine-facing context interface beyond text and vision.

Introduction and Theoretical Foundation

Background and Motivation

Long-running assistants and agents accumulate more history than they can afford to reread. A deployment trace may hold instructions, dialogue, plans, tool calls, observations, and environment feedback, yet a later decision often depends on only a small part of it. The paper argues that while text is convenient for people and interoperable across systems, a model need not require its stored or compressed context to be human-readable.

Theoretical Basis

The paper separates context use into two operations:

• WRITE: maps text to a compact state
• READ: supplies that state to a frozen decoder for downstream QA

This abstraction covers both conversational histories and long documents without attempting to replace retrieval, reflection, update policies, or conflict resolution in a complete memory system.

Positioning vs. Prior Work

The key distinction from prior compression methods (Gist, Auto-Compressor, ICAE, xRAG) is that LatentPress:

1. Leaves the downstream decoder entirely frozen
2. Trains only a small reader-matched adapter (~0.1% of decoder parameters)
3. Consumes vectors directly at the decoder's input-embedding layer
4. Supports variable-length, role-based compression rates

Methodology

Direct-Read Soft Context Interface

Let a context $x = (x_1, \ldots, x_T)$ be a sequence of segments (dialogue turns or document chunks). A frozen decoder $f_\theta$ answers a question $q$ from a compact representation. A small trainable writer maps $x$ to a short sequence of continuous vectors $m$:

$$m = \mathrm{WRITE}_\phi(x; \pi), \quad y = f_\theta([m; \mathrm{emb}(q)]).\tag{1}$$

For each position $i$, the writer fuses the literal input embedding $E_i$ with a context-aware abstraction $c_i$:

$$h_i = H(E_i, c_i).\tag{2}$$

Compression Rate Scheduling

The rule $\pi = (k_1, \ldots, k_T)$ determines how many neighboring token positions are pooled into each soft token:

• Uniform pooling: $k_i = k$ for every segment (used for documents)
• Role-based schedule: $k_{user} = 1$ and $k_{assistant} \in \{8, 16, 32\}$, so user turns retain raw token embeddings while assistant turns are encoded and pooled

Training Objectives

For generic representation learning, the loss combines reconstruction and forward-KL distillation:

$$\mathcal{L}(\phi) = \mathcal{L}_{\mathrm{rec}} + \lambda \mathcal{L}_{\mathrm{fkl}},\tag{3}$$

where for a target sequence $y = (y_1, \ldots, y_N)$:

$$\mathcal{L}_{\mathrm{rec}} = -\frac{1}{N} \sum_{t=1}^{N} \log p_{\mathrm{comp}, t}(y_t),\tag{4}$$ $$\mathcal{L}_{\mathrm{fkl}} = \frac{1}{N} \sum_{t=1}^{N} \mathrm{KL}(p_{\mathrm{full}, t} \parallel p_{\mathrm{comp}, t}).\tag{5}$$

Here $p_{\mathrm{full}, t}$ and $p_{\mathrm{comp}, t}$ are the frozen decoder's teacher-forced next-token distributions given the full and compressed context, respectively. The forward-KL term distills full-context behavior into the writer (with $\lambda = 1.0$).

Writer Sizes by Backbone

Empirical Validation / Results

LongMemEval: Conversational Memory

The writer is trained on 2,000 UltraChat conversations (text only, no QA labels) and evaluated zero-shot on 500 held-out LongMemEval oracle-evidence questions.

Table 2: Zero-shot LongMemEval results (Qwen2.5-7B reader)

Cross-backbone generalization (Table 3): LatentPress beats uniform pooling by +0.34 to +0.45 across all three readers (Qwen2.5-7B, Qwen3-8B, Qwen3-1.7B), and outperforms DeepSeek-OCR on Qwen2.5-7B (0.504 vs. 0.426) and Qwen3-1.7B (0.434 vs. 0.264).

LongBench-QA: Long-Document QA

Table 4: In-domain LatentPress on LongBench-QA English (official overall score in %)

Key findings:

• In-domain adaptation exceeds uncompressed baselines at 4× on all three readers (e.g., Qwen2.5-14B: 57.99 vs. 47.93)
• 16× compression degrades below raw baselines, exposing the cost of losing verbatim detail
• Cross-domain transfer (from LongMemEval-derived QA) partially succeeds, with 4× being the preferred rate

Efficiency Results

Write cost (Qwen3-8B, bfloat16, H100 GPU, batch of 8):

• LatentPress: 43 ms/conversation
• DeepSeek-OCR: 844–1056 ms (~22× longer)
• Text summarization: 407–645 ms (9–15× longer)
• ICAE: 350–700 ms (8–15× longer)

Read cost (Table 5, warm-loaded inference, 30 LongBench-QA examples):

LatentPress is 5.0–9.2× faster than raw inference and 5.5–9.4× faster than the cached OCR route.

Theoretical and Practical Implications

Key Insights

1. More tokens are not automatically better: The uncompressed oracle evidence on LongMemEval reaches only 0.490, while LatentPress achieves 0.504 at 7.70× compression—demonstrating that aggressive compression can actually improve accuracy by filtering noise.

2. Role-based compression is effective: Keeping short user turns lossless while pooling assistant turns preserves answer-bearing facts, suggesting that input structure contains valuable information about information density.

3. The interface is reader-agnostic: LatentPress generalizes across three Qwen backbones spanning two model families and a 4.7× range in scale, with only a small adapter trained per reader.

4. Continuous vectors as a machine-facing interface: The results establish soft tokens as a practical alternative to text and vision for context storage, with significant implications for memory systems, agent architectures, and long-document processing.

Practical Applications

• Conversational memory systems: Complementary to systems like MemGPT, Memory-Bank, and Generative Agents—LatentPress provides the representation layer while those handle retrieval and reflection
• Long-document QA: Enables faster inference with comparable or better accuracy at mild compression rates
• Deployment efficiency: Near-real-time writing (43 ms) and 5–9× faster reading make it suitable for latency-sensitive applications

Conclusion

LatentPress demonstrates that compressed context need not be stored as text or reconstructed from images before a language model can use it. The system satisfies four practical criteria:

1. Accuracy: Compressed readers preserve or improve accuracy on both benchmarks
2. Write cost: Encoded tokens generated in near-real time (43 ms)
3. Read cost: 5–9× faster than raw context or cached OCR
4. Trainable footprint: Only a small adapter (~0.1% of decoder parameters)

Future Directions

• Dynamic compression: Learning a policy to choose compression rates per segment via reinforcement learning against downstream answer reward
• Learned token-wise fusion: Implementing a learned importance-weighted fusion $H$ of literal and contextual features
• Additional readers and modalities: Extending to non-text contexts such as tool, multimodal, or embodied traces
• Integration with retrieval: Pairing LatentPress with a retriever for full memory systems (the current LongMemEval setup uses oracle evidence sessions)