# GigaChat Audio: Time-aware Large Audio Language Model

> Periodic temporal anchors are essential for long-form audio grounding, as removing them collapses mIoU from 53.8 to 14.2 on 20-40 minute recordings.

- **Source:** [arXiv](https://arxiv.org/abs/2607.10387)
- **Published:** 2026-07-22
- **Permalink:** https://picx.dev/p/0y0r4u
- **Whiteboard:** https://picx.dev/p/0y0r4u/image

## Summary

## Summary (Overview)

- **GigaChat Audio** is a time-aware large audio language model that supports up to 120 minutes of audio input and produces answers with explicit timestamps, achieving strong temporal grounding for long recordings.
- The model interleaves continuous audio tokens with periodic **inter-timing markers** (e.g., `hh:mm:ss`) that act as temporal anchors, enabling precise time-localized responses.
- A **cascaded synthetic data pipeline** generates 10k+ hours of temporal supervision from timestamped transcripts, using slicing to reduce front-loading bias and a verifier for global consistency.
- Extensive ablations reveal that (a) **periodic temporal anchors are essential** for long-form grounding (removing them collapses mIoU from 53.8 to 14.2 on 20–40 min audio), (b) **training on a mixture of audio durations** is necessary for length generalization, and (c) even sparse anchors (e.g., once per minute) provide reliable grounding with minimal token overhead.
- The model outperforms open baselines (Qwen3-Omni, TimeAudio) and is competitive with proprietary Gemini 3 Flash on long-form temporal grounding, timed descriptions, and summarization tasks.

## Introduction and Theoretical Foundation

Long recordings (meetings, podcasts, lectures, call-center logs) increasingly require interactive interfaces where users ask questions and navigate to specific evidence. **Temporal grounding** is a verifiability primitive: the system must answer not only *what* happened, but also *when* it happened, enabling users to jump to the supporting audio segment.

Existing audio-conditioned LLMs (GPT-4o, Gemini, Qwen3-Omni, Voxtral) support instruction-following from speech, but temporal grounding in long-form audio remains unreliable—models often produce plausible content with non-parseable timestamps or coarse time references. The core challenge is that **time is not naturally represented in standard audio token streams**, and long recordings exacerbate the mismatch between internal representation and user-facing timestamped outputs.

The paper asks three practical questions:
1. What is the minimal and robust way to represent time in an audio LLM input/output interface?
2. How should temporal supervision be generated at scale for long recordings, where manual annotation is prohibitively expensive?
3. Do temporal models trained on a single duration regime generalize to other lengths?

The authors find strong asymmetry: training only on short audio does not extrapolate to long recordings, while training only on long audio degrades short-audio performance. **Periodic temporal anchors are essential** for long-form grounding.

## Methodology

### Model Architecture
The model is built by attaching an audio front-end to a 10B-A1.8B MoE text checkpoint with 256k-token context and fine-tuning on audio SFT data with explicit time signals.

**Audio processing stack:**  
Encoder → subsampler → projector. Uses FlashAttention with chunk-wise attention in the encoder (8s chunks, 40ms stride), producing continuous audio embeddings aligned to the text embedding space at a **160 ms frame rate**. Audio encoder pretraining follows a HuBERT-like setup with KMeans distillation over 2M hours of unlabeled multilingual audio.

**Time representation:**  
Inter-timings (periodic time markers) are interleaved between continuous audio tokens. They can be either:
- **Plain-text timestamps** (`hh:mm:ss`)
- **Special timing tokens** (dedicated tokens initialized from digit and colon embeddings)

Anchors are inserted every 60 seconds (or other frequencies) and a final marker is appended at the end of the audio sequence.

### Base Audio Post-Training Data
A heterogeneous SFT mixture covering:
- Captioning (general-domain, multilingual, music)
- Multi-turn dialogue (speech-context conversations, interruptions)
- Emotion recognition (discrete affect labels)
- General tasks (multilingual ASR, multi-speaker processing, spotting, TTS-style supervision)

This base mixture is kept fixed across time-aware ablations.

### Time-aware Tasks
Three generation tasks:
1. **Temporal grounding** – localize a text-described event by predicting a time interval.
2. **Fragment description** – given a provided interval, produce a natural-language description of the audio in that window.
3. **Summarization with timings** – produce a multi-part summary anchored in time, where the model must also decide the segmentation.

### Synthetic Data Generation Pipeline
- **Source:** Six English shards of YODAS2 (24k hours), filtered to English (p(English) > 0.7) using VoxLingua107 ECAPA-TDNN, yielding ~16k hours. WhisperX provides word-level timestamps; silence-ratio filtering keeps 14k hours. Bucketed by duration (≤20 min, 20–40 min, ≥40 min) and balanced.
- **Generation:** A text-only GPT-OSS-120B generates QA pairs from ~10-minute transcript slices (to reduce front-loading bias). A separate verifier consumes the full timestamped transcript and filters inconsistent generations.
- **Train/eval split:** At the audio level. For evaluation, five answers are generated per question with high temperature; questions with low median overlap are filtered out.
- **For summarization:** Full timestamped transcript is provided in a single pass.

### Evaluation Metrics
- **Temporal grounding:** mean Intersection over Union (mIoU) and Median Absolute Error (MAE) in seconds (computed at interval midpoint).
- **Fragment descriptions:** LLM-as-a-judge evaluates factual accuracy and hallucination on a 1–5 scale.
- **Summarization with timings:** Four dimensions – Timing structure (Tm), Factual accuracy (Acc), Errors and contradictions (Err), Style (Style). Aggregate content score (AES) = $(Acc + 1 - Err + Style)/3$. Also reports **round segment proportion** (Rd) – segments with duration multiples of 60 seconds.

## Empirical Validation / Results

### Main Comparison Across Tasks and Datasets
**Table 1 (key results):**  
Performance across AudioGrounding (AGr, 7–10s), AMI Corpus (15–50min), DCASE Audio QA (DAQA, 6–10s), and the authors' benchmarks for temporal grounding (TGr), descriptions, and summarization (20–40min). Models compared: Qwen3-Omni-30B, TimeAudio, Gemini 3 Flash, and Ours (with inter-timings at 60s and 7s, and without inter-timings).

| Model | AGr (↑) | AMI (↓) | DAQA (↓) | TGr (mIoU ↑) 0–1m | TGr 2–5m | TGr 20–40m | Descr 0–1m | Descr 2–5m | Descr 20–40m | Summ Tm (↑) | Summ AES (↑) | Summ Rd (↓) |
|-------|---------|---------|----------|---------------------|-----------|-------------|------------|------------|--------------|------------|-------------|-------------|
| Qwen3-Omni-30B | 50.8 | 290.5 | 1.00 | 47.2 | 27.3 | 3.6 | 3.95 | 3.70 | 2.11 | 43.7 | 74.3 | 24.9 |
| TimeAudio | 58.8 | – | 0.12 | 19.6 | – | – | 1.41 | – | – | – | – | – |
| Gemini 3 Flash | 41.7 | 1.00 | 0.9 | 39.3 | 30.2 | 56.1 | 3.63 | 3.05 | 3.75 | 73.6 | 91.3 | 8.2 |
| Ours (inter=60s) | 45.1 | 3.50 | 1.70 | 40.6 | 53.0 | 53.8 | 3.63 | 3.76 | 3.83 | 76.7 | 88.1 | 16.9 |
| Ours (inter=7s) | 39.7 | 1.50 | 1.70 | 54.0 | 64.4 | 65.2 | 3.85 | 3.91 | 3.94 | 79.0 | 89.2 | 10.3 |
| Ours (w/o inter) | 24.5 | 66.0 | 3.20 | 38.1 | 34.0 | 14.2 | 3.54 | 3.49 | 2.67 | 69.2 | 87.5 | 48.4 |

Key findings:
- Strong short-clip performance (e.g., Qwen3-Omni) does not translate to long-form grounding (drops to 3.6 mIoU on 20–40min).
- **Removing inter-timings collapses long-form mIoU from 53.8 to 14.2** and degrades descriptions and summaries.
- With 7s anchors, mIoU reaches 65.2 on 20–40min, surpassing Gemini 3 Flash (56.1).

### Length Extrapolation (Figure 3)
Models trained on specific duration buckets (rows) are evaluated across durations up to 120 minutes. Observations:
- **Training only on short audio fails on long recordings.**
- **Training only on long audio hurts short-audio performance.**
- **The model trained on all durations** outperforms other regimes across nearly all audio lengths.

### Special Timing Tokens vs. Plain-text (Table 2)
Special timing tokens (dedicated tokens for `hh:mm:ss`) require a much larger share of temporal grounding data in the SFT mixture to match plain-text performance. Only at very high ratios (33.4%–50%) do they reach comparable mIoU.

| TG ratio | 2.4% | 4.8% | 9.1% | 16.7% | 33.4% | 50.0% |
|----------|------|------|------|-------|-------|-------|
| regular (plain-text) | 41.4 | 50.9 | 52.5 | 57.5 | – | – |
| extra tokens | – | 13.1 | 37.6 | 50.0 | 55.2 | 56.8 |

### Inter-timing Frequency Trade-off (Table 3)
More frequent anchors improve grounding at the cost of additional tokens. Even with 60s anchors, median error is 3s (vs. 1.5s for 7s anchors), showing effective interpolation between minute-spaced anchors.

| Freq | 7s | 15s | 30s | 60s | 120s | 240s |
|------|----|-----|-----|-----|------|------|
| mIoU (↑) | 63.0 | 59.9 | 55.5 | 50.9 | 41.3 | 31.0 |
| MAE (↓) | 1.5 | 2.0 | 2.5 | 3.0 | 5.0 | 8.5 |
| added ratio | 16.0% | 7.5% | 3.7% | 1.9% | 0.9% | 0.5% |

### Inter-timing Format (Table 4)
With one anchor per minute, `hh:mm:ss` is the best format. Minute-index formats (`m:0`, `m:`) are close with lower token overhead. Pure seconds degrade performance substantially.

| Format | hh:mm:ss | m:0 | m: | m sec |
|--------|----------|-----|-----|-------|
| mIoU | 50.9 | 47.1 | 43.8 | 44.5 | 20.9 |

## Theoretical and Practical Implications

- **Temporal grounding in long recordings is a major bottleneck** for existing multimodal models; they degrade sharply beyond a few minutes of audio. The paper demonstrates that this can be addressed by explicit periodic temporal anchors.
- **Minimal time representation:** Plain-text `hh:mm:ss` or even sparse minute-index formats are sufficient; special timing tokens require disproportionate data to match plain-text performance.
- **Training data composition matters:** A mixture of audio durations is necessary for length generalization. Training on a single duration regime leads to asymmetric failure.
- **Synthetic data at scale:** The cascaded pipeline (transcript slicing + verifier) enables cost-effective generation of temporal supervision for long recordings, addressing the annotation bottleneck.
- **Practical deployment:** The model achieves second-level precision with minute-spaced anchors (median error 3s), suggesting that anchor frequency can be tuned to balance accuracy and computational cost.

## Conclusion

The authors present **GigaChat Audio**, a time-aware audio LLM that supports up to 120 minutes of input and produces answers and summaries explicitly anchored in time. Key findings:

- Periodic temporal anchors (inter-timings) are **essential** for stable long-form grounding; removing them collapses performance.
- Even sparse anchors (e.g., once per minute) provide reliable grounding, enabling a favorable accuracy–compute trade-off.
- Training on a **mixture of audio durations** is necessary for length generalization; models trained on a single regime fail asymmetrically.
- Plain-text `hh:mm:ss` format is effective and robust; special timing tokens require more data.

The model is released as open-weight, along with a 10k+ hours dataset spanning seconds-to-hours recordings for temporal QA and timed summarization, to facilitate further research on time-aware audio understanding.

---

_Markdown view of https://picx.dev/p/0y0r4u, served by PicX — AI-generated visual whiteboard summaries of research papers._