StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring¶

Conference: ECCV2026
Paper: ECCV paper page
Full Text: Paper PDF
Area: Human Understanding
Keywords: streaming generation, co-speech gestures, key-pose anchoring, stochastic anchor masking, flow matching

TL;DR¶

StreamTalk turns streaming gesture generation into a clip-level loop of coarse generation, tail key-pose retrieval, and refinement, achieving FGD 0.383 for one speaker and 0.293 for all speakers on BEAT2 while using sparse destination poses to reduce minute-scale motion drift.

Background & Motivation¶

Speech-driven avatars need more than a convincing short motion segment: they must keep producing coherent body and hand movements as speech continues. Offline models can access the complete utterance, whereas interactive generation must work with incoming audio chunks and typically uses the previous clip's tail frames to initialize the next clip. This rolling pattern appears in both VQ-VAE and diffusion approaches; either family can produce good short clips while still propagating errors in previously generated poses. Increasing overlap can hide a stitching boundary without preventing arms from gradually rising, poses from leaving a natural activity range, or motion from freezing during pauses. Local continuity and long-horizon distributional stability are therefore different requirements: the former concerns smooth joins, while the latter concerns whether motion remains plausible a minute later.

The authors identify the missing ingredient as a forward constraint: a clip knows where it starts but lacks a plausible destination pose. Rather than retrieving an entire database sequence for playback, StreamTalk first generates a candidate from the current speech and then finds a nearby real pose for its tail. This destination is neither future ground-truth motion nor information from future speech; it comes from training data and constrains the current clip's endpoint rather than supplying the next utterance. A real pose alone does not guarantee a plausible trajectory, however: inserting it can cause a discontinuity, and borrowing root translation can incorrectly transfer walking motion into a stationary scene. Retrieval, the generator's ability to use sparse conditions, and the separation of articulation from displacement must consequently work together.

The goal is not to force every frame toward the database, but to supply one credible tail waypoint per clip and let the generator construct a continuous route toward it. This also explains why more anchors need not help: the clip already has historical constraints at its head, and multiple potentially incompatible poses can restrict natural transitions. Ablations over anchor count, placement, and refinement support this interpretation, but they remain dataset-specific evidence rather than a theoretical account of drift in every streaming model. Core Idea: learn motion completion from sparse, independently visible pose and translation prompts during training, then correct each clip's destination with one retrieved tail pose and propagate the corrected history to the next clip.

Method¶

Overall Architecture¶

Inputs comprise the current speech chunk, speaker identity, and motion conditions inherited from the previous clip; the output is a continuous motion clip in the SMPL-X representation. WavLM supplies 1024-dimensional frame-level speech features, while a separate 4-layer Transformer regresses facial features from speech; these are concatenated into audio-facial conditioning. The motion target contains 6D rotations for 55 joints and 3D root translation, giving \(55\times6+3=333\) dimensions per frame; this is not the speech feature dimensionality. During training, Stochastic Anchor Masking constructs partially visible motion conditions, and the Three-Branch DiT learns to reconstruct complete motion from them. At inference, the Three-Branch DiT first generates a coarse clip, the SPG Closed Loop retrieves and refines it, and the refined tail frames condition the next window. Ground-truth motion in the diagram supplies training prompts only; inference tail anchors come from the training pose database, not test-time ground-truth future motion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    GT["Ground-truth training motion"] --> SAM["Stochastic Anchor Masking"]
    SAM -.->|Sparse prompts for training only| DIT["Three-Branch DiT"]
    INPUT["Speech, identity,<br/>motion history, and noise"] --> DIT
    DIT -->|Coarse clip and cached midpoint| SPG["SPG Closed Loop"]
    DB["Training pose database"] -->|Tail-pose retrieval| SPG
    SPG -->|Invoke again with anchor| DIT
    SPG --> OUT["Refined clip<br/>Tail frames seed next window"]

Key Designs¶

1. Stochastic Anchor Masking: handling visible pose with unknown translation

Stochastic Anchor Masking (SAM) separates training motion into joint rotations and root translation, then samples an independent binary frame mask for each. Retained rotation frames form pose prompts and retained translation frames form translation prompts; other positions require completion rather than exposing the full motion answer. With \(P,T\) denoting rotations and translation and \(M_p,M_t\) their visibility masks, the operation in Section 3.4 is:

\[ P_{\mathrm{prompt}}=M_p\odot P,\qquad T_{\mathrm{prompt}}=M_t\odot T. \]

Independent masks allow a frame to contain a pose anchor without a translation anchor, matching SPG's inference-time retrieval of pose without root displacement. Pose prompts feed the body and hand branches, while translation prompts feed the translation branch, teaching completion from incomplete boundary conditions. Gesticulation and locomotion do not necessarily share a rhythm, so forcing both prompts to be visible at the same times would unnecessarily couple their variations.

SAM is not intended to make motion more realistic by itself; it teaches the generator how to use the sparse constraints added at inference. The ablation makes this interpretation testable: in the single-speaker setting, SAM alone changes FGD from 0.478 to 0.503 rather than improving it. Combining it with SPG reaches 0.383, supporting its role as training-inference condition alignment rather than an independently effective general regularizer. The cached text does not specify the full mask sampling probabilities or visible-frame count distribution, so a fixed masking ratio cannot be supplied as an implementation fact.

2. Three-Branch DiT: coordinating parts without letting global displacement dominate articulation

The Three-Branch DiT separates noisy motion into hands, body, and root translation, each processed by residual Transformer blocks. Speaker identity and flow time pass through separate MLPs and are concatenated into a style condition that modulates branch features through FiLM-style scaling and shifting. Each branch reads the corresponding pose or translation prompt through cross-attention and models within-part temporal dependencies through self-attention. The outputs are concatenated and combined with lightweight attention-based fusion, so separate processing does not imply independent hand and body motion.

Separating root translation prevents global displacement from interfering with local finger and arm learning while providing an independent interface for pose retrieval. The same raised-arm pose can occur while standing or walking; the training example's world position should not be copied as an intrinsic property of that pose. This representation allows SPG to update local pose constraints while retaining displacement and global orientation from the current context. Table 3(b) reports FGD 0.612, 0.562, and 0.478 for one-, two-, and three-branch backbones, with parameter counts of 70.0M, 71.0M, and 71.2M, respectively. These are backbone ablations: the three-branch value of 0.478 must not be mislabeled as the final 0.383 result with SAM and SPG.

3. SPG Closed Loop: retrieving a credible destination and regenerating the surrounding motion

Streaming Pose-Guided Generation (SPG) first generates a coarse clip from Gaussian noise, conditioned on current audio, identity, and historical tail frames, while caching intermediate generation states. The default flow-matching step count is \(N=10\); coarse generation lets speech establish a candidate motion direction before retrieval selects a nearby plausible endpoint. The pose database is built offline from per-frame training-set joint rotations, partitioned by speaker, without root translation trajectories that could be directly borrowed. A known identity selects its corresponding partition; when identity is unavailable, the authors propose a merged database of all training speakers as a fallback, not as established zero-shot performance.

Before retrieval, forward kinematics (FK) maps both the generated tail and database candidates into joint space, where the nearest candidate below a distance threshold \(\theta_p\) is selected. Direct rotation-space matching is avoided because a small shoulder-angle error can produce a much larger fingertip displacement along the kinematic chain; joint positions expose this perceptually important difference. The selected pose replaces the tail key-frame condition while the current clip's translation and global rotation are restored, avoiding transplantation of database world-coordinate state. The paper gives the threshold symbol but the inspected main text does not specify its numerical value, the behavior when no candidate passes it, or complete distance-normalization details. The mechanism can therefore be explained, but these omissions should not be filled with an allegedly exact nearest-neighbor implementation.

SPG does not finish by hard-splicing the retrieved frame into the output: it resumes generation from the cached midpoint using the updated anchor condition and completes the second half again. This allows neighboring frames to adapt to the new destination, avoids abrupt single-frame replacement, and saves roughly half the refinement computation compared with restarting from noise. The last 8 refined frames become the next clip's head prompt, so the correction affects subsequent generation rather than merely smoothing the displayed output. The loop operates at clip boundaries and corrects motion endpoints; it is neither a framewise collision solver nor a guarantee that physical constraints are always satisfied.

A Worked Example¶

Under the default setup, a window contains 60 frames, corresponding to the 2 seconds of motion in Table 4, with an 8-frame overlap between adjacent windows. Suppose the preceding clip ends with slightly elevated arms: the next window inherits those 8 frames and generates candidate continuation from current speech; this is a mechanism illustration, not an additional experiment. FK maps the coarse tail into joint space, and matching against the same speaker's training database supplies a nearby acceptable tail pose. Only its local joint configuration is adopted; current translation and global orientation are retained rather than moving the avatar to a database subject's position. Refinement then resumes from the cached midpoint of the 10-step generation process, jointly adjusting body and arm trajectories toward the tail anchor. The next window inherits the corrected last 8 frames; during steady rolling generation, each window contributes \(60-8=52\) new non-overlapping frames. The destination is retrieved from the current coarse candidate, not obtained by looking at real future motion that has yet to arrive.

Loss & Training¶

The paper uses flow-matching training with a clean-motion prediction parameterization, so the network output should not be directly interpreted as a velocity field. Given Gaussian noise \(X_0\) and ground-truth motion \(X_1\), training uses \(X_t=(1-t)X_0+tX_1\), with \(t\sim U(0,1)\). The main regression term recovers \(X_1\); an FK auxiliary term matches predicted and real joint positions; prompt consistency constrains predictions only at visible pose and translation anchors. The FK and prompt-consistency weights are \(\lambda_{fk}=1\) and \(\lambda_{prompt}=0.1\), addressing kinematic structure and adherence to the supplied anchors, respectively. Training uses 4 NVIDIA V100 GPUs with 16GB each, batch size 128, Adam with learning rate \(1\times10^{-4}\), and 1,000 epochs; the authors report approximately 31 hours. Equations (3) through (6) on page 8 have damaged operators, norms, and subscripts in the extracted text, so this note reports verifiable objective meanings and weights without reconstructing exact author equations. The loop endpoints in Algorithm 1 on page 6 should likewise not be mechanically converted into code against the stated \(N=10\) step count; the description here follows the prose's full coarse pass followed by second-half refinement.

Key Experimental Results¶

Main Results¶

BEAT2 contains synchronized audio and SMPL-X motion from 25 speakers, and the experiments follow the official training, validation, and test split. The single speaker is called Scott in the prose and Speaker-2 in Tables 2 and 3; this note uses the single-speaker label without inferring an additional identity mapping. FGD measures the distributional distance between generated and real motion, with lower values preferred; BC measures temporal alignment of motion and speech beats; DIV is the average L1 distance between generated samples. The original tables mark BC and DIV as approaching ground truth (GT), not as unconditionally higher-is-better, and the two speaker settings should be compared separately.

Selected results from Table 1 on page 10, all evaluated on BEAT2.

Setting	Method	FGD, lower is better	BC, closer to GT	DIV, closer to GT
Single speaker	GT	Not applicable	0.703	11.97
Single speaker	GestureLSM	0.409	0.714	13.42
Single speaker	SemTalk	0.428	0.777	12.91
Single speaker	StreamTalk	0.383	0.704	13.18
All speakers	GT	Not applicable	0.477	7.29
All speakers	RAG-GESTURE	0.487	0.514	9.94
All speakers	StreamTalk	0.293	0.616	7.27

StreamTalk has lower FGD in both settings, but its all-speaker BC of 0.616 is farther from GT 0.477 than RAG-GESTURE's 0.514, so the results do not establish superiority on every metric. All-speaker DIV 7.27 is close to GT 7.29, supporting the observation that anchoring does not collapse diversity, without replacing semantic-alignment or cross-domain generalization tests.

Ablation Study¶

Selected results from Table 2(a) on page 12, holding the backbone fixed to examine complementarity between SAM and SPG.

Config	Single-speaker FGD	Single-speaker BC	Single-speaker DIV	All-speaker FGD
Base model	0.478	0.716	12.30	0.391
Add SAM	0.503	0.747	13.72	0.379
Add SPG	0.455	0.695	14.24	0.353
Add SAM and SPG	0.383	0.704	13.18	0.293

Selected results from Table 3(a) on page 12, using single-speaker 60-frame windows to distinguish more anchors from a better endpoint.

Anchor position	Anchor count	FGD, lower is better	BC, closer to 0.703	DIV, closer to 11.97
Random	1	0.426	0.715	13.05
Middle	1	0.408	0.693	13.55
Tail	8	0.575	0.773	13.31
Tail	4	0.443	0.700	13.15
Tail	1	0.383	0.704	13.18

Key Findings¶

Combining SAM and SPG gives the lowest FGD in both settings, but the statement that SAM alone hurts applies only to one speaker; all-speaker FGD improves from 0.391 to 0.379.
Table 2(c) on page 12 reports FGD 0.673, 0.503, 0.471, and 0.383 for random anchors, direct insertion of retrieved anchors, retrieval with linear refinement, and retrieval with the proposed refinement, showing that retrieval quality and regeneration both matter.
The last row of Table 2(c) reports DIV 13.11, whereas the final configurations in Table 1, Table 2(a), and Table 3(a) report 13.18; the paper does not explain the difference, and these values should not be silently unified.
Figure 7(a) on page 13 shows sliding-window FGD over approximately 1,800 frames, or 60 seconds; the text supports reduced drift with SPG, but exact per-window values are not inferred from the extracted plot.
Table 3(c) on page 12 reduces self-intersection frames from 387 without SPG to 39 with SPG; the table does not give a total frame count, so these counts are not converted into incidence rates.
Table 4 on page 14 reports 76 FPS on a default V100-16G configuration, with initial generation taking 0.460 seconds, FK 0.006 seconds, retrieval 0.033 seconds, and refinement 0.230 seconds; these components do not fully reproduce the reported FPS, whose accounting follows the original table.
The default single-speaker database has 156,977 frames and occupies 16.6 MB; scaling it to 20 times its size, with 3.14M frames and 332 MB, yields retrieval time 0.792 seconds and overall throughput 36 FPS in Table 4. Memory and speed are author-reported, not independently measured here.
The user study on page 10 has 27 participants evaluating 15 approximately 60-second segments, producing over 400 ratings; the authors explicitly report Wilcoxon significance of \(p<10^{-3}\) for realism and rhythm consistency only, not for all four criteria.

Highlights & Insights¶

The anchor is feedback on a generated candidate, not an unconditionally supplied real pose. Letting speech establish the candidate before constraining its endpoint respects motion context better than a random pose intervention.
SAM's value depends on the inference interface: training covers both prompt sparsity and asymmetric visibility of pose and translation. This explains the component ablation more precisely than calling random masking generic augmentation.
Midpoint refinement reuses completed computation while allowing the model to reorganize neighboring frames. It is a renewed conditional generation process, not renamed linear interpolation.
A transferable reader insight is to retrieve a low-dimensional or sparse endpoint for long-sequence generation, then let the generator complete the trajectory. Such transfer still requires checking database coordinates, endpoint reachability, and retrieval-failure behavior.

Limitations & Future Work¶

Evidence centers on BEAT2 and roughly one-minute stability analysis; it does not establish stability for arbitrarily long sequences or equate all-speaker evaluation with unseen-identity generalization.
Clip-level real-time throughput is not end-to-end speech-to-motion latency: a 60-frame, 2-second window still involves audio buffering, and Table 4 does not provide a full interactive-latency breakdown.
A speaker-specific database depends on training coverage, and a plausible pose need not match current semantics or yield a collision-free trajectory. The global-database fallback and new-speaker performance require dedicated evaluation.
Exact nearest-neighbor retrieval becomes more expensive as the database grows; indexed search is proposed, but its joint effects on anchor quality, FGD, and latency are not evaluated.
Missing mask-distribution, retrieval-threshold, and failure-policy details, alongside damaged equations, inconsistent DIV values, and FPS accounting, prevent complete implementation reconstruction from this text alone.

Compared with DiffSHEG: the paper describes its inpainting as boundary smoothing; StreamTalk additionally retrieves a tail destination and regenerates motion, shifting the emphasis from join continuity to long-term endpoint constraints.
Compared with RAG-GESTURE: both use retrieval, but StreamTalk retrieves a pose neighbor of the current coarse candidate as feedback; they should not be reduced to an identical retrieval-augmented pipeline.
Compared with GestureLSM and SemTalk: these are major baselines in the quantitative and user studies; StreamTalk's advantages are primarily supported by FGD, dynamic distances, and long-horizon trends rather than a dedicated evaluation of semantic expression.
Research direction: candidate endpoints could be screened jointly for pose distance, speech-semantic compatibility, and reachability, with anchoring frequency adapted to uncertainty; this is a reader proposal, not an implemented component of the paper.

Rating¶

Novelty: 4/5. Tail retrieval becomes clip-level feedback, supported by independent masks and a matching branch structure with clear motivation.
Experimental Thoroughness: 4/5. Component, placement, count, refinement, and database-scale analyses are covered, but cross-domain, unknown-identity, and longer-sequence evidence is limited.
Writing Quality: 3/5. The central mechanism is clear, while numerical accounting and reproducibility details remain unresolved; damaged equations are a limitation of the current text extraction.
Value: 4/5. The method offers a useful endpoint-correction strategy for streaming motion generators without providing a general stability guarantee.