A perceptually grounded and semantics-aware benchmark for holistic co-speech gesture generation that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation.
Holistic gesture quality is not one thing: naturalness, diversity, synchrony, physical plausibility, and semantic appropriateness can move in different directions.
Recent holistic systems synthesize coordinated face, hand, upper-body, lower-body, and global motion, often conditioned on both speech acoustics and linguistic content. Yet evaluation has not kept pace. Different works use different datasets, preprocessing, rendering pipelines, metrics, and subjective protocols, making reported gains hard to compare directly. Even when models are evaluated under the same protocol, commonly used objective metrics may still fail to reflect human perception, particularly for semantic gestures that are sparse, context-dependent, and can be expressed through many valid physical forms.
Prior systems are often evaluated with incompatible training regimes, data subsets, sequence boundaries, renderers, and metric implementations. Apparent model improvements may therefore reflect experimental choices rather than algorithmic progress.
Objective measures are reproducible and scalable, but strong correspondence with human judgments is not guaranteed. Our benchmark explicitly asks which metrics align with which perceptual properties instead of assuming that all scores are equally meaningful.
A gesture can be smooth, plausible, and synchronized while still being semantically empty or mismatched. Reference-pose metrics also penalize alternative gestures that may communicate the same meaning through different motion.
The benchmark compares EMAGE, GestureLSM, SemTalk, and SemConFlow under a shared BEAT2 evaluation setting. The goal is not only to rank systems, but to make those rankings interpretable by controlling the sources of experimental variation.
≈60 hours, 25 speakers, synchronized speech and holistic SMPL-X motion.
Official 85% / 7.5% / 7.5% train / validation / test split across all 25 speakers.
All benchmark experiments are conducted on a single NVIDIA A100 GPU.
A unified holistic framework that jointly synthesizes facial expressions, upper- and lower-body motion, hands, and global translation. It uses a Masked Audio Gesture Transformer and four compositional VQ-VAEs for different body regions. Content Rhythm Attention adaptively combines speech rhythm and transcript semantics.
A flow-matching framework for real-time holistic gesture generation. Region-specific residual vector-quantized tokens are coordinated through spatial and temporal attention, while a latent shortcut model reduces the number of sampling steps.
Separates rhythm-related base motion from sparse semantic motion. A hierarchical coarse-to-fine module establishes rhythmic consistency, while semantic emphasis learning activates content-related motion at key frames using text, CLIP, emotion, and speech features.
Learns region-specific RVQ-VAE motion priors and aligns the resulting holistic motion latent with text and audio. Contrastive flow matching uses mismatched audio-text contexts as negatives, encouraging semantically congruent trajectories and discouraging generic rhythm-dominated gestures.
The benchmark organizes objective evaluation by what each metric is intended to capture. This distinction is important because several metrics that look similar numerically measure fundamentally different properties.
Co-speech gesture realization is one-to-many. The same communicative intent may be expressed with a different hand path, orientation, scale, or even a different valid gesture. Exact geometric agreement can therefore underestimate perceptually appropriate motion.
FGD, Density, and Coverage characterize whether generated motion occupies plausible regions of motion space. In our results, these distributional measures are often more predictive of human ratings than direct pose-space distances.
SRGR and SC provide useful semantic signals, but neither directly resolves the many-to-many relation between meaning and motion. SGP is designed as a complementary event-level measure rather than a replacement for all other metrics.
Participants complete only one condition, so motion-centric judgments can be separated from judgments that require listening to speech. All criteria use a seven-point scale, with higher ratings indicating better perceived quality.
The study compares the four generated systems with BEAT2 motion-capture ground truth. All conditions use the same avatar, camera, crop, renderer, and 10-second temporal window.
Windows are selected to contain many iconic, metaphoric, and deictic gestures. The selected stimuli contain 80 annotated semantic events: 44 iconic, 18 metaphoric, and 18 deictic.
Each study includes practice trials and 16 attention checks. Videos must be watched in full before ratings are submitted. Participants failing two or more checks are excluded.
Existing BEAT annotations specify semantic intervals and coarse categories, but do not fully describe how a gesture is physically realized or what it means in context. We enrich those annotations instead of replacing them.
For each semantic event, the rendered motion segment and target word are provided to Gemini 2.5 Pro. The model is instructed to analyze only the specified interval and ground its output in visible motion rather than assuming a semantic gesture is present.
SGP focuses on whether a semantic interval is preserved as a non-generic gesture rather than collapsing into a beat. For each event, Gemini classifies the observed gesture as iconic, metaphoric, deictic, beat, none, or uncertain.
SGP = 1 means none of the evaluated semantic events are classified as beats. The score is averaged at the clip level, so each clip contributes equally regardless of event count.
A randomly sampled subset of Gemini outputs is manually checked against the rendered motion. An annotation is counted as correct when its visible gesture description captures the principal motion and its semantic interpretation is compatible with the manually assessed communicative function. Under this criterion, 75% of the evaluated LLM-generated annotations are judged correct.
SGP is not semantic ground truth and does not verify exact meaning preservation or exact gesture category. It is specifically a measure of semantic-gesture preservation against beat-gesture collapse, designed to relax strict one-to-one pose matching.
The benchmark results show that distributional similarity, geometry, timing, physical plausibility, and semantic preservation do not improve together automatically. This is exactly why a complementary metric suite is needed.
| Method | FGD ↓ | BC ↑ | Div. ↑ | Dens. ↑ | Cov. ↑ | Dice ↑ | LDLJREL →0 | SRGR ↑ | PCK ↑ | MJD ↓ | Chamfer ↓ | Hausdorff ↓ | Foot Contact ↓ | SC ↑ | SGP ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EMAGE | 5.523 | 0.692 | 88 | 0.8972 | 0.07275 | 0.6407 | 0.0530 | 0.05494 | 0.3309 | 0.2410 | 0.01375 | 0.2908 | 0.5048 | 0.248 | 0.4372 |
| SemConFlow | 2.245 | 0.780 | 120 | 0.01559 | 0.06647 | 0.6793 | -0.4294 | 0.05868 | 0.3552 | 0.2007 | 0.01112 | 0.2530 | 0.5932 | 0.314 | 0.4205 |
| SemTalk | 4.266 | 0.727 | 116 | 0.1123 | 0.06673 | 0.6241 | -0.6020 | 0.05059 | 0.3037 | 0.2303 | 0.13976 | 0.4663 | 0.4040 | 0.268 | 0.4014 |
| GestureLSM | 4.268 | 0.525 | 112 | 0.009239 | 0.009817 | 0.6573 | 1.5386 | 0.04536 | 0.2715 | 0.2391 | 0.01797 | 0.2909 | 0.8883 | 0.248 | 0.2622 |
Achieves the strongest performance on most distributional, geometric, and speech-alignment metrics, including FGD, BC, Diversity, Dice, SRGR, PCK, MJD, Chamfer, Hausdorff, and SC.
Obtains the best Density and Coverage, has LDLJREL closest to zero, and achieves the highest SGP score, indicating fewer semantic intervals collapse into generic beats.
SemTalk achieves the best Foot Contact result, while GestureLSM remains competitive on diversity and geometric fidelity but does not lead an individual metric in the benchmark table.
Objective metrics are computed over 265 shared test sequences, except SGP, which is evaluated on the 15 perceptual-study sequences with Gemini-based semantic annotations.
BEAT2 reference motion receives the highest average rating on all five human-study dimensions. The gap is smaller for motion diversity, suggesting that current systems can approach reference-level variability more easily than they reproduce naturalness and speech-conditioned appropriateness.
Among generated methods, SemConFlow achieves the highest mean subjective score on all five perceptual dimensions.
In the muted study, SemTalk generally ranks second. In the audio study, EMAGE ranks second for speech timing and content match.
A model that looks comparatively good without audio does not necessarily keep the same advantage when timing and semantic correspondence are judged.
GestureLSM consistently receives the lowest mean ratings among the generated systems in the reported perceptual results.
The answer depends strongly on the perceptual target. No individual metric consistently explains all five human judgments, and Semantic Score (SC) shows no statistically meaningful association with any of the five subjective dimensions.
LDLJREL is the strongest correlate for both human-likeness (ρ = −0.61) and absence of animation errors (ρ = −0.62). Coverage and Density are most informative for motion diversity.
Coverage reaches ρ = 0.73 for speech timing and ρ = 0.70 for content match. Density and FGD show similarly strong associations with both speech-dependent targets.
MJD and Hausdorff show little correspondence with subjective dimensions, reinforcing that perceptually appropriate gestures need not reproduce the exact reference geometry.
BC is negatively correlated with all five perceptual dimensions, indicating that stronger beat alignment alone does not guarantee more natural or communicatively appropriate gestures.
SGP shows the intended selectivity toward speech-aware judgments, correlating with both speech timing and content match at approximately ρ = 0.39 while remaining weak or non-significant for muted visual dimensions. SC, in contrast, shows no statistically meaningful correlation with any of the five subjective dimensions.
For each perceptual target, a separate ordinary least-squares regressor combines 13 objective predictors. Chamfer Distance is excluded from the joint model because of its strong redundancy with Hausdorff Distance, while SC is retained as an independent semantics-aware predictor. Models are evaluated with leave-one-out cross-validation over the 60 generated model-sequence observations, with feature standardization performed inside each fold.
Separate regressors are trained for human-likeness, motion diversity, absence of animation errors, speech timing, and content match. This avoids assuming that the same metric mixture should explain every perceptual dimension.
Held-out predictions are compared with human ratings using Spearman correlation, mean squared error, and mean absolute error.
All five composites improve over their strongest individual metric. The largest gains occur for absence of animation errors and content match, while the benefits for motion diversity and speech timing are smaller, showing that metric combination remains strongly target-dependent.
| Target composite | MSE ↓ | MAE ↓ | Composite ρ ↑ | Best individual | Δ|ρ| ↑ |
|---|---|---|---|---|---|
| Human-likeness | 0.2846 | 0.4140 | 0.6568 | LDLJREL | +0.0468 |
| Motion diversity | 0.2035 | 0.3610 | 0.5375 | Coverage | +0.0117 |
| Absence of animation errors | 0.1560 | 0.2984 | 0.7065 | LDLJREL | +0.0819 |
| Speech timing | 0.2671 | 0.4349 | 0.7439 | Coverage | +0.0246 |
| Content match | 0.3665 | 0.4844 | 0.7824 | Coverage | +0.0805 |
Composite predictors are evaluated as dimension-specific automated proxies for human judgments, using out-of-fold predictions on the original seven-point rating scale rather than deriving an additional model-level ranking.
The human correlation analysis is based on 15 unique speech sequences across four generated systems, yielding 60 generated model-sequence observations rather than 60 fully independent utterances.
The selected windows deliberately contain many semantic gestures. This improves sensitivity to semantic differences but may overrepresent semantically dense speech relative to the broader BEAT2 distribution.
Listening environment, distraction, fatigue, and presentation-order effects cannot be fully controlled in an online perceptual study.
Gemini 2.5 Pro provides scalable enrichment, but its outputs should not be treated as semantic ground truth.
SGP measures preservation against beat-gesture collapse; it does not directly verify that the exact intended meaning or semantic gesture category is preserved.
The observed objective–subjective relationships are specific to the evaluated models, sequences, and metric implementations. Future work should extend the benchmark to broader datasets, models, and human studies.
Reliable progress in holistic co-speech gesture generation requires multiple perceptually validated metrics, not a single aggregate score.
Distributional similarity, geometric fidelity, temporal alignment, physical plausibility, and semantic preservation each capture a different slice of gesture quality. The benchmark therefore treats evaluation as a multidimensional problem: standardize the protocol, validate metrics against human perception, and use semantics-aware measures as complementary evidence rather than universal substitutes.