DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning

Conference: ECCV2026
Paper: ECCV paper
Area: Audio & Speech; Multimodal VLM
Keywords: Deception detection, structured reports, multicultural data, feature recalibration, modality consistency distillation

TL;DR

DecepGPT combines structured audiovisual report supervision with SICS and DMC to improve benchmark deception classification, achieving 73.23% accuracy on DOLOs, but these results do not establish reliable real-world lie detection.

Background & Motivation

Audiovisual deception detection typically maps a video clip and its audio to a truthful or deceptive label. Early methods used handcrafted descriptions of expressions, posture, and prosody; later models learned classification through visual encoders and cross-modal fusion. The difficulty is not limited to accuracy: a binary output does not reveal what the model observed or where a reviewer should check its evidence. Replacing the label with a long explanation is insufficient because a large model can invent expressions, acoustic attributes, or causal relationships in fluent prose. The paper therefore seeks to make observations, cross-modal explanations, and final labels separately inspectable, rather than treating explanation length as credibility.

Data scarcity compounds this problem: BoL and MU3D contain only 325 and 320 samples, respectively, and DOLOs contains 1,675. Training can exploit participant style, recording conditions, or correlations in a dominant modality instead of evidence that transfers across settings. Existing datasets concern image narration, social evaluation, or fabricated stories, making task differences difficult to disentangle from cultural differences. T4-Deception consequently uses professional-identity pretense in four national versions of the same television format to keep the task relatively consistent while testing transfer. Here, culture is operationalized through national program versions, which also vary in language, production, and participant composition; it is not a causally isolated factor.

On the modeling side, the authors build on AffectGPT, retaining pretrained audiovisual capabilities while adapting projectors, the language model, and new components. SICS recalibrates fused features using a shared prior and sample-specific residuals, while DMC trains both unimodal branches to match the distribution of the complete audiovisual judgment. Structured reports expose content for inspection; representation learning and distillation address small-data optimization, so these components solve different problems. Even an accurately observed pause or expression is a contextual behavioral description, not independent proof that a speaker is lying. Core Idea: replace an opaque audiovisual classification output with a field-by-field auditable report, while using stabilized fusion and modality consistency distillation for small-data learning, without equating explanatory text with factual evidence.

Method

Overall Architecture

The inputs are synchronized video and audio from a clip; the output is a single-line report with fixed field order and semicolon delimiters: Video Cues; Audio Cues; Reasoning; Prediction. Before training, structured supervision and multicultural data supply report targets and new evaluation settings; during training, frozen encoders extract modality tokens, cross-attention produces fusion tokens, and SICS recalibrates them for the report generator. DMC obtains auxiliary predictions from unimodal tokens and applies a training-only consistency constraint using the multimodal distribution at the final Prediction position as its teacher. The assistants used for dataset construction are therefore not agents called sequentially at deployment, and the DMC auxiliary branches are not retained at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    T["Training annotations<br/>and four-country videos"] --> S["Structured Supervision<br/>and Multicultural Data"]
    I["Input video and audio"] --> E["Frozen encoders<br/>and cross-modal fusion"]
    E --> F["SICS: Stabilized Recalibration"]
    F --> R["Generate four-field report<br/>with final prediction distribution"]
    S -.->|Training report supervision| R
    E -.->|Unimodal tokens| D["DMC: Modality Consistency Distillation"]
    R -.->|Training teacher distribution| D

Key Designs

1. Structured Supervision and Multicultural Data: define inspectable content before expanding task coverage

The report separates video cues from audio cues, asks the reasoning field to integrate those observations, and finally produces a binary label. This ordering gives reviewers an interface for checking whether a cue exists, whether its attributes are described correctly, and whether the conclusion goes beyond the observations. It does not structurally guarantee valid reasoning, nor does it establish that the generation order matches the causal order of the model's internal decision. The training target is the complete corrected report text, rather than only a label; inference does not query an external fact repository. This distinction matters: auditability comes from identifiable fields, whereas factual correctness still requires checking the original video and audio.

During dataset construction, a Qwen-Omni audio assistant extracts prosody and speech patterns, and a GPT-4o video assistant extracts facial dynamics and body behavior. A separate GPT-4o reasoning assistant combines these cues; human annotators correct hallucinations and logical problems in the outputs. A GPT-4o augmentation assistant then paraphrases the text, followed by checks for cue-reasoning contradictions, schema compliance, and near-duplicates, and a final human review. Figure 2 labels the similarity stage as CLIP-similarity, but the main text does not supply its threshold or complete implementation, so a specific deduplication algorithm cannot be asserted. Two annotators completed three pre-annotation calibration rounds, with Cohen's \(\kappa\) of 0.73 on discrete cue and reasoning audit labels. This measures annotation agreement, not prediction accuracy, and does not establish the validity of the audited behaviors as lie-detection cues.

T4-Deception draws from U.S., German, Vietnamese, and Bulgarian versions of “To Tell the Truth,” with professional-identity pretense as the shared task. Table 1 lists 1,695 clips and 1,695 participants, comprising 1,130 deceptive and 565 truthful samples. The 565 identities in Figure 3 refer to shared claimed identities: each is represented by one truthful participant and two impostors, not a total of only 565 participants. The national subsets contain 876, 702, 66, and 51 clips, respectively, and average clip duration is 3.65 seconds, emphasizing immediate interaction rather than complete narratives. The common program format reduces task variation, but performance incentives, editing, and short-clip selection remain far removed from real investigations. The authors describe it as the largest non-laboratory deception dataset at the time; this is a field-specific comparison, not a claim about general truthfulness assessment datasets.

2. SICS: Stabilized Recalibration

Stabilized Individuality-Commonality Synergy first temporally averages the cross-attention fusion tokens to obtain clip-level context. A two-layer fully connected network with an intermediate tanh generates a sample-adaptive residual, from which a gating coefficient is predicted. The gate combines a learnable global vector with the current residual, allowing both a shared reference and a sample-specific adjustment to influence feature recalibration. The global vector is a learned parameter, not a previously validated universal deception rule; the residual is not an identified psychological state of an individual. This is an inductive bias at the representation level: with little data, the model need not learn a completely independent modulation rule for every sample.

Independent parameters then generate positive and negative adjustment weights, which pass through ReLU and act on the input features before the negative branch is subtracted. The polarity adjustment explicitly preserved on page 7 is:

\[ x'_i=x_i\odot\operatorname{ReLU}(w_i^+)-x_i\odot\operatorname{ReLU}(w_i^-). \]

\(x_i\) denotes fusion tokens, \(w_i^+\) and \(w_i^-\) are feature-wise adjustment weights, and \(\odot\) denotes element-wise multiplication. The adjusted features are subsequently combined with the original input; the authors set the balancing hyperparameter to \(\lambda=0.2\). Equations (2) and (3) contain missing symbols in the cached extraction, preventing exact verification of the gated mixture and final residual mixture, so their precise formulas are not reconstructed here. The clearly described mechanism is more than single-branch attention weighting: it introduces a shared reference, sample adaptation, and signed recalibration. However, numerically smoother features do not automatically imply the removal of cultural bias or identity shortcuts; that requires independent experiments.

3. DMC: Modality Consistency Distillation

Distilled Modality Consistency attaches a modality-specific projector to each frozen visual or audio token stream and uses a shared distillation head to produce truthful/deceptive distributions. The paper calls these components Causal Projectors; that name does not demonstrate causal identification or confounder removal. The teacher distribution comes from the language decoder at the final Prediction position, conditioned on the complete audiovisual context and required report schema. Training constrains each unimodal distribution through KL divergence in the teacher-to-student direction, giving the unimodal projectors a learning signal tied to the complete judgment. Intuitively, even when the task can be fitted mostly through vision, the audio branch must try to match the teacher rather than remaining uninformative.

This is neither forced equality of modality features nor random removal of an input branch: consistency is imposed on prediction distributions. The auxiliary branches share a distillation head but retain separate projectors to accommodate different visual and acoustic inputs. The teacher is part of the current model, not an independent expert with additional ground-truth knowledge, so its errors may also propagate. The main text does not fully specify teacher stop-gradient handling, temperature, or label-token normalization; reproducing these details requires checking the implementation. At inference time, the auxiliary components are discarded, leaving audiovisual encoding, fusion, SICS, and report decoding.

Loss & Training

The overall objective combines token-level report cross-entropy with DMC consistency distillation, with distillation weight \(\alpha=0.1\). The report's final field must match the ground-truth sample label, while the other fields learn the human-corrected observations and explanations. Equations (5) and (8) also suffer from extraction damage; this description follows the prose rather than presenting guessed formulas as original equations. The model uses pretrained AffectGPT weights, freezes audiovisual encoders, applies LoRA to the LLM, and fully trains projectors and new components. Training uses AdamW with learning rate \(5\times10^{-5}\) for 200 epochs and batch size 4. The hardware is one NVIDIA H100 80GB, with 8 frames sampled per video at \(224\times224\) resolution. Freezing encoders limits the updated parameter space, but 200 epochs of LoRA training still require careful splits and model selection; parameter efficiency alone does not establish resistance to overfitting.

Key Experimental Results

Main Results

The following selection comes from Table 2 on page 10; Acc. and F1 use the table's percentage scale, with higher values better. BoL and DOLOs follow official 3-fold protocols, while MU3D uses 4-fold evaluation; the provided main text does not specify F1 averaging. GPT-4o is evaluated zero-shot with visual input only; the other models below use audiovisual inputs, and training conditions differ.

Method and setting	DOLOs Acc.	DOLOs F1	BoL Acc.	BoL F1	MU3D Acc.	MU3D F1
PECL, full training	64.75	71.20	59.51	51.06	55.31	60.07
GPT-4o, zero-shot, visual only	66.38	64.21	57.14	56.35	54.12	52.47
VideoLLaMA2, LoRA	53.48	56.12	47.65	49.34	51.84	54.62
DecepGPT, LoRA	73.23	76.13	63.46	63.72	61.25	67.22

In the complete DOLOs comparison in Table 2, AFFAKT achieves 68.10% accuracy, placing DecepGPT ahead by 5.13 percentage points. This supports a classification improvement under that protocol, not the claim that better explanations imply more truthful behavioral inferences.

Ablation Study

The following reproduces Table 6 on page 13 under the corresponding in-domain protocols; Base omits both SICS and DMC.

Config	DOLOs Acc.	DOLOs F1	BoL Acc.	BoL F1	MU3D Acc.	MU3D F1
Base	67.24	71.18	57.69	54.26	53.75	65.34
Base + DMC	68.77	71.94	59.62	58.15	55.00	66.27
Base + SICS	72.68	76.02	62.50	61.43	60.15	67.08
Full, SICS + DMC	73.23	76.13	63.46	63.72	61.25	67.22

Adding SICS alone to Base increases accuracy by 5.44, 4.81, and 6.40 percentage points, respectively, larger than the standalone gains from DMC. Adding DMC on top of SICS yields actual incremental gains of 0.55, 0.96, and 1.10 percentage points. The page 12 prose calls 1.53, 1.93, and 1.25 the “further” DMC gains, but these are Base + DMC minus Base, not the increments on top of SICS.

Key Findings

The following selects entries from Table 1 on page 5 and Table 5 on page 12; in-cultural evaluation uses 3-fold validation, and cross-cultural rows use source-culture training without target-domain fine-tuning.

Training edition → test edition	Total clips in test edition	Acc.	F1
U.S. → U.S.	876	64.65	73.28
Germany → Germany	702	61.11	53.33
Vietnam → Vietnam	66	63.10	68.42
Bulgaria → Bulgaria	51	61.46	66.25
U.S. → Germany	702	59.50	52.14
U.S. → Bulgaria	51	63.00	71.62

Clip counts describe each complete national subset, not the test size of an individual cross-validation fold. Mean in-cultural accuracy is 62.58%, and the mean across 12 cross-cultural directions is 57.69%; the difference of 4.89 is in percentage points, although the prose calls it a relative degradation. Each edition has a 2:1 deceptive/truthful ratio; if test sets preserve that ratio, always predicting deceptive gives 66.67% accuracy, above all four in-cultural accuracies. This is a reader-derived baseline from Table 1 proportions, not an experiment reported by the authors; missing per-fold distributions, balanced accuracy, and majority-class comparisons prevent a claim of reliable cross-cultural discrimination. Table 4 on page 11 also reports an F1 of 61.79 for MU3D + BoL → DOLOs, below PECL's 69.55 despite higher accuracy, so improvements do not cover every metric.

Highlights & Insights

• Separate report fields make errors localizable to observations, attribute descriptions, or inference; their primary benefit is more specific review, not automatic trustworthiness.
• SICS and DMC operate on fused features and prediction distributions, respectively; ablations support complementary gains, although DMC's incremental gain is much smaller than its standalone gain.
• A shared television task is a reusable strategy for reducing task confounding across datasets, but national editions still do not isolate culture alone.

Limitations & Future Work

• Laboratory and entertainment data differ from investigations in incentives, pressure, label meaning, identity performance, and clip selection; these results cannot establish whether a real person is lying.
• Small national subsets, class imbalance, and the majority-baseline issue limit T4 conclusions; balanced accuracy, subgroup false-positive rates, confidence intervals, and calibration should be added.
• The main text refers collection and label-verification details to Appendix 1, but the supplied cache contains only the main paper and references, preventing verification of program-, identity-, and augmentation-level split isolation.
• Page 15 describes human auditing of 100 randomly sampled outputs; exact category proportions in Figure 7 are not clearly available in the cache, so no hallucination rate is supplied here.
• Text similarity and GPT-Score in Table 3 cannot independently verify that audiovisual cues exist, much less establish a causal relationship with deception; future evaluation should separate observation accuracy from decision validity.

Related Work & Insights

• AffectGPT: supplies the explainable audiovisual emotion-understanding backbone; this work adds deception-report supervision, fusion recalibration, and consistency distillation rather than training a general multimodal model from scratch.
• PECL / DOLOs: represent parameter-efficient cross-modal classification and story-fabrication data; this paper adds structured reports and professional-identity pretense, but cross-domain F1 is not uniformly better.
• ModDrop and modality reweighting: address imbalance through simulated missing inputs or optimization weights; DMC instead supervises branches with complete audiovisual predictions and remains vulnerable to teacher errors.
• Transferable insight: other audiovisual behavior tasks can separately audit whether observations are correct and whether conclusions follow; independent evidence should test research hypotheses, rather than relying on report wording.

Rating

• Novelty: 3/5. The supervision, adapter, and distillation combination is task-specific, and multicultural data provide a new evaluation setting.
• Experimental Thoroughness: 3/5. Multiple datasets and ablations are included, but class baselines, split details, and uncertainty reporting remain insufficient.
• Writing Quality: 3/5. The narrative is clear, but some gain descriptions conflict with tables; damaged formula extraction separately limits exact restatement.
• Value: 3/5. Useful for research on auditable audiovisual classification, not as deployment evidence for real-world lie detection or high-stakes judgments about individuals.