Not All Attacks are Learned Equally

Abstract

Speech deepfake detection (SDD) models are trained on multi-attack datasets containing diverse spoofing systems, such as text-to-speech (TTS) and voice conversion (VC). In standard multi-attack training, all attacks are treated as one spoof class, and performance is reported using overall Equal Error Rate (EER). This aggregate view obscures how individual attacks shape learning and generalization. To better understand this attack-level behavior, we first balance TTS and VC exposure using sample and attack omission. We then measure attack-wise EER at inference and analyze attack-wise training loss and entropy-loss to characterize optimization. Results show that attacks contribute unequally: some attacks have high EER sensitivity and concentrated entropy with low loss, indicating strong influence on the decision boundary. We define these as high-impact attacks. To reduce uneven generalization across attacks, we propose a replay-regularized, attack-aware curriculum that stages exposure based on measured attack influence. Experiments on ASVspoof 2019, 2021, ASVspoof 5, and Fake-or-Real show improved overall robustness and reduced attack-level imbalance compared with standard multi-attack training.

Curriculum Training

Using the high-impact and low-impact partition identified above, we design a two-stage curriculum that stages attack exposure based on measured influence. Step 1 pretrains on low-impact attacks to build representations for harder-to-learn attack types. Step 2 adapts to high-impact attacks using a composite loss combining high-impact classification, replay regularization over a low-impact subset, knowledge distillation from a frozen teacher, and a contrastive objective. Code and pretrained models are available at github.com/attackbias-slt/Paper_Codes ↗.

Step 1: Low-Impact Pretraining L_step1 = L_CE(𝒟_low)

Step 2: High-Impact Adaptation with Replay L_step2 = αL_high + α_rL_rep + βL_KD + γL_con

Figure 2. Two-stage curriculum training framework with frozen teacher, knowledge distillation, and replay regularization.

Results & Analysis

We report Equal Error Rate (EER %, ↓) for standard multi-attack training and impact-driven curriculum variants trained on ASVspoof 2019 LA.

EER Comparison Across Retraining Classes

Overall, VC, and TTS EER across leave-two-attacks-out conditions for all three models trained on ASVspoof 2019.

SSL-XLSR

RawNet2

AASIST

Attack-wise Diagnostics

Per-attack loss and entropy-loss across leave-two-attacks-out training conditions.

Attacks included: A1, A2, A3, A4, A5, A6

High-Impact vs. Low-Impact Attacks

We use two diagnostics to characterize attack-level influence. Leave-two-attacks-out EER sensitivity measures how much overall EER changes when an attack pair is withheld from training. Entropy-loss measures output uncertainty per attack type at inference. Together, they dissociate training influence from representational confidence, exposing asymmetry that aggregate EER alone cannot reveal.

High-Impact Attacks

High EER sensitivity: omitting these causes large EER spikes, indicating strong dependence on their gradient signal

Concentrated entropy: confident, peaked outputs indicate well-learned discriminative representations

Low-Impact Attacks

Low EER sensitivity: omitting these has minimal effect on overall EER despite their presence in training

Diffused entropy: uncertain, spread outputs signal systematic underrepresentation in learned features

Curriculum Training Results

EER (%, ↓) across ASVspoof 2019, 2021, ASVspoof 5, and Fake-or-Real. Impact-driven curriculum variants compared against the SSL-XLSR baseline.

Framework	High-impact	Selection Metric	ASV19			ASV21			ASV5			FoR
Framework	High-impact	Selection Metric	Overall	VC	TTS	Overall	VC	TTS	Overall	VC	TTS	Overall
SSL-XLSR [14]	All	—	0.56	0.75	0.55	3.04	4.06	2.92	25.43	24.87	31.80	42.07
SSL-XLSR w/ Curriculum	A4	Both	0.51	0.65	0.54	1.68	2.33	1.62	27.00	27.55	33.66	36.50
SSL-XLSR w/ Curriculum	A1+A4	EER-sensitivity	0.60	0.82	0.58	1.48	2.01	1.45	21.04	19.69	27.82	25.97
SSL-XLSR w/ Curriculum	A3+A4	Entropy-loss	0.40	0.50	0.41	2.09	3.11	1.90	21.39	20.60	26.00	31.89

↓ Lower EER is better. Highlighted values are best per column. Baseline row dimmed for contrast.

Key Findings

Attack-Level Asymmetry

EER sensitivity and entropy-loss analysis reveal that attacks contribute unequally. Some dominate the decision boundary while others are systematically underlearned despite co-training.

Analysis Drives Framework

The high-impact and low-impact partition identified through diagnostics directly determines the curriculum structure. This is not post-hoc analysis but the basis of the training design.

Balanced Learning

Curriculum training reduces unevenness in how attacks are learned across VC and TTS types. Overall EER is reduced, and the gap between VC and TTS EER narrows, a consistent byproduct of more balanced learning.

Better Generalization

When learning is more balanced across attack types, generalization tends to improve. This is a common characteristic observed across datasets, not an absolute guarantee.

Not All Attacks are Learned
Equally in Speech Deepfake Detection