Benchmarking Federated Learning & Knowledge Distillation for Point Cloud Classification¶
Conference: ECCV 2026
Paper: ECCV official page · Paper PDF
Code: https://ezharjan.github.io/FLKD3DBenchmark/
Area: Model Compression
Keywords: Federated learning, knowledge distillation, point cloud classification, non-IID data, evaluation confounding
TL;DR¶
A benchmark of federated teachers and distillation objectives shows that labeled proxy data can produce a 92.94% student from an 8.50% teacher, so a compact student's high accuracy does not necessarily demonstrate successful transfer from federated training.
Background & Motivation¶
Deploying point cloud models across institutions and on edge devices creates two distinct constraints: training data may not be pooled, and the deployed network must be small. Federated learning addresses the first by coordinating local training and parameter aggregation; knowledge distillation addresses the second by training a smaller student from a larger teacher. Combining them does not automatically preserve either the original data-access restriction or the claim that useful knowledge came from the federated model. PointNet++ is also a meaningful stress case because local geometric features learned on heavily skewed client categories may become incompatible under aggregation.
An evaluation that reports only the final student's accuracy misses an important question: what other supervision did that student receive? If its server-side proxy set contains ground-truth labels, cross-entropy can teach classification directly, even when the federated teacher has largely failed. The missing contribution is therefore not another aggregation rule but an evaluation that distinguishes teacher information from proxy-label information. ModelNet40 supplies a relatively demanding multiclass task, while craniosynostosis head shapes provide a complementary setting with an almost saturated centralized reference.
The paper fixes the teacher and student architectures, varies federated algorithms and distillation objectives, and explicitly checks the hard-label coefficient in each objective. The resulting “recovery illusion” concerns attribution: the student's measured accuracy can be genuine while its interpretation as recovered federated knowledge is misleading. Core idea: report teacher quality alongside student performance and group distillation objectives by their use of hard labels to determine whether the student follows the teacher or relearns the centralized proxy labels.
Method¶
Overall Architecture¶
This is a benchmark and diagnostic study, not a new point cloud backbone or aggregation algorithm. Its inputs are labeled three-dimensional point sets, evaluated through centralized teacher training, standalone federated learning, standalone distillation, and a combined federated-learning-then-distillation setting. In the combined setting, federated training produces a PointNet++ SSG teacher; its best checkpoint is frozen and used to supervise a compact SmallPointNet2 student on the centralized training set. Evaluation retains teacher accuracy, student accuracy, and model size rather than reducing the entire pipeline to one student score.
Three choices organize the benchmark: a fixed extreme label-skew partition, a fixed teacher–student capacity difference, and a grouping of distillation objectives by hard-label weight. These control the federated difficulty, compression architecture, and source of supervision, respectively. At inference, only the student is needed to classify a point cloud; neither the teacher nor client collaboration is required. Because the contribution is experimental design rather than a new multicomponent network, a diagram of generic training stages would add little to the explanation.
Key Designs¶
1. Fixed label-skew partition: deliberately expose federated failure
ModelNet40 uses its standard preprocessed split, with 9,843 training shapes and 2,468 test shapes across 40 categories. The craniosynostosis collection contains a healthy control category and three pathology categories, with 100 shapes per category. A deterministic stratified 80/20 split gives 320 training shapes and 80 test shapes. Crucially, these are instances sampled from a patient-derived statistical shape model, not evidence of 400 independent patient scans. Each input contains 1,024 points with three spatial coordinates and no surface normals, fixing input capacity and excluding gains from additional attributes.
For federation, training examples are sorted by class and divided into five consecutive equal slices, one per client. Each client therefore sees a concentrated region of the label space rather than a representative sample of all categories. A slice boundary can split a category, so the construction should not be interpreted as guaranteeing that every class belongs to exactly one client. This is an intentionally severe stress test, not an estimate of typical institution-level heterogeneity. The three seeds vary initialization, minibatch order, and augmentation while holding the partition fixed; reported deviations therefore measure training randomness, not uncertainty across client partitions.
2. Fixed teacher–student capacity difference: separate federation from compression
The teacher is PointNet++ with single-scale grouping, three hierarchical set-abstraction levels, and a two-layer classification head. SmallPointNet2 retains the hierarchy but narrows the set-abstraction and classification layers, exposing a penultimate feature for representation-based distillation. On ModelNet40, parameter and buffer tensors occupy 5.65 MB for the teacher and 1.44 MB for the student, a 74.51% reduction. This reduction primarily comes from choosing a smaller architecture; it is not a compression ratio uniquely achieved by a particular loss. The loss determines how well that fixed compact network preserves accuracy, rather than dynamically designing its structure.
The federated comparison covers thirteen algorithms with different responses to heterogeneous updates. FedAvg uses size-weighted averaging; FedProx, SCAFFOLD, FedNova, and FedDyn introduce proximal regularization, control variates, normalized updates, and dynamic regularization, respectively. FedAvgM and the adaptive FedAdam, FedYogi, and FedAdagrad variants change the server update, while FedMedian uses coordinate-wise medians and FedBN retains local batch-normalization statistics. MOON introduces a model-contrastive objective, and Ditto learns personalized models alongside the global model. All are evaluated with the same teacher backbone, client count, and communication budget, making the current partition's aggregation failures visible. However, a personalized algorithm's global-teacher result should not be treated as a complete assessment of its client-specific deployment benefits.
3. Hard-label grouping: distinguish transfer from supervised retraining
The distillation objectives span output matching, feature and attention matching, relational and contrastive learning, and self-distillation. Vanilla KD matches temperature-scaled class distributions, Logit-MSE matches numerical outputs, Cosine compares output directions, and DKD decouples components of class knowledge. Feature KD, Attention Transfer, SP, and RKD use representations, attention, similarities, or sample relationships; CRD supplies a contrastive representation constraint. Self-distillation first trains a compact student and then uses it to supervise the final reported student. These names denote the implementations and coefficients used in this benchmark, not a guarantee that every implementation of the same method has identical supervision.
The central distinction is the source of the learning signal, not the method's name. Section 4.4 summarizes the relevant loss structure as:
The first term uses ground-truth proxy labels; the transfer term depends on teacher predictions or representations, and \(w_{\mathrm{CE}}\) controls direct hard-label supervision. Vanilla KD, Logit-MSE, Cosine, CRD, and self-distillation use a hard-label weight of 0.5, while DKD uses 1.0. Feature KD, Attention Transfer, SP, and RKD instead use \(1-\alpha-\beta\), which becomes exactly zero under the benchmark's \(\alpha=\beta=0.5\) configuration. Thus, the first group can learn directly from labels, whereas the second relies on the teacher in this setup; feature distillation is not inherently label-free in every formulation.
Stage two deliberately uses the full labeled centralized training set as proxy data to expose this alternative supervision path. When the teacher collapses, cross-entropy can still train the student to recognize classes; without that term, learning depends on the teacher's information. The authors consequently recommend controls that remove cross-entropy or use an unlabeled proxy set. These recommendations should not be mistaken for completed, matched coefficient ablations for every objective: the main evidence compares groups of different objective configurations. Removing labels also does not authorize centralizing otherwise private point clouds or establish a formal privacy guarantee.
Loss & Training¶
Federated training uses five clients, twenty communication rounds, and five local epochs per round, with Adam, learning rate 0.001, and batch size 24. The centralized teacher follows a 200-epoch protocol; the combined-stage diagram likewise specifies 200 distillation epochs, Adam, and learning rate 0.001. Soft targets use temperature \(T=2.0\), with the coefficients above determining whether hard-label supervision remains. Standalone FL accuracy is the best value within twenty rounds for each run, not the final-round value. ModelNet40 distillation and combined tables report peak accuracy during training; some combined runs stop early and are explicitly marked in the source, so their budgets are not interchangeable with completed runs.
The standardized multiseed evaluation comprises 504 training runs. It includes centralized and thirteen federated configurations on ModelNet40, plus centralized training, federated training, ten distillation objectives, and the complete thirteen-by-ten combined grid on the clinical collection, each over three seeds. The main-text ModelNet40 distillation and combined results come from a separate focused round using individual teacher checkpoints, not the three-seed means in Table 1. In particular, the combined table's SCAFFOLD teacher scores 8.50%, whereas its standardized standalone FL mean is 64.26%; these are different experimental statistics. References to a complete supplementary ModelNet40 grid are not fully aligned with the main-text coverage description, and that supplement is absent from the local cache, so this note relies only on verifiable main-text results.
Key Experimental Results¶
Main Results¶
The selected rows below come from Table 1: instance accuracy on the stated test splits, in percent, higher is better, reported as mean ± standard deviation over three seeds. These are standalone federated teachers, not distilled students; centralized and federated optimization schedules are also not identical.
| Training method | ModelNet40 accuracy ↑ | Craniosynostosis accuracy ↑ | Main observation |
|---|---|---|---|
| Centralized teacher | 92.26 ± 0.04 | 100.00 ± 0.00 | Non-federated reference |
| FedNova | 76.32 ± 1.55 | 50.83 ± 8.04 | Best on ModelNet40, but ranking changes |
| FedProx | 71.04 ± 0.87 | 75.83 ± 10.63 | Best on the clinical collection |
| FedAvg | 58.51 ± 0.25 | 69.58 ± 2.60 | Basic weighted averaging |
| SCAFFOLD | 64.26 ± 3.75 | 36.67 ± 1.44 | Not the collapsed focused-round checkpoint |
| FedAvgM | 4.05 ± 0.00 | 25.00 ± 0.00 | Very low accuracy in this configuration |
The best federated results trail centralized training by 15.94 percentage points on ModelNet40 and 24.17 points on the clinical collection. In the separate standalone distillation round, Table 2 reports a 92.44% centralized teacher, a 92.74% Attention Transfer student, and an 87.07% Logit-MSE student. A shared compact architecture therefore does not ensure that every objective succeeds, and model-size reduction alone cannot establish effective transfer.
Ablation Study¶
The main diagnostic is an objective-group analysis rather than a strict one-variable ablation for every loss. The selected Table 4 rows below use the clinical test set and the complete thirteen-teacher-by-ten-objective grid, with three seeds per configuration; accuracy is in percent and higher is better. The chance-level teacher is FedAvgM at 25.00%, and the best teacher is FedProx at 75.83%. Here \(r\) is the Pearson correlation, across thirteen teachers, between mean student accuracy and standalone teacher accuracy; it measures dependence, not an objective that should always be maximized.
| Distillation objective | Hard-label weight | Student with chance-level teacher ↑ | Student with best teacher ↑ | Teacher–student correlation \(r\) |
|---|---|---|---|---|
| Vanilla KD | 0.5 | 97.5 | 99.6 | +0.37 |
| CRD | 0.5 | 100.0 | 99.6 | −0.53 |
| Feature KD | 0 | 27.1 | 84.2 | +0.99 |
| Attention Transfer | 0 | 25.0 | 83.3 | +0.99 |
| SP | 0 | 25.0 | 82.9 | +0.99 |
Table 4 rounds correlations to two decimals; Section 4.5 gives the finer range of 0.986–0.990 for the four pure-transfer objectives. In the ModelNet40 focused round, Table 3 pairs an 8.50% SCAFFOLD teacher with a 92.94% Logit-MSE student: an 84.44-point difference, rounded to 84.4 in the paper. Feature KD reaches only 16.99% with that same teacher; neither of these two student cells is marked as stopped early, but their difference cannot be assigned solely to one coefficient because their transfer objectives also differ.
Key Findings¶
- High student accuracy is not synonymous with a strong teacher: labeled objectives can relearn the proxy labels after teacher collapse.
- Federated algorithm rankings change across datasets; local wins by FedNova or FedProx do not establish universal robustness.
- Pure-transfer objectives expose teacher quality more clearly, but student accuracy need not equal or remain below teacher accuracy: Feature KD reaches 84.2% behind the 75.83% teacher.
- Compression and communication are separate budgets: a smaller student does not eliminate the earlier cost of transmitting the full teacher.
Highlights & Insights¶
- Audit supervision sources, not just objective names. Checking which loss terms access real labels can reveal attribution errors that a final leaderboard hides.
- Use teachers with a broad quality range. Strong teachers place many losses near the ceiling, whereas collapsed checkpoints make dependence on labels much easier to observe.
- Separate utility from evidence of transfer. A successful student may be useful for deployment without demonstrating that federated representations were recovered; the diagnostic concerns the cause of success.
Limitations & Future Work¶
- Author-reported: Two datasets, one extreme label-skew regime, and a fixed PointNet++ teacher–student pair leave milder heterogeneity, larger scanned collections, and other backbones untested.
- Author-reported: Seeds do not vary the partition; clients remain fixed at five, communication at twenty rounds, and client architectures are homogeneous. The results do not invalidate all federated settings.
- Reader assessment: The clinical collection contains statistical-shape-model instances, a small test set, and a saturated centralized reference, so it does not establish real multicenter clinical generalization.
- Reader assessment: The main text lacks a matched hard-label-only student baseline for every combined setting and a complete same-objective cross-entropy on/off table. Label masking is persuasive, but stronger controls are needed to isolate each loss's causal contribution.
- Evidence boundary: The cache contains the complete main Method and Experiments but not the cited Appendices A–D. Per-cell deviations, detailed layer widths, latency measurement conditions, and supplementary grids cannot be independently verified; roughly twofold inference acceleration remains an author-reported claim.
- Evaluation priority: Add paired no-cross-entropy and unlabeled-proxy controls, and disclose proxy-data provenance and authorization. Removing labels alone does not resolve privacy requirements.
Related Work & Insights¶
- Versus FedAvg, FedProx, and FedNova: The paper benchmarks existing aggregation mechanisms with a shared point cloud backbone and label-skew partition rather than proposing another aggregator, revealing unstable cross-task rankings.
- Versus Vanilla KD, FitNets, and Attention Transfer: The contribution is not a new soft-target or feature-matching mechanism, but an account of how implementation-level hard-label coefficients change what evaluation measures.
- Versus unlabeled-proxy distillation: Such approaches better isolate teacher contributions, while proxy privacy, coverage, and distribution shift still require separate assessment.
Rating¶
- Novelty: 3/5 — Existing algorithms; the contribution lies in crossed evaluation and supervision-source diagnosis.
- Experimental Thoroughness: 3/5 — The complete clinical multiseed grid is useful, but the fixed backbone, extreme partition, and focused ModelNet40 round limit generalization.
- Writing Quality: 3/5 — The central phenomenon is clear, but dataset provenance and different experimental reporting regimes need careful reading.
- Value: 4/5 — Supplies a practical audit question for evaluations that connect federated training to labeled distillation.