Knowledge Distillation for Enhancing Interpretability and Efficiency in Complex Machine Learning Models
Article Sidebar
Explainable AI,
Knowledge Distillation,
Machine Learning,
Model Interpretability
Main Article Content
Jaesik Jeong
Kit Ling Chan
Mageswaran Sanmugam
Abstract
Background: Complex machine learning (ML) systems often require substantial computational resources, making them difficult to deploy in real-world environments constrained by hardware limitations, interpretability requirements, and regulatory standards. While knowledge distillation (KD) has traditionally been viewed as a model compression technique, its broader implications for efficiency, interpretability, and regulatory compliance remain underexplored.
Aims: This study aims to reconceptualize knowledge distillation beyond model compression by framing it as a dual strategy for efficiency and interpretability enhancement. The paper proposes a structured distillation protocol that integrates predictive performance assessment, computational profiling, and feature attribution alignment within a unified experimental design.
Methods: The proposed distillation protocol employs a temperature-scaled objective function combining supervised cross-entropy loss and Kullback Leibler divergence to facilitate relational knowledge transfer from teacher to student models. Experiments were conducted across multiple benchmark datasets. Evaluation consisted of three components: (1) predictive performance measurement, (2) computational efficiency profiling including parameter counts and inference latency, and (3) interpretability analysis using feature attribution similarity and perturbation stability metrics. Statistical analyses were performed to assess performance differences.
Result: Across benchmark datasets, distilled student models achieved teacher-level accuracy ranging between 95% and 98%. Parameter counts and inference latency were reduced by more than 60%. Interpretability analyses showed improved explanation consistency, smoother decision structures, and higher feature attribution alignment. Statistical testing confirmed that efficiency and interpretability gains were obtained without significant performance degradation.
Conclusion: The findings support the reconceptualization of knowledge distillation as a dual optimization strategy that enhances both operational efficiency and interpretability while preserving predictive strength. Rather than serving solely as a compression mechanism, KD functions as a scalable and adaptive framework for deployment-ready AI systems that balance performance, computational constraints, and explanation stability.
Article Details
Copyright (c) 2026 Jaesik Jeong, Kit Ling Chan, Mageswaran Sanmugam
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
References
Balakrishnan, V., Shi, Z., Law, C. L., Lim, R., Teh, L. L., Fan, Y., & Periasamy, J. (2022). A Comprehensive Analysis of Transformer-Deep Neural Network Models in Twitter Disaster Detection. Mathematics, 10(24), 1–14. https://doi.org/10.3390/math10244664
Bjerring, J. C., Jakob, B., & Lauritz, M. (2025). Deep learning models and the limits of explainable artificial intelligence. Asian Journal of Philosophy, 4(1), 1–26. https://doi.org/10.1007/s44204-024-00238-8
Bruggeman, F. J., Teusink, B., & Steuer, R. (2023). Trade-offs between the instantaneous growth rate and long-term fitness : Consequences for microbial physiology and predictive computational models. BioEssays, 45(10), 1–20. https://doi.org/10.1002/bies.202300015
Demircioğlu, A. (2025). Reproducibility and interpretability in radiomics: a critical assessment. Artificial Intelligence and Informatics, 31(4), 321–328. https://doi.org/10.4274/dir.2024.242719
Ezzahra, Z. F., Sana, A., Sara, Q., & Said, R. (2025). Multi-objective reinforcement learning for recommender systems: a comprehensive survey of methods, challenges, and future directions. International Journal of Multimedia Information Retrieval, 14(33). https://doi.org/10.1007/s10735-025-00383-7
Gunasekara, S., & Saarela, M. (2025). applied sciences Explainable AI in Education : Techniques and Qualitative Assessment. Applied Sciences, 15(3), 1239. https://doi.org/10.3390/app15031239
Hartmann, J., Heitmann, M., Siebert, C., & Schamp, C. (2023). International Journal of Research in Marketing More than a Feeling : Accuracy and Application of Sentiment Analysis. International Journal of Research in Marketing, 40(1), 75–87. https://doi.org/10.1016/j.ijresmar.2022.05.005
Hohman, F., Kery, M. B., Ren, D., & Moritz, D. (2026). Model Compression in Practice : Lessons Learned from Practitioners Creating On-device Machine Learning Experiences Model Compression in Practice : Lessons Learned from Practitioners Creating On-device Machine Learning Experiences. CHI ’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 645. https://doi.org/10.1145/3613904.3642109
Huang, J., Mishra, A., Kwon, B. C., & Bryan, C. (2022). ConceptExplainer : Interactive Explanation for Deep Neural Networks from a Concept Perspective. IEEE Transactions on Visualization and Computer Graphics, 29(1), 831–841. https://doi.org/10.1109/TVCG.2022.3209384
Kucklick, J., & Muller, O. (2026). Tackling the Accuracy-Interpretability Trade-off : Interpretable Deep Learning Models for Satellite Image- based Real Estate Appraisal Tackling the Accuracy-Interpretability Trade-off : Interpretable Deep Learning Models for Satellite Image-based Real Est. ACM Transactions on Management Information Systems, 14(1), 1–24. https://doi.org/10.1145/3567430
Lan, W., Cheung, Y., Xu, Q., Liu, B., Hu, Z., & Li, M. (2025). Improve Knowledge Distillation via Label Revision and Data Selection. IEEE Transactions on Cognitive and Developmental Systems, 17(6), 1377–1388. https://doi.org/10.1109/TCDS.2025.3559881
Li, C., Cheng, G., & Han, J. (2024). Boosting Knowledge Distillation via Intra-Class Logit Distribution Smoothing. IEEE Transactions on Circuits and Systems for Video Technology, 34(6), 4190–4201. https://doi.org/10.1109/TCSVT.2023.3327113
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022). Interpretable Deep Learning: Interpretations, Interpretability, Trustworthiness, and Beyond. Knowledge and Information Systems, 64(12), 3197–3234. https://doi.org/10.1007/s10115-022-01756-8
Lin, S., Lin, W., Wu, W., Chen, H., & Chen, C. L. P. (2026). SparseTSF: Lightweight and Robust Time Series Forecasting via Sparse Modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(1), 170–183. https://doi.org/10.1109/TPAMI.2025.3602445
Liu, C., Yin, H., & Wang, X. (2025). Theoretical Perspectives on Knowledge Distillation : A Review. Wiley Interdisciplinary Reviews: Computational Statistics, 17(4), 1–17. https://doi.org/10.1002/wics.70049
Mai, N. T., Cao, W., & Liu, W. (2025). Interpretable Knowledge Tracing via Transformer-Bayesian Hybrid Networks : Learning Temporal Dependencies and Causal Structures in Educational Data. Applied Sciences, 15(17), 1–26. https://doi.org/10.3390/app15179605
Malihi, L., & Heidemann, G. (2024). Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression. Applied Sciences Innovation, 7(4), 56. https://doi.org/10.3390/asi7040056
Mandler, H., & Weigand, B. (2026). A review and benchmark of feature importance methods for neural networks. ACM Computing Surveys, 56(12), 1–30. https://doi.org/10.1145/3679012
Meng, C., Trinh, L., Xu, N., Enouen, J., & Liu, Y. (2022). Interpretability and fairness evaluation of deep learning models on MIMIC ‑ IV dataset. Scientific Reports, 12(7166), 1–28. https://doi.org/10.1038/s41598-022-11012-2
Panigrahi, B., Razavi, S., Doig, L. E., Cordell, B., Gupta, H. V., & Liber, K. (2025). On Robustness of the Explanatory Power of Machine Learning Models : Insights From a New Explainable AI Approach Using Sensitivity Analysis. Water Resources Research, 61(3), 1–23. https://doi.org/10.1029/2024WR037398
Rahman, A., Debnath, T., Kundu, D., & Khan, S. I. (2024). Machine learning and deep learning-based approach in smart healthcare : Recent advances , applications , challenges and opportunities. AIMS Public Health, 11(1), 58–109. https://doi.org/10.3934/publichealth.2024004
Salih, A. M., Raisi-estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025). A Perspective on Explainable Arti fi cial Intelligence Methods : SHAP and LIME. Advanced Intelligent Systems, 7(1), 1–8. https://doi.org/10.1002/aisy.202400304
Sonrel, A., Luetge, A., Soneson, C., Mallona, I., Germain, P. L., Knyazev, S., Gilis, J., Gerber, R., Seurinck, R., Paul, D., Sonder, E., Crowell, H. L., Fanaswala, I., Ajami, A. Al, Heidari, E., Schmeing, S., Milosavljevic, S., Saeys, Y., Mangul, S., & Robinson, M. D. (2023). Meta ‑ analysis of ( single ‑ cell method ) benchmarks reveals the need for extensibility and interoperability. Genome Biology, 24(119), 1–11. https://doi.org/10.1186/s13059-023-02962-5
Tan, C., & Liu, J. (2022). Improving Knowledge Distillation With a Customized Teacher. IEEE Transactions on Neural Networks and Learning Systems, 35(2), 1–10. https://doi.org/10.1109/TNNLS.2022.3189680
Wu, C.-J., Acun, B., Raghavendra, R., & Hazelwood, K. (2024). Beyond Efficiency: Scaling AI Sustainably. IEEE Micro, 44(5), 37–46. https://doi.org/10.1109/MM.2024.3409275
Zhang, L., & Ma, K. (2023). Structured Knowledge Distillation for Accurate and Efficient Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12), 15706–15724. https://doi.org/10.1109/TPAMI.2023.3300470
Zhang, Y., Hu, S., Zhang, L. Y., Shi, J., Li, M., & Liu, X. (2024). Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability. Proceedings - IEEE Symposium on Security and Privacy. https://doi.org/10.1109/SP54263.2024.00010