Logo image
Back
Task-Specific Knowledge Distillation for Scalable Sentiment Classification in Low-Resource Settings
Conference proceeding   Peer reviewed

Task-Specific Knowledge Distillation for Scalable Sentiment Classification in Low-Resource Settings

Gobihanath Balasubramaniam, Vadivel Abishethvarman, Banage T.G.S. Kumara, Senthan Prasanth and Banujan Kuhaneswaran
Data Science and Artificial Intelligence, Vol.2726, pp.270-283
Third International Conference on Data Science and Artificial Intelligence, DSAI 2025, 3rd (Sri Lanka, 19/11/2021–21/11/2021)
11/11/2025
DOI: https://doi.org/10.1007/978-981-95-4409-7_19
Appears in  Recent Faculty of Science and Engineering Publications

Metrics

1 Record Views

Abstract

Contextual Understanding Sentiment Analysis Sentimental LLM Task Specific Knowledge Distillation Teacher Student Learning
Knowledge distillation (KD) is a effective method for creating efficient sentiment analysis models by transferring knowledge from a large, high-performing teacher model to a smaller, faster student model. This study compares three models: the teacher, student, and a distilled model that leverages KD to combine the strengths of both. The teacher model achieved the highest accuracy due to its large capacity to capture complex sentiment cues and contextual nuances but required significant computational resources, limiting its use in real-time or resource-constrained environments. Conversely, the student model was lightweight and fast, making it suitable for deployment on devices with limited resources, but it struggled with accuracy, especially in cases involving indirect or subtle sentiment expressions. The distilled model, trained through KD, effectively bridges this gap by approaching the teacher’s accuracy while maintaining the efficiency of the student model. When compared to traditional machine learning models that rely on hard labels, the distilled model better handled mixed, nuanced, and context-dependent sentiments, demonstrating superior generalization. This makes KD-based models practical for scalable sentiment analysis applications, particularly in mobile and edge computing scenarios. Additionally, this approach can be extended to multi-class sentiment classification tasks and integrated with sentiment reasoning models to enhance emotional understanding. The full implementation and code are publicly available at https://github.com/Abishethvarman/KD-Sentiment-Analysis.

Details

Logo image