ADDRESSING SUBJECTIVITY IN CODE SMELL DETECTION BY LEVERAGING HUMAN FEEDBACK IN A DEEP-LEARNING-BASED SOLUTION

Abstract

Code smells, being inherently subjective, can vary based on individual software developers' opinions and perspectives. Though many deep-learning-based models have been proposed for code smell detection, they often overlook the importance of considering each developer's subjective context, undermining the effectiveness of these methods. In this thesis, we address this limitation by (a) constructing a manually annotated dataset for three subjective code smells incorporating multiple annotations for each sample, (b) developing three state-of-the-art deep learning models with different architectures and comparing their performance on the dataset, and (c) conducting an extensive experiment using the best-performing deep-learning model to evaluate the impact of human feedback. We start by building a dataset containing 10,267 annotations for 5,192 code snippets, targeting three code smells at different granularity levels: multifaceted abstraction, complex method, and long parameter list. Additionally, we create a supplementary dataset comprising ‘definitely clean’ and ‘definitely smelly’ samples, identified using the thresholds during dataset construction. To facilitate developers' involvement, we designed a plugin for IntelliJ IDEA and developed a container-based web server to offer services of our baseline deep-learning model within the IDE. This setup lets developers see code smells within the IDE and provide feedback (i.e., whether a snippet has the identified smell and whether a snippet has a smell not identified by our approach). In a controlled experiment, we collected feedback on code smells from 14 participants in two rounds: one with our baseline model and the second with a fine-tuned model. In the second round, we fine-tuned the model based on the participants' feedback and reevaluated the smell detection performance before and after adjustment. The results demonstrate that such calibration improves the smell detection model's performance by 15.49% in the F1 score on average across the experimental group participants. Notably, this improvement is achieved while maintaining a low Krippendorff score, indicating that the smell detection model's ability is enhanced while considering the subjectivity of the feedback.