Files and links (2)
Published (Version of record)CC BY V4.0, Open Access
Published (Version of record)CC BY V4.0, Open
Conference proceeding
FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion
Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp.313-324
ACM Conferences
ISSTA '24: 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (Vienna, Austria, 16/09/2024–20/09/2024)
11/09/2024
Metrics
9 Record Views
Abstract
The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions. To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning. We conducted a comprehensive evaluation of FT2Ra in both token-level and line-level code completions. Our findings demonstrate the remarkable effectiveness of FT2Ra when compared to state-of-the-art methods and its potential to genuine fine-tuning. In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.
Details
- Title
- FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion
- Creators
- Qi Guo - Tianjin UniversityXiaohong Li - Tianjin UniversityXiaofei Xie - Singapore Management UniversityShangqing Liu - Nanyang Technological UniversityZe Tang - Nanjing UniversityRuitao Feng - Singapore Management UniversityJunjie Wang - Tianjin UniversityJidong Ge - Nanjing UniversityLei Bu - Nanjing University
- Publication Details
- Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp.313-324
- Conference
- ISSTA '24: 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (Vienna, Austria, 16/09/2024–20/09/2024)
- Series
- ACM Conferences
- Publisher
- ACM
- Number of pages
- 12
- Grant note
- National Key RD Project: 2021YFF1201102 National Natural Science Foundation of China: 62232008, 62172200, 61872262, 62102283 National Key R&D Program of China: 2021ZD0112903 National Research Foundation, SingaporeCyber Security Agency under its National Cybersecurity RD Programme: NCRP25-PO4-TAICeN Leading edge Technology Program of Jiangsu Natural Science Foundation: BK20202001
This work was partially supported by the National Key R&D Project (2021YFF1201102), the National Natural Science Foundation of China (Key Program, Grant No. 62332005), the National Key R&D Program of China (2021ZD0112903), the National Natural Science Foundation of China (Grant No. 61872262 and No. 62102283), the National Research Foundation, Singapore, and the Cyber Security Agency under its National Cybersecurity R&D Programme (NCRP25-PO4-TAICeN). Lei Bu is supported in part by the Leading edge Technology Program of Jiangsu Natural Science Foundation (No. BK20202001), the National Natural Science Foundation of China (No. 62232008, 62172200). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Cyber Security Agency of Singapore.
- Identifiers
- 991013226413702368
- Copyright
- © 2024 Copyright held by the owner/author(s)
- Academic Unit
- Faculty of Science and Engineering; Information Technology
- Language
- English
- Resource Type
- Conference proceeding