An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks
arxiv(2024)
摘要
Stakeholders constantly make assumptions in the development of deep learning
(DL) frameworks. These assumptions are related to various types of software
artifacts (e.g., requirements, design decisions, and technical debt) and can
turn out to be invalid, leading to system failures. Existing approaches and
tools for assumption management usually depend on manual identification of
assumptions. However, assumptions are scattered in various sources (e.g., code
comments, commits, pull requests, and issues) of DL framework development, and
manually identifying assumptions has high costs (e.g., time and resources). To
overcome the issues of manually identifying assumptions in DL framework
development, we constructed a new and largest dataset (i.e., AssuEval) of
assumptions collected from the TensorFlow and Keras repositories on GitHub;
explored the performance of seven traditional machine learning models (e.g.,
Support Vector Machine, Classification and Regression Trees), a popular DL
model (i.e., ALBERT), and a large language model (i.e., ChatGPT) of identifying
assumptions on the AssuEval dataset. The experiment results show that: ALBERT
achieves the best performance (f1-score: 0.9584) of identifying assumptions on
the AssuEval dataset, which is much better than the other models (the 2nd best
f1-score is 0.6211, achieved by ChatGPT). Though ChatGPT is the most popular
large language model, we do not recommend using it to identify assumptions in
DL framework development because of its low performance on the task.
Fine-tuning ChatGPT specifically for assumption identification could improve
the performance. This study provides researchers with the largest dataset of
assumptions for further research (e.g., assumption classification, evaluation,
and reasoning) and helps practitioners better understand assumptions and how to
manage them in their projects.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要