APPLS: Evaluating Evaluation Metrics for Plain Language Summarization
CoRR(2023)
摘要
While there has been significant development of models for Plain Language
Summarization (PLS), evaluation remains a challenge. PLS lacks a dedicated
assessment metric, and the suitability of text generation evaluation metrics is
unclear due to the unique transformations involved (e.g., adding background
explanations, removing specialized terminology). To address these concerns, our
study presents a granular meta-evaluation testbed, APPLS, designed to evaluate
metrics for PLS. We define a set of perturbations along four criteria inspired
by previous work that a PLS metric should capture: informativeness,
simplification, coherence, and faithfulness. An analysis of metrics using our
testbed reveals that current metrics fail to capture simplification
consistently. In response, we introduce POMME, a new metric designed to assess
text simplification in PLS; the metric is calculated as the normalized
perplexity difference between an in-domain and out-of-domain language model. We
demonstrate POMME's correlation with fine-grained variations in simplification
and validate its sensitivity across 4 text simplification datasets. This work
contributes the first meta-evaluation testbed for PLS and a comprehensive
evaluation of existing metrics. The APPLS testbed and POMME is available at
https://github.com/LinguisticAnomalies/APPLS.
更多查看译文
关键词
summarization,language,meta-evaluation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要