Daniel Deutsch

71 papers A* 6A 1B 3Journal 37Unranked 22

Year	Rank	Type	Title / Venue / Authors
2026	J	`jnl`	TranslateGemma Technical Report. CoRR Mara Finkelstein, Isaac Caswell, Tobias Domhan, Jan-Thorsten Peter, Juraj Juraska, Parker Riley, Daniel Deutsch, Geza Kovacs, Cole Dilanni, Colin Cherry, Eleftheria Briakou, Elizabeth Nielsen, Jiaming Luo, Kat Black, Ryan Mullins, Sweta Agrawal, Wenda Xu, Erin Kats, Stephane Jaskiewicz, Markus Freitag, David Vilar
2025	J	`jnl`	Deconstructing Self-Bias in LLM-generated Translation Benchmarks. CoRR Wenda Xu, Sweta Agrawal, Vilém Zouhar, Markus Freitag, Daniel Deutsch
2025	A*	`conf`	Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation. EMNLP Colten DiIanni, Daniel Deutsch
2025	J	`jnl`	Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation. CoRR Colten DiIanni, Daniel Deutsch
2025	—	`conf`	Enhancing Human Evaluation in Machine Translation with Comparative Judgement. ACL (1) Yixiao Song, Parker Riley, Daniel Deutsch, Markus Freitag
2025	J	`jnl`	Enhancing Human Evaluation in Machine Translation with Comparative Judgment. CoRR Yixiao Song, Parker Riley, Daniel Deutsch, Markus Freitag
2025	A*	`conf`	From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set. ICML Mara Finkelstein, Daniel Deutsch, Parker Riley, Juraj Juraska, Geza Kovacs, Markus Freitag
2025	J	`jnl`	Generating Difficult-to-Translate Texts. CoRR Vilém Zouhar, Wenda Xu, Parker Riley, Juraj Juraska, Mara Finkelstein, Markus Freitag, Daniel Deutsch
2025	J	`jnl`	MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation. CoRR Parker Riley, Daniel Deutsch, Mara Finkelstein, Colten DiIanni, Juraj Juraska, Markus Freitag
2025	J	`jnl`	MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task. CoRR Juraj Juraska, Tobias Domhan, Mara Finkelstein, Tetsuji Nakagawa, Geza Kovacs, Daniel Deutsch, Pidong Wang, Markus Freitag
2025	A*	`conf`	Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation. ICML Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag
2025	J	`jnl`	Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation. CoRR Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag
2025	A*	`conf`	SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? EMNLP Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, David Ifeoluwa Adelani
2025	J	`jnl`	SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? CoRR Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, David Ifeoluwa Adelani
2025	J	`jnl`	Searching for Difficult-to-Translate Test Examples at Scale. CoRR Wenda Xu, Vilém Zouhar, Parker Riley, Mara Finkelstein, Markus Freitag, Daniel Deutsch
2025	—	`conf`	WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects. ACL (Findings) Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, Markus Freitag
2025	J	`jnl`	WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects. CoRR Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, Markus Freitag
2024	—	`conf`	Are LLMs Breaking MT Metrics? Results of the WMT24 Metrics Shared Task. WMT Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frédéric Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, Alon Lavie
2024	—	`conf`	Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data. WMT Zhongtao Liu, Parker Riley, Daniel Deutsch, Alison Lui, Mengmeng Niu, Apurva Shah, Markus Freitag
2024	J	`jnl`	Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data. CoRR Zhongtao Liu, Parker Riley, Daniel Deutsch, Alison Lui, Mengmeng Niu, Apu Shah, Markus Freitag
2024	—	`conf`	Finding Replicable Human Evaluations via Stable Ranking Probability. NAACL-HLT Parker Riley, Daniel Deutsch, George F. Foster, Viresh Ratnakar, Ali Dabirmoghaddam, Markus Freitag
2024	J	`jnl`	Finding Replicable Human Evaluations via Stable Ranking Probability. CoRR Parker Riley, Daniel Deutsch, George F. Foster, Viresh Ratnakar, Ali Dabirmoghaddam, Markus Freitag
2024	J	`jnl`	From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set. CoRR Mara Finkelstein, Daniel Deutsch, Parker Riley, Juraj Juraska, Geza Kovacs, Markus Freitag
2024	—	`conf`	Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy. WMT Brian Thompson, Nitika Mathur, Daniel Deutsch, Huda Khayrallah
2024	J	`jnl`	Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy. CoRR Brian Thompson, Nitika Mathur, Daniel Deutsch, Huda Khayrallah
2024	—	`conf`	LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback. NAACL-HLT (Findings) Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, Markus Freitag
2024	—	`conf`	MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task. WMT Juraj Juraska, Daniel Deutsch, Mara Finkelstein, Markus Freitag
2024	J	`jnl`	MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task. CoRR Juraj Juraska, Daniel Deutsch, Mara Finkelstein, Markus Freitag
2024	—	`conf`	Mitigating Metric Bias in Minimum Bayes Risk Decoding. WMT Geza Kovacs, Daniel Deutsch, Markus Freitag
2024	J	`jnl`	Mitigating Metric Bias in Minimum Bayes Risk Decoding. CoRR Geza Kovacs, Daniel Deutsch, Markus Freitag
2024	—	`conf`	On the Role of Summary Content Units in Text Summarization Evaluation. NAACL (Short Papers) Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Sebastian Gehrmann, Lining Zhang, Saad Mahamood, Miruna Clinciu, Khyathi Raghavi Chandu, Yufang Hou
2024	J	`jnl`	On the Role of Summary Content Units in Text Summarization Evaluation. CoRR Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Raghavi Chandu, Yufang Hou
2023	—	`conf`	A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization. ACL (1) Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Mahamood, Sebastian Gehrmann, Miruna Clinciu, Khyathi Raghavi Chandu, João Sedoc
2023	A	`conf`	Incorporating Question Answering-Based Signals into Abstractive Summarization via Salient Span Selection. EACL Daniel Deutsch, Dan Roth
2023	—	`conf`	MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task. WMT Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, Markus Freitag
2023	J	`jnl`	Pinpoint, Not Criticize: Refining Large Language Models via Fine-Grained Actionable Feedback. CoRR Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, Markus Freitag
2023	—	`ed.`	Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, Eval4NLP 2023, Bali, Indonesia, November 1, 2023 Eval4NLP Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, Andreas Rücklé
2023	—	`conf`	Quality Estimation Using Minimum Bayes Risk. WMT Subhajit Naskar, Daniel Deutsch, Markus Freitag
2023	—	`conf`	Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent. WMT Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, George F. Foster
2023	—	`conf`	The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation. WMT Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, Orhan Firat
2023	J	`jnl`	The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation. CoRR Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, Orhan Firat
2023	—	`conf`	The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics. Eval4NLP Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, Steffen Eger
2023	J	`jnl`	The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics. CoRR Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, Steffen Eger
2023	—	`conf`	There's No Data like Better Data: Using QE Metrics for MT Data Filtering. WMT Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Markus Freitag
2023	J	`jnl`	There's no Data Like Better Data: Using QE Metrics for MT Data Filtering. CoRR Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Markus Freitag
2023	A*	`conf`	Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration. EMNLP Daniel Deutsch, George F. Foster, Markus Freitag
2023	J	`jnl`	Ties Matter: Modifying Kendall's Tau for Modern Metric Meta-Evaluation. CoRR Daniel Deutsch, George F. Foster, Markus Freitag
2023	—	`conf`	Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level. WMT Daniel Deutsch, Juraj Juraska, Mara Finkelstein, Markus Freitag
2023	J	`jnl`	Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level. CoRR Daniel Deutsch, Juraj Juraska, Mara Finkelstein, Markus Freitag
2022	—	`conf`	Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics. ACL (Findings) Daniel Deutsch, Dan Roth
2022	J	`jnl`	Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics. CoRR Daniel Deutsch, Dan Roth
2022	J	`jnl`	GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. CoRR Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir R. Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh D. Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachini, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Stajner, Sébastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin P. AMahidewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
2022	J	`jnl`	Needle in a Haystack: An Analysis of Finding Qualified Workers on MTurk for Summarization. CoRR Lining Zhang, João Sedoc, Simon Mille, Yufang Hou, Sebastian Gehrmann, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Miruna Clinciu, Saad Mahamood, Khyathi Raghavi Chandu
2022	A*	`conf`	On the Limitations of Reference-Free Evaluations of Generated Text. EMNLP Daniel Deutsch, Rotem Dror, Dan Roth
2022	J	`jnl`	On the Limitations of Reference-Free Evaluations of Generated Text. CoRR Daniel Deutsch, Rotem Dror, Dan Roth
2022	—	`ed.`	Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems, Eval4NLP 2022, Online, November 20, 2022 Eval4NLP Daniel Deutsch, Can Udomcharoenchaikit, Juri Opitz, Yang Gao, Marina Fomicheva, Steffen Eger
2022	—	`conf`	Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics. NAACL-HLT Daniel Deutsch, Rotem Dror, Dan Roth
2022	J	`jnl`	Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics. CoRR Daniel Deutsch, Rotem Dror, Dan Roth
2022	J	`jnl`	Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code. CoRR Daniel Deutsch, Dan Roth
2021	J	`jnl`	A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. Trans. Assoc. Comput. Linguistics Daniel Deutsch, Rotem Dror, Dan Roth
2021	J	`jnl`	A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods. CoRR Daniel Deutsch, Rotem Dror, Dan Roth
2021	J	`jnl`	Question-Based Salient Span Selection for More Controllable Text Summarization. CoRR Daniel Deutsch, Dan Roth
2021	J	`jnl`	Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. Trans. Assoc. Comput. Linguistics Daniel Deutsch, Tania Bedrax-Weiss, Dan Roth
2021	B	`conf`	Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries. CoNLL Daniel Deutsch, Dan Roth
2020	B	`conf`	Is Killed More Significant than Fled? A Contextual Model for Salient Event Detection. COLING Disha Jindal, Daniel Deutsch, Dan Roth
2020	J	`jnl`	SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics. CoRR Daniel Deutsch, Dan Roth
2020	J	`jnl`	Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. CoRR Daniel Deutsch, Tania Bedrax-Weiss, Dan Roth
2020	J	`jnl`	Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries. CoRR Daniel Deutsch, Dan Roth
2019	B	`conf`	A General-Purpose Algorithm for Constrained Sequential Inference. CoNLL Daniel Deutsch, Shyam Upadhyay, Dan Roth
2019	—	`conf`	Summary Cloze: A New Task for Content Selection in Topic-Focused Summarization. EMNLP/IJCNLP (1) Daniel Deutsch, Dan Roth
2018	—	`conf`	A Distributional and Orthographic Aggregation Model for English Derivational Morphology. ACL (1) Daniel Deutsch, John Hewitt, Dan Roth