Ian Soboroff - RankMe

180 papers A* 34A 5B 4Misc 1Journal 53Unranked 75

Year	Rank	Type	Title / Venue / Authors
2026	J	`jnl`	Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage. CoRR Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn J. Lawrie, Ian Soboroff, Laura Dietz, Benjamin Van Durme
2026	J	`jnl`	CoverageBench: Evaluating Information Coverage across Tasks and Domains. CoRR Saron Samuel, Andrew Yates, Dawn J. Lawrie, Ian Soboroff, Trevor Adriaanse, Benjamin Van Durme, Eugene Yang
2025	Misc	`conf`	A Large-Scale Study of Relevance Assessments with Large Language Models Using UMBRELA. ICTIR Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Jimmy Lin
2025	A*	`conf`	Assessing Support for the TREC 2024 RAG Track: A Large-Scale Comparative Study of LLM and Human Evaluations. SIGIR Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, Jimmy Lin
2025	—	`conf`	Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance. NAACL (Industry Track) Somnath Banerjee, Avik Halder, Rajarshi Mandal, Sayan Layek, Ian Soboroff, Rima Hazra, Animesh Mukherjee
2025	J	`jnl`	Don't Use LLMs to Make Relevance Judgments. Inf. Retr. Res. J. Ian Soboroff
2025	A*	`conf`	LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? SIGIR Rikiya Takehi, Ellen M. Voorhees, Tetsuya Sakai, Ian Soboroff
2025	A*	`conf`	Nugget-based Annotation Protocol and Tool For Evaluating Long-form Retrieval-Augmented Generation. SIGIR Eugene Yang, Dawn J. Lawrie, Hoa Dang, Ian Soboroff, James Mayfield
2025	J	`jnl`	Overview of the TREC 2022 deep learning track. CoRR Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff
2025	J	`jnl`	Overview of the TREC 2023 deep learning track. CoRR Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff
2025	J	`jnl`	Report from the 4th Strategic Workshop on Information Retrieval in Lorne (SWIRL 2025). SIGIR Forum Johanne R. Trippas, J. Shane Culpepper, Mohammad Aliannejadi, James Allan, Enrique Amigó, Jaime Arguello, Leif Azzopardi, Peter Bailey, Jamie Callan, Rob Capra, Nick Craswell, W. Bruce Croft, Jeff Dalton, Gianluca Demartini, Laura Dietz, Zhicheng Dou, Carsten Eickhoff, Michael D. Ekstrand, Nicola Ferro, Norbert Fuhr, Dorota Glowacka, Faegheh Hasibi, Danula Hettiachchi, Rosie Jones, Jaap Kamps, Noriko Kando, Sarvnaz Karimi, Makoto P. Kato, Bevan Koopman, Yiqun Liu, Chenglong Ma, Joel Mackenzie, Maria Maistro, Jiaxin Mao, Dana McKay, Bhaskar Mitra, Stefano Mizzaro, Alistair Moffat, Josiane Mothe, Iadh Ounis, Lida Rashidi, Yongli Ren, Mark Sanderson, Rodrygo L. T. Santos, Falk Scholer, Chirag Shah, Laurianne Sitbon, Ian Soboroff, Damiano Spina, Paul Thomas, Julián Urbano, Arjen P. de Vries, Ryen White, Abby Yuan, Hamed Zamani, Oleg Zendel, Min Zhang, Shengyao Zhuang, Justin Zobel, Guido Zuccon
2025	A*	`conf`	The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models. SIGIR Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, Jimmy Lin
2025	—	`conf`	The Second Search Futures Workshop at ECIR'25. ECIR (5) Charles Clarke, Paul B. Kantor, Adam Roegiest, Ian Soboroff, Johanne R. Trippas, Zhaochun Ren
2024	J	`jnl`	A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look. CoRR Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, Jimmy Lin
2024	J	`jnl`	Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance. CoRR Somnath Banerjee, Avik Halder, Rajarshi Mandal, Sayan Layek, Ian Soboroff, Rima Hazra, Animesh Mukherjee
2024	A*	`conf`	Browsing and Searching Metadata of TREC. SIGIR Timo Breuer, Ellen M. Voorhees, Ian Soboroff
2024	J	`jnl`	Don't Use LLMs to Make Relevance Judgments. CoRR Ian Soboroff
2024	A*	`conf`	On the Evaluation of Machine-Generated Reports. SIGIR James Mayfield, Eugene Yang, Dawn J. Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Selin Kayi, Kate Sanders, Marc Mason, Noah Hibbler
2024	J	`jnl`	On the Evaluation of Machine-Generated Reports. CoRR James Mayfield, Eugene Yang, Dawn J. Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Selin Kayi, Kate Sanders, Marc Mason, Noah Hibbler
2024	—	`ch.`	Privacy in Information Retrieval. Information Retrieval: Advanced Topics and Techniques Ian Soboroff
2024	—	`ed.`	Proceedings of the Thirty-Third Text REtrieval Conference, TREC 2024, Gaithersburg, MD, USA, November 18-22, 2024 TREC Ian Soboroff, Hoa Dang, George Awad
2024	J	`jnl`	QueryBuilder: Human-in-the-Loop Query Development for Information Retrieval. CoRR Hemanth Kandula, Damianos G. Karakos, Haoling Qiu, Benjamin Rozonoyer, Ian Soboroff, Lee Tarlin, Bonan Min
2024	J	`jnl`	Report on the Search Futures Workshop at ECIR 2024. SIGIR Forum Leif Azzopardi, Charles L. A. Clarke, Paul B. Kantor, Bhaskar Mitra, Johanne R. Trippas, Zhaochun Ren, Mohammad Aliannejadi, Negar Arabzadeh, Raman Chandrasekar, Maarten de Rijke, Panagiotis Eustratiadis, William R. Hersh, Jin Huang, Evangelos Kanoulas, Jasmin Kareem, Yongkang Li, Simon Lupart, Kidist Amde Mekonnen, Adam Roegiest, Ian Soboroff, Fabrizio Silvestri, Suzan Verberne, David Vos, Eugene Yang, Yuyue Zhao
2023	—	`conf`	Evaluating Systems that Generate Content. EVIA Ian Soboroff
2023	—	`conf`	Overview of TREC 2023. TREC Ian Soboroff
2023	—	`conf`	Overview of the NTCIR-17 FairWeb-1 Task. NTCIR Sijie Tao, Nuo Chen, Tetsuya Sakai, Zhumin Chu, Hiromi Arai, Ian Soboroff, Nicola Ferro, Maria Maistro
2023	—	`conf`	Overview of the TREC 2023 Deep Learning Track. TREC Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff
2023	—	`conf`	TREC. NTCIR Ian Soboroff
2023	A*	`conf`	The ACM Multimedia 2023 Deep Video Understanding Grand Challenge. ACM Multimedia Keith Curtis, George Awad, Afzal Godil, Ian Soboroff
2023	A*	`conf`	The BETTER Cross-Language Datasets. SIGIR Ian Soboroff
2023	—	`ed.`	The Thirty-Second Text REtrieval Conference Proceedings (TREC 2023), Gaithersburg, MD, USA, November 14-17, 2023 TREC Ian Soboroff, Angela Ellis
2022	B	`conf`	CHIIR Workshop on Audio Collection Human Interaction (AudioCHI 2022): http: //speechretrievalworkshop.github.io. CHIIR Gareth J. F. Jones, Maria Eskevich, Ben Carterette, Joana Correia, Rosie Jones, Jussi Karlgren, Ian Soboroff
2022	J	`jnl`	Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models? CoRR Ellen M. Voorhees, Ian Soboroff, Jimmy Lin
2022	J	`jnl`	Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks. CoRR Tetsuya Sakai, Sijie Tao, Maria Maistro, Zhumin Chu, Yujing Li, Nuo Chen, Nicola Ferro, Junjie Wang, Ian Soboroff, Yiqun Liu
2022	—	`conf`	Incident Streams 2021 Off the Deep End: Deeper Annotations and Evaluations in Twitter. ISCRAM Cody Buntain, Richard McCreadie, Ian Soboroff
2022	—	`conf`	Overview of the NTCIR-16 We Want Web with CENTRE (WWW-4) Task. NTCIR Tetsuya Sakai, Sijie Tao, Zhumin Chu, Maria Maistro, Yujing Li, Nuo Chen, Nicola Ferro, Junjie Wang, Ian Soboroff, Yiqun Liu
2022	—	`conf`	Overview of the TREC 2022 Deep Learning Track. TREC Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff
2022	—	`ed.`	Proceedings of the Thirty-First Text REtrieval Conference, TREC 2022, online, November 15-19, 2022 TREC Ian Soboroff, Angela Ellis
2022	J	`jnl`	Report on the 1st Workshop on Audio Collection Human Interaction (AudioCHI 2022) at CHIIR 2022. SIGIR Forum Gareth J. F. Jones, Maria Eskevich, Ben Carterette, Joana Correia, Rosie Jones, Jussi Karlgren, Ian Soboroff
2022	B	`conf`	Second International Workshop on Deep Video Understanding. ICMI Keith Curtis, George Awad, Shahzad Rajput, Ian Soboroff
2022	A*	`conf`	The ACM Multimedia 2022 Deep Video Understanding Grand Challenge. ACM Multimedia Keith Curtis, George Awad, Shahzad Rajput, Ian Soboroff
2022	A*	`conf`	What Makes a Good Podcast Summary? SIGIR Rezvaneh Rezapour, Sravana Reddy, Rosie Jones, Ian Soboroff
2021	—	`conf`	Incident Streams 2020: TRECIS in the Time of COVID-19. ISCRAM Cody Buntain, Richard McCreadie, Ian Soboroff
2021	J	`jnl`	Introduction to the second issue on machine translation for low-resource languages. Mach. Transl. Chao-Hong Liu, Alina Karakanta, Audrey Tong, Oleg Aulov, Ian Soboroff, Jonathan Washington, Xiaobing Zhao
2021	—	`conf`	Overview of TREC 2021. TREC Ian Soboroff
2021	A*	`conf`	Podcast Metadata and Content: Episode Relevance and Attractiveness in Ad Hoc Search. SIGIR Ben Carterette, Rosie Jones, Gareth J. F. Jones, Maria Eskevich, Sravana Reddy, Ann Clifton, Yongze Yu, Jussi Karlgren, Ian Soboroff
2021	J	`jnl`	Podcast Metadata and Content: Episode Relevance andAttractiveness in Ad Hoc Search. CoRR Ben Carterette, Rosie Jones, Gareth J. F. Jones, Maria Eskevich, Sravana Reddy, Ann Clifton, Yongze Yu, Jussi Karlgren, Ian Soboroff
2021	—	`ed.`	Proceedings of the Thirtieth Text REtrieval Conference, TREC 2021, online, November 15-19, 2021 TREC Ian Soboroff, Angela Ellis
2021	J	`jnl`	Searching for Scientific Evidence in a Pandemic: An Overview of TREC-COVID. CoRR Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, William R. Hersh
2021	J	`jnl`	Searching for scientific evidence in a pandemic: An overview of TREC-COVID. J. Biomed. Informatics Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, William R. Hersh
2021	A*	`conf`	TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime. SIGIR Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M. Voorhees, Ian Soboroff
2021	J	`jnl`	TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime. CoRR Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M. Voorhees, Ian Soboroff
2020	J	`jnl`	Evaluating Multimedia and Language Tasks. Frontiers Artif. Intell. Ian Soboroff, George Awad, Asad A. Butt, Keith Curtis
2020	J	`jnl`	HLVU : A New Challenge to Test Deep Understanding of Movies the Way Humans do. CoRR Keith Curtis, George Awad, Shahzad Rajput, Ian Soboroff
2020	B	`conf`	HLVU: A New Challenge to Test Deep Understanding of Movies the Way Humans do. ICMR Keith Curtis, George Awad, Shahzad Rajput, Ian Soboroff
2020	A*	`conf`	How to Measure the Reproducibility of System-oriented IR Experiments. SIGIR Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp Schaer, Ian Soboroff
2020	J	`jnl`	How to Measure the Reproducibility of System-oriented IR Experiments. CoRR Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp Schaer, Ian Soboroff
2020	—	`conf`	Incident Streams 2019: Actionable Insights and How to Find Them. ISCRAM Richard McCreadie, Cody Buntain, Ian Soboroff
2020	—	`conf`	Incident Streams 2020: TRECIS in the Time of COVID-19. TREC Cody Buntain, Richard McCreadie, Ian Soboroff
2020	B	`conf`	International Workshop on Deep Video Understanding. ICMI Keith Curtis, George Awad, Shahzad Rajput, Ian Soboroff
2020	J	`jnl`	Introduction to the Special Issue on Machine Translation for Low-Resource Languages. Mach. Transl. Chao-Hong Liu, Alina Karakanta, Audrey Tong, Oleg Aulov, Ian Soboroff, Jonathan Washington, Xiaobing Zhao
2020	—	`conf`	Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task. NTCIR Tetsuya Sakai, Sijie Tao, Zhaohao Zeng, Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu, Maria Maistro, Zhicheng Dou, Nicola Ferro, Ian Soboroff
2020	—	`conf`	TREC 2020 News Track Overview. TREC Ian Soboroff, Shudong Huang, Donna Harman
2020	J	`jnl`	TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. CoRR Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, Lucy Lu Wang
2020	J	`jnl`	TREC-COVID: constructing a pandemic information retrieval test collection. SIGIR Forum Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, Lucy Lu Wang
2020	J	`jnl`	TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19. J. Am. Medical Informatics Assoc. Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, William R. Hersh
2019	—	`conf`	CENTRE@CLEF 2019. ECIR (2) Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Ian Soboroff
2019	—	`conf`	CENTRE@CLEF2019: Overview of the Replicability and Reproducibility Tasks. CLEF (Working Notes) Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Ian Soboroff
2019	—	`conf`	EVIA 2019 Preface. EVIA@NTCIR Nicola Ferro, Ian Soboroff, Min Zhang
2019	—	`conf`	Financial Entity Identification and Information Integration (FEIII) 2019 Challenge: The Report of the Organizing Committee. DSMM@SIGMOD Louiqa Raschid, Douglas Burdick, Cesar de Pablo, Mark D. Flood, John Grant, Joe Langsam, Jay Pujara, Elena Tomas, Ian Soboroff
2019	J	`jnl`	On enhancing the robustness of timeline summarization test collections. Inf. Process. Manag. Richard McCreadie, Shahzad Rajput, Ian Soboroff, Craig Macdonald, Iadh Ounis
2019	—	`conf`	Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. CLEF Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Ian Soboroff
2019	—	`conf`	Overview of TAC-KBP 2019 Fine-grained Entity Extraction. TAC Heng Ji, Avirup Sil, Hoa Trang Dang, Ian Soboroff, Joel Nothman
2019	—	`conf`	Overview of the NTCIR-14 CENTRE Task. NTCIR Tetsuya Sakai, Nicola Ferro, Ian Soboroff, Zhaohao Zeng, Peng Xiao, Maria Maistro
2019	—	`ed.`	Proceedings of the 9th International Workshop on Evaluating Information Access co-located with the 14th NTCIR Conference on the Evaluation of Information Access Technologies (NTCIR 2019), Tokyo, Japan, June 10, 2019. EVIA@NTCIR Nicola Ferro, Ian Soboroff, Min Zhang
2019	—	`conf`	TREC Incident Streams: Finding Actionable Information on Social Media. ISCRAM Richard McCreadie, Cody Buntain, Ian Soboroff
2018	—	`conf`	CENTRE@CLEF2018: Overview of the Replicability Task. CLEF (Working Notes) Nicola Ferro, Maria Maistro, Tetsuya Sakai, Ian Soboroff
2018	—	`conf`	Financial Entity Identification and Information Integration (FEIII) 2018 Challenge: The Report of the Organizing Committee. DSMM@SIGMOD Louiqa Raschid, Douglas Burdick, John Grant, Joe Langsam, Jay Pujara, Elizabeth Roman, Ian Soboroff, Mohammed J. Zaki, Elena Zotkina
2018	J	`jnl`	From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442). Dagstuhl Manifestos Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, Justin Zobel
2018	A	`conf`	Meta-Analysis for Retrieval Experiments Involving Multiple Test Collections. CIKM Ian Soboroff
2018	—	`conf`	Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm. CLEF Nicola Ferro, Maria Maistro, Tetsuya Sakai, Ian Soboroff
2018	J	`jnl`	Overview of the NIST 2016 LoReHLT evaluation. Mach. Transl. Audrey Tong, Lukas L. Diduch, Jonathan G. Fiscus, Yasaman Haghpanah, Shudong Huang, David Joy, Kay Peterson, Ian Soboroff
2018	J	`jnl`	Preface. Mach. Transl. Ian Soboroff, Audrey Tong
2018	J	`jnl`	Report on EVIA 2017. SIGIR Forum Nicola Ferro, Ian Soboroff
2018	J	`jnl`	Report on GLARE 2018: 1st Workshop on Generalization in Information Retrieval. SIGIR Forum Ian Soboroff, Nicola Ferro, Norbert Fuhr
2018	—	`conf`	TREC 2018 News Track Overview. TREC Ian Soboroff, Shudong Huang, Donna Harman
2018	—	`conf`	TREC 2018 News Track. NewsIR@ECIR Shudong Huang, Ian Soboroff, Donna Harman
2018	J	`jnl`	The Dagstuhl Perspectives Workshop on Performance Modeling and Prediction. SIGIR Forum Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, Justin Zobel
2017	A*	`conf`	Building Test Collections: An Interactive Guide for Students and Others Without Their Own Evaluation Conference Series. SIGIR Ian Soboroff
2017	—	`conf`	Financial Entity Identification and Information Integration (FEIII) 2017 Challenge: The Report of the Organizing Committee. DSMM@SIGMOD Louiqa Raschid, Douglas Burdick, Mark D. Flood, John Grant, Joe Langsam, Ian Soboroff
2017	—	`conf`	New Tracks for TREC 2018. NTCIR Ian Soboroff
2017	—	`ed.`	Proceedings of the 8th International Workshop on Evaluating Information Access co-located with the 13th NTCIR Conference on the Evaluation of Information Access Technologies (NTCIR 2017), Tokyo, Japan, December 5, 2017. EVIA@NTCIR Nicola Ferro, Ian Soboroff
2017	—	`conf`	TREC 2017 Dynamic Domain Track Overview. TREC Grace Hui Yang, Zhiwen Tang, Ian Soboroff
2017	J	`jnl`	Using Replicates in Information Retrieval Evaluation. ACM Trans. Inf. Syst. Ellen M. Voorhees, Daniel Samarov, Ian Soboroff
2016	—	`conf`	Financial Entity Identification and Information Integration (FEIII) Challenge: The Report of the Organizing Committee. DSMM@SIGMOD Mark D. Flood, John Grant, Haiping Luo, Louiqa Raschid, Ian Soboroff, Kyungjin Yoo
2016	—	`conf`	News from TREC. NTCIR Ian Soboroff
2016	A*	`conf`	Privacy-Preserving IR 2016: Differential Privacy, Search, and Social Media. SIGIR Grace Hui Yang, Ian Soboroff, Li Xiong, Charles L. A. Clarke, Simson L. Garfinkel
2016	—	`conf`	Promoting Repeatability Through Open Runs. EVIA@NTCIR Ellen M. Voorhees, Shahzad Rajput, Ian Soboroff
2016	—	`conf`	TREC 2016 Dynamic Domain Track Overview. TREC Grace Hui Yang, Ian Soboroff
2016	A*	`conf`	The BOLT IR Test Collections of Multilingual Passage Retrieval from Discussion Forums. SIGIR Ian Soboroff, Kira Griffitt, Stephanie M. Strassel
2015	J	`jnl`	Privacy Preserving IR 2015: A SIGIR 2015 Workshop. SIGIR Forum Grace Hui Yang, Ian Soboroff
2015	A*	`conf`	Privacy-Preserving IR 2015: When Information Retrieval Meets Privacy and Security. SIGIR Hui Yang, Ian Soboroff
2015	—	`conf`	TREC 2015 Dynamic Domain Track Overview. TREC Hui Yang, John R. Frank, Ian Soboroff
2014	J	`jnl`	Building Better Search Engines by Measuring Search Quality. IT Prof. Ellen M. Voorhees, Paul Over, Ian Soboroff
2014	—	`conf`	Computing Confidence Intervals for Common IR Measures. EVIA@NTCIR Ian Soboroff
2014	—	`conf`	Evaluating Stream Filtering for Entity Profile Updates in TREC 2012, 2013, and 2014. TREC John R. Frank, Max Kleiman-Weiner, Daniel A. Roberts, Ellen M. Voorhees, Ian Soboroff
2014	A	`ed.`	Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014 CIKM Jianzhong Li, Xiaoyang Sean Wang, Minos N. Garofalakis, Ian Soboroff, Torsten Suel, Min Wang
2013	A*	`conf`	Building test collections: an interactive tutorial for students and others without their own evaluation conference series. SIGIR Ian Soboroff
2013	—	`conf`	Evaluating Stream Filtering for Entity Profile Updates for TREC 2013. TREC John R. Frank, Steven J. Bauer, Max Kleiman-Weiner, Daniel A. Roberts, Nilesh Tripuraneni, Ce Zhang, Christopher Ré, Ellen M. Voorhees, Ian Soboroff
2013	J	`jnl`	Introduction to the special issue on search intents and diversification. Inf. Retr. Tetsuya Sakai, Noriko Kando, Craig Macdonald, Ian Soboroff
2013	A	`ed.`	Proceedings of the Seventh International Conference on Weblogs and Social Media, ICWSM 2013, Cambridge, Massachusetts, USA, July 8-11, 2013. ICWSM Emre Kiciman, Nicole B. Ellison, Bernie Hogan, Paul Resnick, Ian Soboroff
2012	—	`conf`	Building an Entity-Centric Stream Filtering Test Collection for TREC 2012. TREC John R. Frank, Max Kleiman-Weiner, Daniel A. Roberts, Feng Niu, Ce Zhang, Christopher Ré, Ian Soboroff
2012	J	`jnl`	Information Retrieval on the Blogosphere. Found. Trends Inf. Retr. Rodrygo L. T. Santos, Craig Macdonald, Richard McCreadie, Iadh Ounis, Ian Soboroff
2012	A*	`conf`	On building a reusable Twitter corpus. SIGIR Richard McCreadie, Ian Soboroff, Jimmy Lin, Craig Macdonald, Iadh Ounis, Dean McCullough
2012	—	`conf`	Overview of the TREC-2012 Microblog Track. TREC Ian Soboroff, Iadh Ounis, Craig Macdonald, Jimmy Lin
2011	A	`conf`	A comparative analysis of cascade measures for novelty and diversity. WSDM Charles L. A. Clarke, Nick Craswell, Ian Soboroff, Azin Ashkan
2011	—	`conf`	Overview of the TREC 2011 Microblog Track. TREC Iadh Ounis, Craig Macdonald, Jimmy Lin, Ian Soboroff
2011	—	`conf`	Overview of the TREC 2011 Web Track. TREC Charles L. A. Clarke, Nick Craswell, Ian Soboroff, Ellen M. Voorhees
2011	—	`conf`	Reports from Other Evaluation Campaigns and Panel Discussion: TREC is 20 Years Old, Where Now for Evaluation Campaigns? EVIA@NTCIR Mark Sanderson, William Webber, Ian Soboroff, Gareth J. F. Jones, Andrew Trotman, Shlomo Geva, Nicola Ferro, Hideo Joho
2010	J	`jnl`	Blog track research at TREC. SIGIR Forum Craig Macdonald, Rodrygo L. T. Santos, Iadh Ounis, Ian Soboroff
2010	—	`conf`	Overview of the TREC 2010 Blog Track. TREC Iadh Ounis, Ian Soboroff
2010	—	`conf`	Overview of the TREC 2010 Web Track. TREC Charles L. A. Clarke, Nick Craswell, Ian Soboroff, Gordon V. Cormack
2010	—	`conf`	Test Collection Diagnosis and Treatment. EVIA@NTCIR Ian Soboroff
2010	A*	`conf`	The effect of assessor error on IR system evaluation. SIGIR Ben Carterette, Ian Soboroff
2009	J	`jnl`	A guide to the RIA workshop data archive. Inf. Retr. Ian Soboroff
2009	A*	`conf`	Is spam an issue for opinionated blog post search? SIGIR Craig Macdonald, Iadh Ounis, Ian Soboroff
2009	—	`conf`	Overview of the TREC 2009 Blog Track. TREC Craig Macdonald, Iadh Ounis, Ian Soboroff
2009	—	`conf`	Overview of the TREC 2009 Web Track. TREC Charles L. A. Clarke, Nick Craswell, Ian Soboroff
2009	J	`jnl`	The search and social media workshop at SIGIR 2009. SIGIR Forum Eugene Agichtein, Marti A. Hearst, Ian Soboroff
2008	A*	`conf`	Limits of opinion-finding baseline systems. SIGIR Craig Macdonald, Ben He, Iadh Ounis, Ian Soboroff
2008	A	`conf`	On the TREC Blog Track. ICWSM Iadh Ounis, Craig Macdonald, Ian Soboroff
2008	—	`conf`	Overview of the TREC 2008 Blog Track. TREC Iadh Ounis, Craig Macdonald, Ian Soboroff
2008	—	`conf`	Overview of the TREC 2008 Enterprise Track. TREC Krisztian Balog, Ian Soboroff, Paul Thomas, Peter Bailey, Nick Craswell, Arjen P. de Vries
2008	—	`ed.`	Proceeding of the 2008 ACM Workshop on Search in Social Media, SSM 2008, Napa Valley, California, USA, October 30, 2008 SSM Ian Soboroff, Eugene Agichtein, Ravi Kumar
2008	A*	`conf`	Relevance assessment: are judges exchangeable and does it matter. SIGIR Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, Emine Yilmaz
2008	—	`conf`	TREC, TAC, takeoffs, tacks, tasks, and titillations for 2009. NTCIR Ian Soboroff
2007	—	`conf`	A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track. EVIA@NTCIR Ian Soboroff
2007	A*	`conf`	A comparison of pooled and sampled relevance judgments. SIGIR Ian Soboroff
2007	J	`jnl`	Bias and the limits of pooling for large collections. Inf. Retr. Chris Buckley, Darrin Dimmick, Ian Soboroff, Ellen M. Voorhees
2007	—	`conf`	Overview of the TREC 2007 Blog Track. TREC Craig Macdonald, Iadh Ounis, Ian Soboroff
2007	—	`conf`	Overview of the TREC 2007 Enterprise Track. TREC Peter Bailey, Arjen P. de Vries, Nick Craswell, Ian Soboroff
2007	A*	`conf`	Problems with Kendall's tau. SIGIR Mark Sanderson, Ian Soboroff
2007	A*	`conf`	Reliable information retrieval evaluation with incomplete and biased judgements. SIGIR Stefan Büttcher, Charles L. A. Clarke, Peter C. K. Yeung, Ian Soboroff
2007	J	`jnl`	The CSIRO enterprise search test collection. SIGIR Forum Peter Bailey, Nick Craswell, Ian Soboroff, Arjen P. de Vries
2007	—	`conf`	What's New at TREC: Blog and Legal Discovery Search at TREC-2006. NTCIR Ian Soboroff
2006	—	`conf`	An Exploratory Study of the W3C Mailing List Test Collection for Retrieval of Emails with Pro/Con Argument. CEAS Yejun Wu, Douglas W. Oard, Ian Soboroff
2006	A*	`conf`	Bias and the limits of pooling. SIGIR Chris Buckley, Darrin Dimmick, Ian Soboroff, Ellen M. Voorhees
2006	A*	`conf`	Dynamic test collections: measuring search effectiveness on the live web. SIGIR Ian Soboroff
2006	—	`conf`	Overview of the TREC 2006 Blog Track. TREC Iadh Ounis, Craig Macdonald, Maarten de Rijke, Gilad Mishne, Ian Soboroff
2006	—	`conf`	Overview of the TREC 2006 Enterprise Track. TREC Ian Soboroff, Arjen P. de Vries, Nick Craswell
2006	—	`conf`	The TREC 2006 Terabyte Track. TREC Stefan Büttcher, Charles L. A. Clarke, Ian Soboroff
2005	—	`conf`	Novelty Detection: The TREC Experience. HLT/EMNLP Ian Soboroff, Donna Harman
2005	—	`conf`	Overview of the TREC 2005 Enterprise Track. TREC Nick Craswell, Arjen P. de Vries, Ian Soboroff
2005	J	`jnl`	SIGIR workshop report: predicting query difficulty - methods and applications. SIGIR Forum David Carmel, Elad Yom-Tov, Ian Soboroff
2005	—	`conf`	The TREC 2005 Terabyte Track. TREC Charles L. A. Clarke, Falk Scholer, Ian Soboroff
2005	J	`jnl`	The TREC terabyte retrieval track. SIGIR Forum Charles L. A. Clarke, Nick Craswell, Ian Soboroff
2004	A*	`conf`	On evaluating web search with very few relevant documents. SIGIR Ian Soboroff
2004	—	`conf`	Overview of the TREC 2004 Novelty Track. TREC Ian Soboroff
2004	—	`conf`	Overview of the TREC 2004 Terabyte Track. TREC Charles L. A. Clarke, Nick Craswell, Ian Soboroff
2003	A*	`conf`	Building a filtering test collection for TREC 2002. SIGIR Ian Soboroff, Stephen E. Robertson
2003	—	`conf`	Overview of the TREC 2003 Novelty Track. TREC Ian Soboroff, Donna Harman
2003	J	`jnl`	Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections. SIGIR Forum Ian Soboroff, Ellen M. Voorhees, Nick Craswell
2002	A*	`conf`	Automatic evaluation of world wide web search services. SIGIR Abdur Chowdhury, Ian Soboroff
2002	J	`jnl`	Do TREC web collections look like the web? SIGIR Forum Ian Soboroff
2002	A*	`conf`	Does WT10g look like the web?. SIGIR Ian Soboroff
2002	J	`jnl`	ITtalks: A Case Study in the Semantic Web and DAML+OIL. IEEE Intell. Syst. R. Scott Cost, Timothy W. Finin, Anupam Joshi, Yun Peng, Charles K. Nicholas, Ian Soboroff, Harry Chen, Lalana Kagal, Filip Perich, Youyong Zou, Sovrin Tolia
2002	J	`jnl`	Related, but not Relevant: Content-Based Collaborative Filtering in TREC-8. Inf. Retr. Ian Soboroff, Charles K. Nicholas
2002	—	`conf`	The TREC 2002 Filtering Track Report. TREC Stephen E. Robertson, Ian Soboroff
2001	A*	`conf`	Ranking Retrieval Systems without Relevance Judgments. SIGIR Ian Soboroff, Charles K. Nicholas, Patrick Cahan
2001	—	`conf`	The TREC 2001 Filtering Track Report. TREC Stephen E. Robertson, Ian Soboroff
2000	A*	`conf`	Collaborative filtering and the generalized vector space model. SIGIR Ian Soboroff, Charles K. Nicholas
1999	—	`conf`	Agent Development with Jackal. Agents R. Scott Cost, Timothy W. Finin, Yannis Labrou, Xiaocheng Luan, Yun Peng, Ian Soboroff, James Mayfield, Akram Boughannam
1999	—	`conf`	An Agent-Based Infrastructure for Enterprise Integration. ASA/MA R. Scott Cost, Timothy W. Finin, Yannis Labrou, Xiaocheng Luan, Yun Peng, Ian Soboroff, James Mayfield, Akram Boughannam
1999	J	`jnl`	Interactive Volumetric Information Visualization for Document Corpus Management. Int. J. Digit. Libr. Christopher D. Shaw, James M. Kukla, Ian Soboroff, David S. Ebert, Charles K. Nicholas, Amen Zwa, Ethan L. Miller, D. Aaron Roberts
1999	—	`conf`	TREC-8 Experiments at Maryland: CLIR, QA and Routing. TREC Douglas W. Oard, Jianqiang Wang, Dekang Lin, Ian Soboroff
1999	J	`jnl`	Workshop on Recommender Systems: Algorithms and Evaluation. SIGIR Forum Ian Soboroff, Charles K. Nicholas, Michael J. Pazzani
1997	—	`conf`	Agent Development Support for Tcl. Tcl/Tk Workshop R. Scott Cost, Ian Soboroff, Jeegar Lakhani, Tim Finin, Ethan L. Miller, Charles K. Nicholas
1997	—	`conf`	TKQML: A Scripting Tool for Building Agents. ATAL R. Scott Cost, Ian Soboroff, Jeegar Lakhani, Timothy W. Finin, Ethan L. Miller, Charles K. Nicholas
1997	—	`conf`	Visualizing Document Authorship Using n-grams and Latent Semantic Indexing. Workshop on New Paradigms in Information Visualization and Manipulation Ian Soboroff, Charles K. Nicholas, James M. Kukla, David S. Ebert
1996	J	`jnl`	Head-Tracked Stereo Viewing with Two-Handed 3D Interactionfor Animated Character Construction. Comput. Graph. Forum Russell Turner, Enrico Gobbetti, Ian Soboroff