Ivan Stelmakh

31 papers A* 6B 2Journal 19Unranked 2
YearRankTypeTitle / Venue / Authors
2025 J jnl
A Gold Standard Dataset for the Reviewer Assignment Problem.
Trans. Mach. Learn. Res.
Ivan Stelmakh, John Frederick Wieting, Yang Xi, Graham Neubig, Nihar B. Shah
2024 J jnl
A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions.
CoRR
Charvi Rastogi, Xiangchen Song, Zhijing Jin, Ivan Stelmakh, Hal Daumé III, Kun Zhang, Nihar B. Shah
2024 J jnl
Causal Effect of Group Diversity on Redundancy and Coverage in Peer-Reviewing.
CoRR
Navita Goyal, Ivan Stelmakh, Nihar B. Shah, Hal Daumé III
2024 J jnl
Debiasing Evaluations That Are Biased by Evaluations.
J. Mach. Learn. Res.
Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah
2023 J jnl
A Gold Standard Dataset for the Reviewer Assignment Problem.
CoRR
Ivan Stelmakh, John Wieting, Graham Neubig, Nihar B. Shah
2023 J jnl
Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments.
CoRR
Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, Nihar B. Shah
2022 A* conf
ASQA: Factoid Questions Meet Long-Form Answers.
EMNLP
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang
2022 J jnl
ASQA: Factoid Questions Meet Long-Form Answers.
CoRR
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang
2022 J jnl
Cite-seeing and Reviewing: A Study on Citation Bias in Peer Review.
CoRR
Ivan Stelmakh, Charvi Rastogi, Ryan Liu, Shuchi Chawla, Federico Echenique, Nihar B. Shah
2022 J jnl
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
CoRR
Charvi Rastogi, Ivan Stelmakh, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, Jennifer Wortman Vaughan, Zhenyu Xue, Hal Daumé III, Emma Pierson, Nihar B. Shah
2022 — —
Making Scientific Peer Review Scientific
Ivan Stelmakh
2022 B conf
No Rose for MLE: Inadmissibility of MLE for Evaluation Aggregation Under Levels of Expertise.
ISIT
Charvi Rastogi, Ivan Stelmakh, Nihar B. Shah, Sivaraman Balakrishnan
2022 J jnl
To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online.
CoRR
Charvi Rastogi, Ivan Stelmakh, Xinwei Shen, Marina Meila, Federico Echenique, Shuchi Chawla, Nihar B. Shah
2021 A* conf
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences.
AAAI
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, Hal Daumé III
2021 A* conf
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment.
AAAI
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
2021 — conf
CrowdSpeech and Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription.
NeurIPS Datasets and Benchmarks
Nikita Pavlichenko, Ivan Stelmakh, Dmitry Ustalov
2021 A* conf
Debiasing Evaluations That Are Biased by Evaluations.
AAAI
Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah
2021 J jnl
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review.
J. Mach. Learn. Res.
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
2021 J jnl
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review.
Proc. ACM Hum. Comput. Interact.
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, Hal Daumé III
2021 — ed.
Proceedings of the 2nd Crowd Science Workshop: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale co-located with 47th International Conference on Very Large Data Bases (VLDB 2021), Copenhagen, Denmark, August 20, 2021.
CSW@VLDB
Dmitry Ustalov, Fabio Casati, Alexey Drutsa, Ivan Stelmakh, Nikita Pavlichenko, Daria Baidakova
2021 A* conf
Towards Fair, Equitable, and Efficient Peer Review.
AAAI
Ivan Stelmakh
2021 — conf
VLDB 2021 Crowd Science Challenge on Aggregating Crowdsourced Audio Transcriptions.
CSW@VLDB
Dmitry Ustalov, Nikita Pavlichenko, Ivan Stelmakh, Dmitriy Kuznetsov
2021 J jnl
Vox Populi, Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription.
CoRR
Nikita Pavlichenko, Ivan Stelmakh, Dmitry Ustalov
2020 J jnl
A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions.
CoRR
Ivan Stelmakh, Charvi Rastogi, Nihar B. Shah, Aarti Singh, Hal Daumé III
2020 J jnl
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences.
CoRR
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, Hal Daumé III
2020 J jnl
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment.
CoRR
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
2020 J jnl
Debiasing Evaluations That are Biased by Evaluations.
CoRR
Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah
2020 J jnl
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review.
CoRR
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, Hal Daumé III
2019 A* conf
On Testing for Biases in Peer Review.
NeurIPS
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
2019 B conf
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review.
ALT
Ivan Stelmakh, Nihar B. Shah, Aarti Singh
2018 J jnl
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review.
CoRR
Ivan Stelmakh, Nihar B. Shah, Aarti Singh