Joshua Ainslie

54 papers A* 10B 1Journal 30Unranked 13
YearRankTypeTitle / Venue / Authors
2025 J jnl
Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation.
CoRR
Biao Zhang, Fedor Moiseev, Joshua Ainslie, Paul Suganthan, Min Ma, Surya Bhupatiraju, Federico Lebrón, Orhan Firat, Armand Joulin, Zhe Dong
2025 A* conf
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING.
ICML
Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Kumar Avinava Dubey, Ayzaan Wahid, Sumeet Singh, René Wagner, Tianli Ding, Chuyuan Fu, Arunkumar Byravan, Jake Varley, Alexey A. Gritsenko, Matthias Minderer, Dmitry Kalashnikov, Jonathan Tompson, Vikas Sindhwani, Krzysztof Marcin Choromanski
2025 J jnl
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING.
CoRR
Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, Sumeet Singh, René Wagner, Tianli Ding, Chuyuan Fu, Arunkumar Byravan, Jake Varley, Alexey A. Gritsenko, Matthias Minderer, Dmitry Kalashnikov, Jonathan Tompson, Vikas Sindhwani, Krzysztof Choromanski
2025 A* conf
Linear Transformer Topological Masking with Graph Random Features.
ICLR
Isaac Reid, Kumar Avinava Dubey, Deepali Jain, William F. Whitney, Amr Ahmed, Joshua Ainslie, Alex Bewley, Mithun George Jacob, Aranyak Mehta, David Rendleman, Connor Schenck, Richard E. Turner, René Wagner, Adrian Weller, Krzysztof Marcin Choromanski
2024 A* conf
Functional Interpolation for Relative Positions improves Long Context Transformers.
ICLR
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli
2024 J jnl
Linear Transformer Topological Masking with Graph Random Features.
CoRR
Isaac Reid, Kumar Avinava Dubey, Deepali Jain, Will Whitney, Amr Ahmed, Joshua Ainslie, Alex Bewley, Mithun George Jacob, Aranyak Mehta, David Rendleman, Connor Schenck, Richard E. Turner, René Wagner, Adrian Weller, Krzysztof Choromanski
2024 — conf
MEMORY-VQ: Compression for Tractable Internet-Scale Memory.
NAACL (Short Papers)
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontañón, William W. Cohen, Sumit Sanghai, Joshua Ainslie
2023 A* conf
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding.
EMNLP
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
2023 J jnl
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding.
CoRR
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
2023 A* conf
CoLT5: Faster Long-Range Transformers with Conditional Computation.
EMNLP
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
2023 J jnl
CoLT5: Faster Long-Range Transformers with Conditional Computation.
CoRR
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
2023 A* conf
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference.
NeurIPS
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang
2023 J jnl
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference.
CoRR
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang
2023 — conf
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference.
ACL (Findings)
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William W. Cohen
2023 — conf
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction.
ACL (1)
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolay Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
2023 J jnl
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction.
CoRR
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolai Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
2023 J jnl
Functional Interpolation for Relative Positions Improves Long Context Transformers.
CoRR
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli
2023 J jnl
GLIMMER: generalized late-interaction memory reranker.
CoRR
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W. Cohen, Joshua Ainslie
2023 A* conf
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
EMNLP
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai
2023 J jnl
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.
CoRR
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai
2023 J jnl
MEMORY-VQ: Compression for Tractable Internet-Scale Memory.
CoRR
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontañón, William W. Cohen, Sumit Sanghai, Joshua Ainslie
2023 A* conf
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute.
ICML
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William W. Cohen
2023 J jnl
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute.
CoRR
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William W. Cohen
2023 A* conf
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints.
ICLR
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby
2023 J jnl
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset.
CoRR
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
2023 — conf
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences.
EMNLP (Findings)
David C. Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo
2023 J jnl
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences.
CoRR
David C. Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo
2022 — conf
FNet: Mixing Tokens with Fourier Transforms.
NAACL-HLT
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontañón
2022 J jnl
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference.
CoRR
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William W. Cohen
2022 — conf
FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction.
ACL (1)
Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister
2022 J jnl
FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction.
CoRR
Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister
2022 B conf
Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing.
COLING
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
2022 J jnl
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing.
CoRR
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
2022 J jnl
LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models.
CoRR
Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, Zachary Fisher
2022 — conf
LongT5: Efficient Text-To-Text Transformer for Long Sequences.
NAACL-HLT (Findings)
Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang
2022 — conf
Making Transformers Solve Compositional Tasks.
ACL (1)
Santiago Ontañón, Joshua Ainslie, Zachary Fisher, Vaclav Cvicek
2022 — conf
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT.
EMNLP (Findings)
James Lee-Thorp, Joshua Ainslie
2022 J jnl
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT.
CoRR
James Lee-Thorp, Joshua Ainslie
2022 J jnl
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints.
CoRR
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby
2021 J jnl
FNet: Mixing Tokens with Fourier Transforms.
CoRR
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontañón
2021 — conf
Improving Compositional Generalization in Classification Tasks via Structure Annotations.
ACL/IJCNLP (2)
Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, Santiago Ontañón
2021 J jnl
Improving Compositional Generalization in Classification Tasks via Structure Annotations.
CoRR
Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, Santiago Ontañón
2021 J jnl
Iterative Decoding for Compositional Generalization in Transformers.
CoRR
Luana Ruiz, Joshua Ainslie, Santiago Ontañón
2021 J jnl
LongT5: Efficient Text-To-Text Transformer for Long Sequences.
CoRR
Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang
2021 J jnl
Making Transformers Solve Compositional Tasks.
CoRR
Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, Zachary Fisher
2021 — conf
ReadTwice: Reading Very Large Documents with Memories.
NAACL-HLT
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha
2021 J jnl
ReadTwice: Reading Very Large Documents with Memories.
CoRR
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha
2021 — conf
RealFormer: Transformer Likes Residual Attention.
ACL/IJCNLP (Findings)
Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie
2021 J jnl
ShopTalk: A System for Conversational Faceted Search.
CoRR
Gurmeet Manku, James Lee-Thorp, Bhargav Kanagal, Joshua Ainslie, Jingchen Feng, Zach Pearson, Ebenezer Anjorin, Sudeep Gandhe, Ilya Eckstein, Jim Rosswog, Sumit Sanghai, Michael Pohl, Larry Adams, D. Sivakumar
2020 A* conf
Big Bird: Transformers for Longer Sequences.
NeurIPS
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed
2020 J jnl
Big Bird: Transformers for Longer Sequences.
CoRR
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed
2020 J jnl
ETC: Encoding Long and Structured Data in Transformers.
CoRR
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Philip Pham, Anirudh Ravula, Sumit Sanghai
2020 — conf
ETC: Encoding Long and Structured Inputs in Transformers.
EMNLP (1)
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang
2020 J jnl
RealFormer: Transformer Likes Residual Attention.
CoRR
Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie