Noam Shazeer

65 papers A* 8A 3C 1Misc 1Journal 41Unranked 11
YearRankTypeTitle / Venue / Authors
2025 — conf
Hot ChipsKeynote.
HCS
Noam Shazeer
2023 J jnl
PaLM: Scaling Language Modeling with Pathways.
J. Mach. Learn. Res.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel
2023 J jnl
Scaling Up Models and Data with t5x and seqio.
J. Mach. Learn. Res.
Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Kehang Han, Michelle Casbon, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, Andrea Gesmundo
2022 J jnl
Designing Effective Sparse Expert Models.
CoRR
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus
2022 J jnl
LaMDA: Language Models for Dialog Applications.
CoRR
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Agüera y Arcas, Claire Cui, Marian Croak, Ed H. Chi, Quoc Le
2022 J jnl
PaLM: Scaling Language Modeling with Pathways.
CoRR
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel
2022 J jnl
Scaling Up Models and Data with t5x and seqio.
CoRR
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, Andrea Gesmundo
2022 J jnl
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.
J. Mach. Learn. Res.
William Fedus, Barret Zoph, Noam Shazeer
2021 — conf
Do Transformer Modifications Transfer Across Implementations and Applications?
EMNLP (1)
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
2021 J jnl
Do Transformer Modifications Transfer Across Implementations and Applications?
CoRR
Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
2021 J jnl
GSPMD: General and Scalable Parallelization for ML Computation Graphs.
CoRR
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, Zhifeng Chen
2021 A* conf
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.
ICLR
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen
2021 J jnl
Primer: Searching for Efficient Transformers for Language Modeling.
CoRR
David R. So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V. Le
2021 A* conf
Searching for Efficient Transformers for Language Modeling.
NeurIPS
David R. So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V. Le
2021 J jnl
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.
CoRR
William Fedus, Barret Zoph, Noam Shazeer
2020 J jnl
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
J. Mach. Learn. Res.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
2020 J jnl
Faster Transformer Decoding: N-gram Masked Self-Attention.
CoRR
Ciprian Chelba, Mia Xu Chen, Ankur Bapna, Noam Shazeer
2020 J jnl
GLU Variants Improve Transformer.
CoRR
Noam Shazeer
2020 J jnl
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.
CoRR
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen
2020 — conf
How Much Knowledge Can You Pack Into the Parameters of a Language Model?
EMNLP (1)
Adam Roberts, Colin Raffel, Noam Shazeer
2020 J jnl
How Much Knowledge Can You Pack Into the Parameters of a Language Model?
CoRR
Adam Roberts, Colin Raffel, Noam Shazeer
2020 J jnl
Talking-Heads Attention.
CoRR
Noam Shazeer, Zhenzhong Lan, Youlong Cheng, Nan Ding, Le Hou
2019 — conf
Corpora Generation for Grammatical Error Correction.
NAACL-HLT (1)
Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong
2019 J jnl
Corpora Generation for Grammatical Error Correction.
CoRR
Jared Lichtarge, Christopher Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong
2019 J jnl
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
CoRR
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
2019 J jnl
Fast Transformer Decoding: One Write-Head is All You Need.
CoRR
Noam Shazeer
2019 J jnl
High Resolution Medical Image Analysis with Spatial Partitioning.
CoRR
Le Hou, Youlong Cheng, Noam Shazeer, Niki Parmar, Yeqing Li, Panagiotis Korfiatis, Travis M. Drucker, Daniel J. Blezek, Xiaodan Song
2019 — conf
Music Transformer: Generating Music with Long-Term Structure.
ICLR (Poster)
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck
2018 A* conf
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost.
ICML
Noam Shazeer, Mitchell Stern
2018 J jnl
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost.
CoRR
Noam Shazeer, Mitchell Stern
2018 J jnl
An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation.
CoRR
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Douglas Eck
2018 A* conf
Blockwise Parallel Decoding for Deep Autoregressive Models.
NeurIPS
Mitchell Stern, Noam Shazeer, Jakob Uszkoreit
2018 J jnl
Blockwise Parallel Decoding for Deep Autoregressive Models.
CoRR
Mitchell Stern, Noam Shazeer, Jakob Uszkoreit
2018 A* conf
Fast Decoding in Sequence Models Using Discrete Latent Variables.
ICML
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, Noam Shazeer
2018 J jnl
Fast Decoding in Sequence Models using Discrete Latent Variables.
CoRR
Lukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, Noam Shazeer
2018 — conf
Generating Wikipedia by Summarizing Long Sequences.
ICLR (Poster)
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, Noam Shazeer
2018 J jnl
Generating Wikipedia by Summarizing Long Sequences.
CoRR
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, Noam Shazeer
2018 A* conf
HydraNets: Specialized Dynamic Architectures for Efficient Inference.
CVPR
Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, Kayvon Fatahalian
2018 A* conf
Image Transformer.
ICML
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran
2018 J jnl
Image Transformer.
CoRR
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku
2018 A* conf
Mesh-TensorFlow: Deep Learning for Supercomputers.
NeurIPS
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake A. Hechtman
2018 J jnl
Mesh-TensorFlow: Deep Learning for Supercomputers.
CoRR
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake A. Hechtman
2018 — conf
Tensor2Tensor for Neural Machine Translation.
AMTA (1)
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, Jakob Uszkoreit
2018 J jnl
Tensor2Tensor for Neural Machine Translation.
CoRR
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, Jakob Uszkoreit
2018 — conf
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation.
ACL (1)
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George F. Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, Macduff Hughes
2018 J jnl
Weakly Supervised Grammatical Error Correction using Iterative Decoding.
CoRR
Jared Lichtarge, Christopher Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar
2017 J jnl
Attention Is All You Need.
CoRR
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
2017 — conf
Attention is All you Need.
NIPS
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
2017 J jnl
One Model To Learn Them All.
CoRR
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit
2017 — conf
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
ICLR (Poster)
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean
2017 J jnl
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.
CoRR
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean
2016 Misc conf
End-to-end text-dependent speaker verification.
ICASSP
Georg Heigold, Ignacio Moreno, Samy Bengio, Noam Shazeer
2016 J jnl
Exploring the Limits of Language Modeling.
CoRR
Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu
2016 A conf
NN-Grams: Unifying Neural Network and n-Gram Language Models for Speech Recognition.
INTERSPEECH
Babak Damavandi, Shankar Kumar, Noam Shazeer, Antoine Bruguier
2016 J jnl
NN-grams: Unifying neural network and n-gram language models for Speech Recognition.
CoRR
Babak Damavandi, Shankar Kumar, Noam Shazeer, Antoine Bruguier
2016 J jnl
Sparse Non-negative Matrix Language Modeling.
Trans. Assoc. Comput. Linguistics
Joris Pelemans, Noam Shazeer, Ciprian Chelba
2016 J jnl
Swivel: Improving Embeddings by Noticing What's Missing.
CoRR
Noam Shazeer, Ryan Doherty, Colin Evans, Chris Waterson
2015 J jnl
End-to-End Text-Dependent Speaker Verification.
CoRR
Georg Heigold, Ignacio Moreno, Samy Bengio, Noam Shazeer
2015 A conf
Pruning sparse non-negative matrix n-gram language models.
INTERSPEECH
Joris Pelemans, Noam Shazeer, Ciprian Chelba
2015 — conf
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.
NIPS
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer
2015 J jnl
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.
CoRR
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer
2015 C conf
Sparse non-negative matrix language modeling for geo-annotated query session data.
ASRU
Ciprian Chelba, Noam Shazeer
2015 A conf
Sparse non-negative matrix language modeling for skip-grams.
INTERSPEECH
Noam Shazeer, Joris Pelemans, Ciprian Chelba
2014 J jnl
Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation.
CoRR
Noam Shazeer, Joris Pelemans, Ciprian Chelba
2010 J jnl
Variational Program Inference
CoRR
Georges Harik, Noam Shazeer