Mostofa Patwary

55 papers A* 8A 2Journal 35Unranked 10
YearRankTypeTitle / Venue / Authors
2026 J jnl
LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts.
CoRR
Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, Bita Rouhani
2026 — conf
Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning.
EACL (Volume 1: Long Papers)
Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2025 J jnl
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training.
CoRR
Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan Lin, Jan Kautz, Pavlo Molchanov
2025 J jnl
Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning.
CoRR
Ali Taghibakhshi, Sharath Turuvekere Sreenivas, Saurav Muralidharan, Marcin Chochowski, Yashaswi Karnati, Raviraj Joshi, Ameya Sunil Mahabaleshwarkar, Zijia Chen, Yoshi Suhara, Oluwatobi Olabiyi, Daniel Korzekwa, Mostofa Patwary, Mohammad Shoeybi, Jan Kautz, Bryan Catanzaro, Ashwath Aithal, Nima Tajbakhsh, Pavlo Molchanov
2025 J jnl
Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data.
CoRR
Syeda Nahida Akter, Shrimai Prabhumoye, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Yejin Choi, Bryan Catanzaro
2025 J jnl
Fusing LLM Capabilities with Routing Data.
CoRR
Tao Feng, Haozhen Zhang, Zijie Lei, Pengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jiaxuan You
2025 A* conf
MIND: Math Informed syNthetic Dialogues for Pretraining LLMs.
ICLR
Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2025 J jnl
Multi-Agent Evolve: LLM Self-Improve through Co-evolution.
CoRR
Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, Jiaxuan You
2025 J jnl
Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs.
CoRR
Ali Taghibakhshi, Sharath Turuvekere Sreenivas, Saurav Muralidharan, Ruisi Cai, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Yoshi Suhara, Oluwatobi Olabiyi, Daniel Korzekwa, Mostofa Patwary, Mohammad Shoeybi, Jan Kautz, Bryan Catanzaro, Ashwath Aithal, Nima Tajbakhsh, Pavlo Molchanov
2025 J jnl
Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset.
CoRR
Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2025 — conf
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset.
ACL (1)
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2025 J jnl
Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning.
CoRR
Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2025 J jnl
RLP: Reinforcement as a Pretraining Objective.
CoRR
Ali Hatamizadeh, Syeda Nahida Akter, Shrimai Prabhumoye, Jan Kautz, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi
2025 J jnl
Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning.
CoRR
Ximing Lu, Seungju Han, David Acuna, Hyunwoo Kim, Jaehun Jung, Shrimai Prabhumoye, Niklas Muennighoff, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi
2024 A* conf
Compact Language Models via Pruning and Knowledge Distillation.
NeurIPS
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov
2024 J jnl
Compact Language Models via Pruning and Knowledge Distillation.
CoRR
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov
2024 A* conf
Data, Data Everywhere: A Guide for Pretraining Dataset Construction.
EMNLP
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2024 J jnl
Data, Data Everywhere: A Guide for Pretraining Dataset Construction.
CoRR
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Li, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2024 J jnl
LLM Pruning and Distillation in Practice: The Minitron Approach.
CoRR
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov
2024 A* conf
LLM-Evolve: Evaluation for LLM's Evolving Capability on Benchmarks.
EMNLP
Jiaxuan You, Mingjie Liu, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2024 J jnl
MIND: Math Informed syNthetic Dialogues for Pretraining LLMs.
CoRR
Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2024 J jnl
Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining.
CoRR
Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2024 J jnl
Nemotron-4 15B Technical Report.
CoRR
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu, Patrick LeGresley, Denys Fridman, Jared Casper, Ashwath Aithal, Oleksii Kuchaiev, Mohammad Shoeybi, Jonathan M. Cohen, Bryan Catanzaro
2024 J jnl
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset.
CoRR
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2024 J jnl
Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models.
CoRR
Jupinder Parmar, Sanjeev Satheesh, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2023 J jnl
Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models.
CoRR
Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2023 A conf
Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models.
EACL
Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2023 — conf
Context Generation Improves Open Domain Question Answering.
EACL (Findings)
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
2022 J jnl
Context Generation Improves Open Domain Question Answering.
CoRR
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
2022 A* conf
Evaluating Parameter Efficient Learning for Generation.
EMNLP
Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro
2022 J jnl
Evaluating Parameter Efficient Learning for Generation.
CoRR
Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan J. Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro
2022 A* conf
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models.
NeurIPS
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro
2022 J jnl
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models.
CoRR
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro
2022 A* conf
Factuality Enhanced Language Models for Open-Ended Text Generation.
NeurIPS
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, Bryan Catanzaro
2022 J jnl
Factuality Enhanced Language Models for Open-Ended Text Generation.
CoRR
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2022 — conf
Keynote Talk 2 Training Large Language Models: Challenges and Opportunities.
IPDPS Workshops
Mostofa Patwary
2022 — conf
Multi-Stage Prompting for Knowledgeable Dialogue Generation.
ACL (Findings)
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
2022 J jnl
Multi-Stage Prompting for Knowledgeable Dialogue Generation.
CoRR
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
2022 J jnl
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model.
CoRR
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro
2021 J jnl
Efficient Large-Scale Language Model Training on GPU Clusters.
CoRR
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia
2021 A conf
Efficient large-scale language model training on GPU clusters using megatron-LM.
SC
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia
2021 — conf
End-to-End Training of Neural Retrievers for Open-Domain Question Answering.
ACL/IJCNLP (1)
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro
2021 J jnl
End-to-End Training of Neural Retrievers for Open-Domain Question Answering.
CoRR
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro
2020 — conf
BioMegatron: Larger Biomedical Domain Language Model.
EMNLP (1)
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani
2020 J jnl
BioMegatron: Larger Biomedical Domain Language Model.
CoRR
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani
2020 A* conf
Large Scale Multi-Actor Generative Dialog Modeling.
ACL
Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
2020 J jnl
Large Scale Multi-Actor Generative Dialog Modeling.
CoRR
Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
2020 J jnl
Local Knowledge Powered Conversational Agents.
CoRR
Sashank Santhanam, Wei Ping, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
2020 — conf
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models.
EMNLP (1)
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
2020 J jnl
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models.
CoRR
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
2020 — conf
Training Question Answering Models From Synthetic Data.
EMNLP (1)
Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
2020 J jnl
Training Question Answering Models From Synthetic Data.
CoRR
Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
2019 — conf
DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems.
MLHPC@SC
Adam Rupe, Prabhat, James P. Crutchfield, Nalini Kumar, Vladislav Epifanov, Karthik Kashinath, Oleksandr Pavlyk, Frank Schlimbach, Mostofa Patwary, Sergey Maidanov, Victor W. Lee
2019 J jnl
DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems.
CoRR
Adam Rupe, Nalini Kumar, Vladislav Epifanov, Karthik Kashinath, Oleksandr Pavlyk, Frank Schlimbach, Mostofa Patwary, Sergey Maidanov, Victor W. Lee, Prabhat, James P. Crutchfield
2019 J jnl
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.
CoRR
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro