Samyam Rajbhandari

52 papers A* 9A 7B 3Misc 2Journal 25Unranked 6

Year	Rank	Type	Title / Venue / Authors
2026	—	`conf`	Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads. ASPLOS (2) Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, Samyam Rajbhandari
2025	J	`jnl`	Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI. CoRR Samyam Rajbhandari, Mert Hidayetoglu, Aurick Qiao, Ye Wang, Juncheng Yang, Jeff Rasley, Michael Wyatt, Yuxiong He
2025	J	`jnl`	Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences. CoRR Stas Bekman, Samyam Rajbhandari, Michael Wyatt, Jeff Rasley, Tunji Ruwase, Zhewei Yao, Aurick Qiao, Yuxiong He
2025	J	`jnl`	Fast and Accurate Causal Parallel Decoding using Jacobi Forcing. CoRR Lanxiang Hu, Siqi Kou, Yichao Fu, Samyam Rajbhandari, Tajana Rosing, Yuxiong He, Zhijie Deng, Hao Zhang
2025	J	`jnl`	OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs. CoRR Jaeseong Lee, Seung-won Hwang, Aurick Qiao, Gabriele Oliaro, Ye Wang, Samyam Rajbhandari
2025	J	`jnl`	Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads. CoRR Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, Samyam Rajbhandari
2025	A*	`conf`	SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation. EMNLP Aurick Qiao, Zhewei Yao, Samyam Rajbhandari, Yuxiong He
2024	J	`jnl`	DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. CoRR Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, Yuxiong He
2024	J	`jnl`	SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation. CoRR Aurick Qiao, Zhewei Yao, Samyam Rajbhandari, Yuxiong He
2024	—	`conf`	System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. IPDPS (Workshops) Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminabadi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
2024	A*	`conf`	System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. PODC Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
2024	A*	`conf`	ZeRO++: Extremely Efficient Collective Communication for Large Model Training. ICLR Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
2023	A	`conf`	A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training. ICS Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele
2023	J	`jnl`	A Novel Tensor-Expert Hybrid Parallelism Approach to Scale Mixture-of-Experts Training. CoRR Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele
2023	J	`jnl`	DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. CoRR Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
2023	J	`jnl`	DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. CoRR Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He
2023	J	`jnl`	DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention. CoRR Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qin, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He
2023	J	`jnl`	ZeRO++: Extremely Efficient Collective Communication for Giant Model Training. CoRR Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
2022	Misc	`conf`	1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. HIPC Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
2022	J	`jnl`	DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. CoRR Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
2022	A	`conf`	DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. SC Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, Yuxiong He
2022	A*	`conf`	DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. ICML Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
2022	J	`jnl`	DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. CoRR Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
2022	J	`jnl`	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. CoRR Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro
2021	A*	`conf`	1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. ICML Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He
2021	J	`jnl`	1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. CoRR Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He
2021	J	`jnl`	1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. CoRR Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
2021	J	`jnl`	Scalable and Efficient MoE Training for Multitask Multilingual Models. CoRR Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andrés Felipe Cruz-Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
2021	A*	`conf`	SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement. NeurIPS Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
2021	J	`jnl`	ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. CoRR Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
2021	—	`conf`	ZeRO-Offload: Democratizing Billion-Scale Model Training. USENIX ATC Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
2021	J	`jnl`	ZeRO-Offload: Democratizing Billion-Scale Model Training. CoRR Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
2021	A	`conf`	ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. SC Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
2020	J	`jnl`	APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm. CoRR Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari, Xiangru Lian, Ji Liu, Yuxiong He, Ce Zhang
2020	A*	`conf`	DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. KDD Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He
2020	J	`jnl`	Fast LSTM by dynamic decomposition on cloud and distributed systems. Knowl. Inf. Syst. Yang You, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Cho-Jui Hsieh, Kurt Keutzer, James Demmel
2020	A	`conf`	ZeRO: memory optimizations toward training trillion parameter models. SC Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
2019	—	`conf`	Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft. OpML Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, Elton Zheng, Olatunji Ruwase, Jeff Rasley, Jason Li, Junhua Wang, Yuxiong He
2019	J	`jnl`	AntMan: Sparse Low-Rank Compression to Accelerate RNN inference. CoRR Samyam Rajbhandari, Harsh Shrivastava, Yuxiong He
2019	A*	`conf`	Fast LSTM Inference by Dynamic Decomposition on Cloud Systems. ICDM Yang You, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Cho-Jui Hsieh, Kurt Keutzer, James Demmel
2019	J	`jnl`	ZeRO: Memory Optimization Towards Training A Trillion Parameter Models. CoRR Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
2018	—	`conf`	DeepCPU: Serving RNN-based Deep Learning Models 10x Faster. USENIX ATC Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, Yuxiong He
2018	—	`conf`	Learning Intrinsic Sparse Structures within Long Short-Term Memory. ICLR (Poster) Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li
2017	J	`jnl`	Learning Intrinsic Sparse Structures within Long Short-term Memory. CoRR Wei Wen, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li
2017	A*	`conf`	Optimizing CNNs on Multicores for Scalability, Performance and Goodput. ASPLOS Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul M. Chilimbi
2017	B	`conf`	Optimizing the Four-Index Integral Transform Using Data Movement Lower Bounds Analysis. PPoPP Samyam Rajbhandari, Fabrice Rastello, Karol Kowalski, Sriram Krishnamoorthy, P. Sadayappan
2016	A	`conf`	A domain-specific compiler for a parallel multiresolution adaptive numerical simulation environment. SC Samyam Rajbhandari, Jinsung Kim, Sriram Krishnamoorthy, Louis-Noël Pouchet, Fabrice Rastello, Robert J. Harrison, P. Sadayappan
2016	B	`conf`	On fusing recursive traversals of K-d trees. CC Samyam Rajbhandari, Jinsung Kim, Sriram Krishnamoorthy, Louis-Noël Pouchet, Fabrice Rastello, Robert J. Harrison, P. Sadayappan
2014	A	`conf`	A Communication-Optimal Framework for Contracting Distributed Tensors. SC Samyam Rajbhandari, Akshay Nikam, Pai-Wei Lai, Kevin Stock, Sriram Krishnamoorthy, P. Sadayappan
2014	B	`conf`	CAST: Contraction Algorithm for Symmetric Tensors. ICPP Samyam Rajbhandari, Akshay Nikam, Pai-Wei Lai, Kevin Stock, Sriram Krishnamoorthy, P. Sadayappan
2013	A	`conf`	A framework for load balancing of tensor contraction expressions via dynamic task partitioning. SC Pai-Wei Lai, Kevin Stock, Samyam Rajbhandari, Sriram Krishnamoorthy, P. Sadayappan
2012	Misc	`conf`	International Conference on Computational Science, ICCS 2012. ICCS Pai-Wei Lai, Huaijian Zhang, Samyam Rajbhandari, Edward F. Valeev, Karol Kowalski, P. Sadayappan