Jeffrey Pennington

79 papers A* 25A 3Misc 1Journal 41Unranked 9

Year	Rank	Type	Title / Venue / Authors
2025	J	`jnl`	Homogenization of SGD in high-dimensions: exact dynamics and generalization properties. Math. Program. Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington
2025	J	`jnl`	Modular, focused data science education improves biomedical learners' abilities: A study of the Data and Analytics for Research Training (DART) program. PLoS Comput. Biol. Rose Hartman, Joy Payton, Rose Franzen, Meredith Lee, Elizabeth Drellich, Ali Shokoufandeh, Jeffrey Pennington
2025	A*	`conf`	Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks. ICML Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, Atish Agarwala
2025	J	`jnl`	Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks. CoRR Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, Atish Agarwala
2024	A*	`conf`	4+3 Phases of Compute-Optimal Neural Scaling Laws. NeurIPS Elliot Paquette, Courtney Paquette, Lechao Xiao, Jeffrey Pennington
2024	J	`jnl`	4+3 Phases of Compute-Optimal Neural Scaling Laws. CoRR Elliot Paquette, Courtney Paquette, Lechao Xiao, Jeffrey Pennington
2024	J	`jnl`	Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. Trans. Mach. Learn. Res. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel
2024	J	`jnl`	High dimensional analysis reveals conservative sharpening and a stochastic edge of stability. CoRR Atish Agarwala, Jeffrey Pennington
2024	A*	`conf`	Scaling Exponents Across Parameterizations and Optimizers. ICML Katie E. Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington
2024	J	`jnl`	Scaling Exponents Across Parameterizations and Optimizers. CoRR Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington
2024	A*	`conf`	Small-scale proxies for large-scale Transformer training instabilities. ICLR Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie E. Everett, Alexander A. Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
2024	J	`jnl`	Training LLMs over Neurally Compressed Text. CoRR Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant
2024	J	`jnl`	Training LLMs over Neurally Compressed Text. Trans. Mach. Learn. Res. Brian Lester, Jaehoon Lee, Alexander A. Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant
2024	J	`jnl`	Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability. CoRR Jiri Hron, Laura Culp, Gamaleldin F. Elsayed, Rosanne Liu, Ben Adlam, Maxwell L. Bileschi, Bernd Bohnet, JD Co-Reyes, Noah Fiedel, C. Daniel Freeman, Izzeddin Gur, Kathleen Kenealy, Jaehoon Lee, Peter J. Liu, Gaurav Mishra, Igor Mordatch, Azade Nova, Roman Novak, Aaron Parisi, Jeffrey Pennington, Alex Rizkowsky, Isabelle Simpson, Hanie Sedghi, Jascha Sohl-Dickstein, Kevin Swersky, Sharad Vikram, Tris Warkentin, Lechao Xiao, Kelvin Xu, Jasper Snoek, Simon Kornblith
2023	J	`jnl`	Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. CoRR Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin F. Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel
2023	J	`jnl`	Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5? CoRR C. Daniel Freeman, Laura Culp, Aaron Parisi, Maxwell L. Bileschi, Gamaleldin F. Elsayed, Alex Rizkowsky, Isabelle Simpson, Alex Alemi, Azade Nova, Ben Adlam, Bernd Bohnet, Gaurav Mishra, Hanie Sedghi, Igor Mordatch, Izzeddin Gur, Jaehoon Lee, John D. Co-Reyes, Jeffrey Pennington, Kelvin Xu, Kevin Swersky, Kshiteej Mahajan, Lechao Xiao, Rosanne Liu, Simon Kornblith, Noah Constant, Peter J. Liu, Roman Novak, Yundi Qian, Noah Fiedel, Jascha Sohl-Dickstein
2023	A*	`conf`	Second-order regression models exhibit progressive sharpening to the edge of stability. ICML Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington
2023	J	`jnl`	Small-scale proxies for large-scale Transformer training instabilities. CoRR Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
2023	J	`jnl`	Temperature check: theory and practice for training models with softmax-cross-entropy losses. Trans. Mach. Learn. Res. Atish Agarwala, Samuel Stern Schoenholz, Jeffrey Pennington, Yann N. Dauphin
2022	A	`conf`	A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions. AISTATS Ben Adlam, Jake A. Levinson, Jeffrey Pennington
2022	A*	`conf`	Anisotropic Random Feature Regression in High Dimensions. ICLR Gabriel Mel, Jeffrey Pennington
2022	A*	`conf`	Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions. NeurIPS Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington
2022	J	`jnl`	Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions. CoRR Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington
2022	Misc	`conf`	Online education for data science: Opportunities and challenges. AMIA Jeffrey Pennington, Rose Hartman, Ashwini Davison, Ali Shokoufandeh, Joy Payton, Daniel Chen, André Dietrich
2022	J	`jnl`	Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression. CoRR Lechao Xiao, Jeffrey Pennington
2022	A*	`conf`	Precise Learning Curves and Higher-Order Scalings for Dot-product Kernel Regression. NeurIPS Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, Jeffrey Pennington
2022	J	`jnl`	Second-order regression models exhibit progressive sharpening to the edge of stability. CoRR Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington
2022	A*	`conf`	Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm. ICML Lechao Xiao, Jeffrey Pennington
2022	J	`jnl`	Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm. CoRR Lechao Xiao, Jeffrey Pennington
2022	A*	`conf`	Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling. ICML Jiri Hron, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
2022	J	`jnl`	Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling. CoRR Jiri Hron, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
2021	J	`jnl`	Covariate Shift in High-Dimensional Random Feature Regression. CoRR Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
2021	A*	`conf`	Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit. ICLR Ben Adlam, Jaehoon Lee, Lechao Xiao, Jeffrey Pennington, Jasper Snoek
2021	A*	`conf`	Overparameterization Improves Robustness to Covariate Shift in High Dimensions. NeurIPS Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
2020	A*	`conf`	Disentangling Trainability and Generalization in Deep Neural Networks. ICML Lechao Xiao, Jeffrey Pennington, Samuel Stern Schoenholz
2020	J	`jnl`	Exact posterior distributions of wide Bayesian neural networks. CoRR Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
2020	J	`jnl`	Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit. CoRR Ben Adlam, Jaehoon Lee, Lechao Xiao, Jeffrey Pennington, Jasper Snoek
2020	A*	`conf`	Finite Versus Infinite Neural Networks: an Empirical Study. NeurIPS Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein
2020	J	`jnl`	Finite Versus Infinite Neural Networks: an Empirical Study. CoRR Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein
2020	A*	`conf`	Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. ICLR Wei Hu, Lechao Xiao, Jeffrey Pennington
2020	J	`jnl`	Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. CoRR Wei Hu, Lechao Xiao, Jeffrey Pennington
2020	J	`jnl`	Temperature check: theory and practice for training models with softmax-cross-entropy losses. CoRR Atish Agarwala, Jeffrey Pennington, Yann N. Dauphin, Samuel S. Schoenholz
2020	A*	`conf`	The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. ICML Ben Adlam, Jeffrey Pennington
2020	J	`jnl`	The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. CoRR Ben Adlam, Jeffrey Pennington
2020	A*	`conf`	The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks. NeurIPS Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington
2020	J	`jnl`	The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks. CoRR Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington
2020	A*	`conf`	Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition. NeurIPS Ben Adlam, Jeffrey Pennington
2020	J	`jnl`	Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition. CoRR Ben Adlam, Jeffrey Pennington
2019	—	`conf`	A Mean Field Theory of Batch Normalization. ICLR (Poster) Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
2019	J	`jnl`	A Mean Field Theory of Batch Normalization. CoRR Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
2019	J	`jnl`	A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning. CoRR Ben Adlam, Jake Levinson, Jeffrey Pennington
2019	—	`conf`	Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes. ICLR (Poster) Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
2019	J	`jnl`	Disentangling trainability and generalization in deep learning. CoRR Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz
2019	J	`jnl`	Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs. CoRR Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, Jeffrey Pennington
2019	A	`conf`	KAMA-NNs: Low-dimensional Rotation Based Neural Networks. AISTATS Krzysztof Choromanski, Aldo Pacchiano, Jeffrey Pennington, Yunhao Tang
2019	A*	`conf`	Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. NeurIPS Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey Pennington
2019	J	`jnl`	Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. CoRR Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington
2018	J	`jnl`	Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes. CoRR Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
2018	—	`conf`	Deep Neural Networks as Gaussian Processes. ICLR (Poster) Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
2018	A*	`conf`	Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks. ICML Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington
2018	J	`jnl`	Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks. CoRR Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington
2018	A*	`conf`	Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks. ICML Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz
2018	J	`jnl`	Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks. CoRR Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz
2018	—	`conf`	Sensitivity and Generalization in Neural Networks: an Empirical Study. ICLR (Poster) Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
2018	J	`jnl`	Sensitivity and Generalization in Neural Networks: an Empirical Study. CoRR Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
2018	J	`jnl`	The Emergence of Spectral Universality in Deep Networks. CoRR Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
2018	A*	`conf`	The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network. NeurIPS Jeffrey Pennington, Pratik Worah
2018	A	`conf`	The emergence of spectral universality in deep networks. AISTATS Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
2017	J	`jnl`	A Correspondence Between Random Neural Networks and Statistical Field Theory. CoRR Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
2017	J	`jnl`	Deep Neural Networks as Gaussian Processes. CoRR Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
2017	A*	`conf`	Geometry of Neural Network Loss Surfaces via Random Matrix Theory. ICML Jeffrey Pennington, Yasaman Bahri
2017	—	`conf`	Nonlinear random matrix theory for deep learning. NIPS Jeffrey Pennington, Pratik Worah
2017	—	`conf`	Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. NIPS Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
2017	J	`jnl`	Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. CoRR Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
2016	—	`conf`	Clinical Data Research Network Lessons Learned. CRI Kevin Murphy, Jeffrey Pennington, Aaron N. Browne, Byron Ruth, Ritu Khare, LeMar Davidson, Patricia K. Morris, Levon H. Utidjian, Charles Bailey
2015	—	`conf`	Spherical Random Features for Polynomial Kernels. NIPS Jeffrey Pennington, Felix X. Yu, Sanjiv Kumar
2014	A*	`conf`	Glove: Global Vectors for Word Representation. EMNLP Jeffrey Pennington, Richard Socher, Christopher D. Manning
2011	—	`conf`	Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. NIPS Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, Christopher D. Manning
2011	A*	`conf`	Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. EMNLP Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning