Cyril Zhang

Hi! I'm a researcher at Microsoft AI, based in NYC. My research weaves between the theory, empirical science, and pragmatic design of learning algorithms.

Some cherished research topics (and haunting questions):

The "microstructure" of deep learning. How do the right (and wrong) circuits emerge, mechanistically? What experimental testbeds and methodologies are needed for this new, genre-defying natural science? Which mathematical tools and solution concepts from {statistics, optimization, theoretical computer science} reveal good design spaces for scalable algorithms?
Temporality, dynamical systems, and decision processes. How do we make sense of life's {discrete, continuous} sequences? Towards doing so efficiently and reliably, how do we surmount the perennial challenges of {non-convexity, non-stationarity, misspecification}?

I completed my Ph.D. in Computer Science at Princeton, under the supervision of Prof. Elad Hazan, where we worked on convexifying non-convex problems in filtering and control. For the latter half of that wonderful adventure, I was a student researcher at Google AI, where we worked on the theory and design of large-scale optimizers. Before that, I completed a B.S. in Computer Science at Yale, and grew up in Toronto.

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. Preprint.

With the Microsoft Phi team.

Can Large Language Models Explore In-Context? Preprint.

With Akshay Krishnamurthy, Keegan Harris, Dylan Foster, and Alex Slivkins.

Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression. ICLR 2024.

With Adam Block, Dylan Foster, Akshay Krishnamurthy, and Max Simchowitz.

EyeO: Autocalibrating Gaze Output with Gaze Input. Preprint.

With Akanksha Saran, Jacob Alber, Danielle Bragg, and John Langford.

Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck. NeurIPS 2023 (spotlight).

With Benjamin L. Edelman, Surbhi Goel, Sham Kakade, and Eran Malach.

Exposing Attention Glitches with Flip-Flop Language Modeling. NeurIPS 2023 (spotlight).

With Bingbin Liu, Jordan T. Ash, Surbhi Goel, and Akshay Krishnamurthy.

Learning Hidden Markov Models Using Conditional Samples. COLT 2023.

With Sham Kakade, Akshay Krishnamurthy, and Gaurav Mahajan.

Transformers Learn Shortcuts to Automata. ICLR 2023 (oral).

With Bingbin Liu, Jordan T. Ash, Surbhi Goel, and Akshay Krishnamurthy.

Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms. NeurIPS 2022.

With Surbhi Goel, Sham Kakade, and Adam Kalai.

Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit. NeurIPS 2022.

With Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, and Eran Malach.

Inductive Biases and Variable Creation in Self-Attention Mechanisms. ICML 2022.

With Benjamin L. Edelman, Surbhi Goel, and Sham Kakade.

Understanding Contrastive Learning Requires Incorporating Inductive Biases. ICML 2022.

With Nikunj Saunshi, Jordan T. Ash, Surbhi Goel, Dipendra Misra, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy.

Sparsity in Partially Controllable Linear Systems. ICML 2022.

With Yonathan Efroni, Sham Kakade, and Akshay Krishnamurthy.

Anti-Concentrated Confidence Bonuses for Scalable Exploration. ICLR 2022.

With Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade.

Acceleration via Fractal Learning Rate Schedules. ICML 2021.

With Naman Agarwal and Surbhi Goel.

Machine Learning for Mechanical Ventilation Control. Extended abstract @ ML4H 2021.

With Daniel Suo, Naman Agarwal, Wenhan Xia, Xinyi Chen, Udaya Ghai, Alex Yu, Paula Gradu, Karan Singh, Edgar Minasyan, Julienne LaChance, Tom Zajdel, Manuel Schottdorf, Daniel Cohen, and Elad Hazan.

Deluca—A Differentiable Control Library: Environments, Methods, and Benchmarking. DiffCVGP workshop @ NeurIPS 2020.

With Paula Gradu, John Hallman, Daniel Suo, Alex Yu, Naman Agarwal, Udaya Ghai, Karan Singh, Anirudha Majumdar, and Elad Hazan.

Stochastic Optimization with Laggard Data Pipelines. NeurIPS 2020.

With Naman Agarwal, Rohan Anil, Tomer Koren, and Kunal Talwar.

No-Regret Prediction in Marginally Stable Systems. COLT 2020.

With Udaya Ghai, Holden Lee, Karan Singh, and Yi Zhang.

Calibration, Entropy Rates, and Memory in Language Models. ICML 2020.

With Mark Braverman, Xinyi Chen, Sham Kakade, Karthik Narasimhan, and Yi Zhang.

Robust Guarantees for Learning an Autoregressive Filter. ALT 2020.

With Holden Lee.

Extreme Tensoring for Low-Memory Preconditioning. ICLR 2020.

With Xinyi Chen, Naman Agarwal, Elad Hazan, and Yi Zhang.