Skip to main content

Alistarh Group

Distributed Algorithms and Systems

Distribution has been one of the key trends in computing over the last decade: processor architectures are multi-core, while large-scale systems for machine learning and data processing can be distributed across several machines or even data centers. The Alistarh group works to enable these applications by creating algorithms that scale—that is, they improve their performance when more computational units are available.


This fundamental shift to distributed computing performed puts forward exciting open questions: How do we design algorithms to extract every last bit of performance from the current generation of architectures? How do we design future architectures to support more scalable algorithms? Are there clean abstractions to render high-performance distribution accessible to programmers? The group’s research is focused on answering these questions. In particular, they are interested in designing efficient, practical algorithms for fundamental problems in distributed computing, in understanding the inherent limitations of distributed systems, and in developing new ways to overcome these limitations. One particular area of focus over the past few years has been distributed machine learning.




Team

Image of Alexander Fedorov

Alexander Fedorov

PhD Student

Image of Elias Frantar

Elias Frantar

PhD Student

Image of Eugenia Iofinova

Eugenia Iofinova

PhD Student


Image of Eldar Kurtic

Eldar Kurtic

Research Technician Machine Learning

+43 2243 9000 2081

Image of Ilia Markov

Ilia Markov

PhD Student


Image of Mahdi Nikdan

Mahdi Nikdan

PhD Student

Image of Aleksandr Shevchenko

Aleksandr Shevchenko

PhD Student


Current Projects

Efficient Training and Inference for Massive Models | Distributed machine learning | Concurrent data structures and applications | Molecular computation


Publications

Kurtic E, Hoefler T, Alistarh D-A. 2024. How to prune your language model: Recovering accuracy on the ‘Sparsity May Cry’ benchmark. Proceedings of Machine Learning Research. CPAL: Conference on Parsimony and Learning, PMLR, vol. 234, 542–553. View

Beznosikov A, Horvath S, Richtarik P, Safaryan M. 2023. On biased compression for distributed learning. Journal of Machine Learning Research. 24, 1–50. View

Aksenov V, Alistarh D-A, Drozdova A, Mohtashami A. 2023. The splay-list: A distribution-adaptive concurrent skip-list. Distributed Computing. 36, 395–418. View

Iofinova EB, Peste E-A, Alistarh D-A. 2023. Bias in pruned vision models: In-depth analysis and countermeasures. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR: Conference on Computer Vision and Pattern Recognition, 24364–24373. View

Markov I, Vladu A, Guo Q, Alistarh D-A. 2023. Quantized distributed training of large models with convergence guarantees. Proceedings of the 40th International Conference on Machine Learning. ICML: International Conference on Machine Learning, PMLR, vol. 202, 24020–24044. View

View All Publications

ReX-Link: Dan Alistarh


Career

Since 2017 Assistant Professor, Institute of Science and Technology Austria (ISTA)
2016 – 2017 “Ambizione Fellow”, Computer Science Department, ETH Zurich, Switzerland
2014 – 2016 Researcher, Microsoft Research, Cambridge, UK
2014 – 2016 Morgan Fellow, Downing College, University of Cambridge, UK
2012 – 2013 Postdoc, Massachusetts Institute of Technology, Cambridge, USA
2012 PhD, EPFL, Lausanne, Switzerland


Selected Distinctions

2024 ERC Proof of Concept Grant
2018 ERC Starting Grant
2015 Awarded Swiss National Foundation “Ambizione” Fellowship
2014 Elected Morgan Fellow at Downing College, University of Cambridge
2012 Postdoctoral Fellowship of the Swiss National Foundation
2011 Best Paper Award at the International Conference on Distributed Computing and Networking


Additional Information

Dan Alistarh’s website



theme sidebar-arrow-up
Back to Top